How to use Vault to store ATP Wallet

On a recent project, I created a microservice in OKE where data was persisted in a database. ATP was the obvious choice because it is so easy to provision. The challenge was that to connect to an ATP instance a wallet is needed. This poses a security risk if you build the container image with the wallet in it. A solution is to store the wallet in HashiCorp Vault. This article explains how to store the wallet into Vault, how to setup Kubernetes auth authentication method and how setup the container to read secrets from Vault.

Before following the instructions in this article, make sure you have HashiCorp Vault installed in a Kubernetes cluster or some other VM. I installed Vault in the same cluster as the application, but in vault namespace. Also, you should already have created an ATP database downloaded the wallet.

In production you should have your applications deployed to your own namespace, not default namespace. But for this PoC I deployed the app to default.

Writing ATP Wallet to Vault

You need to setup some environment variables to connect to Vault. Use the root token obtained during Vault initialization

$ export VAULT_ADDR='https://localhost:8200'
$ export VAULT_SKIP_VERIFY="true"
$ export VAULT_TOKEN=<Root Token from initialization>

Then you need to configure port forwarding:

$ kubectl -n namespace get vault service-name -o jsonpath='{.status.vaultStatus.active}'| xargs -0 -I {} kubectl -n namespace port-forward {} 8200

$ kubectl -n vault-ns get vault safe-svc -o jsonpath='{.status.vaultStatus.active}'| xargs -0 -I {} kubectl -n vault-ns port-forward {} 8200

If you haven't created an ATP instance yet, do it now, obtain the ATP wallet zip file and then come back here. Unzip the wallet in a directory of your choice. Just make sure the directory is secure and not available to intruders. Then, using the following create a script to encode and upload the secrets to Vault. The reason for encoding is because some certificates are in binary format

$ vault kv put secret/atp \
   cwallet_sso=`cat cwallet.sso | base64` \
   ojdbc_properties=`cat ojdbc.properties | base64` \
   tnsnames_ora=`cat tnsnames.ora | base64` \
   ewallet_p12=`cat ewallet.p12 | base64` \
   truststore_jks=`cat truststore.jks | base64` \
   atp_password_txt=`cat atp_password.txt | base64` \
   keystore_jks=`cat keystore.jks | base64` \
   sqlnet_ora=`cat sqlnet.ora | base64`

$ vault kv get secret/atp

Kubernetes Configuration

Now you need to perform a series of tasks in OKE/Kubernetes so that the application running in a pod will be authorized to fetch secrets from Vault.

Create a service account

$ kubectl -n default create sa vault-reader

Create cluster role binding

create file named vault-reader-binding.yaml with this content:

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: vault-reader-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault-reader
  namespace: default

Note: the system:auth-delegator is a cluster role, which allows delegated authentication and authorization checks.

$ kubectl -n default create -f vault-reader-binding.yaml

Get Service Account token (jwt) and ca certificate

Run these commands to fetch the service account secret, jwt token (TR_ACCOUNT_TOKEN), and service account name. These will be used later when configuring Vault.

If you want, you could create a script with this content:

export VAULT_SA_NAME=$(kubectl -n default get sa vault-reader -o jsonpath="{.secrets[*]['name']}")
export TR_ACCOUNT_TOKEN=$(kubectl -n default get secret ${VAULT_SA_NAME} -o jsonpath='{.data.token}' | base64 -D)
export SA_CA_CRT=$(kubectl -n default get secret ${VAULT_SA_NAME} -o jsonpath="{.data['ca\.crt']}" | base64 -D; echo)

Get Kubernetes cluster address

Get the cluster address and port so you can use it later

$ kubectl cluster-info

Configure containers

Edit the application deployment yaml and add the following:

service account: to associate the container with serviceAccountName.

VAULT_ADDR environment variable: that points to a running instance of Vault.

volume: setup a shared volume. A script in the init container reads the Vault secrets and write them to the shared volume. The java app in the main container reads the files (the wallet) in the shared volume during authentication to ATP.

WALLET_LOCATION and ATP_CONNECTION_NAME: are used by the application to connect to ATP.

Here is an example:

spec:
      serviceAccountName: vault-reader
      containers:
      - name: priceservice
        image: iad.ocir.io/mytenancy/samplerepo/priceservice:latest
        imagePullPolicy: Always
        env:
        - name: WALLET_LOCATION
          value: /opt/data
        - name: ATP_CONNECT_NAME
          value: myATP_medium
        ports:
        - containerPort: 8080
        volumeMounts:
        - mountPath: /opt/data
          name: wallet
      imagePullSecrets:
      - name: ocir
      volumes:
       - name: wallet
         emptyDir: {}
      initContainers:
      - name: install
        image: iad.ocir.io/mytenancy/samplerepo/init-container:latest
        command: ['sh', '-c', '/root/run_task.sh']
        env:
        - name: VAULT_ADDR
          value: https://safe.vault.svc.cluster.local:8200
        volumeMounts:
        - name: wallet
          mountPath: "/work-dir"

Vault Configuration

Now you need to configure some attributes in Vault to trust the connection from the pod associated with the service account created above.

Enable Kubernetes Auth in Vault

$ vault auth enable kubernetes

Configure Kubernetes Auth to trust service account

These environment variables were set in the steps above. You should also have obtained the address:port as already instructed.

$ vault write auth/kubernetes/config \
    token_reviewer_jwt=${TR_ACCOUNT_TOKEN} \
    kubernetes_host=<address:port> \
    kubernetes_ca_cert=${SA_CA_CRT}

Create a Vault policy to allow capabilities

Create a policy file, atp-pol.hcl,which sets what kind of permissions the entity associated with this policy will have.

path "secret/*" {
    capabilities = ["create", "read", "update", "delete", "list"]

For PoC purposes this policy allows capabilities at the root path, i.e. secret/*. If you want more tight control, and you probably will, define a more specific path such as secret/atp, for example.

Write the policy to Vault:

$ vault policy write atp-pol atp-pol.hcl

Create role in Vault

Create a role which associates the Kubernetes service account with the policy just created.

$ vault write auth/kubernetes/role/atp-role \
   bound_service_account_names=vault-reader \
   bound_service_account_namespaces=default \
   policies=atp-pol \
   ttl=36h

Connecting and getting secrets

The process to obtain secrets now is simple, only two steps. First a script in the init container calls the login API, then a second call is done to fetch the secrets

In the login call, the script passes the service account JWT token, which has been registered with Vault, and the atp-role. The response contains another token, which is used to communicate with Vault in subsequent calls or until the token expires. Here is an example:

export VAULT_TOKEN=$(curl -k \
--request POST \
--data "{\"jwt\": \"`cat /var/run/secrets/kubernetes.io/serviceaccount/token`\", \"role\": \"atp-role\"}" \
https://safe.vault.svc.cluster.local:8200/v1/auth/kubernetes/login | jq -r .auth.client_token)

Then to fetch the wallet secrets, a second API call to /secret/atp will use the token from the login API call.

export WALLET_JSON=$(curl -k -H "X-Vault-Token: $VAULT_TOKEN" \
-X GET $VAULT_ADDR/v1/secret/atp)

The response is a JSON object that contains all the secret strings that were uploaded to /secret/atp. These strings are base64 encoded. So before writing them to the wallet folder they need to be decoded.

echo $WALLET_JSON|jq -r .data.atp_password_txt|base64 -d > /work-dir/atp_password.txt
echo $WALLET_JSON|jq -r .data.cwallet_sso|base64 -d > /work-dir/cwallet.sso
echo $WALLET_JSON|jq -r .data.ewallet_p12|base64 -d > /work-dir/ewallet.p12
echo $WALLET_JSON|jq -r .data.keystore_jks|base64 -d > /work-dir/keystore.jks
echo $WALLET_JSON|jq -r .data.ojdbc_properties|base64 -d > /work-dir/ojdbc.properties
echo $WALLET_JSON|jq -r .data.sqlnet_ora|base64 -d > /work-dir/sqlnet.ora
echo $WALLET_JSON|jq -r .data.tnsnames_ora|base64 -d > /work-dir/tnsnames.ora
echo $WALLET_JSON|jq -r .data.truststore_jks|base64 -d > /work-dir/truststore.jks

Here is a picture that illustrates the entire flow.

I hope this has been useful to provide an alternative method to store ATP wallet secrets in OKE.