Automatically Rotate OCI Secrets using a Custom Function

"The man who can keep a secret may be wise, but he is not half as wise as the man with no secrets to keep." –Edgar Watson Howe

Now, I think it is fairly safe to say that Edgar was not thinking about cybersecurity when he penned this in 1911. However, I do believe there is wisdom to acknowledge as it pertains to a secret in general - regardless if it is digital or not.  When we talk about secrets in the digital world - whether it be our passwords or other sensitive data - it is not a matter of if, rather it is a matter of when they will become exposed. To thwart the enemy of time and staleness, we force users to reset their password every 30/60/90 days - depending on your organization policy.  Fast forwarding to the day when the secret is exposed - it is no longer a threat to the organization because it has long passed its freshness. The same rigor must go into our most secure applications, or at least - we hope.  When it comes to application security and secrets management - one of the best ways to decrease the chances of an inadvertent breach is to take away the human-factor; ergo - the man with no secrets.

A well-defined secrets rotation policy is a non-negotiable requirement for most security and compliance strategies.  In fact, it is such an important practice that several security organizations provide specific guidance to in their publications - such as OWASP and CISA's Zero Trust Model - just to name a couple.  When it comes to human personas, there is a plethora of solutions and protocols that are designed to build layers of assurance during the authentication process - such as biometrics, PKI/FIDO keys, geospatial context, behaviroal patterns, etc. These advanced factors allow us, as humans, to prove our authenticity.

On the other hand, if you have an application or a service - those advanced authentication factors may not make a lot of sense to use for a non-person entity(NPE). Given that we have limited options when proving authenticity of a NPE, we have to apply some primitive solutions in order to keep our applications secure.  This is where we often have to rely on secrets management platforms, secure coding practices and most importantly - discipline - to ensure those secrets are carefully maintained and protected from exposure.  And for applications running on OCI - there is no better option than to leverage our OCI Secrets Management Service - or Secret Service - for short.  Considering the recent release of the automated secrets rotation and secrets generation capabilities, secrets service provides an autonomous capability for managing secrets between both your clients and targets - completely taking the need for humans out of the equation when it comes to generating secrets and rotating them.

About the OCI Secret Service features

As of today, the OCI secrets framework gives you two rotation options available to use.

Autonomous Database - This option supplied does not require the deployment of any pre-built functions. This feature is designed to only rotate the ADMIN credential for the ADB system specified.

With the expectation that your DBAs and applications are hopefully not using the ADMIN credentials for any day-to-day activities, you will want the capability to rotate general database users who are identified by passwords.  For that, OCI Secrets Service has you covered. There exist a couple of Pre-Built functions which will allow you to operate on both Autonomous Databases that use mTLS(Wallets) and non-mTLS(Walletless) connections.  Additionally, the wallet-less PBF can interact with any Oracle database flavor trhough Jdbc connections - not just the autonomous database.

OCI Functions - This is the Swiss army knife option. Creating a custom OCI serverless function gives you flexibility to rotate any type of secret - regardless of the target system.  This blog demonstrates the ability of creating a custom function that can be used with OCI secrets rotation - and will provide an example template designed to show the extensibility of reaching various target systems.

Use Case and Architecture

In order to properly demonstrate the feature, I wanted to find a scenario that I felt could be useful for our customers from day one.  For that, I chose to implement the function with the purpose of rotating the client secret of a OAuth2 Confidential Client/Application configured in OCI IAM domains.

In this scenario - there is a Trusted Web Application/Service that needs to transact with another trusted microservice. This is a common use case for the OAuth2 Client Credentials grant type.  A fellow colleague authored a blog that provides great context around this architecture in more detail - for reference.  To summarize, the red arrows will show the following steps:

1.) Trusted application will first call the OCI Secrets Service to retrieve the ClientID/Secret for WebApp-A

2).  Then, it will call the OAuth2 token endpoint of OCI IAM to get a valid access_token for that transaction

3).  WebAppA uses the access_token to invoke Microservice-B with the Bearer token.

4).  Lastly, the microservice may validate the access_token against the OCI IAM domain.

Designing your OCI Function:

Before we can test that simple scenario above, we must put in some work to create the logic for our secrets rotation function. The OCI function that you must create has several stages that it will iterate through in order to successfully complete the full rotation. 

Once the initial secret has been created, configured and setup with the initial value - the rotation will follow the stages outlined in the above diagram in the blue arrows.

A.) When the next-scheduled rotation time is reached, this will trigger the CustomSecretRotate function to execute.

Custom Function stages:

B.) Verify the existing credentials against the OCI IAM /oauth2/v1/token endpoint to get acknowledge the Current secret version is valid

C.) Upon successful verification, it will read the Current secret version from Secrets Servicc to obtain OAuth2 AccessToken in order to perform a self-service update to regenerate it's own Secret via the OCI IAM Domain API.

D.) The OCI Function will then receive the new Client Secret for the application as a response from OCI IAM Regenerate call.  Then, it will create a new pending secret version of the secret.

B.) Next, it will verify that the new pending secret contents are valid by calling the OCI IAM token endpoint

D.) Once the new credentials are validated, the secret is promoted from Pending to Current Version

Creating a Custom Function:

For this example - I wanted to provide a functional template that can be used when constructing the function.  it is available for example testing only - here.

Below, you can see the 4 stages that the secret rotation function progresses through. Each stage of the code should be designed to ensure there are failure tolerance and fail-back mechanisms built-in in the event one of rotation stages does not complete successfully. Since this is a custom function, the developer needs to account for those mechanisms and properly handle the failures to avoid any secret and target mismatches.

In the case of our OAuth2 Confidential App, our function will need to invoke the target system (OCI IAM) to generate the new secret.  Then, it must return the newly generated secret and construct the object with the updated .content() so that it creates a new secret version for the secretId.
 

Note:  If this was a scenario where the auto-generation feature was turned on for the secret, then we would remove the .content(newSecretPayload) call and allow the Secrets Service to generate the secret as a pending version.

Setup, Deploy and Test

Now that the custom function has been created and deployed as an OCI Function, we can now start to setup the Secret for testing the full end-to-end flow.  Again, this example was created for OCI IAM - so we will need to create a simple OCI IAM confidential application and get the objects to populate our initial secret contents.  Refer to this OCI IAM documentation if required.

Once you have collected the appID, ClientID, and Client Secret - it is time to construct the initial secret contents in the following format.  For simplicity, I opted to use a the same JSON structure as the pre-built functions had used. However, each custom function may have its own json schema defined - that will be up to the developer.

{
"iamDomainInstanceId" : "idcs-xxxxxxxx",
"appId" : "a7xxxxxxxxx4c", 
"clientId" : "05ce9dxxxxxxxxxxx21f",
"appSecret" : "0250axxx-xxxx-xxxx-xxxx-xxxxxxxc74a0"
}

First, create a new secret and provide the typical elements such as name, description and encryption key from the OCI KMS.  Then, select Manual Secret Generation - because our secret will be generated externally and not automatically by the auto-generation capability.

Then, retrieve the OCID of our OCI Function that was deployed and specify it in the Target System Type & Target System Id.

Lastly, the Auto-Rotation is optional, for this I set it to 1 month which is the lowest - however, we will not be waiting a month to test this.  At this point, you should be able to Create Secret successfully.

IAM Policies required for Custom Function

With everything in OCI, we need to first setup the IAM policies to allow our services to have the authorization to communicate with each another.  Since these are cloud native services (functions and secrets), you will need to create a Dynamic Group.  Here is a useful guide explaining the statements used to authorize functions to interact with OCI services.  Additionally, we will also need to authorize the secret to invoke the custom function. 

As a disclaimer, these dynamic groups are relatively all encompassing within the tenancy and could be further constrained further down to specific compartments, secretId or functionId

Rule 1- this signifies any principal of the type vaultsecret - which will be allowing to invoke the custom function

Rule 2 - this specifically identifies our Custom Function, which will be used in policies to allow that function to interact with the secrets

Rule 3 - this statement could be used to include any principal of type fnfunc

Once the Dynamic Group is created - it is time to construct the necessary IAM policies to authorize two key operations.

1.) The ability for a Secrets Service to call our Custom Function

2.) The ability for the Custom Rotate Function to modify the Secrets-Family

Execute the Rotation

At this point, all of the required components have been configured and deployed. It is time to test our secret rotation logic.  There are 2 ways you could go about triggering the rotation.

1.) First and likely the obvious, you can simply select the Rotate button which will kick off the call to the Custom Rotate Function.

2.) Once you have your function working to your liking, you now can take advantage of the scheduling of this secret rotation.  In the OCI console, you will see the minimum selectable value as 1 month.  However, if you use the OCI CLI or the REST Api's to create your secret, you can specify the minimum rotation time down to 24 hours. 

Note: If you wanted to specify a custom rotation interval date, you must use ISO 8601 format.  For example, 1 day would be set as:

Fun Fact: When you trigger a manual Rotation - it will set the Last secret rotation to the current time of successful execution.  Afterwards, it will recalculate and update the Next secret rotation based off of the Rotation Interval specified.

At this point, the secret rotation process will take a up to a minute depending on the size of your function and the target system.  If your rotation is successful, then you should see a completed work item and a new Current Secret Version.

Of course if you are not fully convinced that it actually worked (like me) - you can view the contents of your secret in the OCI Vault Secrets console. Then, compare it to your target system - which in this case is OCI IAM by showing the client secret.

Limits

There is one additional thought to take into account related to OCI limits for OCI Vault.  If you review the limits documentation, a single SecretId is only allow to have a maximum of 30 versions.  Translation - when the initial secret is created, that will account for one secret version.  Each iteration that the secret goes through a successful rotation, there will be an increment of 1 to the secret version count.  To ensure that your secret rotation process continues without interrupt, there needs to be a secret version clean-up that will periodically remove the oldest versions that are no longer needed.

Conclusion

Automated secrets rotation and generation has been a long awaited feature for OCI. This capability fortifies the application deployment and operational controls around handling the most sensitive materials in the IT ecosystem; and takes you one step-closer towards autonomous security operations.  As for your staff - it allows both your application developers and security engineers to focus on the tasks that absolutely require the human-factor while providing the security compliance team the reassurance that their protocols are carried out as defined.