X

Best Practices from Oracle Development's A‑Team

OCI Observability and Management for Networking - Part One - Using the Audit Log to Find Changes

Ben Woltz
Principal Cloud Architect

Introduction

In this blog series we are going to discuss ways to utilize Oracle Cloud Infrastructure (OCI) Observability and Management services and apply them to network resources with examples.  Below are 3 common requests that I've received from customers that we will go over in more detail in this and future blogs in this series.

  • Part One - How can I find who, what, and when something changed in OCI that possibly caused my issue? 
  • Part Two - How can I be notified when changes are made in the future?
  • Part Three - How can I be notified when my network connectivity is down or fails over on the backup path? 

Before we dive in to Part One for this blog, let's briefly review some of the relevant OCI services that we'll be covering.

Audit Service and Audit Log

The OCI Audit service automatically records calls to all supported OCI public Application Programming Interface (API) endpoints and logs them to the Audit Log.  This includes all API calls made by the OCI console, Command Line Interface (CLI),  Software Development Kits (SDK),  other OCI services. Information in the log events include:

  • Event Time
  • User
  • Resource
  • Action (GET, POST, PUT, PATCH, DELETE)
  • Type

The Audit Log supports GET, POST, PUT, PATCH, DELETE actions, which are the five most common Hypertext Transfer Protocol (HTTP) methods for sending and receiving data to a server and map to Create, Read, Update, and Delete (CRUD) operations.

  • POST - Create
  • GET - Read
  • PUT - Update/Replace
  • PATCH - Update/Modify
  • DELETE - Delete    

The audit log events can be viewed using the OCI console,  API, or the SDK for Java and you can also export them to JavaScript Object Notation (JSON) file to use in your own system.

See the below links for more detailed information on the OCI Audit Service and the Audit Log:
https://docs.oracle.com/en-us/iaas/Content/Audit/Concepts/auditoverview.htm
https://docs.oracle.com/en-us/iaas/Content/Logging/Concepts/audit_logs.htm


How can I find who, what, and when something changed in OCI that possibly caused my issue? 

The Audit log can be checked for changes to all types of OCI resources, but for this example we are going to focus on a rule change to a security list.  Let's say that this morning we were able to establish a Secure Shell (SSH) session to our bastion host over the public Internet from our on-premise location, but now it is not working.  We've checked the bastion host instance and it is operational and running, we've checked our Internet connection on-premise and it's working and passing traffic and we're able to ping the bastion host public Internet Protocol (IP) address.  We suspect something has changed but we are not sure what changed.  The steps below will outline how to look at the OCI Audit Log to see if a change happened that may have caused this disruption.  First we'll look finding the change in the Audit Log itself, and then we'll look at using a JSON Diff tool to highlight what has changed for more complicated scenarios.

Checking for the change using the Audit Log

  • From the OCI console go to Observability & Management >> Audit to get to the Audit log page and make sure you select the correct Compartment
  • Since we know the timeframe to search for in our example, let's filter by time for Today
  • Since we only want to look for changes, let's also filter the Request Action Type to "PUT", "POST", "PATCH", and "DELETE" which will only show us new, changed, or delete actions.  The "GET" action type is logged when someone reads a record so we are not interested in viewing those.

As you can see above, the output from our filter has identified a "PUT" action on the security list that happens to be applied to the public subnet the bastion host resides in so this could be the cause.  Note that we also see the User and Event time that this "PUT" action took place so we can tell who and when, in addition to what was changed.

  • To see more details about this change, let's click on the down arrow to the far right to expand the details of this log entry.  You may notice the output is in JSON format and contains a lot of detailed information, including the compartment name and ID, resource ID, etc.

  • To look at the details of what was changed, click on the + next to "stateChange".  Under "stateChange" you will notice a section for "current" and "previous". Under "current" it will show us details about this security list currently, after the change.  Under "previous" it will show us details about this security list prior to this change.  By investigating and comparing the differences between these, we'll be able to tell exactly what was changed.
  • Under "ingressSecurityRules" >> "0" for the "previous" state, which corresponds to the first security rule in the list, we see that it has "0.0.0.0/0" for the source and "tcpOptions" shows port 22 which is the SSH rule allowing our traffic for the bastion host.  This means that TCP port 22 (SSH) traffic was allowed from all source IPs before this change.

  • Looking at that same rule "0" under "current" state, we can see that it now has the source as "10.0.0.0/8" with TCP port 22.  This means that TCP port 22 (SSH) traffic was allowed only from source IPs in the private 10.0.0.0/8 network after this change.    

You can see with the above steps we've determined at 19:34:39 UTC time user ben.woltz@oracle.com changed the security list for the public subnet in which the bastion host resides, restricting the SSH source to 10.0.0.0/8 when it was previously allowing all sources 0.0.0.0/0 and could be the reason why our SSH from the Internet is no longer working.

Checking for the change using a JSON Diff Tool

In the previous example, it was very simple to identify what was changed as it was in the very first security list rule that we checked (labeled "0").  However, there may be scenarios where the amount of data in the audit log entry to check is very large and it will take a long time to manually click through each item to visually see what changed.  For example, if our security list had 100 rules and the rule that was changed was the last one, you would have to click on all 100 rules, both the current and previous, before you see the one that changed.  For this scenario, you can use a JSON Diff tool that will show you very quickly what has changed between two JSON outputs.  JSON Diff tools are typically available online and for free, for the below example I am using one from extendsclass.com but you can use another one if you prefer.  We will copy the "current" section from our audit log, and paste it into the JSON Diff tool, and then copy the "previous" section from our audit log, and paste it into the JSON Diff tool and it will show us exactly which part of the JSON output is different between the two.

  • In the Audit Log, left click on the "current" item and select Copy value to copy the "current" item and everything beneath it to your clipboard

  • Open a browser tab and go to https://extendsclass.com/json-diff.html
  • Paste the output into one of the two boxes.  Since this is the "current" or post change item, I prefer to paste this into the right box and the "previous" or pre change item will be pasted into the left so the before is on the left and after is on the right.
  • In the Audit Log, left click on the "previous" item and select Copy value to copy the "previous" item and everything beneath it to your clipboard
  • Go back to the browser tab with JSON Diff tool
  • Paste the output into the other box

  • Under JSON comparison section below the boxes, click on the Next diff button
  • The different between the two JSON outputs will be highlighted in the boxes below the JSON comparison section.  If there are multiple changes, you can see them all by continuing to cycle through clicking on Next diff.  See below where the same change we identified manually above in the Audit log, is being highlighted for us.

Stay tuned for part two of this series where I will go over how to set up a notification for this change so you can be proactively notified in the future when these changes happen. 

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha