If you have never heard of Postman, please see the official Get Started with Postman -  https://learning.postman.com/getting-started/ or my Part 1: Automate getting an Identity Cloud Service Access Token in Postman -  https://www.ateam-oracle.com/part-1-automate-getting-an-idcs-access-token-in-postman.  If you are familiar with Postman, you may have run into situations where hundreds or even thousands of records are returned from a single RESTful API call, but run into a problem where a limited number of records per page are returned.  For example, a request is sent to say Identity Cloud Service (IDCS) AuditEvents RESTful endpoint, and 9,652 total records are returned, but only a maximum of 1000 records per page are displayed at a time.  As a developer this would be no problem because you could write a program to iterate through the pages of records, but what if you aren’t a developer – Postman comes to the rescue.  Keep in mind this solution could be modified to be used for other APIs with similar requirements such as OCI API Audit events and so forth as an example.  Sound interesting?

How the Postman Collection Works

This solution uses parts from all three of this three-part series on using Postman with IDCS, “Part 1: Automate getting an Identity Cloud Service Access Token in Postman”, “Part 2: Using Postman Runner with Identity Cloud Service”, and “Part 3: Automate Requests against Identity Cloud Service using Newman”.  Then add some new Pre-request and Tests scripts, and leveraging Postman Runner, we can recursively go through and get all the records page by page from the target API.  The example API used is the IDCS REST API, but this could be modified to use it with the OCI API or other APIs.  The following is a sequence diagram that illustrate the logic of the Postman Pre-request and Tests JavaScript plus some various Postman functions.    

The following reference to the block colored sections in the diagram to help explain the flow.

1. Get OAuth2 Bearer Token (Yellow)
We need to start by getting an OAuth2 Bearer token to access the API.  The Pre-request script added in the Postman parent collection “Paginate Records” is responsible for this.  See the article “Part 1: Automate getting an Identity Cloud Service Access Token in Postman” https://www.ateam-oracle.com/part-1-automate-getting-an-idcs-access-token-in-postman, it explains how the Pre-request script can accomplish this task.

2. Make Initial REST Request (Orange)
We first need to make our initial request to our API, which will return a list or results and in the JSON body there is an element “totalResults” we parse to determine if the total is greater than the maximum number of 1000 records allowed to be returned per page.  If the totalResults is greater than 1000 we do some math and calculate the number of pages we have to iterate through to finish getting all the records.

3. Loop Through the Records (Red)
If the number of pages is greater than zero, we set a variable to track the page we have gathered records and then increment up the startIndex by adding the max results value, which that tells us which page of records to be returned.  Each time we get a set of records back we check this in the Tests script, and if we need to get more we use the postman.setNextRequests(“<request name>”) function to call back the request again, this loops until we finish.  When we reach the last page of records the environment variables are cleared.

Importing the Collection and execute it in Runner

This section explains how to implement my custom Postman Collection, tweak some variables, and execute it in Runner.

1. Import and Modify Environment Variables
   1. Download environment Example_Environment.postman_environment.json
   2. In Postman click the top right gear icon Manage Environments
   3. Click the Import button > select Choose Files > and import
   4. Click the environment Example Environment
   5. Update the HOST, CLIENT_ID, and CLIENT_SECRET variables
   6. Click the Update button
   7. Close the MANAGE ENVIRONMENTS
       
2. Import Postman Collection
   1. Download collection Paginate_Records.postman_collection.json
   2. In Postman select from the main menu File
   3. Click Import…
   4. Click the Choose Files
   5. Select the Paginate_Records.postman_collection.json file
   6. Click Open
       
3. Tweak the Collection request Pre-request Script Variables
   1. In Postman expand the imported collection “Paginate Records”
   2. Select the request “HarvestAuditRecords”
   3. Select the Pre-request Script tab
   4. There are a couple variables that can safely be modified:
      1. MAX_RESULTS = <number>
         Note: MAX_RESULTS maximum for IDCS is 1000
      2. UTC_DATE = YYY-MM-DD
         You can optionally hard code a date, or uncomment the alternate option to dynamically get the previous 1 day date if this will be run as a cron job to gather logs every day.
          
4. Execute Collection in Runner
   1. In Postman click on the Runner tab
   2. Under All Collections find and select “Paginate Records”
   3. From the Environment select the Example Environment
   4. Select the Run Paginate Records blue button

Incorporate Postman Newman

Newman is a tool that can run a Postman collection runner from the command-line to put it simply.  The reason I want to bring this tool up is because by incorporating Newman with our collection we could run a cron job (How to Add Jobs to cron Under Linux or UNIX) to collection things like audit records each day.  Using some options in Newman you can export those records to a file, which then that data could be exported to a database as a long-term archive and for analysis.  In the case of IDCS, audit events are purged every 90 days, so if you have a policy to require audit events for longer than 90 days you could use this option to collect the records.

I have already written a blog on Newman Part 3: Automate Requests against Identity Cloud Service using Newman - https://www.ateam-oracle.com/part-3-automate-requests-against-identity-service-cloud-using-newman.  I will defer to this article to explain how to implement Newman and incorporate it into this solution.  The only addition to the article would be to use a few optional parameters per the official github document https://github.com/postmanlabs/newman under the Reporters section.  For example the report is the actual output of the JSON, but that report could include data you do not want such as a statistical summary, verbose things like assertions, and a banner by using some of the CLI Reporter options.

Summary

Combining some of my previous IDCS and Postman articles with this new collection should hopefully help some use cases where you need to iterate through multiple results where the number of records returned exceed a limit.  Be creative and modify the Pre-request and Tests script to also work with OCI by leveraging this great blog “Invoking OCI REST APIs using POSTMAN” - https://www.ateam-oracle.com/invoking-oci-rest-apis-using-postman.  You can leverage the Pre-request script that deals with the OAuth2 Bearer token with OCI to replace the similar script I include in my collection, and then modify the API REST request in the way you need.  What I want as the take away is to understand how Postman can be a powerful tool to build a solution without having to be a hard-core developer.