Oracle GoldenGate Services Architecture: Monitoring Via REST API

Introduction

The Oracle GoldenGate (OGG) REST API may be used to configure, manage, and monitor Oracle GoldenGate Microservices Architecture (OGG-MA) services and deployments. In this article we shall use REST API to monitor an existing OGG-MA deployment.

The concepts, scripts, and information presented in this article are for educational purposes only. They are not supported by Oracle Development or Support, and come with no guarantee or warrant for functionality in any environment other than the test system used to prepare this article. Before applying any changes presented in this article to your environment, you should thoroughly test to assess functionality and performance implications.

Main Article

OGG REST API commands are sent to the OGG-MA Administration Server via the cURL command line tool. cURL (shorthand for Client Uniform Resource Locator) is a library and command line tool for transferring data between a server and the client using the URL syntax. A version of cURL that supports SSL must be installed on the client in order to utilize OGG REST API functions.

The following example cURL command will retrieve a listing of all Replicats configured under the OGG-MA deployment on my virtual server centosora12 and associated with the OGG-MA Admin Server running on port 17100:

[oracle@ora12nomt ~]$ curl -u [Admin Server User]:[Admin Server Password] -k -H “Content-Type: application/json” -H “Accept: application/json” -X GET https://centosora12:17100/services/v2/replicats

Because the centosora12 server is a virtual machine in a test environment, it uses a self-signed SSL certificate; therefore, I must use the -k cURL option to disable certificate validation.

The response from the OGG-MA Admin Server will look similar to this:

{“$schema”:”api:standardResponse”,”links”:[{“href”:”https://centosora12:17100/services/v2/replicats”,”mediaType”:”application/json”,”rel”:”canonical”},{“href”:”https://centosora12:17100/services/v2/replicats”,”mediaType”:”application/json”,”rel”:”self”},{“href”:”https://centosora12:17100/services/v2/metadata-catalog/replicats”,”mediaType”:”application/schema+json”,”rel”:”describedby”}],”messages”:[],”response”:{“$schema”:”ogg:collection”,”items”:[{“$schema”:”ogg:collectionItem”,”links”:[{“href”:”https://centosora12:17100/services/v2/replicats”,”mediaType”:”application/json”,”rel”:”parent”},{“href”:”https://centosora12:17100/services/v2/replicats/R_DW”,”mediaType”:”application/json”,”rel”:”canonical”}],”name“:”R_DW“},{“$schema”:”ogg:collectionItem”,”links”:[{“href”:”https://centosora12:17100/services/v2/replicats”,”mediaType”:”application/json”,”rel”:”parent”},{“href”:”https://centosora12:17100/services/v2/replicats/R_RPTG”,”mediaType”:”application/json”,”rel”:”canonical”}],”name“:”R_RPTG“}]}}

The response is JSON structured data with the field “name” providing the name of each Replicat group configured in the OGG-MA deployment. We can then use cURL to query the OGG-MA Admin Server to obtain Replicat status:

[oracle@ora12nomt ~]$ curl -u [Admin Server User]:[Admin Server Password] -k -H “Content-Type: application/json” -H “Accept: application/json” -X GET https://centosora12:17100/services/v2/replicats/R_DW/info/status

The response contains detailed information about the Replicat; i.e., lag, time last started, and so forth; but for this exercise we’re only interested in the “status” field.

{“$schema”:”api:standardResponse”,”links”:[{“href”:”https://centosora12:17100/services/v2/replicats/R_DW/info/status”,”mediaType”:”application/json”,”rel”:”canonical”},{“href”:”https://centosora12:17100/services/v2/replicats/R_DW/info/status”,”mediaType”:”application/json”,”rel”:”self”},{“href”:”https://centosora12:17100/services/v2/metadata-catalog/replicatStatus”,”mediaType”:”application/schema+json”,”rel”:”describedby”}],”messages”:[],”response”:{“$schema”:”ogg:replicatStatus”,”lag”:0,”lastStarted”:”2018-06-04T14:07:59.625Z”,”position”:{“name”:”ea”,”offset”:1474,”path”:”/u01/oracle/app/goldengate/deployments/centosOra12/var/lib/data”,”sequence”:1},”sinceLagReported”:0,”status“:”abended“}}

 

Scripting Status Checks

We can write automation to check the OGG-MA deployment. The following bash script demonstrates one way to get the status of all Extracts, Replicats, and Distribution Paths in an OGG-MA deployment. If anything is in the “stopped”, “abended”, “killed”, or any state other than “running” state, we’ll report the error condition.

#!/bin/bash

## Get the status of Extracts, Replicats, and Distribution Paths configured
## in OGG-MA deployments and report when anything is not running.

## Servers and OGG-MA Admin Server ports to check
oggmaServerList=("centosora12:17100" "ora12nomt:17010")

## Servers with Distribution Paths configured
## The port is the Distribution Server Port configured on the server
oggmaDistPathList=("centosora12:17101")

## Global array to report down Extracts, Replicats, and Dist Paths
declare -a oggGroupError

getERStatus() {
   local extractList=""
   local replicatList=""
   local adminUID="<em>&lt;MY ADMIN SERVER USERID&gt;</em>:<em>&lt;MY ADMIN SERVER PASSWORD&gt;</em>"
   ## Get a list of all Extracts and Replicats and their status
   for oggmaServer in ${oggmaServerList[@]}; do
     extractList=$(curl -u $adminUID -k -H "Content-Type: application/json" -H "Accept: application/json" -X GET https://$oggmaServer/services/v2/extracts)

     replicatList=$(curl -u $adminUID -k -H "Content-Type: application/json" -H "Accept: application/json" -X GET https://$oggmaServer/services/v2/replicats)

     ## Get the status of each OGG component
     curlGetStatus $oggmaServer $adminUID extracts $extractList
     curlGetStatus $oggmaServer $adminUID replicats $replicatList
 
   done
}

getDistPathStatus() {
   local pathList=""
   local adminUID="<em>&lt;MY ADMIN SERVER USERID&gt;</em>:<em>&lt;MY ADMIN SERVER PASSWORD&gt;</em>"
   ## Get a list of all Distribution Paths
   for oggmaServer in ${oggmaDistPathList[@]}; do
     pathList=$(curl -u $adminUID -k -H "Content-Type: application/json" -H "Accept: application/json" -X GET https://$oggmaServer/services/v2/sources)

     ## Get the status of each OGG component
     curlGetStatus $oggmaServer $adminUID path $pathList

   done
}

curlGetStatus() {
   ## Get the status of each Group

   ## Get number of groups in the provided list
   local groups=$4
   local numGroups=$(gawk 'END { print NR - 1 }' RS="name" &lt;&lt;&lt; "$groups")  

   local groupName=""
   local status=""
   local isPath="0"

   ## Loop through to get all of the names and check their status
   local counter=1
   until [ $counter -gt $numGroups ]; do
     counter=$(expr $counter + 1)
     ## Strip all characters until we get to "name":"
     ## The Group name starts after the last double-quote character
     groups=$(echo ${groups#*,\"name\":\"})

     ## Get the Group name
     groupName=$(cut -d'"' -f 1 &lt;&lt;&lt; $groups)

     ## Get the status
     case "$3" in
       extracts | replicats) 
         status=$(curl -u $2 -H "Content-Type: application/json" -H "Accept: application/json" -X GET http://$1/services/v2/$3/$groupName/info/status)
       ;;
       path)
         isPath="1"
         status=$(curl -u $2 -k -H "Content-Type: application/json" -H "Accept: application/json" -X GET https://$1/services/v2/sources/$groupName)
       ;;
     esac

      ## Check the status and record any stopped or abended states
      case "$status" in 
        *running*) ## do nothing
        ;;
        *) 
           ## Strip out everything except the status
           status=$(echo ${status#*,\"status\":\"})
           if  $isPath -eq "0" ; then
             status=$(echo ${status%\"*})
           else
             status=$(echo ${status%\",*})
             isPath="0"
           fi
           local serverName=$(echo ${1%:*})
           oggGroupError=("${oggGroupError[@]}" "$serverName:$3:$groupName:$status")
         ;;
      esac
   done
}

reportStatus() {
   ## If anything is stopped or abended, report it
   if [[ ${#oggGroupError[@]} -gt "0" ]]; then
     echo $'Server\t\tComponent\tName\tStatus'
     for each in "${oggGroupError[@]}"; do
       set -f; IFS=":"
       set -- $each
       case "$2" in
         path) echo $1$'\t'$2$'\t\t'$3$'\t'$4
         ;;
	 *) echo $1$'\t'$2$'\t'$3$'\t'$4
       esac
       set +f; unset IFS
     done
   fi
}


## Main 

getERStatus
clear
getDistPathStatus
clear
reportStatus

When run from a shell prompt, this script will report the state on anything not running for each deployment specified:

[oracle@ora12nomt ~]$ ./oggmaStatusCheck.sh
Server		Component	Name	Status
centosora12	replicats	R_RPTG	abended
centosora12	path		phadp	stopped

Additional functionality could be added to this script to send email notifications; however, that is beyond the scope of this article as my virtual test systems are not associated with a valid domain and any SMTP connections made to an email server will be rejected for security reasons.

Summary

In this article we demonstrated Oracle GoldenGate REST API functionality that allows us to retrieve the status of Extracts, Replicats, and Distribution Paths configured in OGG-MA deployments. We also demonstrated how this maybe done via automation in a bash script.

For more information on what other articles are available for Oracle GoldenGate please view our index page.

Comments

  1. Bibin John says:

    Could you please let me know if we can implement similar monitoring feature in OGGBD 12.3.2.1 version?

    • OGGBD 12.3.2.1 is part of the Oracle GoldenGate Classic architecture line. OGG Management Pack may be acquired as an add-on for monitoring that line of software.

      Regards,
      Loren Penton
      Oracle Data Integration A-Team

Add Your Comment