ICS On-Premise Monitor – Agent Ping

The Integration Cloud Service (ICS) Connectivity Agent is a feature of ICS that helps with the cloud to ground (on-premise) use case that is common to companies who are moving into cloud technologies. The agent provides an ICS subscriber the ability to create ICS integrations that interact with on-premise resources like legacy applications, databases, etc. without compromising the on-premise resource. In an attempt to make the agent as simple and secure as possible, it is located behind the firewall where the on-premise resource is located and communicates to the Oracle Public Cloud (OPC) via HTTP only. This allows the agent to use existing firewall configurations without added requirements on the “typical” organization network. However, some networks are more secure, complex, and/or stable than others and therefore are less “typical”. When these types of networks are encountered, it prevents the Connectivity Agent from successfully communicating from ground to cloud (or ICS communicating from cloud to ground). This blog focuses on a scenario where the network that the agent is installed on is flaky/unstable.

Although the Connectivity Agent has gone through a fair amount of maturity due to exposure to a wide range of network environments, there are those edge-case networks that can introduce communication challenges from ground to cloud. Sometimes these environments impact communication such that the agent thinks everything is working fine when in fact messaging has broken down between the on-premise network and OPC. What do you do when you know the agent is not receiving messages and the various ICS/agent logs do not indicate any problem? To make the scenario more complex, the communication breakdown does not follow any type of pattern (i.e., it is random). The key to finding the root cause is to narrow the window where the problem surfaces. Once the window is narrowed down, things like tcpdumps and/or tools like Wireshark can be leveraged.

One Approach: Poor Man’s Agent Monitor

When looking at the problem of identifying a random breakdown in messaging between OPC and the on-premise agent, one approach is to periodically send a round-trip message from the agent machine. The idea behind the round-trip message is to track a message that is sent from the agent machine to ICS that flows back down to the agent:

AgentPing_Round-Trip

When the round-trip message stops flowing, an alert is sent so further investigation can be engaged:

AgentPing_Alert

There are 3 pieces to setting up the round-trip message:

  • Bash script located on the agent machine (pingAgent.sh)
  • ICS Integration that appends an line to an on-premise file located on the agent machine (ON_PREMISE_PING_01.00.0000.iar)
  • Cron entry to periodically execute the bash script

Configure pingAgent.sh

When you open the pingAgent.sh script you will see that there is a fair amount of documentation throughout the script. The necessary configurations for your environment can be found following the main header documentation. Here are the descriptions of the variables from the script:

################################################################################
# UPDATE THE FOLLOWING FOR YOUR ENVIRONMENT
################################################################################
# AGENT_PING_DIR      - Recommended to be the agent-domain directory. This value
#                       must match the Specify an Output Directory value in the
#                       file adapter configuration of the ICS On-Premise Ping
#                       integration.
# AGENT_PING_FILE     - Should not have to change this. This value must match
#                       the File Name Pattern in the file adapter configuration
#                       of the ICS On-Premise Ping integration.
# AGENT_PING_PAUSE    - Should not have to change this
# THRESHOLD_SECONDS   - Trigger threshold in seconds for sending an alert email.
#                       For example, if the threshold is set to 300 then an alert
#                       email will be sent when the last message received was
#                       longer than 300 seconds ago.
# EMAIL_TRACKING_FILE - Should not have to change this
# EMAIL_DELAY_MINUTES - Delay between email messages that have been sent. This
#                       controls the frequency of alert emails instead of sending
#                       one every time the script runs.
# EMAIL_RECIPIENT     - Email address of who should received the alert email(s)
# ICS_USERNAME        - Valid ICS user that is required for triggering the ping
#                       integration
# ICS_PASSWORD        - Password for the ICS user
# ICS_REST_URL        - The URL used for the curl command that triggers the ICS
#                       ping integration
################################################################################

It is very important to understand that there is a tight-coupling between two of the bash variables and the ICS On-Premise Ping integration: AGENT_PING_DIR and AGENT_PING_FILE. The values specified for these two variables must match two entry fields in the file adapter configuration wizard for the integration: Specify an Output Directory and File Name Pattern. The reason they need to match is this script will read what is contained in the file written to by the integration, so it must be able to locate the file. There is a screen shot in the next section showing the entry fields in the configuration wizard.

Following the environment variable configuration is a section for setting up command-line email so the script can send the alert email when it is deemed necessary. Linux mailx has been used and is encapsulated in a bash function. This will allow using whatever command-line email that fits your needs in the event mailx is not your preference. Once the email section has been setup, you can test it by manually running the script with the -tm or –testMail arguments. If it tests fine, then you are ready to setup your ICS agent ping integration.

Import ON_PREMISE_PING_01.00.0000.iar

The next part that needs setting up is the ICS integration. This is done using an ICS feature that allows importing an exported integration. Review the online documentation for this feature: Importing an Integration. Then download the ON_PREMISE_PING_01.00.0000.iar and import the IAR per the online documentation. You should see something like the following:

AgentPing-003

Now open the integration and then open the Map Action to review what is being done with the payload that will be sent to the Connectivity Agent:

AgentPing-004

AgentPing-005

You can see that there are two elements that are being set on the target side: Timestamp and POD. Feel free to update the POD to match the POD name for your ICS instance. It’s important to note that this value is not critical to the function of the monitor flow, but is for informational purposes only. The Timestamp mapping should not need changing as it simply uses a function to retrieve the current UTC dateTime:

AgentPing-006

At this point you can Validate and Close the Map Action editor. Now edit the AppendPingTimestamp target (file adapter) to open the configuration wizard. Navigate to the Configure File Write page to set the Specify an Output Directory value and File Name Pattern value. These must match the value specified in the bash script for variables AGENT_PING_DIR and AGENT_PING_FILE:

AgentPing-007

Once these values are set, navigate through the rest of the wizard pages. If you get an error message about the column header POD is invalid, make sure the value specified in the Column Headers for POD is POD only with no proceeding or trailing spaces. Now the integration can be saved, closed, and activated.

Back to the Agent Machine

We almost have all the piece in place to get this round-trip messaging monitor to rock-n-roll. Before we configure the last part, we should test to make sure the script can trigger the integration and we see the timestamp show up on the agent machine. The easiest way to test this is to make the pingAgent.sh executable and run it on the command-line: ./pingAgent.sh

If all goes well, you should now see a file called agentPing.txt on the agent machine and the contents will have an entry like: 2017-08-31T21:41:02.443Z,ics1234

We are now ready to put the wheels in motion by adding an entry in /etc/crontab via root user access. If you add the following to the bottom of the crontab file, linux cron will kick off the pingAgent.sh every minute:

*/1 * * * * oracle /bin/bash /oracle/A-Team/agenthome/user_projects/domains/agent-domain/pingAgent.sh

Please note that “oracle” in the cron entry should be the agent machine user that was used for installing and running the Connectivity Agent. Also, the path to pingAgent.sh must match your environment. Save the crontab file and then reload the cron service for your linux environment (e.g., sudo service crond reload).

At this point you can tail the agentPing.txt file to see if a new entry gets added every minute. If at any time the entries stop, an alert email will be sent based on the threshold value set in the script. When this alert occurs, here are some troubleshooting tips:

pingAgent.sh Unable to Trigger Integration
  • ICS is possibly down, verify via the ICS console or contact Oracle Cloud Operations/Support
  • On-premise network issue preventing outbound HTTP, engage network administrators to investigate further

NOTE: This scenario removes the Connectivity Agent from the troubleshooting effort as it is clearly the environment and not the agent.

pingAgent.sh Not Receiving Message From ICS
  • Issue on ICS, check the ICS console and verify that the On-Premise Ping instances are being created
  • If the On-Premise Ping instances are erroring out, check the ICS logs for details
  • If everything looks good on ICS, then review the Connectivity Agent logs for errors
  • If everything looks good on both ICS and the agent, then engage network administrators to investigate further

If and when there are updates or enhancements to the scripts/integration, they will be updated here and a version entry will be added. I hope you find this helpful in the event you are struggling with identifying sporadic messaging issues with your Connectivity Agent install.

Add Your Comment