ICS Connectivity Agent stdout Log Rotation

When monitoring and/or troubleshooting the ICS Connectivity Agent server, it is important to not only have the standard logs (e.g., AdminServer-diagnostic.log) but also the standard out/error (stdout/stderr).  One of the big challenges of capturing the stdout is not exhausting the disk storage over time.  Since the ICS Connectivity Agent is a WebLogic server installation, there are out-of-the-box options available for redirecting the stdout/stderr (see WebLogic Server logging options).  However, there are limitations for using the built-in features of WebLogic which include not capturing all the stdout/stderr (e.g., initial startup output, thread dumps using kill -3).  This blog focuses on an option available at the OS level and provides a script to help setup your environment.

One of the requirements of the ICS Connectivity Agent is it must run on Linux (see System Requirements and Restrictions). Because of this requirement, we have a Linux command at our disposal that will help manage the rotation of the agent stdout: logrotate (http://www.linuxcommand.org/man_pages/logrotate8.html). There are plenty of examples of how to use logrotate and if you are a seasoned Linux person, this blog is most likely not needed for you.  However, if you want a helper script to get this going for you in the context of the ICS Connectivity Agent then keep reading.

Starting the Connectivity Agent

Part of the ICS Connectivity Agent installation includes a start script (startAgent.sh) that is created in the directory where the agent is installed. In order to make sure the started agent server is not terminated when exiting the shell/terminal, the agent is started using the Linux nohup command.

Example 1:

 $ nohup ./startAgent.sh -u=[ICS User] -p=[ICS User Password] &

When starting the agent as detailed in Example 1, the stdout will be captured in a file called nohup.out in the same directory as the startAgent.sh script.  This is all well and good, but a more descriptive name and location may be desirable. In order to accomplish something that fits within a standard WebLogic installation we can leverage the Linux shell I/O redirect.

Example 2:

 $ nohup ./startAgent.sh -u=[ICS User] -p=[ICS User Password] > agenthome/user_projects/domains/agent-domain/servers/AdminServer/logs/AdminServer.out &

By starting the agent as detailed in Example 2, the AdminServer.out is captured in the same location as all the other WebLogic server logs. This makes finding the logging information for the Connectivity Agent easy since it is all within the same standard location. This approach has worked well for many ICS Connectivity Agent installations and will be used for the following discussion/script for the logrotate command.

Configuring logrotate for the ICS Connectivity Agent

To use the createLogRotateConfig.sh script provided in this blog, please download (createLogRotateConfig.zip) and copy to the machine where the agent is installed. For this discussion, the script will be copied to the same location as the startAgent.sh and has been made executable.

NOTE: This location is not a requirement. The only requirement is to be located somewhere on the agent machine.

This script contains 3 command-line arguments, 2 of which are optional. The optional arguments can be omitted if the agent is started and redirects the stdout to the logs directory as detailed above in the second example. The arguments are:

-ah - This is the path to the agenthome directory
-od - This is the directory where the stdout is being redirected to
-op - This is a filename/pattern for the name of the file containing the stdout (e.g., AdminServer.out or *.out)

NOTE: Review the comments in the script for the most up-to-date details and default configuration values.

The following are screenshots detailing what happens when the script is run:

Screenshot 1: createLogRotateConfig.sh File Location for Example

Screenshot 2: Output From Running createLogRotateConfig.sh

At this point, the only missing piece is to update the /etc/crontab to schedule the execution of the script that was created in the agent-domain directory (agent_stdout_logrotate.sh). The steps for this are provided at the end of the script run and can be seen in Screenshot 2.  The format for the /etc/crontab file is somewhat convoluted, but more details about cron can be found here: Configuring cron Jobs

Also, the agent_stdout_logrotate.sh can be manually run at anytime to force an out-of-cron-cycle logrotate. When agent_stdout_logrotate.sh runs and the stdout file has exceeded the logrotate "size", you will see something like the following where AdminServer.out.1 is created and the AdminServer.out is truncated:

Screenshot 3: Directory Listing Showing Log Rotation for AdminServer.out

The logrotate configuration has been setup to execute a file rotation when the AdminServer.out reaches 1M or larger and to keep a total of 10 rotated files.  It's important to understand that the 1M setting means 1M or larger.  In order to get as close to that 1M as possible, logrotate needs to be run frequently. However, there is always the possibility that a big issue is encountered and the AdminServer.out grows well beyond the 1M before logrotate runs.  You will need to monitor the behavior of the installed agent and adjust the agent_stdout_logrotate.conf and cron accordingly.  The adjustments include changing the "size" value in the agent_stdout_logrotate.conf file and the entry in the /etc/crontab file.

/etc/crontab Examples

If you were to update the /etc/crontab file in an Oracle Enterprise Linux 7 installation with the script recommended setting for every 10 minutes, it would look something like the following:

SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
# For details see man 4 crontabs
# Example of job definition:
# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  *   user-name  command to be executed
  */10 *  *  *  *   oracle     /oracle/A-Team/agenthome/user_projects/domains/agent-domain/agent_stdout_logrotate.sh

If you need to adjust the frequency from every 10 minutes to something else, here are some examples with associated frequencies:

Every Minute, All the Time:

*  *  *  *  *   oracle     /oracle/A-Team/agenthome/user_projects/domains/agent-domain/agent_stdout_logrotate.sh

Every 5 Minutes, All the Time:

*/5  *  *  *  * oracle     /oracle/A-Team/agenthome/user_projects/domains/agent-domain/agent_stdout_logrotate.sh

Every Hour at the Top of the Hour:

0  *  *  *  *   oracle     /oracle/A-Team/agenthome/user_projects/domains/agent-domain/agent_stdout_logrotate.sh

Every 2 Hours at the Top of the Hour:

0 */2 *  *  *   oracle     /oracle/A-Team/agenthome/user_projects/domains/agent-domain/agent_stdout_logrotate.sh

Hopefully you have found this useful and a good option to manage the stdout for your ICS Connectivity Agent.

Disclaimer

This script is provided "AS IS" and without any official support from Oracle. Its use needs to be performed using details from the comments section and/or readme file (if one is included). Any bugs encountered, feedback, and/or enhancement requests are welcome.

No liability for the contents of this script can be accepted. Use the concepts, examples, and information at your own risk. However, great care has been taken to ensure that all technical information is accurate and as useful as possible.

Download: createLogRotateConfig.zip