OCI Logging Analytics Best Practices Series - Management Agent Tuning

January 6, 2024 | 5 minute read
Royce Fu
Principal Database and O&M Solution Architect
Text Size 100%:

Oracle Cloud Infrastructure (OCI) Logging Analytics provides a comprehensive solution for managing and analyzing logs, facilitating easier troubleshooting, auditing, and monitoring of applications and infrastructure. In the cloud environment, it's crucial to use it effectively to optimize both performance and cost.

As a log analysis and log correlation cloud service, we always face requirements to ingest log data into Logging Analytics for log processing and indexing. There are many approaches to ingest your logs into Oracle Logging Analytics using the Oracle Management Agent, uploading logs on-demand, collecting them from the OCI Object Store, or using Service Connector. Based on the log sources and destination, we need to select the proper agent for various log ingestion use cases which ensures efficient collection is configured to filter, parse, and forward logs in a structured manner.

Right Agent for the Right Job

In my blog Demystifying Logging and Monitoring Agent Types in OCI Observability and Management, I have discussed different agent types and their use cases in OCI. It's very important to understand their feature parity and their pros and cons when choosing the agent for the job. As a common strategy, we make our agent decisions based on the destination service we are going to ingest logs into. 

OCI Logging Analytics Smart Analytics and Machine Learning for Technology Stacks
Figure 1. OCI Logging Analytics Smart Analytics and Machine Learning for Technology Stacks

 

Ingesting logs via Unified Monitoring Agent(FluentD) 

Ingesting logs via Management Agent

  • Destination: OCI Logging Analytics
  • Custom Parsers and Log Sources: besides supported out-of-the-box Parsers and Log Sources, you can use the following parser types to create parsers for your log content
    • Regex Type
    • JSON Type
    • XML Type
    • Delimited Type
  • Archive option is integrated into Logging Analytics
  • No export option for the indexed and processed data in Logging Analytics
  • Supported collection types
    • File
    • Syslog listener
    • Database SQL connection
    • Retrieve logs via REST API calls
    • Windows Event Messages via custom event channels
    • Oracle Diagnostic Logs for Middleware and Oracle Applications 

Efficient Log Collection and Ingestion

Once you have successfully picked the right agent for the right job and configured the parser and log source for the agent to ingest and process logs in OCI, you might face a challenge when the cloud resources generate a large amount of log data which may exceed the OCI Management Agent upload limit. In those cases, additional Management Agent performance tuning is required to make sure the Management Agent can push through as much log data as possible in real-time. 

Fun facts about Management Agent configuration and Performance Tuning

  • You can update Agent configuration via the property file ${MGMT_AGENT_HOME}/agent_inst/config/emd.properties
  • Agent upload file size limit is 1.5MB for each log bundle
    • MOS note: OCI: File Size Limit To Uploaded To Logging Analytics (Doc ID 2946101.1)
    • Enable parallel threads to increase the upload throughput
  • Agent Parallel Threads parameter: "_senderManagerPoolSize"
    • Default is 3.
    • Number of threads to work on parallel uploading log bundles
    • Eg. 10MB log data will split into 7 log bundles, if we have 3 parallel upload threads, it takes 3 rounds to upload all log bundles, whereas log bundles will upload all at the same if we have more than 7 upload threads
  • Agent Send Time Frequency: "_senderManagerSendTimerFrequency"
    • Default is 30 seconds. 
    • Minimum tiem frequence is 1 second.
  • Agent force upload without latency: "_forceSendImmediate"
    • Default value is false. 
    • We can set it to true so that sender manager will upload files as soon as they are generated with no delay.

In order to improve the Logging Analytics ingestion performance from the OCI Logging Service and Service Connector Hub, we can reduce Log Volume for logs

  • Filter out unnecessary logs or verbose debug logs in Service Connector Hub that don't add value to the analysis
  • Controls costs and reduces the clutter in the Logging Analytics workspace

Another log collection theme in OCI O&M is to forward your logs to the central syslog server

Conclusion

Optimizing the Management Agent for peak performance in log ingestion is crucial for your infrastructure and operational teams. This enhancement is key to ensuring timely log analysis and detection. By adopting these best practices, you're not just adjusting a system; you're unlocking the full potential of your log data. This enables more informed decision-making, streamlines operational workflows, and boosts security measures. Embrace the power of Management Tuning in your log data analysis to elevate your organization's efficiency and performance, paving the way for operational superiority.

Please check out our Oracle Cloud Customer Connect Observability and Management Community. You can pose questions, connect with experts, and share your successes, thoughts and ideas about Oracle Cloud Observability and Management solutions (including OCI Application Performance Monitoring, Stack Monitoring, Logging Analytics, Database Management and Operations Insights).

Further Reading 

Acknowledgements

  • Contributor: Waymon Whiting

Royce Fu

Principal Database and O&M Solution Architect

Royce Fu is the Principal Database Solution Architect of the North America Cloud Technology and Engineering Team. Royce's area of specialty is core Database Technology and OCI O&M especially in Database Platform Engineering, Architecture, and Integration. He started his career as Java software engineer and spent over a decade in database engineering and architecture.


Previous Post

Fusion Applications Integration with Microsoft Teams : Business Object Change Capture in OIC

Greg Mally | 12 min read

Next Post


OCI Logging Analytics Best Practices Series - Cost Optimization

Royce Fu | 8 min read