Introduction

Oracle Cloud Infrastructure (OCI) is a powerful IaaS, PaaS, and SaaS platform trusted by enterprises worldwide. It offers a broad suite of managed services—from compute and storage to networking and databases.

When it comes to security, having visibility into what’s happening across your infrastructure is essential. That’s where event logs come in. Feeding those logs into a Security Information and Event Management (SIEM) platform allows security teams to correlate data, analyze events, and detect threats in near real time. But to make that happen smoothly, you need a reliable way to collect and forward those logs.

A widely recommended method is to route OCI logs into OCI Streaming, which is compatible with Apache Kafka. From there, SIEM tools that support Kafka consumers can ingest the data directly. This setup brings resilience, reduced latency, and guarantees that logs are retained—even if there’s a temporary hiccup in your SIEM platform.

But here’s the catch:

Not all SIEM platforms natively support data ingestion from Kafka topics or offer out-of-the-box connectors for OCI Streaming. This is where log shippers come to the rescue.

What’s a Log Shipper?

Think of a log shipper as a courier. It picks up logs from one place, optionally repackages or filters them, and then delivers them to their destination—like a SIEM platform, database, or cloud service. In our case, the log shipper fetches logs from OCI Streaming and sends them to third-party SIEMs, using supported protocols like HTTP, TCP, or file-based handoffs.

To ensure seamless communication with both OCI Streaming and third-party SIEM platforms, the log shipper software should run on a machine with internet access. 

High-Level Architecture

At a high level, the architecture looks like this:

Architecture

  • OCI logs are routed into OCI Streaming.
  • The log shipper acts as a Kafka consumer, pulling logs from OCI Streaming.
  • Logs are then either pushed to the SIEM platform using supported integration plugins or written to a file for the SIEM to consume.

Note: While this approach is effective, treat it as a fallback option. Always consult with your SIEM vendor first to explore any native or recommended integrations. If you go with a log shipper, work with your SIEM provider to choose the right one and ensure long-term support and compatibility.

Log Shipper Tutorials

There are plenty of log shippers out there, and our team has put together detailed guides on using several popular ones with OCI Streaming. Each tutorial includes step-by-step instructions and sample examples to help you get started with platforms like Datadog, Rapid7, Elastic Cloud, and more.

  • Filebeat is a lightweight shipper for forwarding and centralizing log data. Filebeat is highly extensible through the use of modules, allowing it to collect logs from sources like Apache Kafka, Amazon Web Services (AWS), and more. It excels in handling substantial data volumes while consuming minimal resources.This tutorial demonstrates how to deploy Filebeat as a log shipper on a compute instance to collect logs from the OCI Logging service and deliver them to SIEM platforms  such as Rapid7 and Datadog.

  • Fluentd is a robust, open-source data collector developed by Treasure Data and now part of the CNCF, designed to streamline log data collection, transformation, and distribution across various systems. It acts as a unified logging layer that gathers logs from diverse sources, processes them using parser and filter plugins, and forwards them to destinations like Elastic, Kafka, or cloud storage.This tutorial shows how to deploy Fluentd as a log shipper on a compute instance to collect logs from the OCI Logging service and deliver them to Elastic for enhanced  monitoring and analysis

  • Fluent Bit is a lightweight, high-performance log shipper, serving as an alternative to Fluentd. Fluent Bit emerged in response to the growing need for an optimal solution capable of collecting logs from numerous sources while efficiently processing and filtering them. Notably, Fluent Bit excels in resource-constrained environments such as containers or embedded systems.This tutorial demonstrates how to deploy Fluent Bit as a log shipper on a compute instance to capture logs from the OCI Logging service and send them to SIEM platforms like Rapid7 and Datadog.

  • Vector is an open-source, high-performance observability pipeline that collects, transforms, and routes logs to various destinations. Vector acts as a log shipper, efficiently processing logs before forwarding them. It can be deployed as an agent-based solution on compute instances or used as a centralized aggregator to handle logs from multiple sources.This tutorial explains how to use Vector to ingest logs from OCI Streaming and forward them to a SIEM platform such as New Relic.

  • Logstash is an open source server-side data processing pipeline that ingests data from a multitude of sources, transforms it, and then sends it to your favorite “stash.” While Elasticsearch is the  go-to output that opens up a world of search and analytics possibilities, it’s not the only one available.Logstash has a variety of outputs that let you route data where you want, giving you the flexibility to unlock a slew of downstream use cases.This tutorial shows how to ingest OCI logs from OCI Streaming using Logstash.

  • Cribl is a next-generation log management platform that empowers organizations to take control of their log data. It offers features such as log routing, transformation, enrichment, and filtering, allowing users to optimize log streams according to their specific requirements. Cribl can easily send log data from various sources, such as OCI to destinations like Google Chronicle and Splunk. Cribl Stream can be deployed on your own infrastructure (self-hosted) or run in Cribl Cloud, which is managed by Cribl. This tutorial demonstrates how you can ingest OCI logs from OCI streaming using Cribl Cloud.

Final Thoughts

With the right log shipper, integrating OCI with third-party SIEM platforms becomes a seamless process—even when native support is missing. But make sure to:

– Understand the capabilities and limitations of each log shipper.
– Evaluate their input/output plugins and protocols.
– Align closely with your SIEM provider for best practices and support.

Happy shipping! With the right tools and guidance, you’re well on your way to secure, scalable, and insightful log integrations.