The widgets on the Security Fundamentals Dashboards (SFD) are saved searches that query log data that has been ingested into the Logging Analytics service. Therefore, it’s important to manage your Logging Analytics storage in a cost-effective way. This blog post describes various options for purging and/or archiving your log data, along with examples, to help you decide what data to keep, where, and for how long.
Please note that data ingested into the Logging Analytics services is stored in storage managed by Oracle and not accessible directly by customers. To view detailed information about your storage usage, navigate in the OCI console to "Observability & Management"->"Logging Analytics"->"Administration", then click on "Storage" (under "Resources") in the left hand-side navigation bar.
Logging Analytics costs are based on the amount of storage used. There’s no additional cost for ingesting or using the data (by searching, analyzing, or visualizing).
Ingested data is first placed in active storage, where it’s available for use. By default, the data remains in this storage indefinitely, unless you archive or purge it.
Archiving the data moves it to a lower cost storage type, but the data is not readily usable. You will need to make a “Recall” request to make archived data available for use. The recall request may take anywhere from a few minutes to a few hours to be completed, depending on the amount of data recalled and other factors. Purging the data on the other hand removes it completely from Logging Analytics.
If you're using only the recent logs for your search and analysis tasks in Oracle Logging Analytics, then enable archiving to optimize the storage cost. Please note you can enable archiving only after you have the minimum specified size of data in active storage. Currently, this is 1 TB. Also, the minimum Active Storage Duration (Days) for logs before they can be archived is 30 days.
To enable log data archiving:
Purging enables you to bring down your usage to reduce charges. Oracle Logging Analytics can purge log data automatically per a set schedule or manually based on your need.
There are different ways to purge log data.
To automate the purge activity, create a purge policy by selecting the log data to purge (based on age of the data, and optionally a query filter), specifying the purge schedule, and enabling the policy.
Keep in mind purging log data, whether on-demand or with a purge policy, purges data in active storage, not archive storage. Data remains in archive for the duration you set under “Archival Storage Duration (Days)”
Open the navigation menu and click Observability & Management. Under Logging Analytics, click Administration. The Administration Overview page opens.
The purge policy is created.
To delete a policy, click Actions icon Actions icon next to the policy name, and click Delete.
To view the purge activities performed, in the Storage page, under Resources, click Activity Report. The Activity Report page is displayed which summarizes all the storage activities. Use the Status and Time filters to view the preferred purge activities.
When deploying the Security Fundamentals dashboards, you first onboarded the Logging Analytics service, and configured ingestion of log data for OCI audit and VCN flow logs into Logging Analytics. We recommend you create a purge policy that meets your company's requirement for data retention. If you believe you may need to query log data at some point in the future, you should consider archiving the log data for a certain period of time.
Consider the following examples assuming you have enabled OCI audit and VCN flow logs only in Logging Analytics (as is the case when you deployed SFD):
In this case, create a purge policy that purges the log data after 2 months:
You do not need to enable archiving in this use case.
In this case, enable archiving with the following parameters:
There's no need to create a purge policy for this use case.
In this case, create a purge policy that purge the VCN Flow log data after 2 months:
Now enable archiving of the remaining log data (in this case it's the audit log data) with the following parameters:
To purge log data, first set up the right permissions by creating the following dynamic group and IAM policies:
Amine is a member of the North America Technology Platform Specialist Team at Oracle Corporation. Amine specializes in Oracle Observability & Management platform (O&M), and Oracle Enterprise Manager (OEM).
Royce Fu is the Principal Database Solution Architect of the North America Cloud Technology and Engineering Team. Royce's area of specialty is core Database Technology and OCI O&M especially in Database Platform Engineering, Architecture, and Integration. He started his career as Java software engineer and spent over a decade in database engineering and architecture.