BAM Design Considerations for Systems with High Volume of Transactions

Dealing with high volume of transactions in Oracle Business Activity Monitoring (BAM) 11g can be challenging. Making the right design decision is critical for the stability and performance of your system. This post covers the common issues and general design recommendations for Oracle BAM 11g in the context of high volume of transactions.

The common issue in the context of  high volume of transactions, is that Active Data processing at the BAM Server Side may not catch up the speed of data received in BAM Active Data Cache (ADC), which in turn may cause slight or serious delays in report refreshing with real-time data. When there are a huge backlog of unprocessed data in BAM, the performance and functionality of the whole system will be significantly impacted. Now, let’s take a look at the following report design aspects which are the key factors impacting the scalability and performance, and our recommendations

Star Schema with High Volume of Data

It is common to use Star Schema as the data model in BAM to store business data. Whenever a message is received by BAM, it will be stored in the schema, and triggers Active Data processing within BAM Server, Under small or median load, say less than 10 – 20 messages per second (the rate of incoming messages received by BAM ADC), Active Data processing can keep up with the incoming data rate, and report should be working as expected. However, if the load goes higher, say over 100 messages per second, Active Data processing will consume higher resources (e.g. CPU, Memory, IO, etc), and the chance of performance degradation will be higher as well. Thus, our main design consideration here is to find out a way to shrink the size and data volumes of the star schema.

Recommendations

1. If you have one single Star Schema that contains a lot of look-up and calculated fields, consider breaking the one big Star Schema into multiple smaller schemas. The benefit of having multiple schemas is that the size and data volumes for each schema is reduced, therefore reducing server side contention and Active Data processing time.

2. If the transaction volume is high, say more than 60 transactions per second, consider adding additional pre-processing for the business data before sending them to BAM. For example, you can use Oracle Event Processing (OEP) to apply filters or aggregation for the business data before sending them to BAM. Using this approach, the data volume in BAM will be dramatically reduced.

3. Another approach for dealing with the high volume of data is to use Poll Mode versus Push Mode for BAM dashboards. BAM Dashboards in Poll Mode bypass Active Data Processing and reload itself at a time interval. Under heavy load, this approach consumes less system resources (CPU, Memory, Threads, etc.), and therefore performs better in terms of throughput and response time. The charts below shows the difference between Push and Poll Mode in terms of CPU usage and thread count.

CPU and Thread Count – Push Mode

jfr03

 

CPU and Thread Count – Poll Mode

jfr04

As you can see, when a report is running in Push Mode, CPU usage is consistently around 20%, and the system creates more threads for handling active data. From the performance perspective, a report in Poll Mode uses less resources and perform better under heavy load.

4. Consider normalizing your current data model to reduce the data volume of a single Data Object. If the main fact table (main Data Object of Star Schema) contains redundant fields that cause the data volume growing dramatically, consider normalizing the data model by moving these fields to separate Data Objects or External Data Objects. If you need to drill down the current view to show the values of these redundant fields, create a new report displaying these fields in Poll Mode and use the drill across feature to link the original view to the new view. 

 

Manipulating Data in Views

BAM reports can include data manipulation functions, such as filters, calculations, drill down, drill across, driving, etc. Data manipulation is expensive in BAM, and can seriously impact performance if overused in BAM reports under heavy load. Applying filters, drill down, or drill across in a BAM report will get all report views reloaded. Reloading a report is expensive as it requires querying database, re-compiling internal classes built for evaluating calculated fields, and establishing persistent HTTP connections between client and server. Frequent reloading will increase the changes of getting contentions for accessing server resources.

Recommendations

1. Minimize the usage of data manipulating functions if possible. If these functions have to be used, ensure the underling Data Object does not contain high volume of data. In the context of high transaction rate, consider normalizing Data Objects instead of using single Star Schema or using Poll Mode for dashboards.

2. If drill down or drill across function is used in the report design, we recommend that you use separate Data Objects for the main and target view. Using separate Data Objects can help to reduce the data volume, which is the key factor for improving performance under heavy load.

 

Comments

  1. Hi,

    I have a few questions regarding BAM in cluster environment.
    We have large volume of data coming in BAM from OSB.

    1. How to change the mode from Push to Poll. As per my understanding based on your article is that during Push mode server itself keep on pushing data to cache whereas in Poll server polls for data at regular intervals. Please correct me in case of wrong understanding.

    2. We have two bam servers in cluster in two nodes with bam server component on bam_server1 on node 1, now both bam servers are running due to which bam web application is running on both nodes and we are passing data to bam using web applications on both nodes from OSB in round robin fashion. Everything is working and bam is updating but we are getting below errors

    2.1 illegal program state detected: Attempted to remove a WLSExeucutionContext from the current WorkContextMap but the object in the Map was not the object to be removed
    2.2 BAM Server singleton enforcement mode is disabled. The BAM Server will not lock BAM schema for exclusive use. Use caution not to start more than one BAM Server against the same BAM schema instance
    2.3 An attempt was made to look up non-versioned global resource “topic.oracle.bam.messaging” from an application version “oracle-bam [Version=11.1.1]”. This can potentially cause conflict of the global resource usages among multiple application versions

    3. We always keep two bam servers in cluster with a configuration of whole server migration in case of failures and at the same time bam server component is also targeted to only one server in the cluster as it is singleton. If during failure, bam_server1 would be migrating to node2 and starts intersepting the requests then why do we keep bam_server2 as it will never be used for any processing.

Add Your Comment