Tuning G1GC For SOA

Garbage-First Garbage Collector (G1GC) is a new GC Algorithm introduced in later version of JDK 1.7. Prior to the introduction of G1GC there have been 2 other GC Algorithms: ParallelGC and Concurrent Mark Sweep (CMS) algorithms. While ParalleGC was the choice for high throughput applications like SOA, CMS was the choice for applications requiring low […]

Oracle Service Cloud Bulk Data Import – Best Practices

This blog is part of the series of blogs the A-Team has been running on Oracle Service Cloud (OSvC). In the previous blog I went through an introduction to the Bulk APIs in OSvC and briefly touched upon throughput considerations(If you haven’t, please read that first). In this blog I build on the previous post […]

EDI Processing with B2B in hybrid SOA Cloud Cluster integrating On-Premise Endpoints

Executive Overview SOA Cloud Service (SOACS) can be used to support the B2B commerce requirements of many large corporations. This article discusses a common use case of EDI processing with Oracle B2B within SOA Cloud Service in a hybrid cloud architecture. The documents are received and sent from on-premise endpoints using SFTP channels configured using […]

SOA Cloud Service – Quick and Simple Setup of an SSH Tunnel for On-Premises Database Connectivity

Executive Overview With the current release of SOA Cloud Service (SOACS) a common requirement often requested is to connect to an on-premise  database from the cloud SOACS instance. This article outlines a quick and simple method to establish the connectivity between the single-node SOACS instance and the on-premise database server using an SSH tunnel. Solution […]

How to find purgeable instances in SOA/BPM 12c

If you are familiar with SOA/BPM 11g purging, after you have upgraded/implemented SOA/BPM 12c, you will not be able to use most of the SQL for 11g to determine the purgeable instances.  This is because SOA/BPM 12c is no longer using composite_instance table for composite instance tracking. In SOA/BPM 12c, a common component is used to […]

Oracle HCM Cloud – Bulk Integration Automation Using SOA Cloud Service

Introduction Oracle Human Capital Management (HCM) Cloud provides a comprehensive set of tools, templates, and pre-packaged integration to cover various scenarios using modern and efficient technologies. One of the patterns is the batch integration to load and extract data to and from the HCM cloud. HCM provides the following bulk integration interfaces and tools: HCM Data […]

HCM Atom Feed Subscriber using SOA Cloud Service

Introduction HCM Atom feeds provide notifications of Oracle Fusion Human Capital Management (HCM) events and are tightly integrated with REST services. When an event occurs in Oracle Fusion HCM, the corresponding Atom feed is delivered automatically to the Atom server. The feed contains details of the REST resource on which the event occurred. Subscribers who consume these […]

Implementing Upsert for Oracle Service Cloud APIs

Introduction Oracle Service Cloud provides a powerful, highly scalable SOAP based batch API supporting all the usual CRUD style operations. We have recently worked with a customer who wants to leverage this API in large scale but requires the ability to have ‘upsert’ logic in place, i.e. either create or update data in Oracle Service […]

REST Adapter and JSON Translator in SOA/OSB 12.1.3

If you are using REST adapter in SOA/OSB 12.1.3, you would probably encounter some requirements that you would need to response with a JSON array format which has no object name or name/value pairs, and must be a valid format according to RFC4627 specification. For example: [“USA”,”Canada”,”Brazil”,”Australia”,”China”,”India”] In SOA/OSB 12.1.3, the REST adapter requires you […]

Submitting an ESS Job Request from BPEL in SOA 12c

Introduction SOA Suite 12c added a new component: Oracle Enterprise Scheduler Service (ESS). ESS provides the ability to run different job types distributed across the nodes in an Oracle WebLogic Server cluster. Oracle Enterprise Scheduler runs these jobs securely, with high availability and scalability, with load balancing and provides monitoring and management through Fusion Middleware Control. ESS […]

Using OSB 12.1.3 Resequencer

Resequencer feature has been added to Oracle Service Bus 12c (12.1.3), it utilises the same resequencer engine as Oracle Mediator.  The objective of this feature is to provide you with the ability to resequence the incoming messages that arrive in random order and send them to the target services in an orderly manner.  In this […]

11g Mediator – Diagnosing Resequencer Issues

In a previous blog post, we saw a few useful tips to help us quickly monitor the health of resequencer components in a soa system at runtime. In this blog post, let us explore some tips to diagnose mediator resequencer issues. During the diagnosis we will also learn some key points to consider for Integration […]

Custom Message Data Encryption of Payload in SOA 11g

Introduction This article explains how to encrypt sensitive data (such as ssn, credit card number, etc ) in the incoming payload and decrypt the data back to clear text (or original form) in the outgoing message. The purpose is to hide the sensitive data in the payload, in the audit trail, console and logs. Main […]

Mediator Parallel Routing Rules

Introduction In 11g, mediator executes routing rules either sequentially or in parallel.  If you are planning to use parallel routing rules, you would need to understand how the mediator queues and evaluates routings in parallel in different threads. This article describes 2 different threads used in the parallel routing rules and the design consideration if […]

White Paper on Message Sequencing Patterns using Oracle Mediator Resequencer

One of the consequences of Asynchronous SOA-based integration patterns is that it does not guarantee that messages will reach their destination in the same sequence as initiated at the source. Ever faced an integration scenario where – an update order is processed in the integration layer before the create order? – the target system cannot […]

Resequencer Health Check

11g Resequencer Health Check In this Blog we will see a few useful queries to monitor and diagnose the health of resequencer components running in a typical SOA/AIA Environment. The first query is a snapshot of the current count of Resequencer messages in their various states and group_statuses. Query1: Check current health of resequencers select […]

How to Recover Initial Messages (Payload) from SOA Audit for Mediator and BPEL components

Introduction In Fusion Applications, the status of SOA composite instances are either running, completed, faulted or staled. The composite instances become staled immediately (irrespective of current status) when the respective composite is redeployed with the same version. The messages (payload) are stored in SOA audit tables until they are purged. The users can go through Enterprise […]

EDN Debugging

I have a customer asked me about how to debug EDN. This blog will show you how to debug EDN and the tools that can be used to debug EDN.

<!–[if !supportLists]–>1.<!–[endif]–>Using EDN-DB-LOG

EDN comes with a useful EDN DB logging servlet to view logging information generated by the EDN component. It is only available for END-DB which is based on AQ, it will not work for EDN with JMS. The servlet uses a table called “EDN_LOG_MESSAGES” in SOA_INFRA schema. It logs the operation on “main” operation of event_queue and oaoo-queue with timestamp information.

The default URL is http://<host_name>:<port_number>/soa-infra/events/edn-db-log.  In this servlet, you can enable, disable and clear logs but you need to have the administrative role in order to access the servlet.  This is a good tool to use to display dynamic counts of un-deq’ed events (potentially “stuck”) in the “main” and “OAOO” queues. The log also provides information of EDN bus when it is being connected to AQ-DB.  In the screenshot below, “EVENT_SEQ:202” shows that the EDN bus is being started.

When the logging is enabled, the EDN_LOG_MESSAGES table will be populated with messages until the logging is disabled, so it is inadvisable to leave logging turned on for large amounts of events. It is recommended to clear the log regularly.

Messages in the log are grouped together. Usually the first line in the group will indicate what operation is being performed, and the event sequence number is used to group the messages together and each group will be highlighted using the same color (e.g. enqueuing an event or handling an event that has been dequeued). In the screenshots below, “EVENT_SEQ:204” is dequeuing an event and “EVENT_SEQ:205” is enqueuing an event.

 

2. Database tables

The second method is to examine the database table. You can check on count of potentially “stuck” events currently in the following queue tables:

 

  • EDN_EVENT_QUEUE_TABLE – This table is for “EDN_EVENT_QUEUE” AQ. Every event published is temporarily enqueued into this table.
  • EDN_OAOO_DELIVERY_TABLE – This table only stores the event with “OAOO” (one-and-only-one) delivery target(s). The event is temporarily enqueued into this table for END_OAOO_QUEUE AQ.

 

For event with OAOO delivery target, it travels through both tables, first it is stored in EDN_EVENT_QUEUE_TABLE and then in EDN_OAOO_DELIVERY_TABLE.

This example shows the event enq’ed in “edn_event_queue”.

 

Another alternative is to check the count from the following database views:

 

  • AQ$EDN_EVENT_QUEUE_TABLE: There are two rows for every event enqueued into “edn_event_queue”.
  • AQ$EDN_OAOO_DELIVERY_TABLE: There is one row for every event enqueued into “edn_oaoo_queue”. 

 

 

This example shows further details about that event which is deq’ed by the subscribers of “edn_event_queue”.

 

The AQ$EDN_EVENT_QUEUE_TABLE.MSG_STATE shows the state of the message.  The states are listed in the table below:

State Code

 

Value

 

Description

 

0

 

Ready

 

The message is ready to be processed, i.e., either the delay
time of the message has passed or the message did not have a delay time specified

 

1

 

Wait

 

The delay specified by message_properties_t.delay while executing dbms_aq.enqueue has not been reached.

 

2

 

Processed

 

The message has been successfully processed (dequeued) but will remain in the queue until the retention_time specified for the queue while executing dbms_aqadm.create_queue has been reached.

 

3

 

Expired

 

The message was not successfully processed (dequeued) in either 1) the time specified by message_properties_t.expiration while executing dbms_aq.enqueue or 2) the maximum number of dequeue attempts (max_retries) specified for the queue while executing dbms_aqadm.create_queue.

 

8

 

Deferred

 

Buffered messages enqueued by a Streams Capture process

 

10

 

Buffered Expired

 

User-enqueued expired buffered messages

 

 

 

If the subscriber type is equal to 2 when there is no subscribers to the message, and there is no transaction id due to invalid transaction, it will be marked as UNDELIVERABLE.

 

 

When the state message is expired, the AQ$EDN_EVENT_QUEUE_TABLE.EXPIRATION_REASON will be populated with one of the following value:

 

  • Messages to be cleaned up later
  • MAX_RETRY_EXCEEDED
  • TIME_EXPIRATION
  • INVALID_TRANSACTION
  • PROPAGATION_FAILURE

 

 

3.<!–[endif]–>Server Logs

 

The third method is using EM log configuration and log viewer. There are few logger names related the EDN:

 

  • oracle.integration.platform.blocks.event
  • oracle.integration.platform.blocks.event
  • oracle.integration.platform.blocks.event.saq
  • oracle.integration.platform.blocks.event.jms

You can set log level to one of the following to capture more details:

  • TRACE:1 (FINE) – Logging the event content details, XPath filter results, event enqueue, dequeue, publish and send operations
  • TRACE:16 (FINER) – Logging the begin, commit and rollback statements of XA transaction (for OAOO) and retry count.
  • TRACE:32 (FINEST)  – All above.

The log level changes take effect immediately without server restart.However, if you want the changes to persist after server restart, make sure to check on the “Persist Log Level State Across Component Restarts” prior to server restart.

 

At FINER or FINEST level, you may see loggings like “Began XA for OAOO.” and “Rolled back XA for OAOO.” These are normal messages of OAOO event delivery when there are no events waiting to be delivered. They are NOT errorred conditions. You may turn off these messages by setting the Java logging level to “TRACE:1 (FINE)” or a higher value. All detailed logging goes into SOA server’s diagnostic.log file configured in EM.  Below is a snippet of the diagnostic log showing the event delivery to an OAOO subscriber:

 


[SRC_METHOD: finerBeganXA] Began XA for OAOO.

 

[SRC_METHOD: fineEventPublished] Received event: Subject: … Sender: …. Event: …

 

[SRC_METHOD: fineFilterResults] Filter [XPath Filter: …] for subscriber “…” returned true/false

 

[SRC_METHOD: fineDequeuedEvent] Dequeued event, Subject: … [source type ..]: business-event…

 

[SRC_METHOD: fineOAOOEnqueuedEvent] Enqueued OAOO event, Subject: … [source: …, target: … ]: business-event…

 

[SRC_METHOD: fineOAOODequeuedEvent] Dequeued OAOO event, Subject: … [source: …, target: …]: business-event…

 

[SRC_METHOD: finerInsertedTryCount] Inserted try count for msgId: …. Status: …

 

[SRC_METHOD: finerRemovedTryCount] Removed try count for msgId: …

 

[SRC_METHOD: fineSentOAOOEvent] Sent OAOO event [QName: … to target: …]: business-event…

 

[SRC_METHOD: fineCommittedOAOODelivery] Committed OAOO Delivery, Subject: … [source: …, target: …]: business-event…

 

[SRC_METHOD: finerBeganXA] Began XA for OAOO.

 

[SRC_METHOD: finerRolledbackXA] Rolled back XA for OAOO.

 

In some cases, more than one method may be necessary to assist in the debugging process. Below is a comparison of server and DB logging that might help you in evaluating and determining which method(s) is/are most suitable in your environment.

 

 

 

Server Logging

  • EDN will generate standard Java logging messages when events are published, when they are pulled from persistent storage and when they are delivered.
  • The logger used by EDN depends on the implementation. For instance, EDN-DB uses “oracle.integration.platform.blocks.event.saq” and EDN-JMS uses “oracle.integration.platform.blocks.event.jms”.
  • As in all Java logging, messages are written at different log levels from ERROR to FINEST. The most detailed messages (including the event body) use FINEST.
  • Loggers can also be configured in logging.xml in your config directory.

 

 

 

DB Logging

  • If you are using EDN-DB, a lot of debugging information may not be accessible due to the many activities that occurred in the database which couldn’t be logged in the server. Hence, a servlet web page that accesses the debug logging table is implemented to assist the debugging process. The page is located at: http://hostname:port/soa-infra/events/edn-db-log and you do need to have administrative role to access the servlet page.
  • There are commands on the servlet web page to enable and disable logging and for clearing the log table. The table will be filled with messages, so it is inadvisable to leave logging turned on for large amounts of events. It is recommended to clear the log regularly.
  • Messages in the log are grouped together. Usually the first line in the group will indicate what operation is being performed (e.g. enqueuing an event or handling an event that has been dequeued).

<!–[if !supportLists]–>

 

AIA/SOA 11g Trips & Tricks: How to Save AIA/BPEL 11g Execution Time Statistics Programmatically in a File

Accessing and saving statistics is quite different in SOA 11g – this is done through JXM MBeans and not anymore by calling a BPEL API.

The following example shows how to retrieve the execution time statistics for all BPEL components deployed to one SOA server.

The example output is:

FOUND 15
Time    BPEL Name    Count    Min    Avg    Max
11:48:19    ProcessFOBillingAccountListRespOSMCFSCommsJMSProducer    6    326    2568.6666666666665    3068
11:48:19    UpdateSalesOrderSiebelCommsProvABCSImplProcess    6    1482    1821.5    2236
11:48:19    CommsProcessFulfillmentOrderBillingAccountListEBF    6    16590    22458.5    29167
11:48:19    ProcessFulfillmentOrderBillingResponseOSMCFSCommsJMSProducer    6    28    166.5    842
11:48:19    AIAAsyncErrorHandlingBPELProcess    4    1459    1758.5    2065
11:48:19    ProcessFulfillmentOrderBillingBRMCommsProvABCSImplProcess    6    1805    2462.8333333333335    4031
11:48:19    QueryCustomerPartyListSiebelProvABCSImplV2    10    640    2639.8    11079
11:48:19    AIASessionPoolManager    20    13    96.0    1344
11:48:19    ProcessSalesOrderFulfillmentOSMCFSCommsJMSProducer    10    94    562.9    1930
11:48:19    ProcessFulfillmentOrderBillingBRMCommsAddSubProcessProcess    6    773    1211.0    1577
11:48:19    SyncCustomerPartyListBRMCommsProvABCSImpl    10    323    2956.0    4045
11:48:19    TestOrderOrchestrationEBF    6    39979    46680.166666666664    52206
11:48:19    ProcessSalesOrderFulfillmentSiebelCommsReqABCSImplProcess    10    1125    2247.1    6522
11:48:19    CommsProcessBillingAccountListEBF    10    7342    12365.5    22876
11:48:19    AIAReadJMSNotificationProcess    4    9    54.5    124

You can easily paste the output in Excel to display charts like:

image

image

You also can periodically retrieve the statistics to determine if there is any performance degrade for some BPEL processes over time.

Lets see how the JMX API is used to achieve this:

First we need to establish a connection to the MBean server – for this we use the same method as we did in our JMXClient:

public static void initConnection(String hostname, String portString,
                                  String username,
                                  String password) throws IOException,
                                                          MalformedURLException {
    String protocol = “iiop”;

    Integer portInteger = Integer.valueOf(portString);
    int port = portInteger.intValue();
    String jndiroot = “/jndi/”;
    String mserver = “weblogic.management.mbeanservers.domainruntime”;

    JMXServiceURL serviceURL =
        new JMXServiceURL(protocol, hostname, port, jndiroot + mserver);

    Hashtable h = new Hashtable();
    h.put(Context.SECURITY_PRINCIPAL, username);
    h.put(Context.SECURITY_CREDENTIALS, password);
    h.put(JMXConnectorFactory.PROTOCOL_PROVIDER_PACKAGES,
          “weblogic.management.remote”);
    // Wait timeout 60 seconds
    h.put(“jmx.remote.x.request.waiting.timeout”, new Long(60000));
    connector = JMXConnectorFactory.connect(serviceURL, h);
    connection = connector.getMBeanServerConnection();
}

After that we retrieve all Mbeans which have the same pattern:

String mBeanName =
    “oracle.dms:Location=” + servername + “,soainfra_composite_label=*,type=soainfra_component,soainfra_component_type=bpel,soainfra_composite=*,soainfra_composite_revision=*,soainfra_domain=default,name=*”;

Set<ObjectInstance> mbeans =
    connection.queryMBeans(new ObjectName(mBeanName), null);
System.out.println(“FOUND ” + mbeans.size());

This matches the display in Enterprise Manager “System MBean Browser”:

EM2

Now, we can query each MBean for the attributes

  • Name
  • successfulInstanceProcessingTime_completed
  • successfulInstanceProcessingTime_minTime
  • successfulInstanceProcessingTime_avg
  • successfulInstanceProcessingTime_maxTime

That’s it!

You can find the complete JDeveloper project here.

The same statistics can of course be retrieved as well programmatically for composites (services) and references.

New BPEL Thread Pool in SOA 11g for Non-Blocking Invoke Activities from 11.1.1.6 (PS5)

Up to release 11.1.1.5 there have been 4 thread pools in Oracle SOA Suite 11g to control parallelism of execution:

  • Invoke Thread Pool (for asynchronous invocations)
  • Engine Thread Pool (i.e. for callback execution)
  • System Thread Pool
  • Audit Thread Pool

Starting with 11.1.1.6 there is one (still undocumented) new thread pool introduced for non-blocking invoke activities.

Here is a view of the System MBean Browser:

image

The MBean name is: 
oracle.dms:Location=soa_server1,name=/soainfra/engines/bpel/requests/non-block-invoke,type=soainfra_bpel_requests

You can change a synchronous invoke activity from a blocking call to non-blocking by using the partnerlink level property:

image

This thread pool is configured in SOA-Administration –> BPEL Service Engine Properties under “More BPEL Configuration Properties…” with the property DispatcherNonBlockInvokeThreads:

image

Be aware that the default is only 2 – so this can be a bottleneck in high load scenarios if not changed. Especially if you have multiple partnerlinks using non-blocking calls – because all of them share this thread pool…

Have fun, Stefan