There is an issue in Oracle SOA Suite Fabric layer in version 11.1.x. and 12.1.3. The issue is documented in Bug# 20517579: "Composites Lost Completely after redeployment and server restart". This bug is fixed in version 12.2.1. A few customers have run into this bug. Once this bug is encountered, BPM server usually shows "MDS 00054: The file to be loaded oramds:/deployed-composites/CompositeName_rev1.0/composite.xml does not exist" error during server startup. The composite is no longer visible in EM and all User Tasks from this composite are not visible in BPM Workspace either. The composite appears to be lost.
One work-around for this issue is deploying the same composite as a higher version. Or customers can manually un-deploy their problematic version of the composite using WLST script and then deploy the same composite again. In either case, customers will lose all running instances of this composite. If this outcome is not desirable, we need to find a way to recover all running instances.
This multi-part blog will present one way to manually recover those instances. To make this process more understandable to readers, a very simple BPM composite is used to take you through the processes of deployment/undeployment, reproduction of the bug# 20517579 and MDS 00054 errors, and finally steps to recover instances. Along the way, we will look at changes in MDS and SOAINFRA tables due to normal life cycles of the sample composite and bug# 20517579.
Even though this bug is fixed in 12.2.1, I think the information documented in this blog will provide a valuable resource for understanding MDS and SOAINFRA schema.
First, let's take a look what happens in MDS when a BPM composite is deployed. To do that, we need a tool like MDS Explorer. Before any BPM composite is deployed, your MDS should look empty similar to the following image.
Now we will deploy a very simple BPM composite to check for changes in MDS. We can use a BPM composite similar to the following for our testing purposes.
Notice that I created two human tasks but only have one interactive activity in my BPM process. We will only use human task 1 for now and the other one will be used to generate the MDS 00054 error later.
After deploy this composite, the MDS should look like this:
Notice that a new folder named "deploymed-composites" is created. This folder contains another folder named "default" which contains our newly deployed BPM composite source project. There is also a very important xml file named "deployed-composites.xml" under the deployed-composites folder. This file contains all composites deployed under the "default" soa-infra domain.
If you look into the "deployed-composites.xml" file, you see the name of our composite "TestBpmProject4" with a version label 1.0. I also highlight the string which starts with "soa_xxxx...xxx". This string is called MDS label or SCA label. This unique label is created for every composite deployment. Later on, we will look at some database tables in schema SOAINFRA, which is the deployment and runtime database for SOA Suite/BPM. The SCA Label is used in many runtime tables to link deployed BPM processes, active instances and audit trail instances to a specific composite deployment. Also highlighted is the SCA entity ID "6", which is also referenced in SOAINFRA schema.
If you removed our deployed composite from the "deployed-composites.xml" file manually, you effectively removed our composite from MDS and SOA runtime. Of course this is not a good way to undeploy a composite since it does not clean up MDS and SOAINFRA tables at all. But this back door does give us a way to recover instances after MDS 00054 error occurs.
To achieve our eventual goal of restoring BPM instances after MDS 00054 error, we need to know what database tables are affected after deployment of our composite. The key tables relevant to our purpose are BPM_CUBE_PROCESS and SCA_ENTITY table.
BPM_CUBE_PROCESS table contains one row per composite deployment. Column SCALABEL contains the newly created the label for this deployment. COMPOSITEDN column contains the location of the deployed composite. PROCESSID is used as a link to run-time tables which we will look at later.
MIGRATIONSTATUS column represents the status of the deployment this label represents. A value of "LATEST" indicates this SCA_LABEL represents the latest deployment for this particular version of the composite. Any rows representing earlier deployments of the same version of the same composite have their MIGRATIONSTATUS changed to "MIGRATED". BPM process instances created based on earlier deployments will continue execution based on the process definition specified as the "LATEST" in MIGRATONSTATUS column.
SCA_ENTITY table contains all entities of a composite. It contains one row per composite entity. These entities include services, references, human workflows, bpmn processes and etc. The key column is the ID which is also referenced as SCA_ENTITY_ID in other runtime tables. The LABEL column is the SCA_LABEL of the deployment.
If you are using flex fields with Human Workflow, tables WFATTRIBUTELABELMAP and WFATTRIBUTELABELUSAGE are populated with flex field mapping information after deployment. There are a few other tables touched by the deployment of our composite. But they are not as relevant as the two tables mentioned above. So we will skip them.
Now let's create an instance of our composite and check which tables are populated with instance data and how they are linked together. You can create an instance in Enterprise Manager. The following is a view of my instance.
If you look into the SOAINFRA schema, you will see the following tables are now populated with our instance data.
Please note that this is not a complete list of tables and columns in each table. Only the tables and columns that are related to our topic are described here.
The following diagram shows the relationship between deployment time and runtime tables. The red lines indicate the relationships that are related to our goal of restoring instances.
It is common that you need to make small changes to your deploy BPM composite. If the changes in BPM process are compatible with the running version of the process, you can redeploy your BPM composite as the same version of currently active BPM composite. Now we will do just that and check to see what happens in MDS and SOAINFRA schema.
We will add a simple script task to the end of our test process. So it looks like this:
After it is successfully redeployed, your MDS should look like this (in MDS Explorer):
As you can see, a new SCA label is created for our redeployment. As a result, the Composite DN attribute is also changed accordingly. However, SCA enity ID stays the same.
Now let's check out the two deployment time tables, BPM_CUBE_PROCESS and SCA_ENTITY.
As you can see in the BPM_CUBE_PROCESS table, a new row is created with the new SCA label and a new process ID. Its MIGRATIONSTATUS column is set to LATEST and that value of the row representing previous deployment is changed to MIGRATED. Our existing instance, which was created based on the first deployment, will be executed based on the process defined in our second deployment (LATEST in MIGRATIONSTATUS). For SCA_ENTITY table, there is no change after the redeployment.
As a final step, let's create another instance, which should now be based on the second deployment. If you check the instance related tables, you will see new rows are created in those tables and they are associated with the new SCA label created for our redeployment. Your EM and Workspace should look like this with the second instance:
This wraps up the first part of this blog. In Part 2, we will first try to reproduce the MDS 00054 error and see the consequence of this error. We will then take a look at what happens in MDS and SOAINFRA tables and gain understanding on why we lost all running instances.