Understanding listen ports and addresses to effectively troubleshoot Fusion Applications

Introduction: To communicate with any process, you need to know at least 3 things about it – IP address, port number and protocol. Fusion Applications comprises many running processes. End users communicate directly with only a handful of them, but all processes communicate with other processes to provide necessary services. Understanding various IP addresses and listen ports […]

WebLogic Server: Saving Disk Space in /tmp

Introduction Many WebLogic Server (WLS) implementations use JRockit 28 as the JVM implementation. JRockit 28 comes with the very useful JRockit Flight Recorder which helps in many troubleshooting situations. Problem In high volume WLS implementations with many domains and many managed servers the Flight Recorder could fill up the disk of the temporary file storage […]

Important log files and their location in Fusion Applications

Overview Log files are critical for the maintenance of system health, debugging performance problems, and functional or technical issues. Fusion applications (FA) is an integrated suite of products and hence it is important to know the log file locations for all the products and components to effectively troubleshoot a problem. A single transaction can span various components […]

Network File System (NFS) Considerations for Fusion Applications

Introduction Network File System (NFS) is used with the majority of Fusion Applications deployments to satisfy the requirement for shared storage. Using NFS correctly can present a challenge, because there is not a single right way to mount NFS shares in Oracle Linux and there are multiple other potential sources for errors like the network. […]

HTTP Server configuration in Fusion Applications

Overview One of the most powerful features provided by Fusion Applications is the out-of-the-box configuration of the HTTP Server at Provisioning time. This feature comes in different flavors, enabling the user to select the most appropriate type of configuration for their environment, including setting it up for use with a load balancer or reverse proxy. […]

Understanding the use of “WebLogic Plugin Enabled”

Introduction: Requests to a WebLogic Server (WLS) usually go through a web server or a load balancer which serve as a proxy for the client requests. When the WLS requests are “front-ended” by either a web server or a load-balancer, the requests are handled via a plugin. It is important for WLS to be aware of the […]

Discovering Fusion Applications in Oracle Enterprise Manager 12c

Introduction Oracle Enterprise Manager Cloud Control 12c offers a comprehensive and convenient way to manage Fusion Applications environments. Enterprise Manager relies on Oracle Management Agents to monitor and manage hosts and applications. These Oracle Management Agents are not to be confused with the Oracle Enterprise Manager Fusion Middleware Control agents that are deployed as part […]

How to Recover Initial Messages (Payload) from SOA Audit for Mediator and BPEL components

Introduction In Fusion Applications, the status of SOA composite instances are either running, completed, faulted or staled. The composite instances become staled immediately (irrespective of current status) when the respective composite is redeployed with the same version. The messages (payload) are stored in SOA audit tables until they are purged. The users can go through Enterprise […]

Scaling out BI with FA Shared Storage Environment

Introduction In a shared storage environment with a BI scale out in Fusion Applications, you may hit the following INST-07005 Validation of Applications or Domain Home location error: This is because the BI Configuration Agent expects an empty location for Domain and Applications Homes, in the scaled out domain. The installer will only allow you […]

Configuring Essbase Cluster for Fusion Applications

If you have provisioned an FA environment using Release 3 (11.1.3) or Release 4 (11.1.4) and followed Chapter 14 of the Enterprise Deployment Guide to cluster BI and Essbase, you will get an error similar to the one below when running Create Cubes ESS …

Tips and Tricks When Upgrading IDM for Fusion Applications – RUP1 through RUP3

In this article, I want to share some lessons learned from doing the IDM portion of the FA upgrade. Overall, the process of upgrading the security components is straightforward if you follow the instructions carefully, but there’s no denying that the IDM part of the upgrade is a little more hands-on and potentially confusing that the FA part.

A Note About Versions

When you start reading the documentation below, you may notice like I did that the naming convention for the upgrades is changing — up to now, the upgrades have been referred to RUP1, RUP2, etc., but the documentation no longer reflects that:
Version              Old Name                   New Name          FA 11gR1 RUP1         FA 11gR1 Update 1          FA 11gR1 RUP2         FA 11gR1 Update 2          FA 11gR1 RUP3         FA 11gR1 Update 3

You will see references to both naming conventions, so you should keep both in mind when searching in My Oracle Support or in your favorite search engine for information.
Important Note: These are incremental and not cumulative patches. You will need to apply RUP1 before you can apply RUP2, and the same goes for RUP3 (RUP1 and RUP2 need to be applied first).

The Documentation

Starting with RUP2/Update 2, the documentation you need to upgrade consists of three separate docs:

For RUP1/Update 1:
Release Notes: MOS Document 1382781.1
Patching Guide: http://docs.oracle.com/cd/E25054_01/fusionapps.1111/e16602/toc.htm

For RUP2/Update 2:
Release Notes: MOS Document 1439014.1
Patching Guide: http://docs.oracle.com/cd/E25178_01/fusionapps.1111/e16602/toc.htm
IDM Upgrade Procedure: MOS Document 1435333.1

For RUP3/Update 3:
Release Notes: MOS Document 1455116.1
Patching Guide: http://docs.oracle.com/cd/E28271_01/fusionapps.1111/e16602/toc.htm
IDM Upgrade Procedure: MOS Document 1441704.1

You should always start with the release notes before looking at the patching guide. The IDM upgrade procedure is a complement to the patching guide — while following the patching guide for the upgrade you’re performing, it will refer you to the IDM upgrade doc, which will then refer you back to the patching guide to continue.
Important Note: These docs have all been updated over time — make sure you check online before starting your upgrade to make sure that you have the most recent version!

Upgrading IDM from RUP1/Update 1 to RUP2/Update 2

Most of this consists of patching, but there are a couple of major changes on the IDM side, namely, the upgrade of the IDM Suite to from which in turn requires the upgrade of WLS in the IDM domain to 10.3.6.

Things You Should Do Before You Start the Upgrade

As of the time I’m publishing this, the RUP2 binaries on eDelivery come with the wrong copy of the IDM Suite installer (version instead of the required This means you will need to separately download the correct version from eDelivery. Once you have everything downloaded and ready to go, it’s worth going through the binaries to identify where the IDM-specific patches are so you don’t have to search for them later. In my case, everything was unpacked to “/u01/backup/rup2” on the machine where I first tried this:

Patch                Location
13797139        /u01/backup/rup2/installers/oracle_common/patch/13797139
13642895        /u01/backup/rup2/installers/oracle_common/patch/13642895
13686287        /u01/backup/rup2/installers/oracle_common/patch/13686287
13579026        /u01/backup/rup2/installers/oracle_common/patch/13579026
13782459        /u01/backup/rup2/installers/pltsec/patch/13782459
13620505        /u01/backup/rup2/installers/pltsec/patch/13620505
13399365        /u01/backup/rup2/installers/iam_patches/13399365
13115859        /u01/backup/rup2/installers/iam_patches/13115859
13684834        /u01/backup/rup2/installers/iam_patches/13684834
13477091        /u01/backup/rup2/installers/webgate/ext

Note that the last patch (13477091) is an OAM 10g patch for WebGate and is not applied with OPatch but run as a standalone executable.
In a similar vein, the doc references some common paths, and the example paths in the doc do not reflect the paths that are specified in the Enterprise Deployment Guide. I went through the paths on one of my lab servers which does follow the EDG:

IDM_ORACLE_HOME                             /u01/app/oracle/product/fmw/idm
OID_ORACLE_HOME                             /u01/app/oracle/product/fmw/idm
IDM_ORACLE_COMMON_HOME        /u01/app/oracle/product/fmw/oracle_common
IAM_ORACLE_HOME                            /u01/app/oracle/product/fmw/iam
OIM_ORACLE_HOME                            /u01/app/oracle/product/fmw/iam
IAM_ORACLE_COMMON_HOME       /u01/app/oracle/product/fmw/oracle_common
SOA_ORACLE_HOME                           /u01/app/oracle/product/fmw/soa
OHS_ORACLE_HOME                           /u01/app/oracle/product/fmw/web
WEB_ORACLE_HOME                          /u01/app/oracle/product/fmw/web
OHS_WEBGATE_ORACLE_HOME      /u01/app/oracle/product/fmw/oam/webgate
OHS_ORACLE_COMMON_HOME      /u01/app/oracle/product/fmw/oracle_common

Again, this is an exercise worth doing before you begin to make the upgrade steps go a little more quickly.

Following the Upgrade Doc

The upgrade doc divides the tasks into 13 steps, and I will highlight some specific comments below for the steps where I encountered an issue or where I felt that some clarification was warranted. For example, there are four patches (13797139, 13642895, 13686287 and 13579026) that you are instructed to apply in Step 6 and then again in Steps 7 and 8 and this made no sense at first. On a closer read, however, it became clear that the doc was covering a very generic deployment where the IDM Suite, the IAM Suite and the Web Tier were on separate WebLogic domains. And that’s not the case for FA installations that follow the EDG. So you really only need to apply the four patches in Step 7 because the WLS-based security components are all in one domain.

Step 4: Create Backups

Do not neglect this step! Upgrading FA is a large and complex effort.

Step 5: Download Required Patches

This step was a bit of a puzzle for me, because it belongs in the “Before Upgrading” section before Step 1. Besides, with the exception of the IDM Suite, everything else is included in the RUP2 binaries that you download from eDelivery.

Step 6: Upgrade the IDM Node

There should be no drama in this section, but remember from above that you can wait until Step 7 to apply those four patches. If you don’t, you will need to roll back and reapply those patches in the next step.

 Step 7: Upgrade the IAM Node

Remember that if you followed the EDG, the IDM and IAM nodes are one and the same and you have already upgraded WLS (and don’t need to do it again).

Apply Oracle Identity Manager patch 13399365

There are two things that happened to me when I tried this the first time. The first thing was an error from OPatch saying that this patch conflicts with patch 12961473. Turns out that this is a known issue and the fix is to simply roll back patch 13399365 and then reapply it. The second thing was deploying the new weblogic.profile file. There were some unfamiliar parameters that needed to be given values, and to save you some time, here is a copy of what it should look like for FA:

# For passwords if you dont want to put password </optional> in this file just comment it out from here, you will be promted for it in rumtime.
#Neccessary env variables [Mandatory]
#DB configuration variables [Mandatory]
# Database password is optional. if you want to  give it on terminal itself leave it commented. Otherwise uncomment it.
# If you have milty-tenancy enabled in your environment
mdsDB.user=<MDS DB Schema owner>
#Password is optional,  if you want to  give it on terminal itself leave it commented. Otherwise uncomment it.
#mdsDB.password=<MDS DB Schema password>
mdsDB.host=<MDS DB Host>
mdsDB.port=<MDS DB port>
mdsDB.serviceName=<MDS DB ServiceName>
#For domain level configurations [Mandatory]
# put here your admin server related credentials
#Password is optional,  if you want to  give it on terminal itself leave it commented. Otherwise uncomment it.
#oim specific domain level parameters [Mandatory]
#SOA specific details [Mandatory]
#put here the name of the targets of taskdetails. in non cluster it will be soa server name and in cluster it will be something like cluster_soa
#Following params is needed only if you have enabled OHS in your env
#If your env is FA, you can set this var false or ignore this if your env is non FA.

The patch_weblogic.sh script is in the same directory as weblogic.profile. You may see an error in the last step that it runs, and this seems to be due to the fact that the script tries to deploy SOA stuff before the server is fully up. This is another known bug, and the easiest fix is to check for undeployed apps in the WebLogic console and manually deploy them.

Upgrading IDM from RUP2/Update 2 to RUP3/Update 3

Most of this consists of patching, but there are a couple of changes on the IDM side, namely, the upgrade of the WebGates to 11g from 10g. The other thing (and it’s not quite spelled out in the doc) is that the FA database is upgraded to, and while this is mandatory for the FA DB it is optional for the IDM DB. For that reason alone, if you are going to upgrade the IDM DB as well, I recommend that you wait until the overall upgrade is complete and smoke tested before doing it.

Things You Should Do Before You Start the Upgrade

Once again, you should map the homes as above to have those paths handy, and locate the patches in the RUP3 binaries before rather than during the upgrade:

Patch 12575078, which contains Oracle Access Manager WebGates (
Patch 13453929 for Oracle Access Manager WebGates
Patch 13735036 for Oracle Identity Manager
Patch 13768278 for IDM Tools. Note: Be sure you download the version of this
patch, which is named p13768278_111150_Generic.zip.
Patch 13787495 for Oracle Access Manager Config Tool
Patch 13879999 for Oracle Internet Directory
Patch 13893692 for Oracle SOA Manager
 /u01/rup3/installers/oracle_common/patch/13893692 < for ORACLE_COMMON_HOME
 /u01/rup3/installers/soa/patch/13893692 < for SOA_HOME
Patch 13901417 for Oracle Access Manager

Important Note: Patch 13893692 has two parts that are applied separately, and the two parts live in different directories.
Finally, when you install the 11g WebGate(s), you will need a couple of specific gcc libraries present, and for me, my server had the 32 bit versions but not the required 64 bit versions. You can build these from scratch by downloading the source from GNU, but if you’re like me and in a hurry sometimes, you can just get them via yum:

 yum update libgcc
 mkdir /u01/app/oracle/product/fmw/gcc_lib
 cd /u01/app/oracle/product/fmw/gcc_lib
 ln -s /lib64/libgcc_s-4.1.2-20080825.so.1 ./libgcc_s.so.1
 ln -s /usr/lib64/libstdc++.6.0.8 ./libstdc++.so.6

You will need this in Step 9.

Following the Upgrade Doc

Same applies here as above — there are 14 steps in this doc with a special section that applies to Solaris.

Step 8: Upgrade the IAM Node

When you apply patch 13893692, you will get a message stating that it is already present. Just to be safe, I did a roll back and reapplication of it.

Step 9: Upgrade the OHS Node

This is where you need those gcc libraries — from the installer, specify the path that contains the symbolic links above (/u01/app/oracle/product/fmw/gcc_lib for me) for the location of the gcc libraries. The file names have to match those above.
In the new httpd.conf file, you may need to add the following lines to ensure that all logins work as they previously did:
You may need to add one line to httpd.conf to ensure that login works correctly:

 #*******Default Login page alias***
 Alias /oamsso “/u01/app/oracle/product/fmw/oam/webgate/access/oamsso”


So that’s it. If you’re careful and methodical, you shouldn’t encounter any issues for the IDM part of the upgrade. The two upgrade docs, while a bit confusing in some respects, are complete and should work for you. If anybody has their own experiences to share, please do.

Changes in SOA Human Task Flow (Run-Time) for Fusion Applications

Recently I was engaged with one of the customers to deploy custom SOA composite to one of the domains in Fusion Applications environment. The SOA composite was basic and had a simple Human Task Flow component. At runtime, the human task was created suc…

Configuring the system for a successful Fusion Application installation – Part 1 – System limits

IntroductionI wanted to share my experience in the installation of Fusion Applications. For those that are not as familiar with it, Fusion application installation goes through several phases after the provisioning plan has been created. These arePr…