Fusion Applications Clone – Troubleshooting a Weblogic Startup Error

Introduction

As noted in the Oracle® Fusion Applications Cloning and Content Movement Administrator’s Guide, the Fusion Applications (FA) Clone tool helps to create another new environment from an existing FA environment relatively quickly. In this blog, we’ll look at how to troubleshoot a common input error regarding user id that results in weblogic startup error.

The Error

The clone process requires the user to populate a response file for the clone tool to make use of. Pre-requisite at this point is that you have read the FA Clone document and have created a clone response file. As with any file of this type, user input errors are common and one we see a lot of, is an incorrect system user ID being placed in the response file. When this occurs, it will cause the clone to fail. Commonly, you will see the following generic error message as the last line on the screen and in the logs themselves.

[input] An exception occurred. PRESS RETURN to proceed with system cleanup.

A word of advice here is to not press enter right away. Take your time to look at the logs and understand the issue. Because, this is an opportunity to look and correct any run-time issues that you may miss if you hit enter. Once you hit enter, the fail cleanup process is activated and performs clean up for next run. Depending on where the error is and clone phase it is executing, you may miss valuable data and an opportunity to fix a few things. So, when you see this error, do not immediately press return. The system will be left in the failed state with all components brought up left running and there you could do the necessary changes as needed.

Above the message indicating that an error has occurred – the information regarding the process that failed can found. This will provide much more information regarding the specific error.

If the error was due to a Weblogic system user ID identification in the clone response file, occurring during the start up of the Weblogic server the lines above the message indicating that an error has occurred will look similar to this:

[execSecure] INFO: Attempting to start the AdminServer …
….
Error message: [LDAP: error code 49 - Invalid Credentials]
….
[execSecure] INFO: The AdminServer failed to start. Please check the AdminServer log and the file /shared/clone_home/FACLONE/11.1.8.0.0/temp/startWlsDomainAdminServerNoNM.2014-07-09T16:50:23.out.

[echo] TASK FAILED: idmmt.wls.main.complete
[echo] ERROR MESSAGE: The following error occurred while executing this line:
[echo] /shared/clone_home/FACLONE/11.1.8.0.0/build/tasks.idmmt.wls.xml:586: The following error occurred while executing this line:
[echo] /shared/clone_home/FACLONE/11.1.8.0.0/build/operations.wls.xml:1053: Process “/bin/bash /shared/clone_home/FACLONE/11.1.8.0.0/build/sh/startWlsDomainAdminServerNoNM.sh -d /u01/app/fusion/idm/config/domains/IDMDomain -t 180 -f /shared/clone_home/FACLONE/11.1.8.0.0/temp/startWlsDomainAdminServerNoNM.2014-07-09T16:50:23.out -u weblogic_idm” exited with non-zero exit code “1”. Input Stream before decrypting for process execution: “ertrHOKsdfgsdfDFGS#$%==”. Environment variables: “”.
[echo] See lines above for the source of the error
[echo] Running writePhaseGuard for clone.idmmt.wls.main fail
[touch] Creating /shared/clone_home/FACLONE/11.1.8.0.0/phaseguards/clone.idmmt.wls.main-123.12.34.345-FAILED.grd
[input] An exception occurred. PRESS RETURN to proceed with system cleanup.

Troubleshooting the Error

Per the error log above, an error occurred while starting the Weblogic ‘Admin Server’. In a typical FA Weblogic configuration, the Weblogic server uses three system user ids to start the server. The list below shows the users in the order Weblogic loads them as it comes up:

  • User 1: the bootstrap user used by the OPSS sub-system. This one is usually identified correctly for both IDM and FA components
  • User 2: In order to talk to the LDAP identity store to validate the user 3 below (weblogic_idm) user credential and role membership, the authenticator(shown in picture below) uses a system user.
  • User 3: the user that is defined to start the server(e.g., weblogic_idm). This weblogic_idm is stored in a LDAP identity store with a DN, (e.g.,: cn=weblogic_idm,cn=Users,dc=mycompany,dc=com). Also, this user needs to be a member of ‘IDM Administrators’ group to posses the privilege which enables the starting of the weblogic server

User 2 is usually where we see a problem. User 1 and User 3 are usually defined correctly, since the clone doc explicitly states which one to use. So for the purposes of this blog, we will focus on an error with user 2.

To debug an issue with user 2, we will focus on the Weblogic authenticator configuration. This can be an OVDAuthenticator or OIDAuthenticator. In our case, the authenticator is defined as OIDAuthenticator. You may login to the source system weblogic console and navigate to:

Home >Summary of Security Realms >myrealm >Providers >OIDAuthenticator (or OVDAuthenticator): Provider Specific and Look at the entry: Principal

A picture of this entry in the WebLogic console is shown here:

CloneBlogImg1

We have set this user – user 2 to the following value: cn=oamSoftwareUser,cn=systemids,dc=mycompany,dc=com

Now look at the response file for these lines:
IDM_OAMLDAP_USERNAME=oamSoftwareUser
IDM_OAMLDAP_DN=cn=oamSoftwareUser,cn=systemids,dc=mycompany,dc=com

The entries should match in these two places – the OID or OVD authenticator user principal(as shown in picture above) and the entry in the clone response file. If these entries do not match then we have found the problem. Please populate the clone response file with the correct value and you may retry the clone process again using steps defined in the FA Clone doc at this link.

Conclusion

In summary, the first time you user FA clone, it is important to ensure you take a detail oriented approach when populating the response file for clone. Or else, input errors like the one described above can happen.

Once all the clone phases are completed successfully and you have verified the new cloned environment working fine, you may save the response file as good reference for your future attempts. Also, with the same master image of source and clone response file, you could repeat the process easily to build out clones with much less effort.

Hope this helps to improve your FA clone experience.

Add Your Comment