Introduction to FA tech stack patching with FASPOT

Introduction

Fusion Applications (FA) patching is a regular administrative task during a Fusion Apps release life-cycle. Patches for Fusion Applications (P4FA) are special collections of one-off fixes and tech stack updates (i.e. for Fusion Middleware, Database, Weblogic Server etc) compiled and certified for Fusion Applications where an installation of them improves system stability and performance substantially. Every FA release has its own set of P4FA patches and it’s mandatory to apply them. P4FA patches are cumulative and administrators must always apply the most recent patch after becoming available on Oracle Support.

Usually P4FA patch installations are major efforts in the maintenance of FA systems, consuming time and resources. Applying all the individual patches manually one by one is also error-prone. This is where FASPOT comes into play to automate the P4FA patch installation. Since Summer 2014 FASPOT scripts are bundled with the P4FA patches. They will simplify the Fusion Applications tech stack patching process. The article describes how to leverage FASPOT for P4FA patching.

Main Article

P4FA Lifecycle Management

As indicated above the P4FA patches are updates for FA tech stack components and valid for a particular FA release. According to the FA certification matrix each update is tied to a specific version of a FMW or RDBMS. After finishing an FA installation or upgrading to a newer version, this certification matrix usually changes at next releases and the former P4FA patches must be renewed for these new releases as required. Hence all P4FA patches are shipped in dependency of FA releases.

Example:

A customer installs Fusion Applications Release 8 (R8). During the life-cycle of R8 several P4FA patches will be shipped for this release and an administrator installs them immediately as soon as they become available. As soon as Release 9 appears, a customer might upgrade to that new version. This consolidates the FA tech stack to the latest certified version including tech release updates and bug fixes. After performing this upgrade the P4FA life-cycle starts again for the new release and new patches have to be applied as they become available on Oracle Support.

Since P4FA patches are cumulative, subsequent patches will also include updates from previous patches although they only need to be applied once.  When applying manually it is important to read every single patch readme doc carefully as it might contain special pre- and post-install instructions. Also, there might be requirements to rollback previously installed patches to resolve conflicts. In general these technology patches are applied by using opatch, adpatch and bsu (Weblogic). As mentioned before FASPOT automates this process.

P4FA Patch Search

As shown in the screenshot below P4FA patches can be identified by a special search request on Oracle Support’s page for Patches & Updates:

  • In field Product must enter: Oracle Fusion Applications Technology Patches
  • In field Release enter the FA release like Oracle Fusion Applications 11.1.8.0.0
  • Specify the Platform like Linux x86-64

 

Screen Shot 2014-04-28 at 15.42.22

A page with search results will show all available P4FA patches according to the search criteria. For the download and installation of these patches follow the instructions in patch readme.

Screen Shot 2014-04-28 at 15.42.58

Support Note 1564176.1 explains the principals of manual P4FA patch installation. Starting with P4FA Patch 19290105 for FA R8, FASPOT is embedded in every single P4FA patch. It can be found after extraction in path <patch-no>/patches4fa/dist/FASPOT as a single zip file containing a patch number in name. Example for P4FA Patch 19290105: p18172284_111180_Generic.zip.
To use the FASPOT scripts this zip file must be extracted and a sub-folder called FASPOT will be created. The scripts can be found in that directory.

FASPOT uses orchestration by ant and provides several targets. Before scripts can run it requires some preparation activities in terms of entering configuration information about the FA topology. Once done the provisioning of patches is a straightforward process, performed by an ant script. FASPOT must run once on every server hosting a component of the FA tech stack.

FASPOT applies the patches to the following categories:

  • Fusion Middleware Components (includes atgpf,odi,oracle_common,soa,webtier,bi,ecm,ses,webcenter,wls)
  • Identity and Access Management (include OID, OIM & OHS)

Worth to mention that P4FA provided RDBMS patches remain a manual task for DBA’s to be applied. Means: these are out of scope for FASPOT. Number of combinations and variants of existing DB topologies would make it too complex to be automated by scripts. Usually customer DBA’s use their own mechanisms and best practices to perform DB life cycle management.

Preparation phase

FASPOT directory structure after extraction looks as follows:

FASPOT

|-   README.txt

|-   build.xml

|-   config

|- …

|-   env

|-   faspot.properties

|-   …

|-   faspot.sh

|-   patch_log_dir

|- …

|-   setEnv.sh

|-   tools

|- …

Meaning of files and directories above or mentioned further in this article:

  • build.xml – script contains all the targets/actions to perform patching in FA environment
  • README.txt – documentation
  • faspot.sh - script handling the ant calls for different targets – to be called to execute the ant targets
  • setEnv.sh – script to set the environment for FASPOT Utility execution. Internally used in certain configurations.
  • env – contains property files that user needs to update before script execution.
  • faspot.properties – Contains details about installed components and patch download location
  • faspot.properties.template – sample template file to fill the values as desired and generate faspot.properties
  • config – contains product specific configurations files and templates generated during runtime.
  • tools – contains ANT Libraries which are used for setting ANT_HOME in setEnv.sh
  • patch_work_dir – external directory created at user desired location during runtime. This directory will contain all the extracted p4fa patches and also laid out in the format recognizable by the FASPOT script. The “patch_work_dir” should be shared across all pods/hosts as required.
  • patch_log_dir – Logs generated during FASPOT script execution are stored in this directory.

The preparation activities self are listed below. Starting point is an extracted FASPOT directory (referred as FASPOT_HOME_DIR below) where files and directory structure can be found as shown above. This directory is located under the P4FA patch directory (i.e. <P4FA_Patch_No>/patches4fa/dist/FASPOT) but can also be copied and used from any other directory in filesystem. The following sections explain what in detail to check, configure or run.

1. Edit file faspot.properties

The next activity is to edit the FASPOT properties file, which contains key information about the FA installation like host names, directory names, user names, passwords etc. Editing this file is usually a one-time activity for this specific P4FA patch run. For this purpose user must copy <FASPOT_HOME_DIR>/env/fasport.properties.template into a new file <FASPOT_HOME_DIR>/env/fasport.properties and start editing that file.

As a mandatory step the both variables for P4FA download directory and patch staging directory must be edited similar to the example below:

PATCH_DOWNLOAD_DIR=<P4FA_PATCH_EXTRACTION_DIR>/patches4fa
PATCH_WORK_DIR=<any_directory_to_run_patches_from>

In the same way provide further details before running first FASPOT target. As a rule of thumb replace all placehoders “%<value>%” with installation-specific values.

Sample values after editing should look like this (paths and passwords are instance-specific and were changed!):

#Modify the following values based on the configuration environment
#############GENERAL#########################################
#PATCH_DOWNLOAD_DIR should contain the following folders
### 1)rel8oneoffs
#############################################################
PATCH_DOWNLOAD_DIR=/u01/fastage/r8_p4fa/19290105/patches4fa/dist
PATCH_WORK_DIR=/u01/fastage/r8_p4fa/19290105/patches4fa/workdir
#############################################################
############FA(FMW COPONENTS)################################
#atgpf - ATGPF
#############################################################
ATGPF_HOST_NAME=fusionhost.mycompany.com
ATGPF_ORACLE_HOME=/u01/app/fa/fusionapps/atgpf
ATGPF_JDK_LOC=/u01/app/fa/fusionapps/jdk6
ATGPF_DEFAULTS_FILE=/u01/app/fa/config/atgpf/admin/defaults.txt
ATGPF_ADPATCH_WORKERS=4
ATGPF_FA_DB_SID=slc01hye
ATGPF_FA_DB_USER=fusion
ATGPF_FA_DB_PASSWD=Welcome1
ATGPF_WEBCENTER_USER=fusion_webcenter
ATGPF_WEBCENTER_PASSWORD=Welcome1
ATGPF_APM_USER=fusion_apm
ATGPF_APM_PASSWORD=Welcome1
ATGPF_FUSION_ORA_ESS_USER=fusion_ora_ess
ATGPF_FUSION_ORA_ESS_PASSWORD=Welcome1
ATGPF_FUSION_RUNTIME_USER=fusion_runtime
ATGPF_FUSION_RUNTIME_PASSWORD=Welcome1
ATGPF_SYSTEM_PASSWORD=Welcome1
ATGPF_SYS_PASSWORD=Welcome1
ATGPF_MIDDLEWARE_HOME=/u01/app/fa/fusionapps
ATGPF_WEBLOGIC_HOME=/u01/app/fa/fusionapps/wlserver_10.3
#############################################################
 
#bi - BI
#############################################################
#For SAAS P4FA patching is done from Primoridal host.
#Enter PRIMORDIAL_HOST_NAME for BI_HOST_NAME
BI_HOST_NAME=fusionhost.mycompany.com
BI_ORACLE_HOME=/u01/app/fa/fusionapps/bi
BI_JDK_LOC=/u01/app/fa/fusionapps/jdk6prepare-patch-stage
BI_WL_ADMIN_USER=FAAdmin
BI_WL_ADMIN_PASSWD=Welcome1
BI_WL_ADMIN_URL=t3://fusionhost.mycompany.com:10201
...

Similar to sample above all remaining sections in file faspot.properties must be edited like:

  • ECM
  • ODI
  • OSN
  • ORACLE COMMON – FA(FMW)
  • SES
  • SOA
  • PFCORE
  • WEBCENTER
  • WEBTIER-ADMIN
  • WEBTIER-APPSOHS
  • WEBLOGIC
  • DATABASE
  • OHS – IDM
  • OID – IDM
  • OIM – IDM

We recommend to save a copy of the properties file for use with future P4FA patches even though minor adjustments might be required.

2. Run preparation targets

With the properties file in place you are ready to run the script in order to to prepare  the patch staging area.

  1. 1. Create staging area – preparation of external staging environment containing patches. This operation extracts P4FA patches into directory specified by PATCH_WORK_DIR in faspot.properties
    Run from within FASPOT directory:
cd <FASPOT_HOME_DIR>
sh ./faspot.sh -Dlogfile=prepare-patch-stage prepare-patch-stage > prepare-patch-stage-stdout.log 2>&1
  1. 2. Prepare the local environment (log directory and property files). This creates instance specific config files under <FASPOT_HOME_DIR>/config. All necessary instance information needed by other patch appliance targets are taken from these config files. To create these configuration information you have to run from within FASPOT directory
cd <FASPOT_HOME_DIR>
sh ./faspot.sh -Dlogfile=prepare-local-env prepare-local-env > prepare-local-env-stdout.log 2>&1

Please notice that running target #1 above will consume time and disk space. Especially during the preparation of staging area all patches as existing in PATCH_DOWNLOAD_DIR are getting extracted into directory PATCH_WORK_DIR. This will  increase the size on disk for patches.

Example: the patches in PATCH_DOWNLOAD_DIR occupy 8.8 GB disk space. After extraction into PATCH_WORK_DIR the size on disk can consume up to ~15 GB more for P4FA.

This emphasizes the importance of providing a certain amount of free disk space before running FASPOT scripts!

Important to remember that the instance specific configuration files under <FASPOT_HOME_DIR>/config are generated by target prepare-local-env and subsequent changes in faspot.properties have no effect for execution of FASPOT targets until this target has been re-run again.

Example: caused by a typo DB connection information were wrong. Changes in  faspot.properties have no effect until target prepare-local-env did run again. This means the connection issue will reappear until environment preparation target has corrected the values in configuration files.

  1. 3. Run environment checking target on Primordial (Admin) and APPOHS hosts by executing the following target
cd <FASPOT_HOME_DIR>
sh ./faspot.sh -Dlogfile=fmw-prereq-check fmw-prereq-check > fmw-prereq-check-stdout.log 2>&1

Verify the log file after target execution for any failures. In case of any wrong values provided in faspot.properties change values and rerun prepare-local-env as explained above.

Repeat the fmw-prereq-check target execution until all the failures are resolved.

  1. 4. Run environment checking target on AUTHOHS, OIM & OID hosts
cd <FASPOT_HOME_DIR>
sh ./faspot.sh -Dlogfile=idm-prereq-check idm-prereq-check > idm-prereq-check-stdout.log 2>&1

Verify the log file after target execution for any failures. In case of any wrong values provided in faspot.properties change values and rerun prepare-local-env as explained above.

Repeat the idm-prereq-check target execution until all the failures are resolved.

Applying patches via FASPOT

Once the preparation steps have been finished the patch installation as is can be performed.

Order of tasks

As a reminder: order of execution is crucial and for certain targets dedicated services must be either down or up and running. Here is an overview of these activities:

  1. 1) Download and extract P4FA patch (ideally on a shared file system accessible by all hosts for FA, IDM and OHS)
  2. 2) Extract FASPOT patch file
  3. 3) Configure your instance specific parameters in file faspot.properties (make it shared across all hosts)
  4. 4) Do RDBMS P4FA patching manually for IDM and FA databases
  5. 5) Start both databases and their services
  6. 6) Follow FASPOT preparation as listed in chapter above
  7. 7) Run the IDM and FMW patch-apply targets as listed below
  8. 8) Startup IDM and FA compeletly
  9. 9) Run the IDM and FMW post-install targets to deploy various artifacts
  10. 10) Validate the results by doing functional tests

Running FASPOT targets to apply P4FA patches

P4FA patching via standard targets provides the most convenient option as it runs all patches per technology in a bulk job. Log file naming can be chosen individually.

A typical script call looks as follows:

 

cd <FASPOT_HOME_DIR>
sh ./faspot.sh -Dlogfile=<target-name> <target-name> > <target-name>-stdout.log 2>&1

The existing standard targets are listed below and must be applied in the correct order:

  • idm-patch-apply – must run on IDM host
  • fmw-patch-apply – must run on FA host
  • fmw-patch-postinstall – must run on FA host
  • idm-patch-postinstall – must run on IDM host

Sample script output looks as shown in screenshot below.

FASPOT_009

To demonstrate the benefits of using FASPOT you can find below some sample timings on an average FA instance containing some test data (duration of each target execution):

  • prepare-patch-stage: 22 min
  • prepare-local-env: 2 min
  • fmw-prereq-check: 1 min
  • idm-prereq-check: 1 min
  • fmw-patch-apply: 87 min
  • idm-patch-apply: 52 min
  • fmw-patch-postinstall: 19 min
  • idm-patch-postinstall: 1 min

Compared to a manual P4FA patch installation it helps saving a tremendous amount of time and makes this patching a reliable, automated process.

Troubleshooting

During various tests the number of issues were minimal. FASPOT scripts ran successfully and fast as shown above.

Sharing my experience:

  • Typos in faspot.properties were corrected but didn’t have an influence – necessary to run target prepare-local-env again as mentioned above
  • If there is an issue with OSN please raise an SR and/or feel free to contact the author. There might be a different treatment between cloud instances and on-premise installations
  • In file faspot.properties might be some parameters which are effectively obsolete like OC_DOMAIN_HOME_LOC_LIST - in case of doubt raise an SR and/or feel free to contact the author.
  • Currently FASPOT script directory is copied to PATCH_WORK_DIR folder. Its NOT required to run FASPOT scripts from there. We’ve been told that this won’t happen anymore in future releases.
  • FASPOT has been developed for Linux 64-bit based installations of Fusion Apps. Its possible to run it on alternate platforms but some restrictions apply: usage of faspot.sh doesn’t work (rather use ant directly), usage of a native OS specific JDK is required and according modifications in setEnv.sh have to be done manually.

The life-cycle of FASPOT has just started and we will cover future updates in this page.

Comments

  1. Fiona zhou says:

    these scripts can be used on solaris sparc64 ?

  2. Johannes Michler says:

    Ulrich,

    thanks a lot for your great blog! Just upgraded our fusion-apps instance to 11.2.0.4 db and latest P4FA patches absolutely flawlessly. Most of the time actually was needed for restart and taking backups! Let’s hope that the functional tests are succesful as well,

    Regards,

    Johannes Michler

Add Your Comment