Introduction

If you experience slow clicks in an FA environment it will be hard to manually generate the needed diagnostic data.

This document is intended to guide you through the steps needed to produce the diagnostic data automatically using existing options in FA software.

We will show these steps for Cloud versions and for the only supported onPremise version.

Enabling auto diagnostics in Cloud as FA user

Prerequisites for Cloud usage

In our actual cloud versions every user that has the roles employee and Business Flow Troubleshooting will be able use the trace option mentioned in this document.

If these roles are not assigned to the user please add them.

Here is a description how to do that.

Connect to your fa using an administrator account that can add users and roles.

In home screen go to Tools and then click on "Security Console"

Click on Users and Security:

Click on Users:

After typing in a search string in search filed and hitting the search button click on the username that you want to check to open the overview

As you can see our user does not have the needed roles. To add them please click on “Edit”

Click on “Add Role” to add the missing roles to user

Type in “EMPLOYEE” in the search field and then select “ORA_PER_EMPLOYEE_ABSTRACT” from the result list

Click on “Add Role Membership”

Click OK to proceed

Type in “Business Flow Troubleshooting” in the search field and then select “ORA_FND_BUSINESS_FLOW_TROUBLESHOOTING_ABSTRACT” from the result list

As you see only 1 result line click on “Add Role Membership”

Click on OK and then Done in the main screen for Add Role Membership

To finish these actions you need to click on "Save and Close" in the userscreen:

In the next screen do not forgot to click on "Done" otherwise your changes will not be saved.

Now we are finished with the prerequisites and we are ready to start with the process to enable tracing.

How to set the log level for end user tracing to individual user

We have a set of log levels inside FA (see picture)

If you do not define it the standard value for tracing will be "Severe" which may not be sufficient for Oracle to diagnose a problem.

Your administrator is able to set the value for a individual user to a higher trace level for example "Finest" - this will write more information in the trace files and help Oracle to do a more detailed analysis.

Here are the steps to activate higher trace levels for an individual user. In this example we will again use : Linus.van Pelt - the user that we have created for logging.

As administrator go to "Setup and Maintenance" then click on the "Task" icon on the right side of the screen.

Click on "Search" and type "Manage Administrator Profile Values" in the search field. Then hit the search button

Click on "Manage Administrator Profile Values" and then type "AFLOG_LEVEL" in "Profile Option Code"  and click "Search"

As we do not see our user mentioned here we need to add him. To do that please click on the "+" . That will create a new line. In Profile Level column choose "User" from dropdown menu. Now click on the dropdown sign in "User Name" column. You will see something like this:

Now click on "Search" - put the username that you want to add in the search field

Now click on "Search" 

Click on OK to move your choice to the original screen. Then choose "Finest" from scroll down menu in "Profile Value" column

To finish your actions to set the log level click on "Save and Close" 

Now you are ready to start tracing with log level finest as the user that you have choosen

Do not forgot to put that value back to "Severe" after you have created the trace - otherwise you may create too much log/trace data in former users sessions

Start tracing as user in FA in cloud

Login as the user that will start the trace and locate yourself on the page where you think you have slow answer times. 

Then click on your name on the upper right side of the page to open a popup window with options.

Click on "Record Issue"

Click on "Start Recording"  your page will change like that

Now click on the action inside of that page that you want to trace. If the action has completed click on "Stop Recording" in the yellow bar at the top. After some seconds you while see a new popup window.

As you can see you can add some information in "Additional Information" field but it is ok if you leave it blank. 

Please note the information given in this screen and note the time of the recording as you need to provide this to support. 

Tip : If you do not remember that values and you want to check it later you can again click on your name then click on "Record Issue" but this time you will have to click on "View My Recordings" that will show you a list of your recordings

Open SR and provide the information to Oracle Support

Note : Please open an SR with Oracle Support immediately after generating the data as described above. Make sure to provide all the information described above to support in this new SR. This will allow Oracle Support to save the files generated by you and upload them to the SR before they disappear from your pod.

Enabling auto diagnosis in on-Premise R12

Prerequisites

You will need a FA R12 onPremise environment to use the tips described here. This is at the moment the only supported version.

Actions needed on FA machine

We will discuss the preparations steps that you have to finish on FA machine based on an real example. But this will work similar for other cases.

In general you have two options to make the needed config changes to create diagnosis data automatically on the FA server.

Option one will be to make the changes in the file dms_config.xml.

Option two will be using wlst commands.

Both options will not need a restart and will be picked up directly from the managed servers.

Regardless which of the two options you use The incident generated will have problem key DFW-99995 [Actual URL] or DFW-99995 [slow requests] (if generateIncidentMinutes is configured).

Config changes using dms_config.xml

You can modify the file dms_config.xml to enable the automatic diagnostics creation. This file should be available in all managed servers across FA. You can find it on $Domain_Home/config/fmwconfig/servers/<your_server>.

For example the path will look similar to this:

/u01/app/fa/instance/domains/fahost1.mycompany.com/FinancialDomain/config/fmwconfig/servers/PayableServer_1/dms_config.xml

You will need to make adjustments to the following configuration options:

• requestThresholdSeconds - Number of seconds above which a request is considered to be slow
• generateIncidentMinutes - If not set then an incident will be generated whenever a request is slower than the requestThresholdSeconds (subject to regular incident flood-control rules). If set to X then a thread will wake up every X minutes and check if there is at least one slow request in the past X minutes. If yes it will create an incident and the readme will contain the ECID, URL and response time of all the slow requests
• incidentDumps - this list of dumps to generate when the slow request incident is created. You can pick any syste-scoped dumps listed. Note that if odl.quicktrace dump is requested on an env where Click History is configured, then the click history dump will also be generated
• maxRequestsReport - Only relevant if generateIncidentMinutes is configured. This is the maximum number of request that will be reported in the slow request incident generated in periodic mode. This option is to prevent too much system resources used to track the slow request if there are too many of them
• skipIncidentCount - Skips the first X incidents based on this setting. This can be used to prevent slow requests due to cold server to trigger an incident

Here is an example which shows edited dms_config.xml configuration:

Change

<destination id = "HTTPRequestTrackerDestination"

name = "HTTP Request Tracker Destination" class = "oracle.dms.event.HTTPRequestTrackerDestination" />

to

<destination id = "HTTPRequestTrackerDestination" 

name = "HTTP Request Tracker Destination" 

class = "oracle.dms.event.HTTPRequestTrackerDestination">

<properties>

<property name="requestThresholdSeconds" value="10"/>

<property name="generateIncidentMinutes" value="20"/>

<property name="incidentDumps" value="jvm.flightRecording,odl.logs,odl.quicktrace,dms.metrics"/>

<property name="maxRequestsReport" value="100"/>

<property name="skipIncidentCount" value="0"/>

</properties>

</destination>

Config Changes using wlst commands

The wlst commands mentioned here will only work if you use wlst.sh script provided in soa. So make sure to use this one for all wlst commands mentioned here. Your command should look similar to the following :

/u01/app/fa/fusionapps/soa/common/bin/wlst.sh

Alternatively WLST commands shown below can be used to configure the settings

updateDMSEventDestination(id='HTTPRequestTrackerDestination', props={'requestThresholdSeconds' : '20'}, server="<server>") - Generate an incident on detection of any request that takes longer than 20 seconds, subject to incident flood control.
updateDMSEventDestination(id='HTTPRequestTrackerDestination', props={'requestThresholdSeconds' : '20', 'generateIncidentMinutes' : '15'}, server="<server>") - Generate an incident after 15 minutes if any requests have taken longer than 20 seconds during that period.
updateDMSEventDestination(id='HTTPRequestTrackerDestination',   props={'requestThresholdSeconds' : '20', 'generateIncidentMinutes' : '15', 'incidentDumps' : 'jvm.flightRecording,dms.metrics'}, server="<server>") - Generate an incident after 15 minutes if any requests have taken longer than 20 seconds during that period, with the incident including only a JFR and DMS metrics dump.

To check if the configuration is picked up, use these WLST commands

listDMSEventFilter(server="server_name")
listDMSEventRoutes(server="server_name")
listDMSEventDestination(server="server_name")

Here is an example how to use these listDMSEvent* commands:

/u01/app/fa/fusionapps/soa/common/bin/wlst.sh

wls:/offline> connect()

Please enter your username :faadmin

Please enter your password :

Please enter your server URL [t3://localhost:7001] :t3://localhost:7401

Connecting to t3://localhost:7401 with userid faadmin ...

Successfully connected to Admin Server 'AdminServer' that belongs to domain 'FinancialDomain'.



Warning: An insecure protocol was used to connect to the

server. To ensure on-the-wire security, the SSL port or

Admin port should be used instead.

wls:/FinancialDomain/serverConfig> listDMSEventFilter(server="PayableServer_1")

Location changed to domainRuntime tree. This is a read-only tree with DomainMBean as the root.

For more help, use help(domainRuntime)

   Id                                   Name

   JFRFilter                            JFRFilter


wls:/FinancialDomain/serverConfig> listDMSEventRoutes(server="PayableServer_1")

Location changed to domainRuntime tree. This is a read-only tree with DomainMBean as the root.

For more help, use help(domainRuntime)

   Filter        :  None

   Destination   :  HTTPRequestTrackerDestination

   Enabled       :  true

   Filter        :  None

   Destination   :  mbeanCreationDestination

   Enabled       :  true

   Filter        :  JFRFilter

   Destination   :  JFRDestination

   Enabled       :  true


wls:/FinancialDomain/serverConfig> listDMSEventDestination(server="PayableServer_1")

   Id            : JFRDestination

   Name          : JRockit Flight Recorder Proxy

   Id            : HTTPRequestTrackerDestination

   Name          : HTTP Request Tracker Destination

   Id            : mbeanCreationDestination

   Name          : MBean Creation Destination

   Id            : LoggerDestination

   Name          : Default Logger Destination

Starting an Explicit JFR Recording

You can also start the flight recording for the given <pid> of the managed server e.g PayableServer_1 directly on the FA machine using:

jrcmd <pid> start_flightrecording duration=30m filename=myrecording.jfr

You will need to identify the ECID to complete the actions needed on the FA machine to sample the diagnostics data. The next chapter will explain how to get this information.

Actions on Client Machine

We will explain the needed actions using Chrome. You can find similar tools for other browsers using Google if you prefer to use a different browser.

On Chrome enable the "Developer Tools" and select the network tab.

Click on the "Network" tab to monitor network traffic in your browser, in the window pane below under the "name" section you will start to see network traffic being recorded for every page load and click

Once you have reproduced the slow click, select the click under the "name" section. In the right-hand pane click on the "timing" tab where the time taken for the click/page load is recorded

Still in the right-hand pane click on the "Headers" tab and search for the text "X-ORACLE-DMS-ECID=" and make a note of the full string

Right click on the selected click under the "Name" section, save the output and email this to your dba group including your username, timing and ECID

Collect Server Logs (finishes actions on FA machine)

As we now have identified the ECID of our request we can run the following commands to identify any server, diagnostics, access logs for the given ECID

cd <app_base>/instance/domains/<hostname>/<domain>/servers/<server_name>/logs

grep -il <ECID> *log*

This will complete the actions on the FA machine.

Actions on database

Run SQL Statements

Get the SQL_ID and SQL execution times for your <ECID>

select distinct a.inst_id,a.sql_id,b.executions,(b.elapsed_time/executions/1000/1000) elapsed_time from gv$active_session_history a,gv$sql b where a.sql_id=b.sql_id and a.force_matching_signature=b.force_matching_signature and a.sql_child_number=b.child_number and a.inst_id=b.inst_id and a.ecid = <ECID>;

Where the <SQL_ID> is identified from the first query

select distinct a.ecid,a.inst_id,a.sql_id,b.executions,(b.elapsed_time/executions/1000/1000) elapsed_time from gv$active_session_history a,gv$sql b where a.sql_id=b.sql_id and a.forcme_matching_signature=b.force_matching_signature and a.sql_child_number=b.child_number and  a.inst_id=b.inst_id and a.sql_id=<SQL_ID>;

AWR Report

Provide the AWR report for the time of the incident

Conclusion

Using the informations given here you should be able to generate diagnosis data for slow clicks in a page in FA regardless if you are working in cloud or onPremise.

As we have shown in this document you are able to achieve that using any user in the cloud and you are not limited to use some account that has admin rights.

The data that you have generated using the steps explained above should be given directly to Oracle support so that Oracle can analyze the data and provide solutions for the problem.