Remedy to Oracle SOA em Console Slowness / Timeout Issues

Introduction

EM console is slow when accessing to view the composite/component details or the instance details as soon as your Oracle SOA database grows.

Here are some remedies which I used to fine tune the Oracle SOA em console performance.


1. Disabling Metrics :

         Disabling metrics will give a dramatic performance on loading composites. This is up to you to configure 10 or 30 mins. But definitely 24 hours will take ages to load.

• Login to em right click soa-infra > SOA Administration > common properties
• It will take you to "SOA Infrastructure Home > Common Properties"Disable the metrics by selecting the check box, "Disable fetching of instance and fault count metrics. Each metric can still be retrieved on demand."  -  By Default fault count metrics is enabled.

• Restrict the display of instances and faults to minutes from hours.

2. Disabling Audit Recovery Status

         Every em console login will pull all the statistics of the BPEL message recovery status. Follow the below steps and disable the recovery status.

• Login to em right click soa-infra > SOA Administration > common properties

• Click on "More SOA Infra Advanced Configuration Properties..."

• Select soa-infra and on the right side click on AuditConfig.

• The default value is 'All'  and disabling the same by entering 'Off' (case - sensitive). Above change will disable 'BPEL Message Recovery Required'

3. Disabling JMX Notifications

         Disable the Managed Server JMX Notifications. This configuration disables the ability to define notifications on MBeans that are contained in the Managed Servers Runtime MBean Servers. This can be done by two ways. 

• By using the Admin server start up parameter like below.

                    -Dweblogic.management.disableManagedServerNotifications=true

• By using WLST

                   cd("JMX/domain-name")
                    cmo.setManagedServerNotificationsEnabled(false)


   This parameter will disable all JMX notification which occur between Admin and Managed servers in a cluster whenever any new component get added, any state get changes etc. This parameter is suggested by Oracle 'A' team. This gave us a huge benefit.

4. Oracle Patches

                    Oracle Support team released some em performance / stability patches from 11.1.1.5.

• Bug 11907158 - LARGE RETURNED DATA SETS CAN HANG EM AND SOMETIMES REQUIRE A RESTART OF EM.
• Bug 13407044 - PSR:PERF:SOA: EM-SOA EXPENSIVE SQLS IN SOA PAGES BASED ON CREATED_TIME PREDICATE.
• Patch 12972254 - ADMIN SERVER MEMORY UNDER FEDERATEDMBEANSERVERDELEGATE.
• Unpublished BUG 9701533 - EM PERFORMANCE IS VERY VERY SLOW.

      Above patches are just for your reference. Please confirm with Oracle Support team before applying as this may be available out of the box in your release or not be applicable for your environment. A big thanks to Oracle 'A' team and Support team for helping us to resolve the issue.

5. Enabling Discovery Cache

            Enterprise Manager Fusion Middleware Control 11g discovery is always performed as part of login.  For installations with thousands of targets, Fusion Middleware Control discovery may take around 1 minute. This time is expected because EM discovery mbeans need to be invoked for every target. Oracle provided a solution to cache the discovery results in the servlet context and use it for subsequent logins. This discovery result will be shared by all the fmwc users. This will still require the entire discovery to be done at least once.  If the initial login is slow due to discovery of large number of targets, changes in this, will not reduce the initial login delay but subsequent login will be fast. By default cache is not enabled.

In FMW 11.1.1.6 (PS5), new parameters are added to enable/disable and to control use of the cached discovery results. Patches are available for some previous FMW releases.

If the caching is enabled, fmwc will use the cached discovery results. The default setting is "not use the cached results".

# Values=true/false   Default=false
oracle.sysman.emas.discovery.wls.FMW_DISCOVERY_USE_CACHED_RESULTS=true
# Time value is in milliseconds.  Default is 7200000 milliseconds.
oracle.sysman.emas.discovery.wls.FMW_DISCOVERY_MAX_CACHE_AGE=7200000 
# Time value is in milliseconds.  Default is 10000 milliseconds.
oracle.sysman.emas.discovery.wls.FMW_DISCOVERY_MAX_WAIT_TIME=10000

• DISCOVERY_MAX_CACHE_AGE - You can change this based on your need, we have updated this to 10 hours (36000000 milliseconds).
• DISCOVERY_MAX_WAIT_TIME - You can change this based on your need, we have updated this to 15 seconds (15000 milliseconds).

Below commands will update the values using WLST. You can implement the same using mbean browser as well.

# Change location to domain custom tree
domainCustom()

# Setting variables
myMBean=ObjectName('emoms.props:Location=AdminServer,name=emoms.properties,type=Properties,Application=em')
types=['java.lang.String', 'java.lang.String']
type=['java.lang.String']

# Set property oracle.sysman.emas.discovery.wls.FMW_DISCOVERY_USE_CACHED_RESULTS=true
params=['oracle.sysman.emas.discovery.wls.FMW_DISCOVERY_USE_CACHED_RESULTS', 'true']
mbs.invoke(myMBean,'setProperty',params,types)

# Set property oracle.sysman.emas.discovery.wls.FMW_DISCOVERY_MAX_CACHE_AGE=36000000
params=['oracle.sysman.emas.discovery.wls.FMW_DISCOVERY_MAX_CACHE_AGE','36000000']
mbs.invoke(myMBean,'setProperty',params,types)

# Set property oracle.sysman.emas.discovery.wls.FMW_DISCOVERY_MAX_WAIT_TIME=15000
params=['oracle.sysman.emas.discovery.wls.FMW_DISCOVERY_MAX_WAIT_TIME','15000']
mbs.invoke(myMBean,'setProperty',params,types)

Note: This can be modified from mbean browser as well.

You can use the setProperty and getProperty to configure and validate the changes.

Note: After enabling discovery cache if you deploy a new composite (or any change in the terget requires Manual refresh) the new composite won't be displayed on FMW Control, perform a manual refresh of the Farm to update the discovery cache.

6. Monitoring Database

                 Housekeeping is very important for Oracle SOA database as it grows everyday in production. Deleting old instances and rebuilding the indexes regularly will help you to keep the database size and performance under control. Proactively monitor Oracle SOA database with the help of AWR and trouble shoot the issues. Following types of statistics need to be collected on regular basis

• Table statistics
• Column statistics
• Index statistics
• System statistics
• Fixed object statistics

                It is recommended to use the default auto stats collection job for all SOA database for stats collection. In case data are being modified / purged which result in significant change to the datashape, manually stats collection is necessary. When investigating slow SQL, use SQL trace or capture execution plan for detailed analysis, for example, check if the sub-optimal operation is related to stale / unrepresentitive stats. Sometimes, SQL performance suffer from change in execution plan all of a sudden, this is all because CBO picked up a new plan that is may not be optimal. The change in execution plan can occur due to stale statistics, changes in data volume or bind peeking. The goal of SQL plan baselines is to preserve the performance of corresponding SQL statements, regardless of changes in the database. A SQL plan baseline contains one or more accepted plans, each of which contains the following information: Set of hints, Plan hash value, and Plan-related information.

              Alternatively try "Partitioning Methodology" as well, I didn't get a chance to implement / use partitioning. But Oracle document says this is another way of keeping the things under control. Here is the link Purging and Partitioning Methodology.

Warning:  

                                      I encourage you to test and experiment with these changes in your development environment before you implement. As some suggestions / solutions may not be applicable to your environment.