LibOVD - How to Use and How to Tune

Introduction: 

LibOVD is a mini version (provides limited features) of OVD (Oracle Virtual Directory). This java library provides virtualization capabilities over LDAP authentication providers in Oracle Fusion Middleware.


Authentication and User/Role Search

Authentication: Fusion Middleware application will use the configured LDAP providers directly for authentication based on control flag. All the configuration parameters like connection pool and timeouts can be configured in the provider configuration which can be viewed/edited on the weblogic -> Providers -> Select Provider -> Provider Specific.

User/Role API search: The user and role API framework allows applications to access identity information (users and roles) in a uniform and portable manner regardless of the particular underlying identity repository. The underlying identity store could be an LDAP directory server such as Oracle Internet Directory.This API framework provides a convenient way to access repositories programmatically in a portable way, freeing the application developer from the potentially difficult task of accounting for the intricacies of particular identity sources. The framework allows an application to work against different repositories seamlessly. An application can switch between various identity repositories without any need for code changes.

What is OPSS Limitation ?

                  Oracle Platform Security Services (OPSS) had a limitation of only one authentication provider on Weblogic. It supports only users / groups from the very first authentication provider in the list. This restriction was removed from 11.1.1.4 onwards. Now you can configure multiple providers by setting the virtualize flag in the idmstore instance.

Note : Oracle Fusion document says another use case for libOVD is split profile where you can keep the user profile in multiple LDAP stores. For more information please read "Configuring Split Profiles"

How libOVD provides solution to OPSS limitation ?

                  The Identity Virtualization feature enables you to query in multiple LDAP directories through OPSS. This java library (Virtualization) creates one view on top of all providers and query on the same. Once you enable Virtualization on Oracle Fusion Middleware, all configured weblogic providers will act as an adapter inside one view.

How to Enable libOVD on Oracle Fusion Middleware

• Login to Oracle Fusion Middleware console and right click on the domain.
• Click on security and then click on Security Provider Configuration. - Below screenshot shows how to navigate.

• Click on configure

• On the Right side panel of Custom Properties, click on Add and enter "virtualize=true".
• The parameter is: virtualize(lower-case) and the value is: true (lower-case).

Above changes will update the $DOMAIN_HOME/config/fmwconfig/jps-config.xml file.

• In the $DOMAIN_HOME/config/fmwconfig/ovd/default/ directory you can see the below files (libOVD config files).

• Click on Ok and restart the domain (All Managed / Admin server).

Note: Now you have enabled libOVD successfully and the real pain starts now. Every then and there you will see "No LDAP Connection" Error.  Let's see how to get rid of this now.

javax.naming.NamingException: No LDAP connection available to process request for DN: cn=Admin.                           
at oracle.ods.virtualization.engine.backend.jndi.JNDIConnectionPool.checkOutContext(JNDIConnectionPool.java:218)        
at oracle.ods.virtualization.engine.backend.jndi.BackendJNDI.getLDAPContext(BackendJNDI.java:1030)                        
at oracle.ods.virtualization.engine.backend.jndi.BackendJNDI.getConnection(BackendJNDI.java:931)                            
at oracle.ods.virtualization.engine.backend.jndi.ConnectionHandle.getHolder(ConnectionHandle.java:416)                      

Problems with libOVD

                   When you use "virtualize=true" to support multiple providers, this makes many ldap calls; which in turn, impacts performance. Since libOVD always searches in the root, unnecessary queries are issued by OPSS and other layers. In adapters.os_xml, every adapter maintains a root element which is the common base of users and groups of every provider. Example, your provider user dn : ou=users,ou=ad,dc=state,dc=com,dc=us and group dn : ou=groups,ou=ad,dc=state,dc=com,dc=us then your libOVD root element will be ou=ad,dc=state,dc=com,dc=us. Even the inclusion / exclusion filters will not work as OPSS and other layers before libOVD triggers these searches and libOVD can't control based on filters. Also we have noticed from the logs and monitoring tool it closes the LDAP connection after the timeout value which is maintained for every adapter in adapters.os_xml.

How to Tune libOVD

                    OPTIMIZE_SEARCH is the default performance tuning parameter for libOVD.  It forces libOVD to search only within the users and groups search bases defined in the Authentication providers instead of root which is maintained in adapters.os_xml. This parameter requires a FMW patch till 11.1.1.6. Below patch is available out of the box on version 11.1.1.7.

The parameter is: OPTIMIZE_SEARCH (upper-case) and the value is: true (lower-case).

Patch Details: Patch 13791927: ATEAM: QUERIES FOR USERS TO EXTERNAL LDAP USE DIRECTORY ROOT AND WHOLESUBTREE.

Note: We are not able to find any Oracle document related to "OPTIMIZE_SEARCH", We got this parameter from Oracle support team. Whenever, you enable virtualize, go for 'OPTIMIZE_SEARCH'.

This should be add in the way you have added virtualize=true (from em console or editing the jps-config.xml).

    em console

     From jps-config.xml

1. Increasing the initialPoolSize based on your requirement / load. (We tuned to 50)
2. Increasing the maxPoolSize based on your requirement / load. Also monitor the CPU load of the backend LDAP, while you increase your pool size. (We tuned to 250)
3. Tune maxPoolWait and maxPoolChecks based on your exception handling scenario as increasing this value will cause slowness in your application during peak load.
4. If your application runs on 11.1.1.7 or higher you can configure "reuseAddress" (adapters.os_xml) to true, if your back end LDAP communication is on non-ssl. This will reuse all the sockets which is in TIME_WAIT state to process new requests. You can check the same using "netstat -na | grep 3060" this will give you the list of sockets in TIME_WAIT and in ESTABLISHED state.
5. In case, your domain has more authentication providers but your application requires to search only in couple of providers then disable the unwanted adapters using active element <active>false</active> as you can't delete an adapter as libOVD recreates on server start up.
6. Make sure that you don't have any overlapping root in your libOVD config file (adapters.os_xml). Example, adapter one root is "ou=ad,dc=state,dc=com,dc=us" and adapter two root is "dc=state,dc=com,dc=us".
7. If you are in libOVD 11.1.1.7.1 then configure inactive connection pool timeout to 1 second more the timeout. Instance, if your timeout is 10 seconds configure 11 seconds as inactive connection pool timeout.
8. libOVD will issue a search query even to Default Authenticator which will end up opening too many sockets on you Admin Server port. So actively monitor your Admin Server open sockets. If not required, disable the Default Authenticator adapter from adapters.os_xml. Otherwise upgrade the libOVD to 11.1.1.7.1 and make use of inclusion/exclusion filter support.

More Bug fixes from Oracle for libOVD issues. Thanks to Oracle A team and Support team for addressing the issues quickly.

16505847 LIBOVD COULD HAVE PERFORMANCE ISSUE
17860953 LIBOVD SENDS SEARCH PAGE CONTROLS DURING BIND WHEN ECID CONTROL NOT SEND
18276672 IDS: AUTHENTICATION ALWAYS FAILS WHEN SEARCH DN IN AN ENTITY IS EMPTY STRING
18900021 STRESS:SIMV3:IDSAPI- ENTITY.GETGUID() THROWS NO LDAP CONN. EXCEPTIONS UNDER LOAD
18037580 MAXIMUM OID CONNECTIONS LIMIT HIT UNDER STRESS TEST
12545418 SUPRESS UNNECESSARY WARNINGS WHEN ROLES ARE FETCHED FROM MULTIPLE LDAPS
12978228 NEED SUPPORT FOR SPECIFYING DYNAMIC ATTRIBUTES IN IDS
12978413 IDS LDAP FILTER SUPPORT
14366006 IDS: NEED CONFIGURABLE INACTIVE CONNECTION TIMEOUT IN LIBOVD CONNECTION POOL
14538213 IDS: ENTITIES TO BE SERIALIZABLE
14538226 IDS: CONSTANTS FOR ALL OOTB USER AND GROUP ATTRIBUTE LOGICAL NAMES
14589983 DO NOT HAVE OPTION TO ADDPLUGIN WITHOUT ANY PARAMETER
14787407 IDS: NEED TO BE ABLE TO UPDATE/OBTAIN OBJECTCLASS FOR USER

All above listed bugs are fixed and included in below mentioned patches.

Patch 17391582 to upgrade libOVD to 11.1.1.7.1
Patch 19939451 on top of 11.1.1.7.1 to get the bug fix

                 After applying the above patches you can configure inactive connection pool timeout and filters support in libOVD.

Note:  Also, please confirm with Oracle Support before deploying any PATCH into your environment, this given information is just for reference and may not be applicable for your environment or this may be available out of the box in your release.

                 Even after applying the patches and tuning parameters, actively capture the tcp packets and monitor the number of LDAP searches getting triggered for one search. Use some monitoring tool or enable the libOVD logs to monitor the connections closure duration.

Changing libOVD configuration

•    Through WLST  - help('OracleLibOVDConfig') or WLST reference

•    Through System MBean Browser.

How to Debug / Trouble Shoot LIB-OVD

1. Edit logging.xml which is under $DOMAIN_NAME/config/fmwconfig/servers/${server_name}/logging.xml and add the below entry.

          <logger name='oracle.ods.virtualization' level='TRACE:32'>
               <handler name='ods-handler'/>
          </logger>

        <log_handler name='ods-handler' class='oracle.core.ojdl.logging.ODLHandlerFactory' level='TRACE:32'>
                 <property name='supplementalAttributes' value='J2EE_APP.name,J2EE_MODULE.name,WEBSERVICE.name,WEBSERVICE_PORT.name'/>
                 <property name='format' value='ODL-Text'/>
                 <property name='path' value='/var/logs/ods.log'/>
                 <property name='maxFileSize' value='10485760'/>
                 <property name='maxLogSize' value='104857600'/>
           </log_handler>

     2. Enabling TCP dump and capture the packets. Which will help you to find number of LDAP searches triggered by libOVD to back end LDAP.

                               "# /usr/sbin/tcpdump -n dst port 3501. "

Note: 3501 is the back end LDAP port. Use some tool like wire shark to analyse the tcp packets.

     3. This java library is located in $ORACLE_COMMON/modules/oracle.ovd_11.1.1/ovd.jar. You can use a de-compiler to debug the code.

Conclusion:

   Avoid using libOVD, until it is really required for your custom application. Instead introduce an OVD(Oracle Virtual Directory) and join all required adapters and restrict unwanted searches with the help of inclusion / exclusion filters. It took months for us to sort this "No LDAP connection" issues.

Warning:  

                            I encourage you to test and experiment with these changes in your development environment before you implement. As some suggestions / solutions may not be applicable to your environment.