Avoiding LibOVD Connection Leaks When Using OPSS User and Role API

The OPSS User and Role API (oracle.security.idm) provides an application with access to identity data (users and roles), without the application having to know anything about the underlying identity store (such as LDAP connection details).

For new development, we no longer recommend the use of the OPSS User and Role API – use the Identity Governance Framework (IGF) APIs instead. However, if you already have code which uses the User and Role API, that code is still supported for the time being.

The OPSS User and Role API supports both LDAP and non-LDAP backends (such as XML files); however, for production use, LDAP backends are the most common, and are strongly recommended. When using an LDAP backend, the OPSS User and Role API retrieves the LDAP connection details from the security providers defined in the WebLogic Security Realm.

Now there are two different modes in which the OPSS User and Role API may operate – virtualize=true, in which all LDAP access is via LibOVD; and virtualize=false, in which LibOVD is not used and the User and Role API’s own LDAP access code is used instead. The difference is that without LibOVD, the User and Role API can only access a single LDAP server (the first defined in the WebLogic security realm); whereas, with LibOVD enabled, the User and Role API uses LibOVD to join the users and roles of two or more LDAP servers into one. A common use case is where you have different LDAPs for different user communities (for example, customers might have user accounts in OUD, while employees might instead be stored in Active Directory).

When using the OPSS User and Role API with LibOVD, you need to be careful not to cause connection leaks. The most common symptom of a connection leak is this error: “NamingException: No LDAP connection available to process request for DN”

Connection leaks can be caused through the use of any of the methods of oracle.security.idm.IdentityStore which return an oracle.security.idm.SearchResponse object. For example, the search(oracle.security.idm.SearchParameters) method. Calling any of these methods, when the SearchResponse object is created, it is associated with one or more backend LDAP connections in the LibOVD connection pools. These connections will not be returned to the pool until one of the following occurs:

• you iterate until the end of the response – i.e. call hasNext() until it returns false
• you call the close() method on the response

If you do not do either of the above, you will leak LibOVD backend LDAP connections. The JavaDoc for the close() method points this out:

Closes this response object and frees all resources associated with it. After close() is called the response object becomes invalid. Any subsequent invocation of its methods will yield undefined results. Close() should be invoked when discarding a response object so that resources are freed. If a response proceeds to its natural end (i.e. when hasNext() returns false) then close() will be called internally. There is no need to call close() explicitly in such case.

Some people decide to use netstat, or packet capture tools such as Wireshark or tcpdump, to try to diagnose this issue. Now, in itself, that is not a bad line of thought; however, you may not be able to detect this type of leak with those tools. That is because, after some time (depending on your configuration), either the LDAP server or else some middle box such as a load balancer may shutdown the underlying network socket; however, the in-memory structure which holds the network socket will still be checked out of the connection pool. There is a cleaner thread which runs inside LibOVD, which clears out closed network sockets from the pool; however, if a connection is marked as in use, the cleaner thread will not remove it from the pool, even though the underlying network socket is closed. In this case, you will not see any build up of actual network connections, even as your LibOVD connection pools are becoming exhausted. One symptom you may see, however, is lack of reuse of network connections.

Eventually, when the pool is completely filled with leaked connections, firstly all clients will hang for the maximum pool wait time, then “NamingException: No LDAP connection available to process request for DN” will occur. At this point, LibOVD may (depending on the version) decide to shutdown and restart the connection pool, thus recovering from the problem. However, the connection leak will continue, and the hang followed by errors will occur again in the future.

All site content is the property of Oracle Corp. Redistribution not allowed without written permission

Note: The above document is from our A-Team chronicles page and not official Oracle documentation:

http://www.ateam-oracle.com/idm/