The number one cause of mystery login problems that I see with customers using LDAP authenticators (be it OID, Sun, or AD) relates to a problem with the search done to determine what groups the user is a member of.
As part of the authentication process, LDAP authenticators do a search to determine what groups the user is a member of which in turn get used in determining the group memberships and roles for the JAAS subject and principals. If there is a problem with this search, then WebLogic will fail the entire authentication even if the user authentication check (username + password) against the directory was successful. Notice that I said problem with the search and not search failure. If the user simply doesn’t exist in any groups then the authentication will succeed. The issue is with the authenticator not even being able to successfully execute the search.
The group search failure can be cause by a couple different things and can be a very nasty situation to recognize and sort out.
Search Configuration Failures
The more straight forward cause of a group search failure is that the search itself. This can be cause by having a bad “Group Base DN” or “All Groups Filter” in the authenticator configuration. If for example you mistyped part of the base DN or the object class in the filter, then the search will fail to execute. What you’ll see in the logs is a message saying authentication has succeeded then one or more error messages about the group search, and then another message saying authentication has failed.
Problems with Nested Groups
The more insidious cause of group search failures is with nested groups. By default the authenticator will search not only what groups a user belongs to, but will go on to search what groups those groups containing the user belong to and then what groups those groups belong to and on and on…
If there are a ton of groups with lots of nesting, the authenticators can time out just in processing nested groups. However, the bigger or more common issue is that authenticators can get in infinite loops processing two groups that contain each other or certain dynamic groups.
The best way to prevent this is to limit the levels of group membership nesting that the authenticator will follow to build up the JAAS subject and principals. You do this by changing “Group Membership Searching” from unlimited to limited and changing “Max Group Membership Search Level” to the desired level of nesting you want to pursue. Leaving it at 0 will mean that the authenticator will not process any nested groups.
My recommendation is that as a best practice you should almost always switch to a limited search even if you want the authenticator to heavily process nested groups. Setting the level of nesting to something high like 5 will almost always give you all the memberships/roles you need but will limit your exposure to infinite loop situations caused by problems in the directory.