Introduction

In Part 3 of 4 – SSSD Linux Authentication: Implementation Step-by-Step Guideline I covered all the necessary step-by-step details on deploying SSSD, but nothing ever seems to go perfect the first time does it.  This is why I have included a final Part 4 that covers known problems I came across, though there could be more, and then troubleshooting tips I learned.  These are all things I learned and want to share with you so that your SSSD implementation is success, but I know there is a lot more out there if you search so don't hesitate to look beyond this article.

Known Problems

The following are known issues that have been discovered. There are no work arounds that I am aware of, so other options need to be used if they are important.

Command “passwd” fails for Active Directory and OID11g

The command “passwd” is used to allow a user or root to change the password. There is a known issue with SSSD using Active Directory 2012 or older and Oracle Internet Directory 11g where executing the passwd command will fail. The issue is during the execution of passwd, SSSD first tries to get password information by querying the LDAP server for a password policy control OID 1.3.6.1.4.1.42.2.27.8.5.1, and second tries to use a LDAP extended feature “Password Modify extended operation” --- OID 1.3.6.1.4.1.4203.1.11.1. Unfortunately, neither Active Directory 2012 or older and OID11g and older do not support these extended LDAP features though with OID11g these features are marked as draft so it may be possible this will be added in the future. There are no known work arounds, so password changes will need to be completely managed in the LDAP server or some other external process.

Example OID11g passwd error –
Below is an example error a user in Oracle Internet Directory will see when trying to use the “passwd” command at the Linux terminal prompt.

[adina@oel6 ~]$ passwd
Changing password for user adina.
Current Password:
New password:
Retype new password:
Password change failed. Server message: Server is Configured to Deny Unauthenticated Binds
passwd: Authentication token manipulation error

Below is a snippet of SSSD logs taken from sssd_OID.log at a debug level 9 while trying to use “passwd” to change a password.

(Tue May 9 11:45:51 2017) [sssd[be[OID]]] [sdap_control_create] (0x0080): Server does not support the requested control [1.3.6.1.4.1.42.2.27.8.5.1].
(Tue May 9 11:45:51 2017) [sssd[be[OID]]] [sdap_exop_modify_passwd_send] (0x0100): Executing extended operation
(Tue May 9 11:45:51 2017) [sssd[be[OID]]] [sdap_exop_modify_passwd_send] (0x2000): ldap_extended_operation sent, msgid = 2
(Tue May 9 11:45:51 2017) [sssd[be[OID]]] [sdap_op_add] (0x2000): New operation 2 timeout 6
(Tue May 9 11:45:51 2017) [sssd[be[OID]]] [sdap_process_result] (0x2000): Trace: sh[0x21b9a90], connected[1], ops[0x21b0020], ldap[0x219b8d0]
(Tue May 9 11:45:51 2017) [sssd[be[OID]]] [sdap_process_result] (0x2000): Trace: ldap_result found nothing!
(Tue May 9 11:45:51 2017) [sssd[be[OID]]] [sdap_process_result] (0x2000): Trace: sh[0x21b9a90], connected[1], ops[0x21b0020], ldap[0x219b8d0]
(Tue May 9 11:45:51 2017) [sssd[be[OID]]] [sdap_process_message] (0x4000): Message type: [LDAP_RES_EXTENDED]
(Tue May 9 11:45:51 2017) [sssd[be[OID]]] [sdap_exop_modify_passwd_done] (0x0200): Server returned no controls.
(Tue May 9 11:45:51 2017) [sssd[be[OID]]] [sdap_exop_modify_passwd_done] (0x0080): ldap_extended_operation result: Protocol error(2), Server is Configured to Deny Unauthenticated Binds
(Tue May 9 11:45:51 2017) [sssd[be[OID]]] [sdap_op_destructor] (0x2000): Operation 2 finished
(Tue May 9 11:45:51 2017) [sssd[be[OID]]] [be_pam_handler_callback] (0x0100): Backend returned: (3, 20, <NULL>) [Internal Error]

Example Active Directory passwd error –

Below is an example error a user in Active Directory will see when trying to use the “passwd” command at the Linux terminal prompt.

-sh-4.1$ passwd
Changing password for user chad.
Current Password:
New password:
Retype new password:
Password change failed. Server message: 0000203D: LdapErr: DSID-0C090EE8, comment: Unknown extended request OID, data 0, v2580
passwd: Authentication token manipulation error

The follow are SSSD logs from sssd_AD.log at a debug level of 9 that illustrate the error “LdapErr: DSID-0C090EE8, comment: Unknown extended request OID, data 0, v2580”, which shows up just after the Password Modify extended operation is attempted without success. Unfortunately there are no work arounds so passwords will need to be managed in Active Directory.

(Tue May 9 15:53:02 2017) [sssd[be[AD]]] [find_password_expiration_attributes] (0x4000): No password policy requested.
(Tue May 9 15:53:02 2017) [sssd[be[AD]]] [simple_bind_send] (0x0100): Executing simple bind as: CN=Chad Eric,CN=Users,DC=cloud,DC=acme,DC=com
(Tue May 9 15:53:02 2017) [sssd[be[AD]]] [simple_bind_send] (0x2000): ldap simple bind sent, msgid = 1
(Tue May 9 15:53:02 2017) [sssd[be[AD]]] [sdap_op_add] (0x2000): New operation 1 timeout 6
(Tue May 9 15:53:02 2017) [sssd[be[AD]]] [sdap_process_result] (0x2000): Trace: sh[0x25ce960], connected[1], ops[0x2692970], ldap[0x267d490]
(Tue May 9 15:53:02 2017) [sssd[be[AD]]] [sdap_process_message] (0x4000): Message type: [LDAP_RES_BIND]
(Tue May 9 15:53:02 2017) [sssd[be[AD]]] [simple_bind_done] (0x1000): Server returned no controls.
(Tue May 9 15:53:02 2017) [sssd[be[AD]]] [simple_bind_done] (0x0400): Bind result: Success(0), no errmsg set
(Tue May 9 15:53:02 2017) [sssd[be[AD]]] [sdap_op_destructor] (0x2000): Operation 1 finished
(Tue May 9 15:53:02 2017) [sssd[be[AD]]] [sdap_auth4chpass_done] (0x1000): user [CN=Chad Eric,CN=Users,DC=cloud,DC=acme,DC=com] successfully authenticated.
(Tue May 9 15:53:02 2017) [sssd[be[AD]]] [sdap_control_create] (0x0080): Server does not support the requested control [1.3.6.1.4.1.42.2.27.8.5.1].
(Tue May 9 15:53:02 2017) [sssd[be[AD]]] [sdap_exop_modify_passwd_send] (0x0100): Executing extended operation
(Tue May 9 15:53:02 2017) [sssd[be[AD]]] [sdap_exop_modify_passwd_send] (0x2000): ldap_extended_operation sent, msgid = 2
(Tue May 9 15:53:02 2017) [sssd[be[AD]]] [sdap_op_add] (0x2000): New operation 2 timeout 6
(Tue May 9 15:53:02 2017) [sssd[be[AD]]] [sdap_process_result] (0x2000): Trace: sh[0x25ce960], connected[1], ops[0x2686780], ldap[0x267d490]
(Tue May 9 15:53:02 2017) [sssd[be[AD]]] [sdap_process_result] (0x2000): Trace: ldap_result found nothing!
(Tue May 9 15:53:02 2017) [sssd[be[AD]]] [sdap_process_result] (0x2000): Trace: sh[0x25ce960], connected[1], ops[0x2686780], ldap[0x267d490]
(Tue May 9 15:53:02 2017) [sssd[be[AD]]] [sdap_process_message] (0x4000): Message type: [LDAP_RES_EXTENDED]
(Tue May 9 15:53:02 2017) [sssd[be[AD]]] [sdap_exop_modify_passwd_done] (0x0200): Server returned no controls.
(Tue May 9 15:53:02 2017) [sssd[be[AD]]] [sdap_exop_modify_passwd_done] (0x0080): ldap_extended_operation result: Protocol error(2), 0000203D: LdapErr: DSID-0C090EE8, comment: Unknown extended request OID, data 0, v2580
(Tue May 9 15:53:02 2017) [sssd[be[AD]]] [sdap_op_destructor] (0x2000): Operation 2 finished
(Tue May 9 15:53:02 2017) [sssd[be[AD]]] [be_pam_handler_callback] (0x0100): Backend returned: (3, 20, <NULL>) [Internal Error]

UID and GID Collisions

UID/GID numbers are how UNIX/Linux systems keep track of file and process ownership. As pointed out earlier in section “SSSD --- UID and GID Numbers”, UNIX/Linux expects that each user has a unique uid ID or uidNumber. A group ID or gidNumber on the other hand can be shared between users as that can grant access to shared files or folders. Where we run into issues are when configuring SSSD with multiple LDAP identity stores and each LDAP store could potentially share the same user or group IDs by accident. Unfortunately, duplicate group IDs can get into security issues because SSSD does not validate that GID numbers are unique across different identity stores. If two group entries are assigned the same ID number, only the first entry is returned in a search for that ID number.

Similar, a user ID, or uidNumber, uniquely identifies a user according to the UNIX operating system. If two people have the same username, jsmith and jsmith, according to UNIX, as long as each user has a unique uid number, say 10001 and 10002, the UNIX operating system will consider them unique and be happy. However, if say jsmith and akhan both have the same uid number 10001, that is a problem. Both jsmith or akhan can then access each other files and folders since they share the same uid number, the UNIX operating system considers both users the same. This is not an issue with SSSD, it is how the UNIX operating system works.

So, if SSSD points to a single LDAP identity store, and GID and UID numbers are managed so they are unique, there will be no issues for users and groups on the UNIX system. However, since in this implementation SSSD is configured for multiple LDAP identity stores, there will be issues since SSSD does not try to determine unique uid numbers at login, nor prevent collisions between user uid numbers between disparate identity stores. Therefore, you could run into a case where two or more users accidently have access to another user’s files and folders --- not good.

With this in mind, when SSSD is configured to authenticate and authorize users from multiple disparate LDAP identity stores, there needs to be a method to assign UID and GID numbers in a way to avoid number collisions. A simple approach would be to prefix the number differently between identity stores; e.g. Active Directory and OID. For example, users and groups in Active Directory could prefix their numbers differently than users and groups in OID11g. For example, in one LDAP identity store, a number prefix for users could be 95000 where another users UIDs could start with 72000. The number prefix will help mitigate a collision and security concerns, simple, but effective though needs planning and ongoing effort when each user or group is created.

Duplicate UID and GID Number Security Issues

Limiting administrative access to the LDAP identity stores used by SSSD is also important because an administrator in say Active Directory could modify a user’s uid number of jsmith to match the same uid number of a user akhan OID11g. If this is done, then jsmith could relogin and gain access to the same files and folders owned by akhan because jsmith now has the same UID number. However, this may not automatically happen because remember, SSSD caches data in the LDB Cache, and if SSSD still finds data for jsmith at login even if it is a relogin, it will use that data and not the updated data changed in Active Directory; though eventually that cache will expire . Also something to keep in mind for troubleshooting. To clear the cache, you would either run one of the following commands as root to remove the LDB Cache database, and have jsmith login again.

[root@host] sss_cache –u jsmith
[root@host] rm –f /var/lib/sss/db/*

I am not explaining this as a hack, but to inform anyone of this security issue and a tip for troubleshooting.

Troubleshooting

There will be cases where nothing works correctly and troubleshooting can be tricky. The following are some recommended tips from lessons learned.

Verify SSL/LDAPS Certificates

The most common issue is SSL certificates are not working or configured incorrectly. This can be caused by anything from the certificates on the LDAP server or load balancer are not correct, the certificates on the Linux server are missing or wrong, maybe the wrong type of certificate, or a certificate could be expired. The following are some tips to validate certificate issues.

1. Verify the SSL certificates exist under the parameter location defined by ldap_tls_cacertdir in the sssd.conf.

2. Run the following command to validate the certificate. One key thing to look for is the Subject: CN=<value in red> -- value match the LDAP hostname end point, and the certificate date have not expired.

[root@host] openssl x509 -text -noout -in acme.cert.pem
Certificate:
Data:
Version: 1 (0x0)
Serial Number: 0 (0x0)
Signature Algorithm: md5WithRSAEncryption
Issuer: CN=my.acme.com
Validity
Not Before: May 1 19:32:43 2017 GMT
Not After : Apr 30 19:32:43 2022 GMT
Subject: CN=my.acme.com
<more stuff below this>

3. Test that you can retrieve the SSL certificate from the LDAP hostname by running the following command on the SSSD Linux host you are trying to login. A certificate should be returned and match what was run in the previous test (2.).

[root@host] openssl s_client -connect \
my.acme.com:636 -showcerts </dev/null

4. If the previous steps pass, you can verify LDAPS is working by running the following command on the SSSD Linux host the user is trying to login to. You will get a similar response as the one shown below, if not there may be something wrong with the implantation of the SSL certificate.

[root@host] ldapsearch -x -H ldaps://my.acme.com:636 -b "dc=my,dc=acme,dc=com"

# extended LDIF
#
# LDAPv3
# base <dc=my,dc=acme,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#
#
# search result
search: 2
result: 1 Operations error
text: 000004DC: LdapErr: DSID-0C090728, comment: In order to perform this opera
tion a successful bind must be completed on the connection., data 0, v2580
#
# numResponses: 1

Debug Log Location

There are a couple locations where logs can be found in order to debug issues with SSSD. The following are locations and example logs you can use to trace problems.

      • /var/log/sssd
         o sssd_pam.log
         o sssd_nss.log
         o sssd_<domain name>.log
      • /var/log
         o messages
         o secure

Tips on Debugging

Earlier in Part 1 of 4 - SSSD Linux Authentication: Introduction and Architecture, SSSD Architecture was explained and how SSSD communicates with several modules. This is important to know, especially the flow. Since SSSD does not have a global debug setting, debugging needs to be enabled and turned up in each section of the sssd.conf file. For example, the debugging has to be set in each section; e.g. [nss], [pam], [AD], [OID], etc. where AD or OID is the domain name of the section. Debugging should be added to each section in the sssd.conf and turned up to 9. The range of the value is 1 – 9, where 9 being the highest and most verbose.

[nss]
reconnection_retries = 3
debug_level = 9

[pam]
reconnection_retries = 3
debug_level = 9

[domain/AD]
# AD General
description = AD 2012R2 DC
debug_level = 9

Once the debug levels have been set, as “root” restart sssd as follows.

[root@host] service sssd restart

When SSSD is initially started, several things are identified that can also help identify issues which show up in the domain logs such as connections with other modules, parameter settings, etc. It can be very helpful to review the details in the log as the services start up to see if there are any immediate issues.

Verify a user as UNIX attributes and values

Before a user can login to a UNIX server they must minimally have the following attributes and values in LDAP.

✓ uidNumber

✓ gidNumber

The Linux operating system will expect these and if they don’t exist there will be an issue logging in. Check the backend LDAP identity store and verify the user has values for each of these attributes. Below is an example showing a user in Active Directory using Active Directory Users and Computers. It shows the tab UNIX Attributes, which shows up after installing Microsoft Identity Management for UNIX (ignore this if you did not use this package).

[caption id="attachment_47849" align="aligncenter" width="801"] Example Active Directory User Properties[/caption]

Summary

So finally Part 4 of 4 is complete.  Hopefully, these articles have provided a solid solution to a common problem to solve authenticating and authorizing users to an Oracle IaaS server in OPC that leverages one or more Identity stores whether it be Active Directory or a standard LDAP service. Since SSSD has many great features, you may find other ideas that can be done that expand beyond these articles.  Good luck!