sudo + sssd + ldap

In my quest to implement sssd, my focus turned towards sudo.  Centralizing the sudo rules to an LDAP server (or cluster) simplifies management of users and access.  Rather than /etc/sudoers files on each machine, sudo can look in to LDAP for a specific user’s rules.

The path of a query is:
sudo command requested -> network service switch -> sssd -> sssd_sudo -> ldap

OpenLDAP does not have the schema installed by default, and many have converted to the slapd.d configuration file format which leaves a lot to be desired. You have to take the sudoers.schema file and convert it to slapd.d format (I’ve converted it, so if you just want to use this file, feel free), which comes out like this (name your file accordingly) /etc/openldap/slapd.d/cn=config/cn=schema/cn={6}sudoers.ldif.

Of course, with any schema change you need to restart ldap:

systemctl restart slapd

Next, the local /etc/sudoers file needs to be converted and then imported.  There’s a helpful utility that comes with the sudo rpm called sudoers2ldif.  It does a good job at getting your started, and you must keep in mind that the resulting ldif file will need manual adjusting before importing:

sudo SUDOERS_BASE=ou=SUDOers,dc=company,dc=com perl
/usr/share/doc/sudo*/sudoers2ldif /etc/sudoers > /tmp/sudoers.ldif

Again, be sure to look the file over and remove any unusual entries.  Also, add the SUDOers ou definition to the top of the file:

dn: ou=SUDOers,dc=company,dc=com
description: SUDOers
objectclass: organizationalUnit
objectclass: top
ou: SUDOers

With the file cleaned up and the ou defined, import the ldif file.  With it imported, the data is there, but nothing is using it.

Update /etc/sssd/sssd.conf to include “sudo” in the services line for the [sssd] stanza, and at the end of the file create a blank [sudo] stanza.  Restart sssd.

systemctl restart sssd

Update nsswitch.conf to include the line:

sudoers: files sss

Remove all sudo entries from /etc/sudoers for users with a UID >= 1000. Those are now handled via LDAP ,and all of the other entries (for users with a UID < 1000) will be managed by the local /etc/sudoers file.  This is a limitation hardcoded in sssd that cannot be overridden.

sssd and tmpfs

After my previous efforts to migrate to sssd, it was discovered after a reboot that sssd and a tmpfs /var/log did not play well together.  sssd has a couple of minor faults:

  • If the default /var/log/sssd directory does not exist, sssd will not start
  • It does not create the default /var/log/sssd directory if one does not exist first.
  • You cannot change the default log directory location (without recompiling the binary, and those of us with packaged based distributions don’t want to be doing that custom step if not necessary)

Searching the web, there are a number of discussions surrounding this very issue, with the majority of them dying off with responses like “Why would you want to run tmpfs for /var/log?” “What would be the benefit?” “Seems like this isn’t a good idea”, and other useless responses.  I say useless because those types of responses are far from helpful, distract from the original question/problem, and don’t address the problem at hand that the individual is asking about.

Systemd installations (such as RHEL7 and the current Fedora tree) use the file /usr/lib/systemd/system/sssd.service as part of the sssd rpm.  Sure, you can try modifying that file, but “Type = forking” only allows for a single command in “ExecStart”.  You can go down the path of tinkering with it, writing another systemd script to create the directory structure for you, but, as I found out, that would be re-inventing the wheel.

Thankfully, I found one reference elsewhere to tmpfiles.d which is part of the systemd.  It plugs the gap where programs don’t or can’t build the directory structure you require.  Simply create the file /etc/tmpfiles.d/sssd.conf with the following contents:

# sssd needs a directory in /var/log/sssd to store its log files
d /var/log/sssd 0755 root root

The tmpfiles process is kicked off at the systemd “sysinit” level, which happens after the filesystems are mounted but before services are started.  So, you are all set.

You might ask: Why run /var/log on tmpfs?

I happen to run a number of mythtv frontend systems that are not much more than dummy clients, all of which have ssd internal drives (to limit the power usage at the TV).  I’ve done some basic optimizations to preserve the life of the SSD while still retaining debugging functionality (it is mythtv after all, which requires a lot of TLC).  Having /var/log contents can be very helpful in a pinch when logging on to the box, but I understand that both the logs and the machines are relatively disposable and I’m ok with losing the logs after a reboot.

Embracing SSSD in Linux

When RHEL6 came around and I saw that sssd was a new way to sync up to the LDAP server, I cringed in horror. The PADL tool set that was shoehorned into RHEL5 was painful, and it took a lot to get it working correctly.  The last thing I wanted to do was to re-open that wound.

Caching and why you want to do it
With the advent of centralized services, the IT world took a great step forward.  It wasn’t perfect, though. It still required the client to be able to talk to the service (ldap, hosts, automount maps, etc) in real time.  If the centralized service were down, clients would break as well.  Introducing redundancy in to the environment (active/passive LDAP pairs) sort of solved this problem but not entirely.  Those with laptops taking their machines off the wire either to go home or to wander to a new location, would lose access.  Individual services might cache credentials locally, but those eventually have to phone home, fail, and then lock the user out.

The goal, really, is centralized management. Make it easier to manage the complex linux environment.  Everything from user accounts and permissions, to domain name resolution.

Single Sign-On SSO
As we get closer to a Single-Sign-On (SSO) environment for Linux, funneling all requests through a master intermediary which would handle the connection to the various backend solutions and cache the data locally looked to be a decent option. The PADL tools were the beginnings of that, but their implementation was clunky and hack-ish.  Despite that, the results were worth the headache.

With enough griping, when RHEL5 hit, it was all re-wrapped in to nslcd.  Slightly updated from previous incarnations, those familiar with the configuration files of old were able to easily implement nslcd.  Combined with nscd, you had a workable solution yet still filled with headaches of its own.

System Security Services Daemon (SSSD)
RHEL6 came in to place and introduced sssd. Aside from the awful name, it was yet another hideous beast fighting to control. Like many others, I had reluctantly embraced nslcd because it was close to the old ldap.conf file, the configurations were generally the same, and it took a short amount of time to get implemented.  SSSD was a completely different beast that requires some time to learn and understand before diving fully in.

Unlike other people, I didn’t hold an angry grudge against the PADL tools.  It was a means to an end, and, when all was said and done, it delivered what it promised.  I had a bunch of integration to OpenLDAP through PAM with caching with nslcd. Things worked.

Why implement SSSD?
Despite what the administrator in you feels (“do it once, do it right” creedo), the Admin/Engineer’s world is about doing things better. SSSD combines the functionality of nslcd and nscd without the array of bugs, without the odd “third wheel” product support, and it expands the scope of what can be managed easily.

Admit it: the LDAP world is changing, and Single Sign On (SSO) is continually evolving. The tools to support that have grown in functionality and scope, and new options are out there. For example, one of the key centralized backend tools, OpenLDAP, is admittedly unfriendly on a good day.  Alternatives have come along, such as the Fedora Directory Server (FDS) – now known as 389-ds -, and have matured significantly.  When bundled with SSSD and IPA, you have the makings of the Windows Active Directory equivalent in Linux.  389-ds is the supported RedHat tool, and it is actively being developed in the Fedora world, too.  IPA is a RedHat only option, but there is always FreeIPA for the rest of us.

In other words: you have gone as far as you can go with nslcd/nscd, and implementing sssd prepares you for the future.

BEWARE of documentation caveats
SSSD is still growing and evolving. Changes to the content and format of the sssd.conf file are happening quite frequently, and here are a few items to be aware of:

  • SSSD fires off a separate daemon handler for each service. If you want to debug a service, you cannot turn debugging on in the [sssd] stanza, but you must define it within the service stanza. For example, “debug_level = 9” would go under “[autofs]” to debug autofs.
  • The RedHat documentation is good but it gets confusing with regard to setting autofs attributes. To clarify, those settings must be made at the [domain/default] stanza otherwise they will be quietly ignored.
  • Debug settings are no longer numbers from 0-10.  They are now a crazy scheme of 0x4000 type of numbering.
  • SSSD’s debugging is a bit painful. It doesn’t always log what you want where you want it to. If you get close to the end of your rope, it is very helpful to run sssd in the foreground in one window while testing in another to watch the output live.
    sssd -d 9 -c /etc/sssd/sssd.conf -i

The actual migration
Migrating from nslcd to sssd can be a bit of a pain as it is likely you already have a number of dependencies in place.  For me, my edge hosts only needed to talk back to the LDAP servers for local authentication and automount tables.  Documentation from sites such as RedHat are very complete yet are still lacking in some key areas.

Migrating basic authentication is rather easy:

  1. yum -y install sssd
  2. Build your /etc/sssd/sssd.conf:
    [sssd]
    config_file_version = 2
    reconnection_retries = 3
    services = nss, pam, autofs, sudo
    # SSSD will not start if you do not configure any domains.
    # Add new domain configurations as [domain/] sections, and
    # then add the list of domains (in the order you want them to be
    # queried) to the "domains" attribute below and uncomment it.
    domains = LDAP
    #debug_level = 10
    
    [nss]
    filter_users = root,ldap,named,avahi,haldaemon,dbus,radiusd,news,nscd
    reconnection_retries = 3
    
    [pam]
    reconnection_retries = 3
    
    [autofs]
    
    [sudo]
    
    [domain/LDAP]
    ldap_tls_reqcert = never
    # Note that enabling enumeration will have a moderate performance impact.
    # Consequently, the default value for enumeration is FALSE.
    # Refer to the sssd.conf man page for full details.
    enumerate = true
    auth_provider = ldap
    # ldap_schema can be set to "rfc2307", which stores group member names in the
    # "memberuid" attribute, or to "rfc2307bis", which stores group member DNs in
    # the "member" attribute. If you do not know this value, ask your LDAP
    # administrator.
    #ldap_schema = rfc2307bis
    ldap_schema = rfc2307
    ldap_search_base = dc=ourdomain,dc=com
    ldap_group_member = uniquemember
    id_provider = ldap
    ldap_id_use_start_tls = False
    chpass_provider = ldap
    ldap_uri = ldap://ldap1.ourdomain.com/,ldap://ldap2.ourdomain.com/
    ldap_chpass_uri = ldap://ldap.ourdomain.com/
    # Allow offline logins by locally storing password hashes (default: false).
    cache_credentials = True
    ldap_tls_cacertdir = /etc/openldap/cacerts
    entry_cache_timeout = 600
    ldap_network_timeout = 3
    autofs_provider = ldap
    ldap_autofs_search_base = dc=ourdomain,dc=com
    ldap_autofs_map_object_class = automountMap
    ldap_autofs_map_name = ou
    ldap_autofs_entry_object_class = automount
    ldap_autofs_entry_key = cn
    ldap_autofs_entry_value = automountInformation
    sudo_provider = ldap
    ldap_sudo_search_base = ou=Sudoers,dc=ourdomain,dc=com
    ldap_sudo_full_refresh_interval=86400
    ldap_sudo_smart_refresh_interval=3600
    # Enable group mapping otherwise only the user's primary group will map correctly. Without this
    # defined group membership won't work
    ldap_group_object_class = posixGroup
    ldap_group_search_base = ou=group,dc=ourdomain,dc=com
    ldap_group_name = cn
    ldap_group_member = memberUid
    
    # An example Active Directory domain. Please note that this configuration
    # works for AD 2003R2 and AD 2008, because they use pretty much RFC2307bis
    # compliant attribute names. To support UNIX clients with AD 2003 or older,
    # you must install Microsoft Services For Unix and map LDAP attributes onto
    # msSFU30* attribute names.
    ; [domain/AD]
    ; id_provider = ldap
    ; auth_provider = krb5
    ; chpass_provider = krb5
    ;
    ; ldap_uri = ldap://your.ad.example.com
    ; ldap_search_base = dc=example,dc=com
    ; ldap_schema = rfc2307bis
    ; ldap_sasl_mech = GSSAPI
    ; ldap_user_object_class = user
    ; ldap_group_object_class = group
    ; ldap_user_home_directory = unixHomeDirectory
    ; ldap_user_principal = userPrincipalName
    ; ldap_account_expire_policy = ad
    ; ldap_force_upper_case_realm = true
    ;
    ; krb5_server = your.ad.example.com
    ; krb5_realm = EXAMPLE.COM
    
  3. Start sssd
    systemctl enable sssd; systemctl start sssd
  4. Update your nsswitch.conf file by replacing all references to ldap with sss
  5. RESTART your services.  I cannot stress this one enough. Most of the services do some form of internal connection caching on their own and will still be attempting to communicate to the back-end service as it was previously defined in nsswitch.conf.
  6. Stop and disable nscd:
    systemctl stop nscd; systemctl disable nscd
  7. Test each service
    users accounts: getent passwd
    user authentication: ssh atestuser@localhost
    POP3/IMAP access via dovecot: fire up a mail client (your phone, a web client) to see if you can access your email
    Sendmail authentication via SASL: make sure you can SEND an email
    Email filtering via MailScanner: this should work because it is not connected to users, but it won't hurt it to restart
    Web authentication via LDAP or tools such as owncloud, roundcube, etc: try logging in
    Filesystem mounts via NFS: cd /home/atestuser
  8. Stop and disable nslcd:
    systemctl stop nslcd; systemctl disable nslcd