CentOS 7, Active Directory, and Kerberos, oh my!

For the longest time, I have utilized ldap authentication and nfs shares to loosely bind my Linux servers to my Active Directory domain. I did stretch one aspect of my ldap authentication to use a load balanced URL so that should one or the other of my local DCs go offline, I still had a legitimate and responsive way to authenticate. I did this over the Samba method of joining because I hated the very idea of using DOMAIN\user as a way to log into my linux servers. Tack onto that, I used nfs and autofs to mount home directories off of a Windows file server. All this required some helping hands from Windows, such as the NIS service available from Microsoft as well as NFS server plugins for the file server. It all worked fairly well, when it could be convinced to do so, thus it was for a while.

Recently, I had the odd experience of watching a coworker pull up a simple list of instructions for joining a linux server to an AD domain. Explaining my experience to him, we stepped through the process and I had to say I was taken back at the easy success of the process and the implications of what it had done. Instead of simply relying on some cobbled together, loose tether to Active Directory, it was leveraging Kerberos and even, as the coworker suggested to me, some limited forms of Group Policy processing. Wanting to know more at a way that may simplify or strengthen my own environment, I started poking around and while I still have quirks, it seems Just That Easy to gain some significant benefits in a better bind to Windows AD.

Getting Connected

First things first. After the usual post-install update and upgrades for CentOS and finding documentation on RootUsers.com on how to join Linux to AD, I began by installing the requisite packages. One package was not called out that I had seen from the work with my coworker (PackageKit) is added in here, to simplify the process. These commands are all run as root or via sudo.

yum install sssd realmd oddjob oddjob-mkhomedir adcli samba-common samba-common-tools krb5-workstation openldap-clients policycoreutils-python PackageKit -y

This enabled me to begin the process of the domain join. The process is simple enough, by using the realm join command, it leverages the packages, sssd and samba most likely, to set everything up. You’ll even get a legitimate computer object in AD to work with! Now, I’ve replaced specifics I used for my domain with generic examples. The –user needs to be an account in AD with authority to join computers, and contoso.com will be the AD domain name that you are joining to.

realm join --user=Administrator contoso.com

Afterwards, you will be able to leverage the commands in the above documentation to validate your domain join, and probe for user accounts to verify the ability to search the directory and get valid information. I was somewhat surprised that the sssd service was happily generating Unix-compatible uids for accounts. To simplify my environment and to ensure I could continue using short names to log in, I made a few edits to my /etc/sssd/sssd.conf file.

• Setting use_fully_qualified_names from True to False
• Setting fallback_homedir from /home/%u@%d to /home/%u

Restart the sssd service and you can give logins a try!

What about home directories?

Thanks to the packages installed above, if you do nothing else, a default home directory will be created for you by the pam_oddjobs_mkhomedir plugin. This is referenced in the usual PAM locations (system-auth and password-auth within /etc/pam.d). I chose to make things more complicated than usual. Since my users already had a home directory living on a Windows file server, it made sense to go ahead and mount that directory. As I have started using DFS instead of simple straight-up file servers, this added another hurdle to making this work. Thankfully, someone came up with at least part of my solution over on the DomainAtHome blog. I say part of my solution because, sometime between 2015 when the article was written and now, the applications must have changed enough, it was written for Ubuntu 14 anyways, and it doesn’t completely provide a working solution on CentOS 7.

I had to begin by finding the libpam-mount package they reference. It isn’t in the CentOS official repositories. While I did find it on the Nux Desktop repository, it is also on Epel and since I am more familiar with Epel, I chose to install it from there.

yum install epel-release -y
yum install pam_mount cifs-utils keyutils -y
vi /etc/security/pam_mount.conf.xml

Updating the configuration of pam_mount.conf.xml with the volume definitions I needed. I discovered from troubleshooting this later that the uid=%(USER),gid=100 options are unnecessary for CentOS and thus removed them. Additionally, the cruid=%(USERID) option informs mount to leverage Kerberos tickets from that user’s CCache.

<volume user="*" sgrp="domain users" fstype="cifs" server="contoso.com" path="dfs/homes/%(USER)" mountpoint="/home/%(USER)" options="cruid=%(USERUID),sec=krb5,dir_mode=0700" />

Then, the module must be enabled within PAM so that it will be run when logins occur. This module does not seem to integrate with authconfig, so that method of updating the PAM stack has to be abandoned since manual edits of the files will be required. Since we are using Kerberos for authentication, we do not need to get pam_mount.so the user’s password, so we can skip adding anything into the auth stack and simply add an entry into the session stack. I added this just after the oddjob create home directory for reasons explained below, and in both the system-auth and password-auth processes within /etc/pam.d.

session     optional      pam_oddjob_mkhomedir.so umask=0077
session     optional      pam_mount.so disable_interactive

Creating the mount point is a must, too, and while pam_mount.so has the capacity, I found I would get Access Denied errors and assumed that the logon program had already transitioned to the users rights.

The problem with this, however, is DFS is different. I kept running into problems mounting the home directory over and over again, often with the dreaded error of (mount.c:72): mount error(126): Required key not available. I had to dig quite some time to sort out that during mount and using the Kerberos methods, in order to obtain that ticket, mount had to use cifs.upcall to do so. I had to set the system to log all debugging messages for cifs.upcall in order to see what was happening. In short, it was taking the contoso.com server literally and not chasing the various referrals for DFS to find the target host it was on. Problem. It seems this might be solved somewhat simply, and a tip from yet another blog helped (notably, his step 4). By changing the command slightly within the /etc/request-key.d/cifs.spnego.conf file (where the settings for that reside on CentOS 7) adding option -t (the option -c was not used in CentOS 7) so that it would trust DNS, suddenly it works! Within /var/log/debug where I dumped those messages, I can see it chasing referrals.

Oct 13 23:43:25 centos7 cifs.upcall: key description: cifs.spnego;0;0;39010000;ver=0x2;host=contoso.com;ip4=192.168.1.251;sec=krb5;uid=0x40872e90;creduid=0x40872e90;user=user;pid=0x13e6a
Oct 13 23:43:25 centos7 cifs.upcall: ver=2
Oct 13 23:43:25 centos7 cifs.upcall: host=contoso.com
Oct 13 23:43:25 centos7 cifs.upcall: ip=192.168.1.251
Oct 13 23:43:25 centos7 cifs.upcall: sec=1
Oct 13 23:43:25 centos7 cifs.upcall: uid=1082601104
Oct 13 23:43:25 centos7 cifs.upcall: creduid=1082601104
Oct 13 23:43:25 centos7 cifs.upcall: user=user
Oct 13 23:43:25 centos7 cifs.upcall: pid=81514
Oct 13 23:43:25 centos7 cifs.upcall: krb5_get_init_creds_keytab: -1765328174
Oct 13 23:43:25 centos7 cifs.upcall: handle_krb5_mech: getting service ticket for contoso.com
Oct 13 23:43:25 centos7 cifs.upcall: cifs_krb5_get_req: unable to get credentials for contoso.com
Oct 13 23:43:25 centos7 cifs.upcall: handle_krb5_mech: failed to obtain service ticket (-1765328377)
Oct 13 23:43:25 centos7 cifs.upcall: ip_to_fqdn: resolved 192.168.1.251 to controller2.contoso.com
Oct 13 23:43:25 centos7 cifs.upcall: handle_krb5_mech: getting service ticket for controller2.contoso.com
Oct 13 23:43:25 centos7 cifs.upcall: handle_krb5_mech: obtained service ticket
Oct 13 23:43:25 centos7 cifs.upcall: Exit status 0
Oct 13 23:43:25 centos7 cifs.upcall: key description: cifs.spnego;0;0;39010000;ver=0x2;host=Controller1;ip4=192.168.1.250;sec=krb5;uid=0x40872e90;creduid=0x40872e90;user=user;pid=0x13e6a
Oct 13 23:43:25 centos7 cifs.upcall: ver=2
Oct 13 23:43:25 centos7 cifs.upcall: host=Controller1
Oct 13 23:43:25 centos7 cifs.upcall: ip=192.168.1.250
Oct 13 23:43:25 centos7 cifs.upcall: sec=1
Oct 13 23:43:25 centos7 cifs.upcall: uid=1082601104
Oct 13 23:43:25 centos7 cifs.upcall: creduid=1082601104
Oct 13 23:43:25 centos7 cifs.upcall: user=user
Oct 13 23:43:25 centos7 cifs.upcall: pid=81514
Oct 13 23:43:25 centos7 cifs.upcall: krb5_get_init_creds_keytab: -1765328174
Oct 13 23:43:25 centos7 cifs.upcall: handle_krb5_mech: getting service ticket for controller1
Oct 13 23:43:25 centos7 cifs.upcall: handle_krb5_mech: obtained service ticket
Oct 13 23:43:25 centos7 cifs.upcall: Exit status 0
Oct 13 23:43:25 centos7 cifs.upcall: key description: cifs.spnego;0;0;39010000;ver=0x2;host=fileserver;ip4=192.168.1.21;sec=krb5;uid=0x40872e90;creduid=0x40872e90;user=user;pid=0x13e6a
Oct 13 23:43:25 centos7 cifs.upcall: ver=2
Oct 13 23:43:25 centos7 cifs.upcall: host=fileserver
Oct 13 23:43:25 centos7 cifs.upcall: ip=192.168.1.21
Oct 13 23:43:25 centos7 cifs.upcall: sec=1
Oct 13 23:43:25 centos7 cifs.upcall: uid=1082601104
Oct 13 23:43:25 centos7 cifs.upcall: creduid=1082601104
Oct 13 23:43:25 centos7 cifs.upcall: user=user
Oct 13 23:43:25 centos7 cifs.upcall: pid=81514
Oct 13 23:43:25 centos7 cifs.upcall: krb5_get_init_creds_keytab: -1765328174
Oct 13 23:43:25 centos7 cifs.upcall: handle_krb5_mech: getting service ticket for fileserver
Oct 13 23:43:25 centos7 cifs.upcall: handle_krb5_mech: obtained service ticket

Finally!

Wrapping up

One lingering issue seems to be sticking, however, and that is that when using PuTTY from a domain-joined workstation, I can obtain a password-less login because PuTTY supports GSSAPI logins. That doesn’t seem to be passing my tickets over in order to perform the fileserver login, and thus denying me my home directory. Come to find out that this is because PuTTY and perhaps Windows is not willing to do the same Kerberos ticket sharing that OpenSSH is when it plays both client and server. What PuTTY does is permit delegation, and this is an entirely different mechanism and requires a few extra steps. A post over at Serverfault details what they are.

First, within Active Directory, you need to enable delegation on the computer object. I would like to limit this to simply SSH, since that’s the only service I’m providing at the moment, but I’m not quite sure how to accomplish that so full delegation for the moment is acceptable.

Second, within PuTTY you need to save either for the session or defaults that delegation should be permitted.

Once that was all done, I was able to provide only my user name and viola!