28. August 2018

Windows authentication in Docker Swarm

As I very recently wrote, Windows authentication for Docker containers on Windows Server 2019 made a huge step in the right direction by losing the need for identically named containers and gMSAs. But that unfortunately will be reality only some time in the future and with Windows Server 2012 (R2) still very much in use and even the odd Server 2008 R2 here and there, we can’t expect to only work on 2019 in the near future even when it is released. As Jakub Vanak told me that dynamically scaled services in Docker Swarm are also very much able to reuse a single gMSA for multiple containers (with different names) I was very intrigued and had to try it myself.

The TL;DR

It works 🙂 Ok, maybe a bit more detail even for the lazy reader: A Docker Swarm service is a collection of Docker Containers from the same image and with the same configuration but running on potentially multiple hosts. The beauty of this concept is that you can dynamically scale it with a simple command line operation. So with a simple docker service scale nav=5 we could scale a service called „nav“ to have 5 containers (the correct term in this case would be „replicas“) on potentially 5 or more hosts. With that we can easily spread load for e.g. peaks and also downscale after that peak with something like docker service scale nav=2. Those containers will have different names which I wrongly assumed would break gMSA support but as Jakub correctly pointed out and I verified, this is not true. Windows auth works well for all containers in a service!

How to test

I decided to test this with simple Windows Server Core containers. As you know if you’ve been following my blog at all, my main focus is Dynamics NAV / BC, so the next step will be that image, but for the moment let’s stay with an easier route: We will run Windows Server Core containers and validate the gMSA usage with nltest and Add-LocalGroupMember as I have involuntarily verified quite often, that this won’t work if something is wrong with the gMSA and / or container using it… So how can we test this:

First we need a Docker Swarm, which actually is a lot easier than you might think. I will use two machines (t00-k8s-w1 and t00-k8s-w2) because a swarm with only one host is not much fun, but technically it would also be possible. Those two machines both are Windows Server 2016, version 1607, OS build 14393.2125, so no Insider Preview or anything like that. The only thing „special“ is that they have fixed IPs which makes the Swarm handling easier as you will see soon. Creating a swarm consisting of two hosts (one „manager“ and one „worker“¹ only takes two commands:
1. docker swarm init --advertise-addr=IP --listen-addr IP:2377, which in my case is docker swarm init --advertise-addr=10.111.77.51 --listen-addr 10.111.77.51:2377. This should return some output similar to this
```
Swarm initialized: current node (v5r8foxx1grk1f7f8quaxyefb) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-4qjjvorpta5j1l2vg1ui16h7m4zoa9m4erwv2lnwqggof91szc-dx6t0p5xyz0fnsdnckf8ogqu9 10.111.77.51:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
```
2. We easily use exactly that join command on the second host which should give you something like this
```
docker swarm join --token SWMTKN-1-4qjjvorpta5j1l2vg1ui16h7m4zoa9m4erwv2lnwqggof91szc-dx6t0p5xyz0fnsdnckf8ogqu9 10.111.77.51:2377
This node joined a swarm as a worker.
```
3. To verify that this has worked, run docker node ls which shows you the swarm and the participating nodes with something like the following:
```
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS
l8iuwdqsbm9wa38hpnil65h5l     t00-k8s-w2          Ready               Active
v5r8foxx1grk1f7f8quaxyefb *   t00-k8s-w1          Ready               Active              Leader
```
Now that we have our swarm, we can deploy a service. Make sure that you have the gMSA in place (New-ADServiceAccount and New-CredentialSpec) and then reference it on the service create command. You still need to make sure that service name, hostname and gMSA name match:
```
docker service create --credential-spec file://tst19.json --name tst19 --hostname tst19 microsoft/windowsservercore ping -t localhost
```
The ping command is only added to make sure the container keeps running as it otherwise would immediately stop (and get restarted in an endless loop by the swarm).

In that state state we have only one container on one of the two hosts running but we want to make sure it works on both machines, so we scale the service with docker service scale tst19=2. With docker service ps tst19 we can check what happened and should find something like this which shows two containers, one on each of our hosts

ID                  NAME                IMAGE                                NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
jcmtjwsgk2q1        tst19.1             microsoft/windowsservercore:latest   t00-k8s-w2          Running             Running 43 minutes ago
z7r7sudh5yeo        tst19.2             microsoft/windowsservercore:latest   t00-k8s-w1          Running             Running 40 minutes ago

And we get to the critical part: Validating the gMSA usage. For that we find the ID of the container with docker ps and then check with nltest that everything is fine:
```
PS C:\Windows\system32> docker exec 80 nltest /parentdomain
global.fum. (1)
The command completed successfully
PS C:\Windows\system32> docker exec 80 nltest /query
Flags: 0
Connection Status = 0 0x0 NERR_Success
The command completed successfully
```
As we have seen before, the container name has an appended number and as you can see in the output of docker ps and even an appended id, so we definitely no longer have identical container and gMSA names, but the AD connection still works
To make 100% sure it works, we also try to add an AD user to the local admin group. This works as follows:
1. First we need an interactive PowerShell session inside of the container which is easily done with docker exec -ti ID powershell
2. In that session we need to install the remote admin tools with Add-WindowsFeature RSAT-AD-PowerShell
3. With that in place we can add an AD user to the local admin group: Add-LocalGroupMember -Group Administrators -Member tfenster. If that doesn’t return an error, then everything has worked as expected!

With all that in place, we have verified the gMSA usage in a Windows Docker Container Swarm which gives us very much flexibility for dynamic scaling and also support for configurations and secrets. The next step will be authentication inside the NAV Docker image

and yes, a „manager“ can actually „work“ as well in Swarm :)