While working with Docker the other day, I ran into an undesirable interaction between Docker and systemd services that utilize the PrivateTmp directive.

The PrivateTmp directive, if true, “sets up a new file system namespace for the executed processes and mounts private /tmp and /var/tmp directories inside it that is not shared by processes outside of the namespace”. This is a great idea from a security perspective, but can cause some unanticipated consequences.

The problem in a nutshell

1. Start a Docker container:

    # cid=$(docker run -d larsks/thttpd)
    # echo $cid
    e68df3f45d6151259ce84a0e467a3117840084e99ef3bbc654b33f08d2d6dd62
   

2. See the devicemapper mountpoint created by Docker for the container:

    # grep devicemapper/mnt /proc/mounts
    /dev/mapper/docker-253:6-98310-e68df3f45d6151259ce84a0e467a3117840084e99ef3bbc654b33f08d2d6dd62 /var/lib/docker/devicemapper/mnt/e68df3f45d6151259ce84a0e467a3117840084e99ef3bbc654b33f08d2d6dd62 ext4 rw,context="system_u:object_r:svirt_sandbox_file_t:s0:c261,c1018",relatime,discard,stripe=16,data=ordered 0 0
   

3. Now restart a service – any service! – that has PrivateTmp=true:

    # systemctl restart systemd-machined
   

4. Get the PID for that service:

    # systemctl status systemd-machined | grep PID
     Main PID: 18698 (systemd-machine
   

5. And see that the mount created by the Docker “devicemapper” storage driver is visible inside the mount namespace for this process:

    # grep devicemapper/mnt /proc/18698/mounts
    /dev/mapper/docker-253:6-98310-e68df3f45d6151259ce84a0e467a3117840084e99ef3bbc654b33f08d2d6dd62 /var/lib/docker/devicemapper/mnt/e68df3f45d6151259ce84a0e467a3117840084e99ef3bbc654b33f08d2d6dd62 ext4 rw,context="system_u:object_r:svirt_sandbox_file_t:s0:c261,c1018",relatime,discard,stripe=16,data=ordered 0 0
   

6. Attempt to destroy the container:

    # docker rm -f $cid
   

7. Watch Docker fail to destroy the container because it is unable to remove the mountpoint directory:

    Jan 17 22:43:03 pk115wp-lkellogg docker-1.4.1-dev[18239]:
    time="2015-01-17T22:43:03-05:00" level="error" msg="Handler for DELETE
    /containers/{name:.*} returned error: Cannot destroy container e68df3f45d61:
    Driver devicemapper failed to remove root filesystem
    e68df3f45d6151259ce84a0e467a3117840084e99ef3bbc654b33f08d2d6dd62: Device is
    Busy"
   

8. Because while that mount is gone from the global namespace:

    # grep devicemapper/mnt /proc/mounts
   

9. It still exists inside the mount namespace for the service we restarted:

   # grep devicemapper/mnt /proc/18698/mounts
   /dev/mapper/docker-253:6-98310-e68df3f45d6151259ce84a0e467a3117840084e99ef3bbc654b33f08d2d6dd62 /var/lib/docker/devicemapper/mnt/e68df3f45d6151259ce84a0e467a3117840084e99ef3bbc654b33f08d2d6dd62 ext4 rw,context="system_u:object_r:svirt_sandbox_file_t:s0:c261,c1018",relatime,discard,stripe=16,data=ordered 0 0
   

10. To resolve this problem, restart the service holding the mount open:

    # systemctl restart systemd-machined