I wrote earlier about systemd-nspawn, and how it can take much of the fiddly work out of setting up functional chroot environments. I’m a regular Ansible user, and I wanted to be able to apply some of those techniques to my playbooks.

Ansible already has a chroot module, of course, but for some situations – such as targeting an emulated chroot environment – that just means a lot of extra work. Using systemd-nspawn makes this trivial.

systemd-nspawn has been called “chroot on steroids”, but if you think of it as Docker with a slightly different target you wouldn’t be far wrong, either. It can be used to spawn containers on your host, and has a variety of options for configuring the containerized environment through the use of private networking, bind mounts, capability controls, and a variety of other facilities that give you flexible container management.

There are many different ways in which it can be used. I’m going to focus on one that’s a bit of a corner use case that I find particularly interesting. In this article we’re going to explore how we can use systemd-nspawn to spawn lightweight containers for architectures other than that of our host system.

While working with Docker the other day, I ran into an undesirable interaction between Docker and systemd services that utilize the PrivateTmp directive.

The PrivateTmp directive, if true, “sets up a new file system namespace for the executed processes and mounts private /tmp and /var/tmp directories inside it that is not shared by processes outside of the namespace”. This is a great idea from a security perspective, but can cause some unanticipated consequences.

The problem in a nutshell

1. Start a Docker container:

Recently, I’ve been playing around with Fedora Atomic and Kubernetes. I ran into a frustrating problem in which I would attempt to start a service from within a script launched by cloud-init, only to have have systemctl block indefinitely because the service I was attempting to start was dependent on cloud-init finishing first.

It turns out that systemctl has a flag meant exactly for this situation:

   --no-block
       Do not synchronously wait for the requested operation to finish. If
       this is not specified, the job will be verified, enqueued and
       systemctl will wait until it is completed. By passing this
       argument, it is only verified and enqueued.

Replacing systemctl start <service> with systemctl start --no-block <service> has solved that particular problem.

I ran into an odd problem the other day: I was testing out some configuration changes for a web application by dropping files into /tmp and pointing the application configuration at the appropriate directory. Everything worked out great when testing it by hand…but when starting up the httpd service, the application behaved as if it was unable to find any of the files in /tmp.

My first assumption was that had simply missed something obvious like file permissions or that I had a typo in my configuration, but after repeated checks and lots of testing it was obvious that something else was going on.