A recent update to Arch Linux replaced the qemu-kvm package with an updated version of qemu. A side effect of this change is that the qemu-kvm binary is no longer available, and any libvirt guests on your system utilizing that binary will no longer operate. As is typical with Arch, there is no announcement about this incompatible change, and queries to #archlinux will be met with the knowledge, grace and decorum you would expect of that channel:

2013-04-08T18:00 < gtmanfred> USE --enable-kvm for fucks sake
2013-04-08T18:00 < gtmanfred> DO I HAVE TO SAY IT AGAIN?

The emulator binary is hardcoded into your domain in the <emulator> emulator, and typically looks something like this:

<emulator>/usr/bin/qemu-kvm</emulator>

In order to get your guests working again after the upgrade you'll need to replace this path with an appropriate selection from one of the other binaries provided by the qemu package, which include qemu-system-i386 and qemu-system-x86_64. You'll want to select the one appropriate for your guest architecture. You can do this manually running virsh edit for each affected guest, but if you have more than a couple that rapidly becomes annoying.

We can use XSLT to write a transformation that will set the <emulator> to an appropriate value, and we can set things up to run this automatically across all of our guests. The following stylesheet will replace the <emulator> tag with a path to an appropriate qemu (by extracting the arch attribute of the domain/os/type element:

<?xml version="1.0"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

  <!-- copy all elements verbatim... -->
  <xsl:template match="@*|node()">
    <xsl:copy>
      <xsl:apply-templates select="@*|node()"/>
    </xsl:copy>
  </xsl:template>

  <!-- ...except for the 'emulator' element. -->
  <xsl:template match="emulator">
    <emulator>/usr/bin/qemu-system-<xsl:value-of select="/*/os/type/@arch"/></emulator>
  </xsl:template>

</xsl:stylesheet>

We're going to apply this to all of our (inactive) guests via the virsh edit subcommand. This command runs an editor (selected based on your VISUAL or EDITOR environment variables) on your domain XML. We need to create an "editor" that will apply the above transformation to its input file. Something like this will work:

#!/bin/sh

tmpfile=$(mktemp "$1.patched.XXXXXX")
xsltproc -o "$tmpfile" patch-emulator.xsl "$1"
mv "$tmpfile" "$1"

Assuming the above script has been saved as "patch-emulator.sh" (and made executable), we can run this across all of our inactive guests like this:

#!/bin/sh

VISUAL=./patch-emulator.sh
export VISUAL

virsh list --inactive --name | while read vm; do
        [ "$vm" ] || continue
        virsh edit $vm
done
Mar 12, 2013

I2C on the Raspberry Pi

I've set up my Raspberry Pi to communicate with my Arduino via I2C. The Raspberry Pi is a 3.3v device and the Arduino is a 5v device. While in general this means that you need to use a level converter when connecting the two devices, you don't need to use a level converter when connecting the Arduino to the Raspberry Pi via I2C.

The design of the I2C bus is such that the only device driving a voltage on the bus is the master (in this case, the Raspberry Pi), via pull-up resistors. So when "idle", the bus is pulled to 3.3v volts by the Pi, which is perfectly safe for the Arduino (and compatible with it's 5v signaling). To transmit data on the bus, a device brings the bus low by connecting it to ground. In other words, slave devices never drive the bus high. This means that the Raspberry Pi will never see a 5v signal from the Arduino...unless, of course, you make a mistake and accidentally digitalWrite a HIGH value on one of the Arduino's I2C pins. So don't do that.

Note that the built-in pull-up resistors are only available on the Pi's I2C pins (Pins 3 (SDA) and 5 (SCL), aka BCM GPIO0 and GPIO1 on a Rev. 1 board, GPIO2 and GPIOP3 on a Rev. 2 board):

Raspberry Pi Pins

On the Arduino Uno, the I2C pins are pins A4 (SDA) and A5 (SCL):

Arduino Uno Pins

For information about other boards and about the Arduino I2C API, see the documentation for the Wire library.

There are several Python libraries out there for interacting with the GPIO pins on a Raspberry Pi:

All of them are reasonably easy to use, but the Quick2Wire API provides a uniquely useful feature: epoll-enabled GPIO interrupts. This makes it trivial to write code that efficiently waits for and responds to things like button presses.

The following simple example waits for a button press attached to GPIO1 (but refer to the chart in this document to see exactly what that means; this is pin 12 on a Raspberry Pi v2 board) and lights an LED attached to GPIO0 when the button is pressed:

#!/usr/bin/env python3

import select
from quick2wire.gpio import pins, In, Out, Rising, Falling, Both

button1 = pins.pin(0, direction=In, interrupt=Both)
led = pins.pin(1, direction=Out)

with button1,led:
    epoll = select.epoll()
    epoll.register(button1, select.EPOLLIN|select.EPOLLET)
    while True:
        events = epoll.poll()
        for fileno, event in events:
            if fileno == button1.fileno():
                print('BUTTON 1!', button1.value)
                led.value = button1.value

There is also a Selector class that makes the epoll interface a little easier to use. The following code is equivalent to the above epoll example:

#!/usr/bin/env python3

from quick2wire.gpio import pins, In, Out, Both
from quick2wire.selector import Selector

button1 = pins.pin(0, direction=In, interrupt=Both)
led = pins.pin(1, direction=Out)

with button1, led, Selector(1) as selector:
    selector.add(button1)
    while True:
        selector.wait()
        if selector.ready == button1:
            print('BUTTON 1!', button1.value)
            led.value = button1.value

The selector module includes a Timer class that lets you add one-shot or repeating timers to a Selector. The following example will light the LED for one second after the button is pressed, unless the button is pressed again, in which case the LED will go out immediately:

#!/usr/bin/env python3

from quick2wire.gpio import pins, In, Out, Both
from quick2wire.selector import Selector, Timer

button1 = pins.pin(0, direction=In, interrupt=Both)
led = pins.pin(1, direction=Out)
active = False

with button1, led, \
        Selector(1) as selector, \
        Timer(offset=2) as timer:

    selector.add(button1)
    selector.add(timer)

    while True:
        selector.wait()
        if selector.ready == button1:
            print('BUTTON 1!', button1.value, active)

            if button1.value:
                if active:      
                    active = False      
                    led.value = 0       
                    timer.stop()        
                else:           
                    active = True       
                    led.value = 1       
                    timer.start()       

        if selector.ready == timer:
            if active:  
                active = False  
                led.value = 0   

All of these examples rely on Python's with statement. If you're unfamiliar with with, you can find more information here.

I've recently started playing with an Arduino kit I purchased a year ago (and only just now got around to unboxing). I purchased the kit from SparkFun, and it includes a motley collection of resistors, LEDs, a motor, a servo, and more.

I was fiddling around with this exercise, which uses the SoftwareServo library to control a servo. Using this library, you just pass it an angle and the library takes care of everything else, e.g. to rotate to 90 degrees you would do this:

myservo.write(90);

The exercise suggests trying to control the servo without using the library:

While it is easy to control a servo using the Arduino’s included library sometimes it is fun to figure out how to program something yourself. Try it. We’re controlling the pulse directly so you could use this method to control servos on any of the Arduino’s 20 available pins (you need to highly optimize this code before doing that).

It took me a few tries, and it looks as if the upper and lower limits for the servo pulses given in that documentation may not be 100% accurate. This is what I finally came with. As an added bonus, it writes position information to the serial port:

int incomingByte = 0;
int servo0 = 600;
int servo180 = 2100;
int inc = 20;
int pos = servo0;
int servoPin = 9;
int pulseInterval=2000;

void setup() {
  Serial.begin(9600);     // opens serial port, sets data rate to 9600 bps
  pinMode(servoPin, OUTPUT);
}

void loop() {
  int i;

  pos += inc;

  if (pos > servo180) {
    Serial.println("REVERSE!");
    pos = servo180;
    inc *= -1;
    delay(500);
  } else if (pos < servo0) {
    Serial.println("FORWARD!");
    pos = servo0;
    inc *= -1;
    delay(500);
  }

  Serial.print("pos = ");
  Serial.println(pos, DEC);

  digitalWrite(servoPin, HIGH);
  delayMicroseconds(pos);
  digitalWrite(servoPin, LOW);
  delay(20);
}

Under Linux or OS X, you could view the serial output using screen like this:

screen /dev/tty.usbmodemfd12441 9600
Feb 25, 2013

A quote about XMLRPC

I've been reading up on Puppet 3 lately, and came across the following:

XMLRPC was the new hotness when development on Puppet started. Now, XMLRPC is that horrible thing with the XML and the angle brackets and the pain and sad.

(from http://somethingsinistral.net/blog/the-angry-guide-to-puppet-3/)

...which also accurately sums up my feelings when I come across yet another piece of software where someone has decided that XML (or even JSON) is a good user-facing configuration syntax.

Feb 21, 2013

A systemd unit for ucarp

In Fedora 17 there are still a number of services that either have not been ported over to systemd or that do not take full advantage of systemd. I've been investigating some IP failover solutions recently, including ucarp, which includes only a System-V style init script.

I've created a template service for ucarp that will let you start a specific virtual ip like this:

systemctl start ucarp@001

This will start ucarp using settings from /etc/ucarp/vip-001.conf. The unit file is on github and embedded here for your reading pleasure:

Feb 1, 2013

Running dhcpcd under LXC

I've been working with Arch Linux recently, which uses dhcpcd as its default DHCP agent. If you try booting Arch inside an LXC container, you will find that dhcpcd is unable to configure your network interfaces. Running it by hand you will first see the following error:

# dhcpcd eth0
dhcpcd[492]: version 5.6.4 starting
dhcpcd[492]: eth0: if_init: Read-only file system
dhcpcd[492]: eth0: interface not found or invalid

This happens because dhcpcd is trying to modify a sysctl value. Running dhcpcd under strace we see:

open("/proc/sys/net/ipv4/conf/eth0/promote_secondaries", O_WRONLY|O_CREAT|O_TRUNC, 0666) = -1 EROFS (Read-only file system)

This happens because /proc is typically mounted read-only in a container environment (to prevent the container from modifying things that would potentially affect the host system).

We can use a "bind mount" to solve this problem. A "bind mount" allows you to mount part of a filesystem on another part of the filesystem. In this case, we're going to mask that value in /proc by bind mounting a file on top of it.

  • First, we create the file we'll use as a mask:

    # echo 0 > /var/tmp/promote_secondaries
    
  • Then we mount in on top of the /proc entry:

    # mount -o bind /var/tmp/promote_secondaries \
        /proc/sys/net/ipv4/conf/eth0/promote_secondaries
    

And now that /proc value is "writable" from the perspective of dhcpcd. If we try to run dhcpcd now, we see:

# dhcpcd eth0
dhcpcd[770]: version 5.6.4 starting
dhcpcd[770]: eth0: sending IPv6 Router Solicitation
dhcpcd[770]: eth0: rebinding lease of 192.168.117.53
dhcpcd[770]: eth0: acknowledged 192.168.117.53 from 192.168.117.1
dhcpcd[770]: eth0: checking for 192.168.117.53
dhcpcd[770]: eth0: sending IPv6 Router Solicitation
dhcpcd[770]: eth0: leased 192.168.117.53 for 3600 seconds
dhcpcd[770]: forked to background, child pid 796

If you are running dhcpcd via the dhcpcd@.service unit, then you can automate this masking with the following service unit:

[Unit]
Description=Mask read-only /proc entries for %I.
RequiredBy=dhcpcd@%I
Before=dhcpcd@%I

[Service]
ExecStartPre=/bin/dd if=/proc/sys/net/ipv4/conf/%I/promote_secondaries \
  of=/var/tmp/promote_secondaries_%I
ExecStart=/bin/mount -o bind /var/tmp/promote_secondaries_%I \
  /proc/sys/net/ipv4/conf/%I/promote_secondaries
RemainAfterExit=yes
ExecStop=/bin/unmount /proc/sys/net/ipv4/conf/%I/promote_secondaries

[Install]
WantedBy=multi-user.target

If you see...

/usr/lib/dhcpcd/dhcpcd-hooks/30-hostname: line 17: /proc/sys/kernel/hostname: Read-only file system

...you may need to do something similar to mask the kernel.hostname entry in /proc, although this will need to be done once rather than per-interface. Alternatively, you can modify the hook script responsible for setting the hostname (/usr/lib/dhcpcd/dhcpcd-hooks/30-hostname).

I was intrigued by this post on socket activated containers with systemd. The basic premise is:

  • systemd opens a socket on the host and listens for connections.
  • When a client connections, systemd spawns a new container.
  • The host systemd passes the connected socket to the container systemd.
  • Services in the container receive these sockets from the container systemd.

This is a very neat idea, since it delegates all the socket listening to the host and only spins up container and service resources when necessary.

An interesting corollary to this is that the service container doesn't actually need any networking: since the host is responsible for opening the socket and listening for connections, and the container receives an already connected socket, you can create containers that have no network interfaces other than the loopback interface and still connect to them remotely.

The example presented in Lennarts article will work just fine if you change this:

ExecStart=/usr/bin/systemd-nspawn -jbD /srv/mycontainer 3

To this:

ExecStart=/usr/bin/systemd-nspawn --private-network -jbD /srv/mycontainer 3

After this change, if you connect to this container you'll see:

# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever

This opens up a variety of interesting possibilities for creating "endpoint" containers that offer services over the network but are able to limit the scope of a compromised service. Because systemd-nspawn has been designed as more of a convenice tool than a full container solution, we'll need to wait for libvirt and lxc to introduce this socket-passing feature before it's more than an interesting idea.

Jan 28, 2013

Cleaning up LXC cgroups

I spent some time today looking at systemd (44) under Fedora (17). When stopping an LXC container using lxc-stop, I would always encounter this problem:

# lxc-stop -n node0
lxc-start: Device or resource busy - failed to remove cgroup '/sys/fs/cgroup/systemd/node0

This prevents one from starting a new container with the same name:

# lxc-start -n node0 
lxc-start: Device or resource busy - failed to remove previous cgroup '/sys/fs/cgroup/systemd/node0'
lxc-start: failed to spawn 'node0'
lxc-start: Device or resource busy - failed to remove cgroup '/sys/fs/cgroup/systemd/node0'

You can correct the problem manually by removing all the child cgroups underneath /sys/fs/cgroup/systemd/<container>, like this:

# find /sys/fs/cgroup/systemd/node0/ -type d |
  tac |
  xargs rmdir

The call to tac (which will output lines in reverse order) is necessary because we need to start with the "deepest" directory and work our way back up.

This appears to be a version-specific problem. I do not see the same behavior with systemd 197 under Arch.

Jan 28, 2013

How do I LXC console?

It took me an unreasonably long time to boot an LXC container with working console access. For the record:

When you boot an LXC container, the console appears to be attached to a pts device. For example, when booting with the console attached to your current terminal:

# lxc-start -n node0
...
node0 login: root
Last login: Mon Jan 28 16:35:19 on tty1
[root@node0 ~]# tty
/dev/console
[root@node0 ~]# ls -l /dev/console
crw------- 1 root tty 136, 12 Jan 28 16:36 /dev/console

This is also true when you attach to a container using lxc-console:

# lxc-start -n node0 -d
# lxc-console -n node0
Type <Ctrl+a q> to exit the console

node0 login: root
Last login: Mon Jan 28 16:36:00 on console
[root@node0 ~]# tty
/dev/tty1
[root@node0 ~]# ls -l /dev/tty1
crw------- 1 root tty 136, 6 Jan 28 16:37 /dev/tty1

In both cases, the devices have major number 136, which is the pts driver. This means that if your LXC configuration file has this:

lxc.cgroup.devices.deny = a

Then your LXC configuration file will also need:

lxc.cgroup.devices.allow = c 136:* rwm
Return button