Let's say that you have set up an environment using tripleo-quickstart and you would like to add another virtual machine to the mix that has both "external" connectivity ("external" in quotes because I am using it in the same way as the quickstart does w/r/t the undercloud) and connectivity to the overcloud nodes. How would you go about setting that up?

For a concrete example, let's presume you have deployed an environment using the default tripleo-quickstart configuration, which looks like this:

  - name: control_0
    flavor: control

  - name: compute_0
    flavor: compute

extra_args: >-
  --neutron-network-type vxlan
  --neutron-tunnel-types vxlan
  --ntp-server pool.ntp.org

network_isolation: true

That gets you one controller, one compute node, and enables network isolation. When your deployment is complete, networking from the perspective of the undercloud looks like this:

  • eth0 is connected to the host's brext bridge and gives the undercloud NAT access to the outside world. The interface will have an address on the network.

  • eth1 is connected to the host's brovc bridge, which is the internal network for the overcloud. The interface is attached to the OVS bridge br-ctlplane.

The br-ctlplane bridge has the address

And your overcloud environment probably looks something like this:

[stack@undercloud ~]$ nova list
| ID    ...| Name                    | Status |...| Networks           |
| 32f6ec...| overcloud-controller-0  | ACTIVE |...| ctlplane= |
| d98474...| overcloud-novacompute-0 | ACTIVE |...| ctlplane= |

We want to set up a new machine that has the same connectivity as the undercloud.

Upload an image

Before we can boot a new vm we'll need an image; let's start with the standard CentOS 7 cloud image. First we'll download it:

curl -O http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2

Let's add a root password to the image and disable cloud-init, since we're not booting in a cloud environment:

virt-customize -a CentOS-7-x86_64-GenericCloud.qcow2 \
  --root-password password:changeme \
  --run-command "yum -y remove cloud-init"

Now let's upload it to libvirt:

virsh vol-create-as oooq_pool centos-7-cloud.qcow2 8G \
  --format qcow2 \
  --allocation 0
virsh vol-upload --pool oooq_pool centos-7-cloud.qcow2 \

Boot the vm

I like to boot from a copy-on-write clone of the image, so that I can use the base image multiple times or quickly revert to a pristine state, so let's first create that clone:

virsh vol-create-as oooq_pool myguest.qcow2 10G \
  --allocation 0 --format qcow2 \
  --backing-vol centos-7-cloud.qcow2 \
  --backing-vol-format qcow2

And then boot our vm:

virt-install --disk vol=oooq_pool/myguest.qcow2,bus=virtio \
  --import \
  -r 2048 -n myguest --cpu host \
  --os-variant rhel7 \
  -w bridge=brext,model=virtio \
  -w bridge=brovc,model=virtio \
  --serial pty \

The crucial parts of the above command are the two -w ... arguments, which create interfaces attached to the named bridges.

We can now connect to the console and log in as root:

$ virsh console myguest
localhost login: root

We'll see that the system already has an ip address on the "external" network:

[root@localhost ~]# ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:7f:5c:5a brd ff:ff:ff:ff:ff:ff
    inet brd scope global dynamic eth0
       valid_lft 3517sec preferred_lft 3517sec
    inet6 fe80::5054:ff:fe7f:5c5a/64 scope link 
       valid_lft forever preferred_lft forever

And we have external connectivity:

[root@localhost ~]# ping -c1 google.com
PING google.com ( 56(84) bytes of data.
64 bytes from lga25s40-in-f14.1e100.net ( icmp_seq=1 ttl=56 time=20.6 ms

--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 20.684/20.684/20.684/0.000 ms

Let's give eth1 an address on the ctlplane network:

[root@localhost ~]# ip addr add dev eth1
[root@localhost ~]# ip link set eth1 up

Now we can access the undercloud:

[root@localhost ~]# ping -c1
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=64 time=0.464 ms

--- ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.464/0.464/0.464/0.000 ms

As well as all of the overcloud hosts using their addresses on the same network:

[root@localhost ~]# ping -c1
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=64 time=0.464 ms

--- ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.464/0.464/0.464/0.000 ms

Allocating an address using DHCP

In the above instructions we've manually assigned an ip address on the ctlplane network. This works fine for testing, but it could ultimately prove problematic if neutron were to allocate the same address to another overcloud host. We can use neutron to configure a static dhcp lease for our new host.

First, we need the MAC address of our guest:

virthost$ virsh dumpxml myguest |
  xmllint --xpath '//interface[source/@bridge="brovc"]' -
<interface type="bridge">
  <mac address="52:54:00:42:d6:c2"/>
  <source bridge="brovc"/>
  <target dev="tap9"/>
  <model type="virtio"/>
  <alias name="net1"/>
  <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0"/>

And then on the undercloud we run neutron port-create to create a port and associate it with our MAC address:

[stack@undercloud]$ neutron port-create --mac-address 52:54:00:42:d6:c2 ctlplane

Now if we run dhclient on our guest, it will acquire a lease from the neutron-managed DHCP server:

[root@localhost]# dhclient -d eth1
Internet Systems Consortium DHCP Client 4.2.5
Copyright 2004-2013 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/eth1/52:54:00:42:d6:c2
Sending on   LPF/eth1/52:54:00:42:d6:c2
Sending on   Socket/fallback
DHCPREQUEST on eth1 to port 67 (xid=0xc90c0ba)
DHCPACK from (xid=0xc90c0ba)
bound to -- renewal in 42069 seconds.

We can make this persistent by creating /etc/sysconfig/network-scripts/ifcfg-eth1:

[root@localhost]# cd /etc/sysconfig/network-scripts
[root@localhost]# sed s/eth0/eth1/g ifcfg-eth0 > ifcfg-eth1
[root@localhost]# ifup eth1
Determining IP information for eth1... done.

Provider external networks (in an appropriate amount of detail)

In Quantum in Too Much Detail, I discussed the architecture of a Neutron deployment in detail. Since that article was published, Neutron gained the ability to handle multiple external networks with a single L3 agent. While I wrote about that back in 2014, I covered the configuration side of it …

read more

Four ways to connect a docker container to a local network

Mon 11 August 2014 by Lars Kellogg-Stedman Tags networking docker openvswitch

Update (2018-03-22) Since I wrote this document back in 2014, Docker has developed the macvlan network driver. That gives you a supported mechanism for direct connectivity to a local layer 2 network. I've written an article about working with the macvlan driver.

This article discusses four ways to make a …

read more

Video: Configuring OpenStack's external bridge on a single-interface system

I've just put a video on Youtube that looks at the steps required to set up the external bridge (br-ex) on a single-interface system:

read more

Open vSwitch and persistent MAC addresses

Normally I like to post solutions, but today's post is about a vexing problem to which I have not been able to find a solution.

This started as a simple attempt to set up external connectivity on an all-in-one Icehouse install deployed on an OpenStack instance. I wanted to add eth0 to br-ex in order to model a typical method for providing external connectivity, but I ran into a very odd problem: the system would boot and work fine for a few seconds, but would then promptly lose network connectivity.

read more