In this post, we’ll walk through the process of getting virtual machines on two different hosts to communicate over an overlay network created using the support for VXLAN in Open vSwitch (or OVS).

The test environment

For this post, I’ll be working with two systems:

• node0.ovs.virt at address 192.168.122.107
• node1.ovs.virt at address 192.168.122.174

These hosts are running CentOS 8, although once we get past the package installs the instructions will be similar for other distributions.

I recently acquired a managed switch for my home office in order to segment a few devices off onto their own isolated vlan. As part of this, I want to expose these vlans on my desktop using Open vSwitch (OVS), and I wanted to implement the configuration using NetworkManager rather than either relying on the legacy /etc/sysconfig/network-scripts scripts or rolling my own set of services. These are my notes in case I ever have to do this again.

Introduction

A long time ago, I wrote an article all about OpenStack Neutron (which at that time was called Quantum). That served as an excellent reference for a number of years, but if you’ve deployed a recent version of OpenStack you may have noticed that the network architecture looks completely different. The network namespaces previously used to implement routers and dhcp servers are gone (along with iptables rules and other features), and have been replaced by OVN (“Open Virtual Network”). What is OVN? How does it work? In this article, I’d like to explore a minimal OVN installation to help answer these questions.

Let’s say that you have set up an environment using tripleo-quickstart and you would like to add another virtual machine to the mix that has both “external” connectivity (“external” in quotes because I am using it in the same way as the quickstart does w/r/t the undercloud) and connectivity to the overcloud nodes. How would you go about setting that up?

For a concrete example, let’s presume you have deployed an environment using the default tripleo-quickstart configuration, which looks like this:

In Quantum in Too Much Detail, I discussed the architecture of a Neutron deployment in detail. Since that article was published, Neutron gained the ability to handle multiple external networks with a single L3 agent. While I wrote about that back in 2014, I covered the configuration side of it in much more detail than I discussed the underlying network architecture. This post addresses the architecture side.

The players

This document describes the architecture that results from a particular OpenStack configuration, specifically:

Update (2018-03-22) Since I wrote this document back in 2014, Docker has developed the macvlan network driver. That gives you a supported mechanism for direct connectivity to a local layer 2 network. I’ve written an article about working with the macvlan driver.

This article discusses four ways to make a Docker container appear on a local network. These are not suggested as practical solutions, but are meant to illustrate some of the underlying network technology available in Linux.

I’ve just put a video on Youtube that looks at the steps required to set up the external bridge (br-ex) on a single-interface system:

Normally I like to post solutions, but today’s post is about a vexing problem to which I have not been able to find a solution.

This started as a simple attempt to set up external connectivity on an all-in-one Icehouse install deployed on an OpenStack instance. I wanted to add eth0 to br-ex in order to model a typical method for providing external connectivity, but I ran into a very odd problem: the system would boot and work fine for a few seconds, but would then promptly lose network connectivity.

In my previous post I discussed a problem I was having setting a persistent MAC address on an OVS bridge device. It looks like the short answer is, “don’t use ip link set ...” for this purpose.

You can set the bridge MAC address via ovs-vsctl like this:

ovs-vsctl set bridge br-ex other-config:hwaddr=$MACADDR

So I’ve updated my ifconfig-br-ex to look like this:

DEVICE=br-ex
DEVICETYPE=ovs
TYPE=OVSBridge
ONBOOT=yes
OVSBOOTPROTO=dhcp
OVSDHCPINTERFACES=eth0
MACADDR=fa:16:3e:ef:91:ec
OVS_EXTRA="set bridge br-ex other-config:hwaddr=$MACADDR"

The OVS_EXTRA parameter gets passed to the add-br call like this:

I run OpenStack on my laptop, and I’ve been chasing down a pernicious problem with OVS bridge interfaces under both F19 and F20. My OpenStack environment relies on an OVS bridge device named br-ex for external connectivity and for making services available to OpenStack instances, but after rebooting, br-ex was consistently unconfigured, which caused a variety of problems.

This is the network configuration file for br-ex on my system:

DEVICE=br-ex
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROT=static
IPADDR=192.168.200.1
NETMASK=255.255.255.0
ONBOOT=yes
NM_CONTROLLED=no
ZONE=openstack

Running ifup br-ex would also fail to configure the interface, but running ifdown br-ex; ifup br-ex would configure things appropriately.