In Quantum in Too Much Detail, I discussed the architecture of a Neutron deployment in detail. Since that article was published, Neutron gained the ability to handle multiple external networks with a single L3 agent. While I wrote about that back in 2014, I covered the configuration side of it in much more detail than I discussed the underlying network architecture. This post addresses the architecture side.

The players

This document describes the architecture that results from a particular OpenStack configuration, specifically:

  • Neutron networking using VXLAN or GRE tunnels;
  • A dedicated network controller;
  • Two external networks

The lay of the land

This is a simplified architecture diagram of the network connectivity in this scenario:

Everything on the compute hosts is identical to my previous article, so I will only be discussing the network host here.

For the purposes of this article, we have two external networks and two internal networks defined:

$ neutron net-list
+--------------------------------------+-----------+----------...------------------+
| id                                   | name      | subnets  ...                  |
+--------------------------------------+-----------+----------...------------------+
| 6f0a5622-4d2b-4e4d-b34a-09b70cacf3f1 | net1      | beb767f8-... 192.168.101.0/24 |
| 972f2853-2ba6-474d-a4be-a400d4e3dc97 | net2      | f6d0ca0f-... 192.168.102.0/24 |
| 12136507-9bbe-406f-b68b-151d2a78582b | external2 | 106db3d6-... 172.24.5.224/28  |
| 973a6eb3-eaf8-4697-b90b-b30315b0e05d | external1 | fe8e8193-... 172.24.4.224/28  |
+--------------------------------------+-----------+----------...------------------+

And two routers:

$ neutron router-list
+--------------------------------------+---------+-----------------------...-------------------+...
| id                                   | name    | external_gateway_info ...                   |...
+--------------------------------------+---------+-----------------------...-------------------+...
| 1b19e179-5d67-4d80-8449-bab42119a4c5 | router2 | {"network_id": "121365... "172.24.5.226"}]} |...
| e2117de3-58ca-420d-9ac6-c4eccf5e7a53 | router1 | {"network_id": "973a6e... "172.24.4.227"}]} |...
+--------------------------------------+---------+-----------------------...-------------------+...

And our logical connectivity is:

+---------+    +----------+    +-------------+
|         |    |          |    |             |
|  net1   +----> router1  +---->  external1  |
|         |    |          |    |             |
+---------+    +----------+    +-------------+

+---------+    +----------+    +-------------+
|         |    |          |    |             |
|  net2   +----> router2  +---->  external2  |
|         |    |          |    |             |
+---------+    +----------+    +-------------+

Router attachments to integration bridge

In the legacy model, in which an L3 agent supported a single external network, the qrouter-... namespaces that implement Neutron routers were attached to both the integration bridge br-int and the external network bridge (the external_network_bridge configuration option from your l3_agent.ini, often named br-ex).

In the provider network model, both interfaces in a qrouter namespace are attached to the integration bridge. For the configuration we've described above, the configuration of the integration bridge ends up looking something like:

Bridge br-int
    fail_mode: secure
    Port "qvoc532d46c-33"
        tag: 3
        Interface "qvoc532d46c-33"
    Port br-int
        Interface br-int
            type: internal
    Port "qg-09e9da38-fb"
        tag: 4
        Interface "qg-09e9da38-fb"
            type: internal
    Port "qvo3ccea690-c2"
        tag: 2
        Interface "qvo3ccea690-c2"
    Port "int-br-ex2"
        Interface "int-br-ex2"
            type: patch
            options: {peer="phy-br-ex2"}
    Port "tapd2ff89e7-16"
        tag: 2
        Interface "tapd2ff89e7-16"
            type: internal
    Port patch-tun
        Interface patch-tun
            type: patch
            options: {peer=patch-int}
    Port "int-br-ex1"
        Interface "int-br-ex1"
            type: patch
            options: {peer="phy-br-ex1"}
    Port "qr-affdbcee-5c"
        tag: 3
        Interface "qr-affdbcee-5c"
            type: internal
    Port "qr-b37877cd-42"
        tag: 2
        Interface "qr-b37877cd-42"
            type: internal
    Port "qg-19250d3f-5c"
        tag: 1
        Interface "qg-19250d3f-5c"
            type: internal
    Port "tap0881edf5-e5"
        tag: 3
        Interface "tap0881edf5-e5"
            type: internal

The qr-... interface on each router is attached to an internal network. The VLAN tag associated with this interface is whatever VLAN Neutron has selected internally for the private network. In the above output, these ports are on the network named net1:

Port "qr-affdbcee-5c"
    tag: 3
    Interface "qr-affdbcee-5c"
        type: internal
Port "tap0881edf5-e5"
    tag: 3
    Interface "tap0881edf5-e5"
        type: internal

Where qr-affdbcee-5c is router1's interface on that network, and tap0881edf5-e5 is the port attached to a dhcp-... namespace. The same router is attached to the external1 network; this attachment is represented by:

Port "qg-09e9da38-fb"
    tag: 4
    Interface "qg-09e9da38-fb"
        type: internal

The external bridges are connected to the integration bridge using OVS "patch" interfaces (the int-br-ex1 on the integration bridge and the phy-br-ex1 interface on the br-ex1).

From here to there

Connectivity between the qg-... interface and the appropriate external bridge (br-ex1 in this case) happens due to the VLAN tag assigned on egress by the qg-... interface and the following OpenFlow rules associated with br-ex1:

# ovs-ofctl dump-flows br-ex1
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=794.876s, table=0, n_packets=0, n_bytes=0, idle_age=794, priority=1 actions=NORMAL
 cookie=0x0, duration=785.833s, table=0, n_packets=0, n_bytes=0, idle_age=785, priority=4,in_port=3,dl_vlan=4 actions=strip_vlan,NORMAL
 cookie=0x0, duration=792.945s, table=0, n_packets=24, n_bytes=1896, idle_age=698, priority=2,in_port=3 actions=drop

Each of these rules contains some state information (like the packet/byte counts), some conditions (like priority=4,in_port=3,dl_vlan=4) and one or more actions (like actions=strip_vlan,NORMAL). So, the second rule there matches packets associated with VLAN tag 4 and strips the VLAN tag (after which the packet is delivered to any physical interfaces that are attached to this OVS bridge).

Putting this all together:

  1. An outbound packet from a Nova server running on a compute node enters via br-tun (H)

  2. Flow rules on br-tun translate the tunnel id into an internal VLAN tag.

  3. The packet gets delivered to the qr-... interface of the appropriate router. (O)

  4. The packet exits the qg-... interface of the router (where it is assigned the VLAN tag associated with the external network). (N)

  5. The packet is delivered to the external bridge, where a flow rule strip the VLAN tag. (P)

  6. The packet is sent out the physical interface associated with the bridge.

For the sake of completeness

The second private network, net2, is attached to router2 on the qr-b37877cd-42 interface. It exits on the qg-19250d3f-5c interface, where packets will be assigned to VLAN 1:

Port "qr-b37877cd-42"
    tag: 2
    Interface "qr-b37877cd-42"
        type: internal
Port "qg-19250d3f-5c"
    tag: 1
    Interface "qg-19250d3f-5c"
        type: internal

The network interface configuration in the associated router namespace looks like this:

# ip netns exec qrouter-1b19e179-5d67-4d80-8449-bab42119a4c5 ip a
30: qg-19250d3f-5c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:01:e9:e3 brd ff:ff:ff:ff:ff:ff
    inet 172.24.5.226/28 brd 172.24.5.239 scope global qg-19250d3f-5c
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe01:e9e3/64 scope link 
       valid_lft forever preferred_lft forever
37: qr-b37877cd-42: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:4c:6c:f2 brd ff:ff:ff:ff:ff:ff
    inet 192.168.102.1/24 brd 192.168.102.255 scope global qr-b37877cd-42
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe4c:6cf2/64 scope link 
       valid_lft forever preferred_lft forever

OpenFlow rules attached to br-ex2 will match these packets:

# ovs-ofctl dump-flows br-ex2
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=3841.678s, table=0, n_packets=0, n_bytes=0, idle_age=3841, priority=1 actions=NORMAL
 cookie=0x0, duration=3831.396s, table=0, n_packets=0, n_bytes=0, idle_age=3831, priority=4,in_port=3,dl_vlan=1 actions=strip_vlan,NORMAL
 cookie=0x0, duration=3840.085s, table=0, n_packets=26, n_bytes=1980, idle_age=3742, priority=2,in_port=3 actions=drop

We can see that the second rule here will patch traffic on VLAN 1 (priority=4,in_port=3,dl_vlan=1) and strip the VLAN tag, after which the packet will be delivered to any other interfaces attached to this bridge.


OpenStack Networking without DHCP

In an OpenStack environment, cloud-init generally fetches information from the metadata service provided by Nova. It also has support for reading this information from a configuration drive, which under OpenStack means a virtual CD-ROM device attached to your instance containing the same information that would normally be available via the …

read more

Cloud-init and the case of the changing hostname

Wed 10 December 2014 by Lars Kellogg-Stedman Tags openstack rdo neutron

Setting the stage

I ran into a problem earlier this week deploying RDO Icehouse under RHEL 6. My target systems were a set of libvirt guests deployed from the RHEL 6 KVM guest image, which includes cloud-init in order to support automatic configuration in cloud environments. I take advantage of …

read more

Booting an instance with multiple fixed addresses

This article expands on my answer to Add multiple specific IPs to instance, a question posted to ask.openstack.org.

In order to serve out SSL services from an OpenStack instance, you will generally want one local ip address for each SSL virtual host you support. It is possible to …

read more

Multiple external networks with a single L3 agent

In the old days (so, like, last year), Neutron supported a single external network per L3 agent. You would run something like this...

$ neutron net-create external --router:external=true

...and neutron would map this to the bridge defined in external_network_bridge in /etc/neutron/l3_agent.ini. If you wanted to support …

read more