Normally I like to post solutions, but today’s post is about a vexing problem to which I have not been able to find a solution.
This started as a simple attempt to set up external connectivity on
an all-in-one Icehouse install deployed on an OpenStack instance. I
wanted to add
br-ex in order to model a typical method for
providing external connectivity, but I ran into a very odd problem:
the system would boot and work fine for a few seconds, but would then
promptly lose network connectivity.
The immediate cause was that the MAC address on
br-ex was changing.
I was setting the MAC explicitly in the configuration file:
# cat ifcfg-br-ex DEVICE=br-ex DEVICETYPE=ovs TYPE=OVSBridge ONBOOT=yes OVSBOOTPROTO=dhcp OVSDHCPINTERFACES=eth0 MACADDR=fa:16:3e:ef:91:ec
This was required in this case in order to make the MAC-address filters on the host happy. When booting an instance, Neutron sets up a rule like this:
-A neutron-openvswi-s55439d7d-a -s 10.0.0.8/32 -m mac --mac-source FA:16:3E:EF:91:EC -j RETURN -A neutron-openvswi-s55439d7d-a -j DROP
But things quickly got weird. Some testing demonstrated that the MAC
address was changing when starting
neutron-openvswitch-agent, but a
thorough inspection of the code didn’t yield any obvious culprits for
I liberally sprinkled the agent with the following (incrementing the
echo each time to uniquely identify each message):
os.system('echo 1 >> /tmp/ovs.log; ip link show dev br-ex >> /tmp/ovs.log')
It turns out that the MAC address on
br-ex was changing…when
Neutron was deleting a port on
br-int. Specifically, at this
After some additional testing, it turns out that just about any OVS operation causes an explicit MAC address to disappear. For example, create a new OVS bridge:
# ovs-vsctl add-br br-test0 # ip link show dev br-test0 9: br-test0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether ba:cb:48:b9:6a:43 brd ff:ff:ff:ff:ff:ff
Then set the MAC address:
# ip link set br-test0 addr c0:ff:ee:ee:ff:0c # ip link show br-test0 8: br-test0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether c0:ff:ee:ee:ff:0c brd ff:ff:ff:ff:ff:ff
Now create a new bridge:
# ovs-vsctl add-br br-test1
And inspect the MAC address on the first bridge:
# ip link show dev br-test0 9: br-test0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether ba:cb:48:b9:6a:43 brd ff:ff:ff:ff:ff:ff
In other words, creating a new bridge caused the MAC address on
br-ex to revert. Other operations (e.g., deleting a port on an
unrelated switch) will cause the same behavior.
I’ve seen this behavior on both versions
So far everyone I’ve asked about this behavior has been stumped. If I am able to figure out what’s going on I will update this post. Thanks for reading!