typedef int (*funcptr)();

An engineers technical notebook

Neutron L3 agent with multiple provider networks

Due to requirements outside of my control, there was a requirement to run multiple "provider" networks each with each providing their own floating address pool from a single network node, I wanted to do this as simply as possible using a single l3 agent rather than having to figure out how to get systemd to start multiple with different configuration files.

Currently I've installed and configured an OpenStack instance that looks like this:

|                     |
|                  +--+----+
|                  |       |
|      +-----------+-+  +--+----------+
|      | Compute     |  | Compute     |
|      |     01      |  |     02      |
|      +------+------+  +-----+-------+
|             |               |
|             |               +----------+
|             +------------+--+          |
|                          |             |
| +-------------+    +-----+-------+     |
| | Controller  |    |   Network   |     |
| |             |    |             |     +---+  Tenant Networks (vlan tagged) (vlan ID's 350 - 400)
| +-----+----+--+    +------+----+-+
|       |    |              |    |
|       |    |              |    +-----------+  Floating Networks (vlan tagged) (vlan ID's 340 - 349)
|       |    |              |
|       |    |              |
+------------+--------------+----------------+  Management Network (
        +------------------------------------+  External API Network (

There are two compute nodes, a controller node that runs all of the API services, and a network node that is strictly used for providing network functions (routers, load balancers, firewalls, all that fun stuff!).

There are two flat networks that provide the following:

  1. External API access
  2. A management network that OpenStack uses internally to communicate between instances and to manage it, which is not accessible from the other three networks.

The other two networks are both vlan tagged:

  1. Tenant networks, with the possibility of 50 vlan ID's
  2. Floating networks, with existing vlan ID's for existing networks

Since the OpenStack Icehouse release, the l3 agent has supported the ability to use the Open vSwitch configuration to specify how traffic should be routed rather than statically defining that a single l3 agent routes certain traffic to a single Linux bridge. Setting this up is fairly simple if you follow the documentation, with one caveat, variables you think would be defined to no value, actually have a value and thus need to be explicitly zeroed out.

On the network node

First, we need to configure the l3 agent, so let's set some extra variables in /etc/neutron/l3-agent.ini:

gateway_external_network_id =
external_network_bridge =

It is important that these two are set, not left commented out, unfortunately when commented out they have some defaults set and it will fail to work, so explicitly setting them to blank will fix that issue.

Next, we need to set up our Open vSwitch configuration. In /etc/neutron/plugin.ini the following needs to be configured:

  • bridge_mappings
  • network_vlan_ranges

Note, that these may already be configured, in which case there is nothing left to do. Mine currently looks like this:

bridge_mappings = tenant1:br-tnt,provider1:br-ex

This basically specifies that any networks created under "provider name" tenant1 are going to be mapped to the Open vSwitch bridge br-tnt and any networks with "provider name" provider1 will be mapped to br-ex.

br-tnt is mapped to my tenant network and on the switch has vlan ID's 350 - 400 assigned, and br-ex has vlan ID's 340 - 349 assigned.

Following the above knowledge, my network_vlan_ranges is configured as such:

network_vlan_ranges = tenant1:350:400,provider1:340:349

Make sure to restart all neutron services:

openstack-service restart neutron

On the controller (where neutron-server lives)

On the controller we just need to make sure that our network_vlan_ranges matches what is on the network node, with one exception, we do not list our provider1 vlan ranges since we don't want to make those available to accidentally be assigned when a regular tenant creates a new network.

So our configuration should list:

network_vlan_ranges = tenant1:350:400

Make sure that all neutron services are restarted:

openstack-service restart neutron

Create the Neutron networks

Now, as an administrative user we need to create the provider networks.

source ~/keystonerc_admin

neutron net-create "" \
--router:external True \
--provider:network_type vlan \
--provider:physical_network provider1 \
--provider:segmentation_id 340

neutron net-create "" \
--router:external True \
--provider:network_type vlan \
--provider:physical_network provider1 \
--provider:segmentation_id 341

Notice how we've created two networks, given them each individual names (I like to use the name of the network they are going to be used for) and have been attached to the provider1. Note that provider1 is completely administratively defined, and could just as well have been physnet1, so long as it is consistent across all of the configuration files.

Now let's create subnets on this network:

neutron subnet-create "" \
--allocation-pool start=,end= \
--disable-dhcp --gateway

neutron subnet-create "" \
--allocation-pool start=,end= \
--disable-dhcp --gateway

Now that these networks are defined, you should be able to have tenants create routers and set their gateways to either of these new networks by selecting from the drop-down in Horizon or by calling neutron router-gateway-set <router id> <network id> on the command line.

The l3 agent will automatically configure and set up the router as required on the network node, and traffic will flow to either vlan 340 or vlan 341 as defined above depending on what floating network the user uses as a gateway.

This drastically simplifies the configuration of multiple floating IP networks since no longer is there a requirement to start up and configure multiple l3 agents each with their own network ID configured. This makes configuration less brittle and easier to maintain over time.

OpenStack resizing of instances

One thing that is not always adequately explained in the OpenStack documentation is how exactly instance resizing works, and what is required, especially while using KVM as the virtualisation provider, with multiple compute nodes.

You might find something similiar to the following in your logs, and no good documentation on how to fix it.

ERROR nova.compute.manager [req-7cb1c029-beb4-4905-a9d9-62d488540eda f542d1b5afeb4908b8b132c4486f9fa8 c2bfab5ad24642359f43cdff9bb00047] [instance: 99736f90-db0f-4cba-8f44-a73a603eee0b] Setting instance vm_state to ERROR
TRACE nova.compute.manager [instance: 99736f90-db0f-4cba-8f44-a73a603eee0b] Traceback (most recent call last):
TRACE nova.compute.manager [instance: 99736f90-db0f-4cba-8f44-a73a603eee0b]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5596, in _error_out_instance_on_exception
TRACE nova.compute.manager [instance: 99736f90-db0f-4cba-8f44-a73a603eee0b]     yield
TRACE nova.compute.manager [instance: 99736f90-db0f-4cba-8f44-a73a603eee0b]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3459, in resize_instance
TRACE nova.compute.manager [instance: 99736f90-db0f-4cba-8f44-a73a603eee0b]     block_device_info)
TRACE nova.compute.manager [instance: 99736f90-db0f-4cba-8f44-a73a603eee0b]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4980, in migrate_disk_and_power_off
TRACE nova.compute.manager [instance: 99736f90-db0f-4cba-8f44-a73a603eee0b]     utils.execute('ssh', dest, 'mkdir', '-p', inst_base)
TRACE nova.compute.manager [instance: 99736f90-db0f-4cba-8f44-a73a603eee0b]   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 165, in execute
TRACE nova.compute.manager [instance: 99736f90-db0f-4cba-8f44-a73a603eee0b]     return processutils.execute(*cmd, **kwargs)
TRACE nova.compute.manager [instance: 99736f90-db0f-4cba-8f44-a73a603eee0b]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py", line 193, in execute
TRACE nova.compute.manager [instance: 99736f90-db0f-4cba-8f44-a73a603eee0b]     cmd=' '.join(cmd))
TRACE nova.compute.manager [instance: 99736f90-db0f-4cba-8f44-a73a603eee0b] ProcessExecutionError: Unexpected error while running command.
TRACE nova.compute.manager [instance: 99736f90-db0f-4cba-8f44-a73a603eee0b] Command: ssh mkdir -p /var/lib/nova/instances/99736f90-db0f-4cba-8f44-a73a603eee0b
TRACE nova.compute.manager [instance: 99736f90-db0f-4cba-8f44-a73a603eee0b] Exit code: 255
TRACE nova.compute.manager [instance: 99736f90-db0f-4cba-8f44-a73a603eee0b] Stdout: ''
TRACE nova.compute.manager [instance: 99736f90-db0f-4cba-8f44-a73a603eee0b] Stderr: 'Host key verification failed.\r\n'
TRACE nova.compute.manager [instance: 99736f90-db0f-4cba-8f44-a73a603eee0b] 
ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: Unexpected error while running command.
Command: ssh mkdir -p /var/lib/nova/instances/99736f90-db0f-4cba-8f44-a73a603eee0b
Exit code: 255
Stdout: ''
Stderr: 'Host key verification failed.\r\n'

When OpenStack's nova is instructed to resize an instance it will also change the host it is running on, almost never will it schedule the instance on the same host and do the resize on the same host it already exists. There is a configuration flag to change this, however in my case I would rather the scheduler be run again, especially if the instance size is changing drastically. During the resize process, the node where the instance is currently running will use SSH to connect to the instance where the resized instance will live, and copy over the instance and associated files.

There are a couple of assumptions I will be making:

  1. Your nova, and qemu user both have the same UID on all compute nodes
  2. The path for your instances is the same on all of your compute nodes

Configure the nova user

First things first, let's make sure our nova user has an appropriate shell set:

cat /etc/passwd | grep nova

Verify that the last entry is /bin/bash.

If not, let's modify the user and make it so:

usermod -s /bin/bash nova

Generate SSH key and configuration

After doing this the next steps are all run as the nova user.

su - nova

We need to generate an SSH key:

ssh-keygen -t rsa

Follow the directions, and save the key WITHOUT a passphrase.

Next up we need to configure SSH to not do host key verification, unless you want to manually SSH to all compute nodes that exist and accept the key (and continue to do so for each new compute node you add).

cat << EOF > ~/.ssh/config
Host *
    StrictHostKeyChecking no

Next we need to make sure we copy the the contents of id_rsa.pub to authorized_keys and set the mode on it correctly.

cat ~/.ssh/id_rsa.pub > .ssh/authorized_keys
chmod 600 .ssh/authorized_keys

This should be all the configuration for SSH you need to do. Now comes the import part, you will need to tar up and copy the ~nova/.ssh directory to every single compute node you have provisioned. This way all compute nodes will be able to SSH to the remote host to run the commands required to copy an instance over, and resize it.

Reset state on existing ERROR'ed instances

If you have any instances that are currently in the ERROR state due to a failed resize, you will be able to issue the following command to reset the state back to running and try again:

nova reset-state --active <ID of instance>

This will start the instance, and you will be able to once again issue the resize command to resize the instance.