ML2 Mechanism driver and small control plane for the [VPP forwarder](What is VPP)

This is a Neutron mechanism driver to bring the advantages of VPP to OpenStack deployments.

It's been written to be as simple and readable as possible while offering either full Neutron functionality or a simple roadmap to it. While the driver is not perfect, we're aiming for

• robustness in the face of failures (of one of several Neutron servers, of agents, of the etcd nodes in the cluster gluing them together)
• simplicity
• testability - having failure cases covered is no good if you don't have a means to test the code that protects you from them

As a general rule, everything is implemented in the simplest way, for three reasons: we get to see it working faster, we can test it, and anyone that wants to join the project can make sense of it.

There's a devstack plugin. You can add this plugin to your local.conf and see it working. The devstack plugin now takes care of

• installing the networking-vpp code
• installing VPP itself (version 18.04)
• installing etcd
• using a QEMU version that supports vhostuser well

To get the best performance, this will use vhostuser sockets to talk to VMs, which means you need a modern version of your OS (CentOS 7 and Ubuntu 16.04 work). It also means that you need to run your VMs with a special flavor that enables shared memory - basically, you need to set up hugepages for your VMs, as that's the only supported way Nova does this today. Because you're using pinned shared memory you are going to find you can't overcommit memory on the target machine. The devstack plugin converts all the default flavours to use shared memory.

If you want to build from components yourself, you can certainly get this working with your own VPP build or with a newer QEMU version, but you may want to read the files in devstack/ to work out how we choose to configure the system.

We've made some effort to make this backward-compatible so that it will work with older stable branches as well as the current master branch of Neutron. You should find this will work with Newton, Mitaka and Liberty.

Before you start, add the following to your kernel options and reboot.

iommu=pt intel_iommu=on

You will need to set some configuration items in your local.conf to get the system running.

One of the important things you need to know before you start is that VPP is an entirely user-space forwarder. It doesn't live in the kernel, it doesn't use kernel network devices; using DPDK, it steals NICs away from the kernel for its own nefarious uses. So - to test VPP - you can do one of two things:

• choose a device to give to VPP, remembering that it will not be useful for anything else (so it's a really good idea not to use the interface with the host IP address)
• use a loopback or TAP interface in VPP to keep it quiet (perfect for one node test systems).

I recommend the following bits of configuration and use them when I'm testing. Make a local.conf in your devstack directory that looks something like this:

[[local|localrc]]
# We are going to use memory in the system for 2M hugepages.  Pick
# a number you can afford to lose.  Here we're taking 2500 pages
# (about 5GB of memory) which works well in my 8GB test VM.
NR_HUGEPAGES=2500

disable_service q-agt # we're not using OVS or LB
enable_plugin networking-vpp https://github.com/openstack/networking-vpp
Q_PLUGIN=ml2
Q_USE_SECGROUP=True
Q_ML2_PLUGIN_MECHANISM_DRIVERS=vpp
Q_ML2_PLUGIN_TYPE_DRIVERS=vlan,flat
Q_ML2_TENANT_NETWORK_TYPE=vlan
ML2_VLAN_RANGES=physnet:100:200
MECH_VPP_PHYSNETLIST=physnet:tap-0

[[post-config|$NOVA_CONF]]
[DEFAULT]
# VPP uses some memory internally.  reserved_huge_pages
# tells Nova that we cannot allocate the last 512 pages
# (1GB) of memory to VMs, because in practice it will be
# used already and those VMs won't start.  If you tweak
# the VPP options you can reduce this number, and 1GB is
# probably too much for a test VM, but it's the default
# for a 2 core machine.
reserved_huge_pages=node:0,size:2048,count:64

and a startup.conf file like this:

unix {
  nodaemon
  log /tmp/vpp.log
  full-coredump
  startup-config /etc/vpp-startup.conf
}

api-trace {
  on
}

dpdk {
        socket-mem 128
}

There are a few settings up there you might want to tweak.

Firstly, it's important that you get the memory allocation right -we're going to take the memory in your system, make some of it into hugepages, and then hand those hugepages to VPP and OpenStack.

Above, the NR_HUGEPAGES setting says how many 2MB hugepages are allocated from the system. This is a balancing act - you need a number that leaves normal memory behind for the OS and the OpenStack processes, but VPP and the VMs you run will all come out of the hugepage allocation. 2500 pages - about 5GB - works well on an 8GB system.

From that memory, VPP will use some. The socket-mem line says how much memory in MB it will use for each core. The above line tells it to give one core 128MB of memory (64 pages). You can change this number or make it a comma separated list to add memory to additional cores, but again that's a good place to start.

VMs that run in VPP systems have to use hugepages for their memory, so we have a little under 5GB of memory remaining in this example to give to the VMs we run.

The reserved_huge_pages is a count of hugepages that OpenStack will not be allowed to give out to VMs - it works out there are 2500 pages available, and this line tells it that 64 of those pages are not its to give away (because VPP has used them). If you get this line wrong, you will end up with scheduling problems.

Secondly, you need to sort out an 'uplink' port. This is the port on your VM that is used to connect the OpenStack VMs to the world. The above local.conf has the line:

MECH_VPP_PHYSNETLIST=physnet:tap-0

That tap-0 is the name of a VPP interface, and you can change it to suit your setup.

VPP is designed specifically to take one whole interface from the kernel and use it as the uplink. If you have a DPDK compatible 1Gbit card, the interface is typically GigabitEthernet2/2/0 - but this does depend a bit on your hardware setup, so you may need to run devstack, then run the command 'sudo vppctl show int' - which will list the interfaces that VPP found - fix the local.conf file and try again. (If your situation is especially unusual, you will need to go look at VPP's documentation at <https://wiki.fd.io/> to work out how VPP chooses its interfaces and things about how its passthrough drivers work). If you're setting up a multinode system, bridge this between the servers and it will form the Neutron dataplane link.

Another option is to use loop0 - this is a loopback device. Using this. you can get things up and running, but you won't get access to the tenant networks from outside of VPP (though you can still use the 'ip netns exec' trick through router namespaces). You can run two VMs and talk between them by logging in on the console, for instance.

If you need a 'loop0' interface, you have to make VPP create it at startup. Add the following line to your startup.conf file:

unix {
...
    startup-config /etc/vpp-commands.txt
}

And create that /etc/vpp-commands.txt containing the line:

create loopback interface

A third option is half way between the other two. You can use tap-0 in your configuration, and make a Linux kernel TAP device to connect your host kernel to your VMs. This means you can easily run a one node setup without needing an extra NIC port, but you can still connect to the networks inside OpenStack using that interface and any VLAN subinterfaces you care to create. You can even set up masquerade rules so that your VMs can talk to the world though your machine's kernel NIC.

To use a TAP device, set up the vpp-commands.txt file as above but put in the line:

tap connect uplink

When VPP runs, it will create a new TAP interface uplink, which you can being up, address, bridge, etc. as you see fit. That device is bridged to the VLANs that the VMs are attached to.

After all this, run ./stack.sh to make devstack run.

NB: A number of the important options are set by default to allow out-of-the-box operation. Configuration defaults (including ETCD settings and VPP branch specification) are found in devstack/settings.

VPP, and the VMs it runs, need hugepages, and the plugin will make you some automatically - the default setting for the number of hugepages is 1024 (2GB).

If the specified VPP uplink interface in the physnet list is tap-0, the plugin will create it in VPP to use if it's not already present (and you won't have to give a physical interface up to VPP and work out the configuration steps, which can be quite involved). This will turn up on your host as an interface named 'test', which you should be able to use normally -you can give it an address, add routes, set up NAT or even make VLAN subinterfaces.

Take a peek into the init_networking_vpp function of devstack/plugin.sh (executed at stack-time) to see some of what's happening.

To check whether VPP has started run ps -ef and look for:

/usr/bin/vpp -c /etc/vpp/startup.conf

You may need to add the kernel command line option:

iommu=pt

to your kernel before VPP starts. It depends on the Linux deployment you're using. Refer to the VPP documentation if you need more help.

If running on VirtualBox you will need to use an experimental option to allow SSE4.2 passthrough from the host CPU to the VM. Refer to the VirtualBox Manual for details.

Today, it supports VLANs, VXLAN-GPE and flat networks.

networking-vpp provides the glue from the Neutron server process to a set of agents that control, and the agents that turn Neutron's needs into specific instructions to VPP.

The glue is implemented using a very carefully designed system using etcd. The mechanism driver, within Neutron's API server process, works out what the tenants are asking for and, using a special failure tolerant journalling mechanism, feeds that 'desired' state into a highly available consistent key-value store, etcd. If a server process is reset, then the journal - in the Neutron database -contains all the records that still need writing to etcd.

etcd itself can be set up to be redundant (by forming a 3-node quorum, for instance, which tolerates a one node failure), which means that data stored in it will not be lost even in the event of a problem.

The agents watch etcd, which means that they get told if any data they are interested in is updated. They keep an eye out for any changes on their host - so, for instance, ports being bound and unbound - and on anything of related interest, like security groups. If any of these things changes, the agent implements the desired state in VPP. If the agent restarts, it reads the whole state and loads it into VPP.

This mechanism driver doesn't do anything at all until Neutron needs to drop traffic on a compute host, so the only thing it's really interested in is ports. Making a network or a subnet doesn't do anything at all.

And it mainly interests itself in the process of binding: the bind calls called by ML2 determine if it has work to do, and the port postcommit calls push the data out to the agents once we're sure it's recorded in the DB. (We do something similar with security group information.)

In our case, we add a write to a journal table in the database during the same transaction that stores the state change from the API. That means that, if the user asked for something, Neutron has agreed to do it, and Neutron remembered to write all of the details down, it makes it to the journal; and if Neutron didn't finish saving it, it doesn't get recorded, either in Neutron's own records or in the journal. In this way we keep etcd in step with the Neutron database -both are updated, or neither is.

The postcommit calls are where we need to push the data out to the agents - but the OpenStack user is still waiting for an answer, so it's wise to be quick. In our case, we kick a background thread to push the journal out, in strict order, to etcd. There's a little bit of a lag (it's tiny, in practice) before etcd gets updated, but this way if there are any issues within the cloud (a congested network, a bad connection) we don't keep the user waiting and we also don't forget what we agreed to do.

Once it's in etcd, the agents will spot the change and change their state accordingly.

To ensure binding is done correctly, we send Nova a notification only when the agent has definitely created the structures in VPP necessary for the port to work, and only when the VM has attached to VPP. In this way we know that even the very first packet from the VM will go where it's meant to go - kind of important when that packet's usually asking for an IP address.

Additionally, there are some helper calls to determine if this mechanism driver, in conjunction with the other ones on the system, needs to do anything. In some cases it may not be responsible for the port at all.

NOTE: As of release 18.04, the native L3 service plugin (vpp-router) is

fully supported in L3 HA configuration for VLAN, Flat and VXLAN-GPE type networks.

To enable the vpp-router plugin add the following in neutron.conf:

service_plugins = vpp-router

And make sure the Openstack L3 agent is not running. You will need to nominate one or more hosts to act as the Layer 3 gateway host(s) in ml2_conf.ini:

[ml2_vpp]
l3_hosts = <my_l3_gateway_host.domain>

The L3 host(s) will need L2 adjacency and connectivity to the compute hosts to terminate tenant VLANs and route traffic properly.

The vpp-agent acts as a common L2 and L3 agent so it needs to be started on the L3 host as well.

In the 18.04 release, we fully support Layer3 HA for VPP.

First, ensure that the vpp-router plugin is enabled.

Next, enable Layer3 HA by adding enable_l3_ha=True in the ml2.ini configuration file (as shown below).

Lastly, you need to provide the list of Layer3 hosts in your deployment, using a comma separated notation (as shown below) in the ml2.ini configuration file.

[ml2_vpp]
l3_hosts = node1,node2
enable_l3_ha = True

We use the keepalived to do the ACTIVE/BACKUP router election using VRRP. You can find these sample keepalived config files and scripts in the /tools directory.

First, you need to decide which node will become the master. Use the keepalived.master.conf file on the master node, and the keepalived.backup.conf on all the backup nodes. Just copy these files from the tools directory to /etc/keepalived.conf on the respective nodes.

Now, when there is a state transition to MASTER, keepalived will run the script named master.sh, to notify the master election, and it will run the backup.sh, when the node state transitions to BACKUP. These scripts are also in the /tools directory.

The argument required for each of these scripts is the ${hostname} of the node. These scripts update a key in etcd used to control the router state.

The etcd key is: "/networking-vpp/nodes/${HOSTNAME}/routers/ha" The value of this key is set to 1 on the MASTER node and 0 on the BACKUP. If the key is not present or it's value unset, the router will become a BACKUP.

NOTE: If you are on a non-HA environment, i.e., there's a single network node, set enable_l3_ha=False in ml2_conf.ini

The vpp-agent listens for watch events on this key. On the master node, the router BVI interfaces are enabled and they are disabled on all the backup nodes.

This uses the Python API module that comes with VPP (vpp_papi). VPP has an admin channel, implemented in shared memory, to exchange control messages with whatever agent is running. The Python bindings are a very thin layer between that shared memory system and a set of Python APIs. We add our own internal layer of Python to turn vpp's low level communications into something a little easier to work with.

For now, assume it moves packets to where they need to go, unless they're firewalled, in which case it doesn't. It also integrates properly with stock ML2 L3, DHCP and Metadata functionality. In the 17.01 release, we supported the ACL functionality added for VPP 17.01. This includes security groups, the anti-spoof filters (including the holes for things like DHCP), the allowed address pair extension and the port security flag.

VXLAN-GPE is an overlay encapsulation technique that uses the IP routed underlay network to transport Layer2 and Layer3 packets (a.k.a overlay) sent by tenant instances.

At this point, we only support Layer2 overlays between bridge domains using the included "gpe" type driver.

Following are some key concepts that will help you set it up and get going.

First, it's much easier than what you think it is! Most of the complexities are handled in the code to make the user experience and service deployment much easier. We will walk you through all of it.

If you are just interested in setting it up, you only need to understand the concept of a locator. VPP uses this name to identify the uplink interface on each compute node as the GPE underlay. If you are using devstack, just set the value of the variable "GPE_LOCATORS" to the name of the physnet that you want to use as the underlay interface on that compute node.

Besides this, set the devstack variable "GPE_SRC_CIDR" to a CIDR value for the underlay interface. The agent will program the underlay interface in VPP with the IP/mask value you set for this variable.

In the current implementation, we only support one GPE locator per compute node.

These are the only two new settings you need to know to get GPE working.

Also ensure, that you have enabled vxlan as one of the tenant_network_type settings and allocated some vni's in the vni_ranges. It is a good practice to keep your VLAN and VXLAN ranges in separate namespaces to avoid any conflicts.

We do assume that you have setup IP routing for the locators within your network to enable all the underlay interfaces to reach one-another via either IPv4 or IPv6. This is required for GPE to deliver the encapsulated Layer2 packets to the target locator.

These are some GPE internals to know if you are interested in contributing or doing code reviews. You do not need to know about these if you are just primarily interested in deploying GPE.

Within VPP, GPE uses some terms that you need to be aware of. 1. GPE uses the name EID to denote a mac-address or an IP address. Since we support Layer2 overlays at this point, EID refers to a mac-address in our use-case. 2. GPE creates and maintains a mapping between each VNI and its corresponding bridge-domain. 3. GPE maintains mappings for both local and remote mac addresses belonging to all the VNIs for which a port is bound on the compute node. 4. To deliver an L2 overlay packet, GPE tracks the IP address of the remote locator that binds the Neutron port.The remote mac addresses are pushed into VPP by the vpp-agent each time a port is bound on a remote node only if that binding is interesting to it. So the way this works is that the agents communicate their bound mac-addresses, their VNI and the underlay IP address using etcd watch events. A directory is setup within etcd for this at /networking-vpp/global/networks/gpe. An eventlet thread on the vpp-agent watches this directory and adds or removes the mappings within VPP iff it binds a port on that VNI. All other notifications, including its own watch events are uninteresting and ignored. 5. GPE uses a "locator_set" to group and manage the locators, although in the current implementation, we only support one locator within a pre-configured locator_set.

In general, check the bugs at <https://bugs.launchpad.net/networking-vpp> - but worth noting:

• SNAT setting may use the uplink interface on certain occasions besides the BVI interfaces.
• Some failure cases (VPP reset) leave the agent wondering what state VPP is currently in. For now, in these cases, we take the coward's way out and reset the agent at the same time. This adds a little bit of thinking time (maybe a couple of seconds) to the pause you see because the virtual switch went down. It's still better than OVS or LinuxBridge - if your switch went down (or you needed to upgrade it) the kernel resets and the box reboots.

We also keep our job list in <https://bugs.launchpad.net/networking-vpp> anything starting 'RFE' is a 'request for enhancement'.

We'll be dealing with TaaS and a plugin code manager in the upcoming 18.04.1 release.

At the least, just use it! The more you try things out, the more we find out what we've done wrong and the better we can make it.

If you have more time on your hands, review any changes you find in our gerrit backlog. All feedback is welcome.

And if you want to pitch in, please feel free to fix something - bug, typo, devstack fix, massive new feature, we will take anything. Feel free to ask for help in #openstack-neutron or in the openstack-dev mailing list if you'd like a hand. The bug list above is a good place to start, and there are TODO comments in the code, along with a handful of, er, 'deliberate' mistakes we put into the code to keep you interested (ahem).

Neutron's agent framework is based on communicating via RabbitMQ. This can lead to issues of scale when there are more than a few compute hosts involved, and RabbitMQ is not as robust as it could be, plus RabbitMQ is trying to be a fully reliable messaging system - all of which work against a robust and scalable SDN control system.

We didn't want to start down that path, so instead we've taken a different approach, that of a 'desired state' database with change listeners. etcd stores the data of how the network should be and the agents try to achieve that (and also report their status back via etcd). One nice feature of this is that anyone can check how well the system is working - both sorts of update can be watched in real time with the command:

etcdctl watch --recursive --forever /

The driver and agents should deal with disconnections across the board, and the agents know that they must resync themselves with the desired state when they completely lose track of what's happening.

We have unit tests written by developers, and are also doing system tests by leveraging the upstream Openstack CI infrastructure. Going forward, we will be increasing the coverage of the unit tests, as well as enhancing the types of system/integration tests that we run, e.g. negative testing, compatibility testing, etc.

In the 19.03 release, we made a few changes to add functionality:

• To complement the GPE overlay, there is now also a L3 solution for port mirroring (OpenStack TaaS) connections. When you're using GPE, the overlay is L3 and you may have routers between your compute nodes, so moving the TAP traffic around also requires an L3 overlay. We use VXLAN point to point connections to get that copy of traffic to the VM destination, and these can be routed over L3.

  We automatically use VXLAN connections when you're tapping ports on GPE networks and VLAN connections for L2 networks so this should all Just Work.

• We've worked on out Python 3 compatibility in preparation for the EOL of Python 2.7. Both the Neutron server's driver components and the agent components that control VPP should now support Python 3.5, but this work is ongoing and if you do find any issues please report them on the Launchpad page.
• We've worked on our L3 Neutron routers support and added functionality that allows you to use NAT from multiple routers simultaneously -ideal if, for instance, you like to use one router per tenant and in more unusual cases routers with different external networks.
• We've been doing the usual round of bug fixes and updates - the code will work with both VPP 18.10 and 19.01 and has been updated to keep up with Neutron Rocky and Stein.