Transparent VLAN Tagging with libvirt and Open vSwitch

As of version 0.10.0 of libvirt, transparent VLAN tagging is fully supported with Open vSwitch (OVS). This feature allow to transparently forward tagged VM traffic to the ethernet network, providing functionality similar to a switch's VLAN trunk. Transparent VLAN tagging is particularly interesting if you want to directly and easily terminates VLANs on your virtual machines (say virtual routers or gateways), while keeping your host configuration as simple as possible, and so without messing up with tens or even hundreds of bridges.

Libvirt is a well-known virtualization API that is supporting a large numbers of system and networking features. Open vSwitch, is at the other end, a very-popular multilayer Open Source virtual switch, that is used by projects like OpenStack and that is actually shipped with most popular Linux distributions. Unfortunately, at this time of writing, some distributions like Debian and Ubuntu still ships with rather old versions of libvirt. To run newer features like transparent VLAN tagging, you can overwrite the old version of libvirt with the latest stable version or with a development build. This solution is pretty straightforward, but unfortunately not very well documented.

If you run Debian, Ubuntu, or a derivative distribution, first check the installed version of libvirt. If the following is inferior to 0.10.0, then you can manually build libvirt from source while setting the current version of libvirt on 'hold', so your system will never update, nor overwrite the version your manually installed.

Check installed version of libvirt

First, check the current version of libvirt installed on your host, by using the following command:

# dpkg -l | grep libvirt
i  libvirt-bin            0.9.8-2ubuntu17.19        programs for the libvirt library
i  libvirt0               0.9.8-2ubuntu17.19        library for interfacing with different virtualization systems
i  python-libvirt         0.9.8-2ubuntu17.19        libvirt Python bindings
Install the build dependencies

Prior to running the source's configuration script, install the necessary build dependencies:

# apt-get install build-essentials libyajl-dev libyajl1 libxml2 libxml2-dev libdevmapper1.02.1 libdevmapper-dev libnl-3-dev libnl-route-3-dev pkg-config libgnutls-dev libpciaccess-dev
Install libvirt from source

Run the configuration script, compile and install all utilities and libraries using the standard system paths.

# ./configure --prefix=/usr --localstatedir=/var --sysconfdir=/etc
# make && make install
Freeze/hold libvirt system packages

Make sure the libvirt system packages won't get overwritten in the future by marking them on hold.

# apt-mark hold libvirt-bin
# apt-mark hold libvirt0
# apt-mark hold python-libvirt
Restart libvirt system service

Once the newest version is installed, you must restart the libvirt system service for the new features to be available.

# /etc/init.d/libvirt-bin restart
Create the Open vSwitch Bridge

Prior to configuring the libvirt network, you must create a native OVS bridge. Then, you must add the physical interface(s) to your bridge. In the example below I created a bridge named 'br1' and I added a bonding interface, the latter can be created using Open vSwitch utilities as well.

# ovs-vsctl add-br br1
# ovs-vsctl add-port br1 bond0
Create a network configuration file with Transparent VLAN tagging

Below, an example of a libvirt network configuration, using transparent VLAN tagging. A bridge, here 'br1' must be first created using Open vSwitch configuration utilities, and your physical interface(s) must be added to it. You must also lists every VLAN ID (VID) to be allowed over the virtual bridge you just created.

<network>
 <name>ovs-net-vlan</name>
 <forward mode='bridge'/>
 <bridge name='br1'/>
 <virtualport type='openvswitch'/>
 <portgroup name='vlan-all' default='yes'>
   <vlan trunk='yes'>
     <tag id='802'/>
     <tag id='803'/>
     <tag id='804'/>
     <tag id='805'/>
   </vlan>
 </portgroup>
</network>
Define the network in libvirt

Once the network configuration file has been created, you must create (or define) the network using the following set of self-explanatory commands:

# virsh net-define /var/tmp/ovs-net-vlan.xml
# virsh net-start ovs-net-vlan
# virsh net-autostart ovs-net-vlan
Create a new virtual machine using virt-manager

Once a libvirt network has been defined, you can refer to it using libvirt API's clients. However, some utilities like virt-install, don't support some specific network types, therefore you must first bootstrap a minimal configuration using the --nonetworks command-line option and then edit the VM configuration file by hand. Here's an example using virt-install:

# virt-install -n lab-csr1000v-01 --vcpus sockets=4,cores=1,threads=1 --ram=2480 --cdrom=csr1000v-universalk9.03.12.00.S.154-2.S-std.iso --disk path=/var/lib/libvirt/images/ab-csr1000v-01.qcow2,size=3,format=qcow2 --graphics=vnc,listen=0.0.0.0,password=letmevnc --nonetworks

Then edit the VM configuration file using the virsh edit <domain> command:

<domain type='kvm'>
  <name>lab-csr1000v-01</name>
  <uuid>f69b1077-d446-cf67-1287-993f8ff66c71</uuid>
  <memory unit='KiB'>2539520</memory>
  <currentMemory unit='KiB'>2539520</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-1.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu>
    <topology sockets='4' cores='1' threads='1'/>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/ab-csr1000v-01.qcow2'/>
      <target dev='hda' bus='ide'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:d8:98:db'/>
      <source network='ovs-net-vlan' portgroup='vlan-all'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:fd:ff:a9'/>
      <source network='ovs-net-vlan' portgroup='vlan-all'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:f0:19:f9'/>
      <source network='ovs-net-vlan' portgroup='vlan-all'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0' passwd='letmevnc'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </memballoon>
  </devices>
</domain>

As you can see on the guest's xml configuration above, there is three interfaces of type 'network' and of model 'virtio', nothing fancy here, excepted the addition of the 'source' block, and its 'network' and 'portgroup' attributes. The former attribute specifies the name of the libvirt network, the second defines the port group, as you defined in the network's xml configuration. By specifying the network name and the port group, you can respectively tie a VM's interface to a bridge and to a set of allowed VLANs.

As you can see from the example above, the transparent VLAN tagging features of libvirt, and its use with OVS, is a convenient mean of dealing with a large number of VLANs. You do not have to define bridges individually for every VLANs, and you can directly terminates a range of VLANs on your guest's vNICs, without having to use SR-IOV, PCI pass-through, or any not-so-well supported configurations like nested linux bridges.

If you know more interesting features of libvirt or you want to share configuration examples, feel free to share them as comment in this article.

About the author Nicolas Chabbey

Nicolas Chabbey is a Network Engineer certified with Cisco Systems and Juniper Networks. He has begun his career in 2003, and has designed, implemented and maintained networks for enterprises and service providers. When he is not behind a computer, he is riding his mountain bike across the Swiss alps.

Previous Home Next

Comments

blog comments powered by Disqus