Xen Networking

I recently started using Xen and needed to do something which I thought many other would have done before. The difficulty was that extensive research found many examples but almost all were based on earlier versions of Xen. I'm using XenServer versions 5.5 and 5.6 and the configuration process is vastly different from earlier versions where you could simply code everything in flat files. These files and directories don't even exist in contemporary versions of XenServer.

What I'm trying to accomplish seems so straightforward. I have a server which has one ethernet interface. What I want is to forward requests to port 80 to one virtual machine (VM) and forward port 8080 to another. Here's a diagram of the configuration:

I'm using a reserved class C address on my internal network and the link to the 'net is provided by a commercial cable modem service. I use fixed IP addresses since I only run seven servers, a colour laser printer and a wireless router.

I'm not going to document how to install XenServer and configure the primary interface: there are excellent resources available on the Citrix site. Ditto for the VM installation. Of course you also need to install XenCenter on an available Windows® machine in order to perform most of the following directions.

In a way I'm fortunate to have considerable networking knowledge and experience. I'm familiar with iptables since almost all of my servers run Linux variants. I can also decode the output from tcpdump, a tool I made extensive use of while arriving at the solution. Seeing how things like ICMP was working (or not, as the case may be) and seeing how ARP (Address Resolution Protocol) requests were being routed was essential to determining the correct configuration.

Connect via XenCenter, select the server and select the Network tab. The default network is Network 0 so select that and click on the Properties button.

Select Properties -> Network Settings and you should see that, by default, the network is automatically added to new VMs:

At this point you have a choice: either deselect the button and click the OK button or remove the virtual interfaces (VIFs) created automatically when you create new VMs. At this point we need to discuss some of the commands we'll be using to configure the network. The commands are fully documented in the Administration Guide. If the VIFs are created automatically then you will need to remove them. First you need to get a list of the VIFs so click on the Console tab and follow the example below. In this and all subsequent examples of console interaction the output generated by the system is shown in regular font while the user input is shown in bold.

[root@sudsyxen ~]# xe vif-list
uuid ( RO)            : 434ec2fc-7582-f5a6-cadb-714d918443a4
         vm-uuid ( RO): c9f91447-5b84-f62e-6530-d746c512d5a6
          device ( RO): 0
    network-uuid ( RO): 5d59181a-360d-ec1f-b791-c20cc234cd33

uuid ( RO)            : 6f178b46-a48a-cf1f-f6f5-05ad853a0a21
         vm-uuid ( RO): 13a258d7-1142-5073-50a2-f7d3ddec76a4
          device ( RO): 0
    network-uuid ( RO): 5d59181a-360d-ec1f-b791-c20cc234cd33

[root@sudsyxen ~]#

The first thing you should notice is the extensive use of Universally Unique IDentifiers (UUIDs) in Xen. You'll often need to enter these values when performing tasks such as creating new interfaces. Xen makes it somewhat easier since you can hit the tab key and get automatic completion as long as what you've entered to that point is unique. I keep a notepad handy when performing configuration in order to record the first six characters of UUIDs. That's typically enough to ensure uniqueness.

The problem with the previous output is that it doesn't immediately provide the information you need to conveniently correlate the interfaces to the VMs. You could execute a xe vm-list and xe network-list and cross-correlate the values but there's an easier way.

[root@sudsyxen ~]# xe vif-list params=vm-name-label,network-name-label,uuid,device
uuid ( RO)                  : 434ec2fc-7582-f5a6-cadb-714d918443a4
         vm-name-label ( RO): Fedora server 1
                device ( RO): 0
    network-name-label ( RO): Pool-wide network associated with eth0

uuid ( RO)                  : 6f178b46-a48a-cf1f-f6f5-05ad853a0a21
         vm-name-label ( RO): Fedora server 2
                device ( RO): 0
    network-name-label ( RO): Pool-wide network associated with eth0

[root@sudsyxen ~]#

Now we can delete the interfaces.

[root@sudsyxen ~]# xe vif-destroy uuid=434ec2fc-7582-f5a6-cadb-714d918443a4
[root@sudsyxen ~]#

I only had to type the first four characters of the UUID then hit tab. The remainder of the UUID was completed by Xen. Delete other unneeded interfaces before creating a new network.

[root@sudsyxen ~]# xe network-create name-label="Internal network" \
> name-description="Internal network"
[root@sudsyxen ~]#

In order to cut the length of the commands and make it easier to diplay on browsers I'm using standard *NIX semantics. That is, end a line with a backslash in order to be able to continue the command on a new line. You'll see this used a lot further on since many of the configuration commands are quite lengthy. Note that the output of this command is the UUID of the newly created network. Now we have to create the VIFs in the VMs. First we need to get a list of the UUIDs of the VMs.

[root@sudsyxen ~]# xe vm-list
uuid ( RO)           : c9f91447-5b84-f62e-6530-d746c512d5a6
     name-label ( RW): Fedora server 1
    power-state ( RO): halted

uuid ( RO)           : 7057f439-be04-45fe-ba1f-c6f67b94773a
     name-label ( RW): Control domain on host: xenserver
    power-state ( RO): running

uuid ( RO)           : 13a258d7-1142-5073-50a2-f7d3ddec76a4
     name-label ( RW): Fedora server 2
    power-state ( RO): halted

[root@sudsyxen ~]#

We now have all the information required in order to create the VIFs on the new internal network.

[root@sudsyxen ~]# xe vif-create vm-uuid=c9f91447-5b84-f62e-6530-d746c512d5a6 \
> network-uuid=194de49e-115c-948b-2f6b-bf0ac910f70c device=0

This last example demonstrates an idiosyncrasy in Xen. While UUID completion works when the command is entered on a single line, it's not available when using command continuation. In this case, I had to cut the complete UUID from a display command in order to paste it into the second line of the command. Once again, the output is the UUID of the new component (a VIF in this case).

The next step is very important and needs to be performed on the guest VMs. You have to hard-code the IP address and default route on the domU machines. The process will vary based on the particular operating system so refer to the appropriate documentation. Keep in mind that some operating systems have a firewall configured by default; this has to be disabled for the remaining steps to work.

You now have to modify the configuration file /etc/sysctl.conf in order to enable forwarding and proper ARP (Address Resolution Protocol) behaviour. Ensure that the following settings are specified:

# Controls IP packet forwarding
net.ipv4.ip_forward = 1
# Enable ICMP redirects
net.ipv4.conf.all.send_redirects = 1

You'll also need to add the following to the end of the file:

# Enable ARP filter
net.ipv4.conf.default.arp_filter = 1

You'll have to run sysctl -p after making any changes to the configuration file in order to force a reload.

The behavior of the internal network interface in dom0 (xapi1) is unusual. If you execute netstat -i before the first virtual machine is started then xapi1 will not appear as an interface. Only after the first virtual machine has started does the interface appear. There are a number of commands which need to be executed once the xapi1 interface is started:

[root@sudsyxen ~]# ip link set xapi1 arp on1
[root@sudsyxen ~]# ip addr add dev xapi12
[root@sudsyxen ~]# sysctl -w net.ipv4.conf.xapi1.proxy_arp=13
[root@sudsyxen ~]# iptables -I FORWARD -i xapi1 -d -o xenbr0 -j ACCEPT4
[root@sudsyxen ~]# iptables -I FORWARD -i xenbr0 -d -o xapi1 -j ACCEPT5
[root@sudsyxen ~]#


  1. Enable ARP on xapi1
  2. Specify the address of the network connected to xapi1
  3. Enable ARP proxy on xapi1
  4. Enable forwarding from the internal to the external network (see below)
  5. Enable forwarding from the external to the internal network (see below)

Note that we're using the insert (-I) flag to iptables. This is because, by default, packets are routed through a RedHat firewall. See the sample output from listing the FORWARD chain:

[root@sudsyxen ~]# iptables -L FORWARD
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target               prot opt in     out     source               destination         
    0     0 RH-Firewall-1-INPUT  all  --  any    any     anywhere             anywhere
[root@sudsyxen ~]#

If we used the append flag (-A) then the forwarding would not take place since the packets would be dropped by the firewall rule. But the firewall rules aren't applied to the NAT (Network Address Table) chains so append can be used there. Now we need to route requests for port 8080 to our internal server:

[root@sudsyxen ~]# iptables -A PREROUTING -t nat -p tcp -i xenbr0 --dport 8080 \
> -j DNAT --to
[root@sudsyxen ~]#

This adds a rule to the prerouting chain (before routing takes place) which alters the destination IP address (Destination NAT or DNAT) of packets with a destination port of 8080. The inverse is enabled by default so this is all that is required to redirect requests to our internal VM. It does not, however, enable access from that machine to the outside world. If you wanted to run a web browser or other application in the VM which required Internet access then there is one final step:

[root@sudsyxen ~]# iptables -A POSTROUTING -t nat -o xenbr0 \
> -j SNAT --to-source
[root@sudsyxen ~]#

This rule enables source NAT (SNAT) so that the source address of packets sent out through the xenbr0 interface will be coerced to which is the IP address of the external interface. Note also that this rule is added to the POSTROUTING chain; it is applied after other packet processing, i.e. forwarding.

The only remaining challenge is how to execute the required commands when the xapi1 interface is brought up. Remember the cron task: it fires up once a minute in order to run the tasks in /etc/crontab (plus others, but that's beside the point). Add a shell script which monitors for the existence of the xapi1 interface and runs the necessary commands when found. Use a marker file so that the commands are only run once. Call it something like /usr/local/bin/config_xapi and remember to make it executable, i.e. chmod +x the file. You'll end up with something like this:

if [ ! -f $MARKER_FILE ]
  COUNT=`netstat -i | grep $INTERFACE | wc -l`
  if [ $COUNT -gt 0 ]
    ip link set xapi1 arp on
    ip addr add dev xapi1
    sysctl -w net.ipv4.conf.xapi1.proxy_arp=1
    iptables -I FORWARD -i xapi1 -d -o xenbr0 -j ACCEPT
    iptables -I FORWARD -i xenbr0 -d -o xapi1 -j ACCEPT
    echo $$ > $MARKER_FILE

I use a marker file in /var/tmp to prevent the script from running more than once; I simply populate it with the process ID which created it. Note, however, that the /var/tmp directory is not cleaned on boot so you'll have to add a line to /etc/rc.d/rc.local to remove the marker. It will look like this:

rm -f /var/tmp/xapi_configured

Now you just need to add the following line to the /etc/crontab file:

* * * * * root /usr/local/bin/config_xapi

Copyright © 2011, 2012, 2013, 2014, 2015 by Phil Selby. All rights reserved.