Ever since 2004 or 2005 I've wanted to see real virtual router functionality on Linux.  It would make setting up a lightweight networking lab with Quagga (or BIRD) a pinch and also allow me to leverage multiple Internet connections for VPN isolation, amongst other things.

You might say, "Hey, Linux has that in the form of tables and rules that can be manipulated by ip(8)!"  It does, sort-of.  It's possible to setup another routing table (optionally naming it in /etc/iproute2/rt_tables), add arbitrary routes, then setup rules to always tell the kernel to use a specific routing table for all packets coming from a certain IP address (or many more things, if you use iptables MARK).  This only "sort-of" works because there's no way (from what I can tell) to actually bind interfaces to a particular routing table.  There's also the issue with overlapping IP space—how do I tell ssh(1), for example, to use a particular routing table if there are two interfaces with the same IP address?  The -b argument won't do me much good.  Also, DHCP is problematic because heavy modifications are needed in dhclient-script and they'd be mostly implmentation-specific.

So, although I've used multiple routing tables with rules, they don't really fit my definition of virtual routers (VRF-lite, in Cisco parlance), which I define as a isolated construct has exclusive access to a set of interfaces and their addresses.

Linux Containers

I've been messing with Linux containers (LXC) recently and I think they might provide the exact functionality I've been looking for all these years.  With LXC it's possible to fire up a small instance that has its own routing table, interfaces, and applications.  There's no need to use rules or -b arguments anymore.  DHCP and Quagga don't require any hacks and work the way they should.

Networking with LXC is about what I expect; interfaces are dedicated to the container.  Overlapping IP addresses are certainly possible, if there's a need for that.  Connecting a container to the host or other containers can be achieved using a virtual Ethernet interface with a bridge.  This makes it easy to setup multiple Linux "virtual routers" without ever having to mess with VMs, routing tables or rules.  A good article on various LXC networking modes can be found here.

If you're familiar with Junos, Linux containers are almost analgous to logical systems in this type of role.  Logical routers, unlike VRFs, have their own copy of RPD, which is a daemon that handles all dynamic routing protocols.  If you're using Quagga with containers, the architecture is similar.

A slight drawback I can see with containers as virtual routers is the disk space usage.  Each container has, by default, its filesystem stored in a separate directory in /var/lib/lxc.  There's quite a bit of redundant data if you fire up many containers using the same distribution (e.g., Debian).  I'm sure there is some way to de-duplicate this (which would help with package upgrades, too!) but I haven't really looked into it because storage is so cheap nowadays and most of us have plenty of it.  A fairly fully-featured Debian container I've got is not that large, anyway:

(vega:17:04)# du -hcs /var/lib/lxc/*
4.0K	/var/lib/lxc/lxc-monitord.log
488M	/var/lib/lxc/soran
488M	total

So, in summary, for general-use virtual routers, I think LXC is pretty great.  The best part is that the only thing required to use LXC is a recent kernel with cgroups enabled and mounted properly.

New comments are currently disabled for this entry.