http:// www.jms1.net / xen /

Xen

I have always been interested in the idea of running multiple virtual machines on one physical box. I have a co-located server, and offer hosting services to my friends. Some of them have special needs and either need root access to the machine, or need special work-arounds to allow them to do things like restart apache without root access.

I looked at UML (User-Mode Linux) and could only get it to work once, and only while I was logged directly into the physical console of the machine, and after shutting down the virtual machine I couldn't bring it back up again, even with the same commands. I suspect I may have been using a buggy version of the UML code, but at the time it claimed to be stable and it didn't work for me- so I gave up on it.

I had heard about a program called Xen, which does the same basic thing but goes about it in a different manner- it runs a low-level "hypervisor", and your existing OS runs as a client- a client with special privileges, but still just a client. The hypervisor is actually capable of running other operating systems than Linux- there are client kernels for Linux 2.4 and 2.6, NetBSD 2.0 and current, FreeBSD 5, and Plan 9. A future version Xen will be able to take advantage of hardware virtualization features and run ANY x86 operating system.

This was true when I originally wrote this page, back in 2005-11. Xen is now capable of doing full hardware virtualization, if your CPU supports it. (Mine didn't when this page was written, but I have since upgraded to newer hardware, and full hardware virtualization is working now.)

So I played around with it and picked up a few things, and then a friend asked me to do a presentation about it at a Linux User Group meeting. I'm currently looking at putting something together, so this may end up being the plan for such a presentation- if not, it will at least be a fairly good walk-through of how to set things up.

Background Information

Before we start installing things, there are a few basics which you need to understand about how Xen works.

Xen itself consists of a hypervisor, which is the program running "below" everything else (like the Linux kernel on a non-Xen machine) and one or more "sessions" (also known as "clients") which run a modified (or in the case of hardware virtualization, a normal) operating system, such as a Linux kernel. The hypervisor itself has no real user interface (other than logging output.) The only way to send it commands is to use a session- which of course means that at least one session needs to exist in the first place.

There are two types of sessions- a xen0 session, which has access to the hardware (through the hypervisor) and can also send commands to the hypervisor itself, and a xenU session, which has very limited access to the hardware and cannot send commands to the hypervisor. A running hypervisor will always have exactly one xen0 session, and may have any number of xenU sessions.

The name "xen0" refers to the fact that the sessions are numbered by the hypervisor, with session 0 being the only one with full access. The name "xenU" means (as near as I can tell) "Xen User". Both types of session require that the kernel running within the session be compiled with support for working through the hypervisor in order to access the hardware- a standard Linux kernel without any Xen patches would not run correctly under the hypervisor, since the hypervisor would block the kernel from accessing the hardware directly.

When the hypervisor boots, it will immediately start a xen0 session. If the xen0 session cannot start, the hypervisor stops. When the xen0 session shuts down, the last thing it does is tells the hypervisor to either shut down the hardware, or to reboot (depending on whether the OS in the xen0 session is shutting down or rebooting.)

The xen0 session normally runs a program called xend, which sends commands to, and listens for messages from, the hypervisor. The developers of Xen are big fans of the Python programming language, and wrote xend and the xm command-line utility in Python.

After the xen0 session is up and running, it is responsible for starting any xenU sessions which may need to run. The Xen package comes with a set of "sysvinit" scripts which handle starting both xend and any xenU sessions you wish to have started automatically whenever the system boots.

Each xenU session is defined using a text file containing information about the session- the session's name, information about how its virtual disk partitions should map to physical resources as seen by the xen0 session (which may be any block device- a disk partition, an LVM logical volume, a file connected to a loopback device, etc.)

The amount of memory available for each xenU session is limited to the physical RAM in the machine. Each individual session is free to add its own swap space, but the "physical RAM" available to all machines is limited to the physical RAM in the machine, less about 35MB for the hypervisor and whatever the system uses for BIOS purposes (a shadow copy of the video ROM, shared video RAM for the cheap video cards, etc.)

Networking is normally handled using bridge interfaces on the xen0 session, which the xenU sessions see as being connected to their ethernet interfaces. Any xenU session which needs to access the network must be configured to access a bridge interface which also connects to a physical ethernet interface in the xen0 session. If you have multiple ethernet cards, or want to set up a private network segment which is shared by some or all of the xen0 and xenU sessions, or even something like a xenU session running a firewall and the xen0 cannot access the Internet except through the xenU session, these can be handled by creative use of bridge interfaces.

About this web site

The directions on this site were originally written for a simple machine- a P3-866 with 512MB RAM, a 30GB HD, a single ethernet card, and a video card capable of supporting a framebuffer console (which almost every card on the market, as well as the embedded cards on most motherboards, will do.) The framebuffer console is not really needed (and actually causes a few minor problems with xenU sessions which don't know it's there), I use it because I find it easier to read a 128x48 screen than I do a standard 80x25 or 80x50 screen.

Since I originally wrote the page, I have decided to upgrade my server (the one serving the page you're reading) to CentOS 5.0, which includes Xen as a standard option in the installer. As I write this, I have built and deployed the new hardware (dual P3-1GHz, 3.5GB RAM, dual 147GB HDs in a hardware RAID-1 configuration) and have spent several days experimenting with the scripting to build client sessions. Once I have it all figured out, I plan to move this web site to a Xen client session on the new machine- so if you're reading this after 2007-12-15 (or somewhere in that neighborhood), chances are the page was served by the machine about which the CentOS 5.0 directions below were written.

2009-10-11 The machine has been rebuilt again, for those who are keeping score. It's now an Intel Core-2 Duo with 8GB RAM and to 1TB hard drives on a 3ware 9650SE-2LP hardware RAID controller. So the current machine is actually the third instance of my Xen server.

The systems I have built, both CentOS 4.2 and CentOS 5.0, use LVM (Linux Volume Manager) to handle allocating disk space to the various sessions. Because not many people are familiar with LVM, the page which explains how to install CentOS for the xen0 session also includes a detailed walk-through of how to set up LVM containers, including how to install CentOS itself within LVM logical volumes. I was not able to find an official "home page" for LVM, however this page at RedHat and this page at TLDP seem to cover things fairly well.

The first version of this web page covered CentOS 4.2, which at the time was the "current" version of CentOS. It did not include Xen as a standard option, so the pages I wrote about it cover downloading and installing Xen as an add-on package.

When CentOS 5.0 was released in 2007, I started looking into Xen again, using the tools provided by CentOS (which originally came from RedHat, since CentOS is built using RedHat's source code.) While I do think it can be useful, I don't particularly like their "virt-manager" tool, for a few reasons:

The CentOS "virt-manager" tool is a GUI, which means that even if you don't install X, you still have to install a bunch of X and Gnome libraries just to use the tool. Granted, it's not the end of the world, but on a dedicated server which will never have a display attached to it, it just seems rather a waste. (Of course, I am biased towards command-line tools- my use of Linux is 95% servers, so most of the GUI stuff is wasted on me. My own desktop and laptop machines are running Mac OSX.)
When it builds a child session, it uses a single container and presents that to the child session as a "disk" upon which it will create partitions, rather than presenting multiple containers which the child sees as partitions and uses for individual filesystems. I prefer to have each filesystem as a separate entity, so I can work on them while the child is not running, and so I can resize them when/if needed.

My own strategy, since I will be SSH'ing into the server 99% of the time (I've only physically seen my current server twice in the past two years), is to write scripts which I can run from an SSH session to set up the child sessions.

CentOS 4.2

Our xen0 session will be running CentOS 4.2, because that's what I normally use to build servers. The instructions on the Xen web site show how to set up xen0 on Debian and on RedHat/Fedora, these directions are very similar to the RedHat/Fedora instructions (which makes sense, since CentOS is a clone of RedHat Enterprise.) They also explain how to build Xen from source, and those directions include an option to build a xenU NetBSD kernel from within Linux.

If you are interested in using other operating systems with Xen, either as a xen0 or as a xenU, Google is your friend- try a search for "xen freebsd", or "xen" combined with the name of whatever other operating system you're looking for.

The pages listed below may not be complete. I started writing the site but never finished it. I will either finish the pages or remove the links at some point in the future. Also realize that I'm concentrating on CentOS 5 now- chances are any unfinished pages below will remain unfinished. Sorry for any inconvenience.

CentOS 5.0

Base OS Install
Set up CentOS Repository (for client use)
Client Setup (CentOS 5.0 client)
Changing a session after it's been created
Running a 32-bit domU on a 64-bit dom0