The Deliverator – Wannabee

So open minded, my thoughts fell out…

Archive for March, 2014

Dharma / Maya – A New Virtualization Platform for Silverfir to Replace Minimus

Posted by Deliverator on 22nd March 2014

Back in 2009, most of’s services were migrated from an aging behemoth of a Compaq server named Frankenputin onto what was hoped would be a much more manageable platform which I christened Minimus. Minimus was designed to be a server that could run contentedly in a closet for year’s on end. It was based around a dual core atom motherboard which sipped power and featured completely passive cooling and used a solid state drive as its boot drive for greater reliability. The host OS was Windows 7 running VMware Server 1.x to host a Ubuntu Linux virtual machine. Eventually VMware stopped supporting the free VMware Server 1.x line and we were forced to upgrade to VMware Server 2.x, which featured a barely functional web based management interface.

Several months ago, Silverfir started experiencing unexplained lockups every few days that required the virtual machine to be rebooted. This became annoying (especially for a box-in-the-closet) and very inconvenient for both Ryan and I. Eventually, the VM failed to boot entirely and the chunk of Silverfir hosted by Minimus was down entirely. This coincided with an extended trip by me to Rarotonga, an island in the middle of the pacific ocean with very minimal internet access. While Ryan and I were jointly remotely investigating the causes of this misbehavior, we found several VM metadata files that were 0 bytes and backup copies of these files were zero bytes as well. I reinstalled VMware Server and Ryan recreated the VM definition files from scratch. During this process, the web interface was very difficult to work with, as it wasn’t working properly in modern versions of IE and Firefox. Eventually, Ryan was able to get everything working again, and took the opportunity to upgrade from Ubuntu 9.04 to the current Long Term Support edition of Ubuntu, 12.04. Unfortunately, this did not solve the problem with the VM locking up every couple days. I decided it was time to modernize Minimus.

After spending a week experimenting with a number of modern hardware assisted hypervisors, I eventually decided to use Xenserver 6.2 Xenserver is a free (in multiple senses of the word) minimal footprint hypervisor similar to VMware ESXi. Unlike VMWare Server, which required a full fledge host OS be installed, Xenserver’s host footprint is very minimal, leaving more of the system’s resources (especially ram) free to be allocated to guest virtual machines. Because Xenserver relies on hardware level support for virtualization (a cpu feature called VT-x), guest virtual machines run much closer to the “bare metal” and feel a lot snappier as a result. Xenserver also has support for another newer hardware virtualization feature called VT-d which allows for hardware devices to be directly shared with guest VMs. This allows for devices like GPUs to be directly accessible to virtual machines. Citrix likes to show off this feature by running demanding, modern games like Skyrim in a VM and playing the game through a thin client device like an Ipad. Neat, but not very relevant to our particular use case.

The main things that I liked about Xenserver were:

  1. Free in multiple senses of the word. Based around an open source project, Xen, with widespread adoption both within the OS community and among major commercial users such as Amazon. Xenserver stores its VM metadata in standardized formats that are directly importable into other virtualization environments. Because of this, I have some confidence that I am not going to be bitten by a product discontinuation or lack of an easy forward migration path as occurred with VMware Server.
  2. After a week of hammering at it, Xenserver feels very mature. I only experienced one bug, relating to migrating VMs with associated snapshots, in a week of testing oddball cases. The windows based management tool, called Citrix Xencenter, is a pleasure to use.
  3. Xenserver allows me to create redundancy “pools,” clusters of  Xenserver hosts and shared storage resources that allow for guest virtual machines to be moved back and forth between multiple physical servers without needing to be taken offline. It is VERY cool to have a server being run off one physical box one moment and 30 seconds later having it be running off a different server with less than a single second’s network downtime. This should allow for Silverfir to stay online while hardware maintenance is being performed, something that wasn’t possible under the previous VMware Server environment.
  4. Xenserver allows for easy snapshot backups of running VMs, allowing backups to be created while the server is in use. With VMware Server, the guest had to be shutdown to backup the virtual disks, a process which took hours even using an eSATA based external backup drive.

Migrating Minimus to Xenserver was a fairly straightforward process. I was able to import the primary Minimus VMware virtual disk .vmdk file directly into a newly created VM guest in Xenserver. I had to edit grub, fstab and network interfaces to get the VM working in Xenserver and also had to add an remove a few kernel modules and install the old VMware tools, but all told I probably spent less than 2 hours getting Minimus running happily under Xenserver. What wasn’t so painless was getting the second data volume .vmdk to import. This second .vmdk stored all of Silverfir’s websites, photo galleries, etc…you know…the things that people actually care about. I received errors trying to import this file using a wide variety of virtual disk management/manipulation tools. I think this vmdk file had been created in an earlier version of VMware and probably used an older version of the .vmdk format. Eventually, after almost a whole day of trying to import this file I threw up my arms in disgust. As a workaround, I put both the old Minimus virtual machine and the new Xenserver virtual machine, which I am calling Maya, on the same network segment and created a new virtual disk container in Xenserver. Ryan then copied the data from the old virtual server to the new using the magic of rsync. This took quite a while, as almost 500 GB of data needed to be copied over at 100 mbit speeds. After almost a full day of copying and some adjusting of permissions, Maya was substantially complete and took over hosting duties from Minimus. Maya has been happily hosting Silverfir without incident for over a week.

In the near future, I plan to decommission Minimus entirely and replace it entirely. Maya’s current primary Xenserver host is a Core 2 Duo with 7 GB of ram. I plan on using most of the guts of Minimus to create a new server based around an Intel Avoton C2750 motherboard. This new system, which I am calling Dharma will be the primary Xenserver host for Maya, with the current Core 2 Duo host serving as a high availability backup server. Hosting of Maya’s data volumes will be via a Readynas Ultra 6 with a 12 TB Raid 6 array. Hopefully this new setup will allow for greater reliability and fault tolerance than what was  achieved with Minimus and Maya can continue serving Silverfir’s users for years to come.

Posted in Blogging, Emulation and Virtualization, General, Linux, Operating Systems, Tech Stuff | No Comments »