The Deliverator – Wannabee

So open minded, my thoughts fell out…

Archive for the 'Emulation and Virtualization' Category

Upgrades to Silverfir’s Xenserver hosting environment

Posted by Deliverator on 11th February 2015

It has been almost a year since my last post about Xenserver. Silverfir’s Xenserver environment has remained relatively static over that time. I ended up purchasing the Asrock Avoton C2750 motherboard mentioned in the previous post and using it as the basis for a Xenserver pool master / redundant server. I also topped up the Core 2 Duo machine with its max 8 GB of ram. The older Core 2 Duo machine actually turned out to be a better choice for primary host for the Silverfir VM, as the Avoton C2750 based machine had a tendency to lock up hard every 2 weeks or so. Plenty reliable for desktop purposes, but not for a computer intended as a low maintenance server. One nice thing about the Asrock motherboard is that it features an IPMI interface which allows one to perform basic management tasks like bouncing a box remotely, as well as IP KVM type stuff. The IPMI is really quite impressive, but is small consolation if the underlying platform it is designed to manage isn’t stable.

Recently, Citrix released a new version of Xenserver, version 6.5. I decided to upgrade all the cluster’s boxes to the new version. Installation was as simple as burning a CD, popping it in and doing an in place upgrade over the existing installation. While I was at it, I checked Asrock’s website and they had released a fresh bios for the motherboard as well as a new firmware for the IPMI. The release notes mentioned numerous stability improvements, so I figured what the hell and went for it. So far, the Asrock motherboard has been 100% rock solid since the upgrades. If this stability continues, I will switch to using the Avoton machine as the primary host and keep the Core 2 powered off and use it just as a cold backup. The Avoton should be much more power efficient.

Before doing the upgrades, I took the precautionary measure of making a full backup of the Silverfir VM using the excellent and inexpensive Xackup software. I’ve been backing up Silverfir semi-regularly over the last year, usually getting around to it every couple months. I’d like to backup Silverfir more often, but it was taking upwards of 22 hours to complete a backup at ~2MB/s. Also,  because I am a cheapskate and am using the free version of Xackup, each backup has to be triggered manually. Investigating potential causes, I found that the Atom 330 box I use to run the Xenserver’s Xencenter management software and Xackup, was just too low end to handle the task. Running Xackup was maxing the Atom 330’s CPU cores at 100% all throughout the backup, even at the lower compression level settings. Not applying compression resulted in much larger backup files while not significantly reducing completion time. I decided that the management computer needed an upgrade as well, but wanted to keep it low power and relatively inexpensive.

I ended up settling on a Asrock Q2900-ITX Mini ITX board with an integrated, passively cooled Intel J2900 CPU. This board comes with a CPU that is significantly faster than the Atom 330 and sips power, which is nice in a management server that has to be online all the time. I gutted the old Atom 330 system, plopped in the new motherboard and 8 GB of DDR3 ram and while I was at it put in a 3TB HDD to have more space for backups. After a fresh install of Win 7, Xencenter 6.5 and Xackup, I fired up the backup job, which previously took 22 hours. It now takes around 5 hours at maximum compression and the CPU cores stay below 70% the whole time. Mission accomplished.

Posted in Emulation and Virtualization, Tech Stuff | No Comments »

Dharma / Maya – A New Virtualization Platform for Silverfir to Replace Minimus

Posted by Deliverator on 22nd March 2014

Back in 2009, most of’s services were migrated from an aging behemoth of a Compaq server named Frankenputin onto what was hoped would be a much more manageable platform which I christened Minimus. Minimus was designed to be a server that could run contentedly in a closet for year’s on end. It was based around a dual core atom motherboard which sipped power and featured completely passive cooling and used a solid state drive as its boot drive for greater reliability. The host OS was Windows 7 running VMware Server 1.x to host a Ubuntu Linux virtual machine. Eventually VMware stopped supporting the free VMware Server 1.x line and we were forced to upgrade to VMware Server 2.x, which featured a barely functional web based management interface.

Several months ago, Silverfir started experiencing unexplained lockups every few days that required the virtual machine to be rebooted. This became annoying (especially for a box-in-the-closet) and very inconvenient for both Ryan and I. Eventually, the VM failed to boot entirely and the chunk of Silverfir hosted by Minimus was down entirely. This coincided with an extended trip by me to Rarotonga, an island in the middle of the pacific ocean with very minimal internet access. While Ryan and I were jointly remotely investigating the causes of this misbehavior, we found several VM metadata files that were 0 bytes and backup copies of these files were zero bytes as well. I reinstalled VMware Server and Ryan recreated the VM definition files from scratch. During this process, the web interface was very difficult to work with, as it wasn’t working properly in modern versions of IE and Firefox. Eventually, Ryan was able to get everything working again, and took the opportunity to upgrade from Ubuntu 9.04 to the current Long Term Support edition of Ubuntu, 12.04. Unfortunately, this did not solve the problem with the VM locking up every couple days. I decided it was time to modernize Minimus.

After spending a week experimenting with a number of modern hardware assisted hypervisors, I eventually decided to use Xenserver 6.2 Xenserver is a free (in multiple senses of the word) minimal footprint hypervisor similar to VMware ESXi. Unlike VMWare Server, which required a full fledge host OS be installed, Xenserver’s host footprint is very minimal, leaving more of the system’s resources (especially ram) free to be allocated to guest virtual machines. Because Xenserver relies on hardware level support for virtualization (a cpu feature called VT-x), guest virtual machines run much closer to the “bare metal” and feel a lot snappier as a result. Xenserver also has support for another newer hardware virtualization feature called VT-d which allows for hardware devices to be directly shared with guest VMs. This allows for devices like GPUs to be directly accessible to virtual machines. Citrix likes to show off this feature by running demanding, modern games like Skyrim in a VM and playing the game through a thin client device like an Ipad. Neat, but not very relevant to our particular use case.

The main things that I liked about Xenserver were:

  1. Free in multiple senses of the word. Based around an open source project, Xen, with widespread adoption both within the OS community and among major commercial users such as Amazon. Xenserver stores its VM metadata in standardized formats that are directly importable into other virtualization environments. Because of this, I have some confidence that I am not going to be bitten by a product discontinuation or lack of an easy forward migration path as occurred with VMware Server.
  2. After a week of hammering at it, Xenserver feels very mature. I only experienced one bug, relating to migrating VMs with associated snapshots, in a week of testing oddball cases. The windows based management tool, called Citrix Xencenter, is a pleasure to use.
  3. Xenserver allows me to create redundancy “pools,” clusters of  Xenserver hosts and shared storage resources that allow for guest virtual machines to be moved back and forth between multiple physical servers without needing to be taken offline. It is VERY cool to have a server being run off one physical box one moment and 30 seconds later having it be running off a different server with less than a single second’s network downtime. This should allow for Silverfir to stay online while hardware maintenance is being performed, something that wasn’t possible under the previous VMware Server environment.
  4. Xenserver allows for easy snapshot backups of running VMs, allowing backups to be created while the server is in use. With VMware Server, the guest had to be shutdown to backup the virtual disks, a process which took hours even using an eSATA based external backup drive.

Migrating Minimus to Xenserver was a fairly straightforward process. I was able to import the primary Minimus VMware virtual disk .vmdk file directly into a newly created VM guest in Xenserver. I had to edit grub, fstab and network interfaces to get the VM working in Xenserver and also had to add an remove a few kernel modules and install the old VMware tools, but all told I probably spent less than 2 hours getting Minimus running happily under Xenserver. What wasn’t so painless was getting the second data volume .vmdk to import. This second .vmdk stored all of Silverfir’s websites, photo galleries, etc…you know…the things that people actually care about. I received errors trying to import this file using a wide variety of virtual disk management/manipulation tools. I think this vmdk file had been created in an earlier version of VMware and probably used an older version of the .vmdk format. Eventually, after almost a whole day of trying to import this file I threw up my arms in disgust. As a workaround, I put both the old Minimus virtual machine and the new Xenserver virtual machine, which I am calling Maya, on the same network segment and created a new virtual disk container in Xenserver. Ryan then copied the data from the old virtual server to the new using the magic of rsync. This took quite a while, as almost 500 GB of data needed to be copied over at 100 mbit speeds. After almost a full day of copying and some adjusting of permissions, Maya was substantially complete and took over hosting duties from Minimus. Maya has been happily hosting Silverfir without incident for over a week.

In the near future, I plan to decommission Minimus entirely and replace it entirely. Maya’s current primary Xenserver host is a Core 2 Duo with 7 GB of ram. I plan on using most of the guts of Minimus to create a new server based around an Intel Avoton C2750 motherboard. This new system, which I am calling Dharma will be the primary Xenserver host for Maya, with the current Core 2 Duo host serving as a high availability backup server. Hosting of Maya’s data volumes will be via a Readynas Ultra 6 with a 12 TB Raid 6 array. Hopefully this new setup will allow for greater reliability and fault tolerance than what was  achieved with Minimus and Maya can continue serving Silverfir’s users for years to come.

Posted in Blogging, Emulation and Virtualization, General, Linux, Operating Systems, Tech Stuff | No Comments »

Notice Anything Different?

Posted by Deliverator on 20th October 2009

Ryan pulled a late night and did a final sync of data from Frankenputin (old server) to Minimus. I mainly sat back and let Ryan do the heavy lifting, just acting as cheerleader, head scratcher in chief and occasional googler of error messages. There were some struggles with Mysql and the usual Gallery puking, but eventually the beast was wrestled into submission. A few port forwarding changes and now everything is being served up by Minimus. Frankenputin has been powered down, quite possibly for good. My garage no longer sounds like a jet taking off and I kinda miss it.

If you are a user, please poke your head into infrequently visited dark corners and see if you find anything growing there. I plan to consign Frankenputin to its new role as boat anchor and kick Minimus into a closet as soon as it is verified that the new server is stable and no additional settings/data need importing from Frankenputin.

Posted in Blogging, Emulation and Virtualization, Linux, Tech Stuff | No Comments »


Posted by Deliverator on 4th May 2006

Well, I have virtually run out of Lucasarts Scumm games to play using ScummVM. I came to wondering whether there was any equivalent project for Sierra’s classic adventure games. Unfortunately, unlike Lucasarts, with their unified “Scumm” scripting system, Sierra games were implemented with a large number of tools. The chief systems used in Sierra adventure games were AGI for their earlier works and various versions of SCI for their later adventure games. While there are some interpreters to allow these games to be played on modern platforms, such as NAGI and FreeSCI, neither is nearly as mature and polished as ScummVM. So, what can one do, short of busting a 486 out of the closet? One solution that I have found is DOSBox. DOSBox is an opensource project that implements a large subset of the operating system and hardward environment of an old DOS computer, including all the oddball sound and video cards in use 15-25 years ago (Wow, has it really been that long?). DOSBox in itself is a bit of a pain to use, but couple it with a nice graphical front end like D-Fend and running old Dos games becomes a breeze. One of the nice things about D-Fend is allows one to create environment profiles for each game and create a direct icon to launch each game. So, if a game runs better with a Sound Blaster 16 or a EGA graphics card, then give it a EGA graphics card. So far, I haven’t found an old Dos game which DOSBox will not play. So far I have tested:

-Doom 2
-Gabriel Knight
-Goblins 1 and 2
-Hero’s Quest (later renamed Quest for Glory)
-King’s Quest I, II and III
-Wing Commander: Privateer
-Quest for Glory II
-Quest for Glory I – VGA
-Space Quest I – EGA and VGA
-Space Quest II

Posted in Emulation and Virtualization, Gaming, General, Operating Systems | 6 Comments »

OSX86 10.4.6 running on VMware Server

Posted by Deliverator on 19th April 2006


At last week’s hacknight, I got to see OSX running on commodity Intel hardware. In particular, it was running on a Dell XPS M170 gaming notebook (one of the new XPS series ones that glow). Unofficial support for non-apple hardware has come along quite a bit, with support for more video cards, non-SSE3 machines (still requires SSE2 minimum), and in the very latest release, support for SATA hard drives. I decided to try it out for myself, so I downloaded an pre-patched image created by someone named Myzar. I ran into one problem after another, trying to install using this iso image in VMware. I managed to resolve one problem after another, but finally bumped into an haulting ACPI error that prevented the system from fully booting after install. I found that Myzar created a new image and had released a PPF differential patch update to patch the previous iso image to OSX version 10.4.6 and also added the option of using a recompiled SSE3 kernel. The default kernel didn’t work, but the recompiled one did the trick. OSX is now running happily in VMware. All the hardware is working (network, sound, usb, etc), except for the video card, which is running with the unaccelerated VGA driver. I ended up having to tweak a lot of VMware settings to get it OSX working properly. Here are some settings I would recommend if you choose to try this at home…

  • Added the line paevm=true to the virtual machine’s .vmx config file. By default, VMware disables support for PAE (physical address extension), a technique for accessing ram > 4 GB on 32 bit CPUs. OSX requires support for PAE to run.
  • Mount the OSX cd image (iso file) using Daemon Tools, and then tell VMware to use the virtual cd-rom created by Daemon Tools. Many PC optical drives have problems with the OSX cds and VMware’s own .iso image mounting doesn’t work well with the HFS+ format cd.
  • Allocate all the hard disk space for your virtual hard disk drive immediately (instead of choosing the option allowing it to grow as space is used).
  • Specify the guest OS type as FreeBSD.
  • Add a USB controller to the list of virtual hardware, but disable the autoconnect option
  • In “host settings” choose the option to fit all guest os memory into reserved memory, rather than allowing for swapping. Produces quite a speedup.

Here are some more pictures of OSX86 running in VMware, as well as some of the error messages I received.

Posted in Emulation and Virtualization, General, Mac, Operating Systems, Tech Stuff | 15 Comments »

Is that a monkey in your pocket?

Posted by Deliverator on 12th April 2006

I have been playing around with ScummVM quite a bit this week. I have played 3 Lucasarts games to completion, so far, and have started on my fourth – Full Throttle. I have been scouring eBay and p2p for more games. One of the problems I ran into is that many of the newer Scumm games include full digital soundtracks and voice-overs. Earlier games simply had subtitles for the dialogue and used fm synthesis/midi files or something similar for the sound effects. As a result, some of the earlier games had data files ammounting to only 5-10 MB, while later games like Full Throttle might total a gig. The large size of the datafiles makes playing these larger games on a PDA rather difficult, as these devices tend to use fairly small flash cards for storage. One solution, other than going and plunking down those hard earned $ on a large flash card is to make use of an auxilary project to ScummVM called Scumm Tools. Scumm tools allows one to compress the Scumm audio files using MP3 or Ogg formats. Compressed data files are supported by most newer versions of ScummVM, although you will need to ensure that your platform has enough processing power to decode the audio and do everything else, as well. The tools are a nice start, but many games spread their data across a large number of files, making for a lot of time spent at a command prompt to compress them all. Thankfully, a user on the ScummVM forums created a series of game-specific batch scripts that help automate most of the process. The benefits of compressing are sizeable (haha). Most “talkie” games compressed to about 40% of their previous size, with no perceptible loss of audio quality.

Posted in Emulation and Virtualization, General | No Comments »

A computer on your keychain

Posted by Deliverator on 9th November 2005

VMware recently introduced VMware Player, a free piece of software that allows you to run virtual machines created in any of their commercial products. The software requires you to have administrator rights on any machine you use it on, as it installs several services, but it is still extremely cool to be able to take your virtual machines with you.

Another option for those wishing to take a vm along for the ride is Qemu, which unlike VMware’s software, supports access to network hardware without needing to install any special services. Sticking qemu and a vm on a keychain is a great way to have a secure, network capable computing environment with you should you need to access the internet from a untrusted/public computer.

Still another option, and perhaps your best bet if you don’t have a 1 or 2 GB USB keychain is Portable CE which allows one to run a complete Windows CE environment from a small keychain.

Portable CE

Posted in Emulation and Virtualization, General, Tech Stuff | No Comments »

Ubuntu, now fortified with silicon

Posted by Deliverator on 16th August 2005

I have been liking Ubuntu enough that I decided to devote some actual hardware to it and not simply run in a virtual machine. Running it in the VM was fine, but I was unable to get VMware tools to install, using the older version of VMware that I own (the new version supports it just fine). Without the VMware tools, the client OS is running without much in the way of graphics acceleration. I saw that VMware was releasing a toolkit, to allow OS makers to build in support for VMware’s “virtual hardware,” so that the tools install becomes unnecessary. in other VMware news, the x86 version of OS X has apparently been hacked to run within VMware. I will definitely have to check this out further. I have been running the PPC version of OS X in PearPC for some time now, but due to all the cross-instruction set translation needed to run PPC code on a x86 machine, running OSX on x86 hardware is currently very slow. Now, with OSX tunning natively on x86, emulation should be much faster. There is some DRM built into the developers release right now that tries to restrict OSX from running on anything but the official developer boxes, so a virtual machine may actually be the ideal environment for those wishing to run OSX on generic, non-apple x86 boxes.

Anyways, back to the original point. I now have a Ubuntu install on a Celeron 500 w 256MB of ram. Despite the relatively modest hardware, Ubuntu runs just great. Certainly much faster than XP on similar hardware. I am a little disappointed by how many things have been left out of the OS for “political” reasons. They can all be added back with a little effort, but it would be nice if they had been included on the install disks by default (even if not installed in the default installation configurations). I have found that I have needed to download far more packages with Ubuntu than with Fedora to get a nicely rounded system.

Posted in Emulation and Virtualization, General, Tech Stuff | 1 Comment »

Exercise while Ubuntu

Posted by Deliverator on 12th August 2005

Azureus, my BitTorrent client of choice finished downloading the DVD .ISO for Ubuntu Linux this evening. Although it was getting rather late, I blasted the image to a dvd-r using my new Plextor 12x SATA burner at full speed. I don’t normally burn at the faster speeds, as I have found that disks burned at slower speeds have better archival life and wear resistance. I attribute this to the burner being able to create more well-defined pits at slower speeds. Anyways, the Plextor handled the high-speed burn like a champ and a few scant minutes later, I had a Ubuntu installation DVD. I set up a VM in VMware with which to play around with it and popped in the disk. Installation went smoothly, although Ubuntu seems to be one of the last major distros still using a ascii text based installation wizard by default. The only hitch in the installation process came when after booting into the Gnome Desktop, I tried to get all available package updates. It popped up a dialogue saying I had to be logged in as root to do the updates. So, I logged out and proceeded to try and log back in as root. I typed in the password I normally use for super user level accounts and was presented with a bad password error message. I was sure I was asked for a password during the install, so I tried a couple minor variations, but to no avail. So, I was left to reinstall the OS from scratch. I did so, only to encounter the same problem. I googled for a solution. It turns out Ubuntu disables the root account by default and gives the user account you create during the install process super user status. It is easy to enable the root account, and I did so promptly. I really dislike having to use the sudo command every time I try and do something of substance. Anyways, outside of this minor issue, so far I really like Ubunto.

In other news, I am trying to get back into some semblance of shape that well…resembles a shape. I have been biking 15+ miles almost every day on my Lifefitness Lifecycle these last few weeks. I recently mounted a compact keyboard and a handheld trackball to the handlebars. This, coupled with my newly improved media center pc (Thanks go out to Colin for the mini-itx case!) makes for a decent web browsing, email and media platform while biking. I composed this entire entry while biking, although I will probably go back and edit in some links afterwards on my main pc. Using a keyboard as a text entry solution is not ideal for biking, as it requires me to take both hands off the handlebars and move into a more upright position. I might invest in an alternative text entry/pointing device like a Twiddler if I persist in exercising while I blog. I would also like to pick one up for eventual incorporation into a wearable computer rig.

Posted in Emulation and Virtualization, General, Tech Stuff | No Comments »

Lines in the sand

Posted by Deliverator on 10th August 2005

After a conversation with Matt Westervelt yesterday at hacknight, I have decided to try out Ubuntu Linux. I have been playing around with Fedora for a while now in VMware, and have been running OS X Tiger pretty regularly in PearPC. This is all part of an effort to become more proficient in non-microsoft OSes in anticipation of an eventual switch.

Despite being a long time advocate of Microsoft based solutions (and having worked there for 3 years), I have become more and more displeased of late with Microsoft’s heavy handedness towards consumers. In particular, I greatly dislike Windows Genuine Advantage, Paladium related initiatives and a general trend I see within Microsoft of not looking at what individuals actually want, but instead treating consumers as “The Product.” This is much in the same vein as media companies, where individuals are seen as “The Product” to be sold to advertisers, and not the reverse. A number of previously *Rock of Gibraltar* stable XP machines (and yes, they are legal licensed copies) have started bluescreening and displaying other distressing behaviour since the forcible installation of Windows Genuine Advantage a few weeks back. To prove that WGA was indeed the cause of these problems, I restored several of these boxes to disk images (I image the OS partitions of my most crucial machines regularly) taken prior to WGA and immediately turned off windows update, even going so far as to disable the service entirely. Unsurprisingly, these boxes returned to their previous high level of stability. I have just about had it with XP. Microsoft’s increasing attempts to exert control over users (DRM, Paladium, Windows Genuine Advantage, etc.) and post-sale crippling of features (like raw sockets support in SP2) has lost them my support as a user and more importantly, as an advocate. I am in the process of migrating several boxes back to W2K Server in the interim, but my longterm plan is for a much less Microsoft-centric infrastructure.

Microsoft still has time to regain my trust, but as it stands, I will not be purchasing Longhorn (recently rechristened Windows Vista) or any other Microsoft products that dares to disrespect the customer. This will be a difficult transition for me. One can tolerate a little malfeasance, but when push comes to shove, I vote with my wallet and am vocal about it. If a business does something you dislike, call them on it. Make sure the decision makers know the reason for you ire. Equally, make sure to let them know what you value about their business/product/service. All institutional power ultimately stems from the consent/acquiescence of individuals. It is often easier to let things slide and let some small portion of your life be governed/controlled by others, but at some point you have to fight for what you believe. To not do so is inhuman.

Posted in Emulation and Virtualization, General, Tech Stuff | 4 Comments »