The Deliverator – Wannabee

So open minded, my thoughts fell out…

Archive for the 'Blogging' Category

Thoughts on blogging

Dharma / Maya – A New Virtualization Platform for Silverfir to Replace Minimus

Posted by Deliverator on 22nd March 2014

Back in 2009, most of’s services were migrated from an aging behemoth of a Compaq server named Frankenputin onto what was hoped would be a much more manageable platform which I christened Minimus. Minimus was designed to be a server that could run contentedly in a closet for year’s on end. It was based around a dual core atom motherboard which sipped power and featured completely passive cooling and used a solid state drive as its boot drive for greater reliability. The host OS was Windows 7 running VMware Server 1.x to host a Ubuntu Linux virtual machine. Eventually VMware stopped supporting the free VMware Server 1.x line and we were forced to upgrade to VMware Server 2.x, which featured a barely functional web based management interface.

Several months ago, Silverfir started experiencing unexplained lockups every few days that required the virtual machine to be rebooted. This became annoying (especially for a box-in-the-closet) and very inconvenient for both Ryan and I. Eventually, the VM failed to boot entirely and the chunk of Silverfir hosted by Minimus was down entirely. This coincided with an extended trip by me to Rarotonga, an island in the middle of the pacific ocean with very minimal internet access. While Ryan and I were jointly remotely investigating the causes of this misbehavior, we found several VM metadata files that were 0 bytes and backup copies of these files were zero bytes as well. I reinstalled VMware Server and Ryan recreated the VM definition files from scratch. During this process, the web interface was very difficult to work with, as it wasn’t working properly in modern versions of IE and Firefox. Eventually, Ryan was able to get everything working again, and took the opportunity to upgrade from Ubuntu 9.04 to the current Long Term Support edition of Ubuntu, 12.04. Unfortunately, this did not solve the problem with the VM locking up every couple days. I decided it was time to modernize Minimus.

After spending a week experimenting with a number of modern hardware assisted hypervisors, I eventually decided to use Xenserver 6.2 Xenserver is a free (in multiple senses of the word) minimal footprint hypervisor similar to VMware ESXi. Unlike VMWare Server, which required a full fledge host OS be installed, Xenserver’s host footprint is very minimal, leaving more of the system’s resources (especially ram) free to be allocated to guest virtual machines. Because Xenserver relies on hardware level support for virtualization (a cpu feature called VT-x), guest virtual machines run much closer to the “bare metal” and feel a lot snappier as a result. Xenserver also has support for another newer hardware virtualization feature called VT-d which allows for hardware devices to be directly shared with guest VMs. This allows for devices like GPUs to be directly accessible to virtual machines. Citrix likes to show off this feature by running demanding, modern games like Skyrim in a VM and playing the game through a thin client device like an Ipad. Neat, but not very relevant to our particular use case.

The main things that I liked about Xenserver were:

  1. Free in multiple senses of the word. Based around an open source project, Xen, with widespread adoption both within the OS community and among major commercial users such as Amazon. Xenserver stores its VM metadata in standardized formats that are directly importable into other virtualization environments. Because of this, I have some confidence that I am not going to be bitten by a product discontinuation or lack of an easy forward migration path as occurred with VMware Server.
  2. After a week of hammering at it, Xenserver feels very mature. I only experienced one bug, relating to migrating VMs with associated snapshots, in a week of testing oddball cases. The windows based management tool, called Citrix Xencenter, is a pleasure to use.
  3. Xenserver allows me to create redundancy “pools,” clusters of  Xenserver hosts and shared storage resources that allow for guest virtual machines to be moved back and forth between multiple physical servers without needing to be taken offline. It is VERY cool to have a server being run off one physical box one moment and 30 seconds later having it be running off a different server with less than a single second’s network downtime. This should allow for Silverfir to stay online while hardware maintenance is being performed, something that wasn’t possible under the previous VMware Server environment.
  4. Xenserver allows for easy snapshot backups of running VMs, allowing backups to be created while the server is in use. With VMware Server, the guest had to be shutdown to backup the virtual disks, a process which took hours even using an eSATA based external backup drive.

Migrating Minimus to Xenserver was a fairly straightforward process. I was able to import the primary Minimus VMware virtual disk .vmdk file directly into a newly created VM guest in Xenserver. I had to edit grub, fstab and network interfaces to get the VM working in Xenserver and also had to add an remove a few kernel modules and install the old VMware tools, but all told I probably spent less than 2 hours getting Minimus running happily under Xenserver. What wasn’t so painless was getting the second data volume .vmdk to import. This second .vmdk stored all of Silverfir’s websites, photo galleries, etc…you know…the things that people actually care about. I received errors trying to import this file using a wide variety of virtual disk management/manipulation tools. I think this vmdk file had been created in an earlier version of VMware and probably used an older version of the .vmdk format. Eventually, after almost a whole day of trying to import this file I threw up my arms in disgust. As a workaround, I put both the old Minimus virtual machine and the new Xenserver virtual machine, which I am calling Maya, on the same network segment and created a new virtual disk container in Xenserver. Ryan then copied the data from the old virtual server to the new using the magic of rsync. This took quite a while, as almost 500 GB of data needed to be copied over at 100 mbit speeds. After almost a full day of copying and some adjusting of permissions, Maya was substantially complete and took over hosting duties from Minimus. Maya has been happily hosting Silverfir without incident for over a week.

In the near future, I plan to decommission Minimus entirely and replace it entirely. Maya’s current primary Xenserver host is a Core 2 Duo with 7 GB of ram. I plan on using most of the guts of Minimus to create a new server based around an Intel Avoton C2750 motherboard. This new system, which I am calling Dharma will be the primary Xenserver host for Maya, with the current Core 2 Duo host serving as a high availability backup server. Hosting of Maya’s data volumes will be via a Readynas Ultra 6 with a 12 TB Raid 6 array. Hopefully this new setup will allow for greater reliability and fault tolerance than what was  achieved with Minimus and Maya can continue serving Silverfir’s users for years to come.

Posted in Blogging, Emulation and Virtualization, General, Linux, Operating Systems, Tech Stuff | No Comments »

Social media integration for long winded posts.

Posted by Deliverator on 18th November 2012

I’ve been forsaking the WordPress blog here for quite some time. Most of my ruminations seem to be too short for me to be bothered with writing a blog entry, so I’ve largely shifted to using my Twitter account. At the same time, I’m also finding Twitter’s 140 character limit a bit too limiting. I am often writing 3 or 4 back to back tweets on a subject, which I am sure does not endear me to followers uninterested in said subject.

There do exist 3rd party services like TwitLonger that work around Twitter’s forced brevity problem/feature, but I like to keep my data in-house to avoid many of the snafus that are part and parcel of using cloud services. Having lost or lost control of data important to me in the past, I don’t like trusting my content/making myself dependent on companies whose operational procedures are opaque to me and whose terms of service, business model, etc. might change with the blowing of the wind. It is one of the chief reasons I’ve yet to join Facebook, Google+, etc.

I am going to start testing various plugins for WordPress that allow me to automatically cross-post to Twitter as well as archive my tweets here in case Twitter’s business model becomes too onerous (the promoted tweets are already getting obnoxious).

Posted in Blogging, General, Mobile Blogging, Rants and Raves | No Comments »

Notice Anything Different?

Posted by Deliverator on 20th October 2009

Ryan pulled a late night and did a final sync of data from Frankenputin (old server) to Minimus. I mainly sat back and let Ryan do the heavy lifting, just acting as cheerleader, head scratcher in chief and occasional googler of error messages. There were some struggles with Mysql and the usual Gallery puking, but eventually the beast was wrestled into submission. A few port forwarding changes and now everything is being served up by Minimus. Frankenputin has been powered down, quite possibly for good. My garage no longer sounds like a jet taking off and I kinda miss it.

If you are a user, please poke your head into infrequently visited dark corners and see if you find anything growing there. I plan to consign Frankenputin to its new role as boat anchor and kick Minimus into a closet as soon as it is verified that the new server is stable and no additional settings/data need importing from Frankenputin.

Posted in Blogging, Emulation and Virtualization, Linux, Tech Stuff | No Comments »

Galleries Back Up (sorta)

Posted by Deliverator on 24th March 2008

I took some time this evening to reinstall Gallery. A large number of sub-galleries had some level of corruption, and although I could have restored many of them to working order, the degree of time needed to do so was more than I was willing to accommodate. So, I am starting out fresh. I have virtually all the images contained in my original gallery, but naming and organization will change somewhat. Due to Silverfir’s limited bandwidth and my being too lazy to physically visit the server, it will likely take a few weeks to get all the pictures up again. If I am feeling particularly bored some time, I may go back through all my blog entries and re-link to the new image urls. Going forward, Silverfir now has a backup drive, of which I intend to make frequent use.

The first gallery to go up is of pictures from the 2008 Seattle FIRST Robotics Competition.

Posted in Blogging, Rants and Raves, Titan Robotics Club | 1 Comment »

Are You Human?

Posted by Deliverator on 29th July 2007

Discerning whether a comment comes from a human being or a spambot is a surprisingly difficult problem and a large number of automated solutions have sprung up to keep bots from filling up the blogoverse with blogorhea (at least the type generated by bots, to say nothing about human inanities). Up until recently, I have used a combination of Akismet and a plugin called Did You Pass Math? to automatically discard the vast majority of comment spam. If a potential comment passed Akismet and demonstrated basic math skills, WordPress throws the comment in a moderation queue for further examination by a meat filter (myself). WordPress emails me and I can discard or approve the comment in short order. Recently, I have started receiving comments in my final stage moderation queue that indicate that robots have learned to add and subtract, or an equally startling possibility, that human beings have done the same! Today addition, tomorrow the world! Surely the apocalypse is nigh!

I decided to swap out Did You Pass Math? for a more robust Captcha based solution. I have not been a big fan of Captcha based solutions, in part due to their almost universally poor implementations. Many Captcha implementations are extremely difficult for the average human being to “solve,” but surprisingly easy for special purpose OCR software. In other cases, spammers looking to circumvent Captcha based solutions will cleverly relay the Captcha images to the login pages of high traffic porn sites and use porn starved human beings to solve the Captcha for them. Additionally, many Captcha based solutions make vital Internet servicesa…like my blog…inaccessible to blind users. Enter reCAPTCHA, a free service from Carnegie Mellon University, the guys who quite literally invented the term CAPTCHA (or at least hold all the trademarks).

reCAPTCHA places a couple twists on the CAPTCHA concept:

-Make it difficult to impossible to redirect the CAPTCHA to another site to be solved by an unwitting human.
-Get your initial source material for generating the CAPTCHA image from books being scanned for the Internet Archive. Use only snippets which are given the Archive’s OCR software problems. This text is by definition difficult for automated OCR software to solve.
-Distort the image in ways that make it even more difficult for OCR software, but which don’t increase the difficulty for human identification.
-Use human beings as “proof readers,” making every solving of a Captcha a meaningful contribution towards the preservation of human knowledge and not simply a task which wastes 15 seconds of you time and has you swearing under your breath.
-Provide an audio CAPTCHA system for blind users

Anyways, I installed the WordPress plugin for reCAPTCHA today. Give it a try and let me know what you think….or at least try. I apologize in advance to any readers of this site who are both deaf and blind. I have a system in the works to address the problem based on Smell-O-Vision.

Posted in Blogging, General, Tech Stuff | 3 Comments »

New Contact Form

Posted by Deliverator on 29th March 2007

I added a contact form to the site. If you wish to contact me about a personal matter, site concern, consulting inquiry, etc., you may use the form found to the right under site links. Please do not simply comment in a random post in hopes of getting my attention.

Posted in Blogging | No Comments »

Upgraded Gallery

Posted by Deliverator on 18th March 2007

I upgraded my Gallery installation to the latest test release. Gallery 1.x has been in development for many years and even the alpha releases seem very stable to me. I contemplated migrating to the database based 2.2, which was recently released, but I am not yet at the point where performance is suffering from the flat file nature of my gallery and the feature set in 2.2 vs 1.x isn’t enough to make me want to perform the rather involved upgrade procedure. 1.x also offers at least one really compelling advantage over 2.2 due to its flat file nature; It is possible to backup a gallery 1.x install with a single short command line and it is possible to build an offline version of a gallery for distribution on CDs/DVDs. I did some minor tweaks to the gallery after upgrade to keep my thumbnail sizes more consistent. Please note any wonkiness you might encounter.

While I was at it, I did a fresh gallery install for my brother, Scott. He doesn’t have much up yet, but hopefully this will compel him to stop using Flickr and Picasa.

Posted in Blogging, Media, Photography | 1 Comment »

Terra Bite Lounge – Kirkland

Posted by Deliverator on 5th March 2007

So, I’ve been meaning to check out the Terra Bite Lounge in Kirkland for a while now. Today, thanks to a rare break in my schedule, I finally got the chance. Terra Bite is a cafe which has no set prices and payment is strictly voluntary. They have a drop box near the counter in which you can contribute anything you choose, and can also pay online via paypal. Payment is neither encouraged nor discouraged. Supposedly, Terra Bite started out as a bet as to whether, in the absence of compulsion, people are inherently good or evil and the cafe is the means of testing the proposition. I really like the fact that I can pay whatever I like, pay online, pay weekly, etc. The atmosphere is really nice and low key and the service is better, as the Barristas aren’t spending half their time making change and processing credit cards (although you can pay by card as well). I am really intrigued by this experiment, as it is really similar in philosophy to much that is at the heart of the DRM debate.

The cafe itself seems to have good food and drink, music and nice furniture. I am sitting here relaxing on a big plush leather couch while watching someone play “Gears of War” for the XBOX 360 on a big plasma screen. Needless to say, and as evidenced by this post, Terra Bite has free WiFi as well.

I highly recommend you check it out. It is at the corner of Kirkland and State street.

Posted in Mobile Blogging, Wireless | No Comments »

Explosive Growth in N800 Development

Posted by Deliverator on 7th February 2007

Since the recent release of the N800, Maemo related development has really taken off. Nokia has really helped foster this development with the sourceforge-like development site, well documented SDK´s and even a preconfigured dev environment released as a VMWare virtual computer image. They have also given away some 500 N800´s to select OSS developers. Whatever the reason, the dev community seems to be growing by leaps and bounds. Here are a few tidbits of dev news:

-Quake 2 has been demonstrated running on the N800 at quite acceptable frame rates. This really demonstrates the drastic improvement the more modern 330mhz Omap chip in the N800 offers over the older OMAP chip in the N770.

-Dual boot from a SD/MMC card is now available for the N800. This really makes it possible for developers to muck about with the OS internals without worring about needing to reflash or possibly brick their device.

-Probably as a direct result of the previous news item, some people are fooling around with full blown windows managers and producing windows manager switching scripts. The Hildon UI is well adapted to the constraints of a mobile tablet, but it also requires existing applications to be tweaked to work properly. Having a full window manager available to run unmodified applications will allow the use of a much wider range of OSS without anything more complicated than a recompile.

-Developers discovered that the N800 contains two previously undocumented chips on the motherboard, hidden under metal RF shielding. The first is an FM radio tuner. A desktop applet to control the FM radio was released by Nokia shortly after the discovery. The radio works quite well, using the headphones as an antenna. The applet has the ability to switch the sound between the internal stereo speakers and headphones when using the headphones as an antenna. With the new alarm framework, this should make it easy for a FM radio alarm clock to be developed.

The other chip discovered in the N800 is a USBOTG power controller. While currently unsupported, many people hope that this will allow for powered hostmode for external devices. Currently, plugging in things like USB keyboards requires the use of a powered USB hub or the construction of a USB power injector.

-Nokia released an updated version of their Mediastreamer application, which supports streaming of various forms of content such as pictures and music from uPnP enabled media servers. The new version adds support for video as well.

-There are now instructions available to get Mono, a OSS implementation of the .NET Framework, running on the N800 and N770. There are still no MAEMO specific bindings in MONO yet, but this is a good step towards having applications that can run on the desktop and on the Nokia Internet Tablets without recompilation.

Posted in Linux, Mobile Blogging, Operating Systems, Portable Computing/Gadgets, Tech Stuff | No Comments »

WordPress Upgrade – 2.06

Posted by Deliverator on 5th January 2007

Upgraded the software which runs this blog, WordPress, to version 2.06

Please let me know if you notice anything out of sorts.

Posted in Blogging, General | No Comments »