One of my customers is in a very unhappy situation: He bought a piece of proprietary software that runs on RedHat’s Enterprise Linux 3, and now the company he bought it from is no longer available. What’s worse, they’ve tied their software to a USB copy protection dongle that uses a homemade Linux kernel module that only works on Linux 2.4.

Add insult to injury: Their entire business depends on this piece of software. And now the server it’s running on is slowly falling apart, entire CPUs have burned due to failing fans, and there’s a proprietary RAID controller doing dangerous things to old and strange SCSI disks that have been unavailable on the market for several years already.

So they can’t upgrade their software, and they can’t upgrade their hardware either. They have to watch as the old hardware collapses bit by bit, until it destroys their business.

Tough luck, eh? They shouldn’t have worked with proprietary software, then this might not have happened. They could just upgrade all the way through to the 21st century and be done with it. But since they didn’t do that, now it’s too late for them to switch, so I thought I’d “just” virtualize that server.

Here are all the associated problems that needed to be solved:

• The software runs only on RedHat Enterprise Linux 3, release 2003. That’s ten years ago, just in case you’re not counting.
• The software depends on a hardware dongle that only works with certain versions of Linux 2.4. It doesn’t work on any 2.6 or 3.x kernels, so simply using a modern version of CentOS is out of the question.
• Using RHEL 3 means you can’t really transfer the system to modern hardware, since RHEL 3 doesn’t even support SATA.
• Even if you do manage to find some hardware with ATA disks, who’d want that, it would be almost as old as the server it’s replacing!

I ended up creating a blank virtual machine in VirtualBox, and using a live CD to do these special things:

Create an ext3 filesystem that RHEL 3 can mount. That means using only 128 byte inodes and getting rid of some special features:

mkfs.ext2 -j -I 128 -L / /dev/hda1

Notice the label “/”, which RedHat really likes to have. Then disable the features RHEL 3 can’t deal with:

debugfs -w /dev/hda1
 debugfs 1.40.8 (13-Mar-2008)
 debugfs:  features -ext_attr -resize_inode -dir_index +large_file +needs_recovery
 Filesystem features: has_journal filetype needs_recovery sparse_super large_file
 debugfs:  quit

Thanks, this article, for the hint. Notice the “hda” there? That’s right, you need to mount the thing as parallel ATA device, not SATA, not SCSI.

As next step, I SCP’d all the files from the physical server to the root of the new super-compatible virtual hard disk. Then I adapted /etc/fstab to point at the new hda1 (it used to point at sda1, the RAID device exported via SCSI on the hardware server).

Finally, I booted Super Grub 2 Disk, selected the legacy Grub 1 configuration from the hard drive, booted into the new virtual hard drive, adapted /boot/grub/grub.cfg and device.map to point at the new devices, ran grub-install. And so I had RHEL 3 running virtualized.

If you ever need to do data archaeology of this kind, maybe this article is useful for you. You could likely automate it all in a little shell script if you need to virtualize many old systems.

The final step is to install an up to date GNU/Linux distribution on brand new hardware for my customer, stick that virtual machine on and pass the USB copy protection dongle through to the guest system.

Here’s a use case: You want to run an application server that only speaks HTTP, but securely, over HTTPS. The problem is that the application server won’t know that it’s being accessed via HTTPS, so any URLs and redirects it generates might point to HTTP. Here’s an example virtual host entry that takes care of that by rewriting the header.

You need Apache, mod_proxy and mod_headers.

<VirtualHost *:443>
  ServerName foo.bar.example.com

  SSLEngine on
  SSLCertificateFile    /etc/ssl/certs/ssl-cert-snakeoil.pem
  SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key
  Header edit Location "^http\:(.*)$" "https:$1"

  PassengerEnabled off
  ProxyPass / http://127.0.0.1:3000/
  ProxyPassReverse / http://127.0.0.1:3000/

  DocumentRoot /var/www/foo/bar
  <Directory /var/www/foo/bar>
    AllowOverride none
    Options -MultiViews
  </Directory>
</VirtualHost>

The magical line is the one with “Header edit…”. This makes sure any request your app server would have sent to HTTP are rewritten to HTTPS.

I was quite annoyed that the Steam Cloud seems to misread file timestamps some of the time, resulting in game saves being overwritten with older versions. And since Steam doesn’t offer a way to reliably find out which save is the most recent one, I needed something better.

The things I need:

1. Connects over SSH, works with an SFTP/SSH server
2. Transmits only file differences
3. Reliably determines which copy of the pair is the newest
4. Takes any required actions to make the older copy be identical to the newer one
5. Works on GNU/Linux and Windows, since some games still aren’t available on Linux
6. Should be able to save sets of pre-defined sources and targets so I can sync with one click or command

If you have similar needs, I might have a recommendation for you: Unison. Unison is not new, but directory synchronization (especially cross-platform) is not a trivial problem, and Unison has had a solution since 1998 and has only improved since then.

I have tried WinSCP’s synchronize mode on Windows, which worked quite well, but with Unison I now have a solution for Linux, Windows and Mac OS X (not that I’d touch that with a pole).

I didn’t actually need to do anything to get Unison to work. I have an SSH account on a server, and I just had to make sure that server has Unison installed as well (apt-get install unison-all on Debian GNU/Linux). Then I simply create one profile per game. A profile contains the source directory (where the savegames are stored on the local machine) and one target (the SSH server and subdirectory for that particular game).

With this set up, I can double-click a profile, it shows me the changes it needs to sync, and I just have to send them off with the “Go” button. Trivial!

So far, Unison has made no mistakes whereas Steam Cloud made two, twice assuming that the remote files were newer when in fact they were older. Maybe it’s a bug with Steam’s timezone conversion vs. DST? Anyhow, it feels nice to have my save game syncs under my own control, so if I fuck up the timezone, Unison will look at the file’s actual contents and do the right thing.

The hardest part now is to find out where each game saves its savegame files, but that’s something you can sort out with a search engine.