IT archaeology: Virtualizing RedHat Enterprise Linux 3 (Taroon) with VirtualBox in the year 2013

One of my customers is in a very unhappy situation: He bought a piece of proprietary software that runs on RedHat’s Enterprise Linux 3, and now the company he bought it from is no longer available. What’s worse, they’ve tied their software to a USB copy protection dongle that uses a homemade Linux kernel module that only works on Linux 2.4.

This customer’s entire business depends on this piece of software. And now the server it’s running on is slowly falling apart, entire CPUs have burned due to failing fans, and there’s a proprietary RAID controller doing dangerous things to old and strange SCSI disks that have been unavailable on the market for several years already.

So they can’t upgrade their software or hardware. They have to watch as the old system collapses bit by bit.

You’d think you could just virtualize the server, but I ran into a couple of snags:

  • The software runs only on RedHat Enterprise Linux 3, release 2003. That’s ten years ago, just in case you’re not counting.
  • The software depends on a hardware dongle that only works with certain versions of Linux 2.4. It doesn’t work on any 2.6 or 3.x kernels, so simply using a modern version of CentOS is out of the question.
  • Using RHEL 3 means you can’t really transfer the system to modern hardware, since RHEL 3 doesn’t even support SATA.
  • Even if you do manage to find some hardware with ATA disks, who’d want that, it would be almost as old as the server it’s replacing!

I ended up creating a blank virtual machine in VirtualBox (so the customer can control it via GUI), and using a live CD to do these special things:

Create an ext3 filesystem that RHEL 3 can mount. That means using only 128 byte inodes and getting rid of some special features:

mkfs.ext2 -j -I 128 -L / /dev/hda1

Notice the label “/”, which RedHat really likes to have. Then disable the features RHEL 3 can’t deal with:

debugfs -w /dev/hda1
 debugfs 1.40.8 (13-Mar-2008)
 debugfs:  features -ext_attr -resize_inode -dir_index +large_file +needs_recovery
 Filesystem features: has_journal filetype needs_recovery sparse_super large_file
 debugfs:  quit

Thanks, this article, for the hint.

As next step, I SCP’d all the files from the physical server to the root of the new virtual hard disk. Then I adapted /etc/fstab to point at the new hda1 (it used to point at sda1, the RAID device exported via SCSI on the hardware server).

Finally, I booted Super Grub 2 Disk, selected the legacy Grub 1 configuration from the hard drive, booted into the new virtual hard drive, adapted /boot/grub/grub.cfg and to point at the new devices, ran grub-install. And so I had RHEL 3 running virtualized.

If you ever need to do data archaeology of this kind, maybe this article is useful for you.

The final step is to install an up to date GNU/Linux distribution on brand new hardware for my customer, put Virtual Box and that VM on there and pass the USB copy protection dongle through to the guest system.

Proxying from Apache HTTPS to some backend server that only speaks HTTP

Here’s a use case: You want to run an application server that only speaks HTTP, but securely, over HTTPS. The problem is that the application server won’t know that it’s being accessed via HTTPS, so any URLs and redirects it generates might point to HTTP. Here’s an example virtual host entry that takes care of that by rewriting the header.

You need Apache, mod_proxy and mod_headers.

<VirtualHost *:443>

  SSLEngine on
  SSLCertificateFile    /etc/ssl/certs/ssl-cert-snakeoil.pem
  SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key
  Header edit Location "^http:(.*)$" "https:$1"

  PassengerEnabled off
  ProxyPass /
  ProxyPassReverse /

  DocumentRoot /var/www/foo/bar
  <Directory /var/www/foo/bar>
    AllowOverride none
    Options -MultiViews

The magical line is the one with “Header edit…”. This makes sure any request your app server would have sent to HTTP are rewritten to HTTPS.

The slow and painful act of ungoogling yourself

With Google’s questionable treatment of privacy, you might want to gain some distance from that company.

I’ve done that myself a couple of steps at a time, and now I’m at the point where only one or two unhappy circumstances keep me nailed to the crucifix of Google systems and services.

Here’s what worked well:

  • Replacing Google Reader with my own TinyTinyRSS instance.
  • Replacing Picasa with my own Gallery instance.

Here’s what didn’t work so well:

  • Finding a new mobile operating system. I’ll be trying out Firefox OS and Sailfish on the Galaxy Nexus as soon as both have images available. I’m tied to WhatsApp and I think Sailfish is more likely to get a WhatsApp port, so my money’s on that. But ZTE has recently announced that they’ll be making off-the-shelf phones running Firefox OS this year, and Samsung is reviving Tizen, so I’ll be looking at those as well.

    I don’t know how much cross-pollination there will be between TIzen and Sailfish (as they’re both based on a Meego fork), but if there is any, it could be important for the market.

  • Replacing Google Docs. There seems to be no direct competition for the realtime collaboration aspect, except if you’re content with very basic formatting and using PiratePad.

    I know no reasonably easy to maintain Free Software browser-based office suite I could install. For more structured text editing with less of a realtime requirement, an instance of DokuWiki will do just fine.

  • Finding a good search engine. Sure, there is DuckDuckGo, but even though I love that thing and all its features, the results sometimes lag behind Google’s.

    I remember one episode where I asked something about a recent bug in some piece of software on a developer IRC channel and was asked, “why didn’t you Google it? It’s the second result on Google!” Of course I’m not a stupid arsehole, I always research existing solutions before bothering people on IRC. But DuckDuckGo simply didn’t find any information about that issue, even days later. Google really did have it on the first page (not as the second result, due to the way the search bubble works, but on the first page anyhow).

    How could we let it get so far that only one single company can provide good search results anymore?

So for me, it’s mostly WhatsApp keeping me with Google’s products. I realize WhatsApp is its own privacy nightmare. The main reasons I use it is that I refuse to pay € 0.08 per 160 characters of text when sending text messages on my phone, and the fact that it does group communication and attachments, whereas SMS is stuck in 1989. What are the telcos thinking?

Telcos are working on rolling out a replacement for SMS, and I hope it will at least dethrone WhatsApp. Of course it would be even better if people just used networks of interconnected XMPP servers, but I don’t think the average user can be arsed to do that.

Yay, degoogling!


Creating your own Steam Cloud (or how to reliably sync any two directories)

I was quite annoyed that the Steam Cloud seems to misread file timestamps some of the time, resulting in game saves being overwritten with older versions. And since Steam doesn’t offer a way to reliably find out which save is the most recent one, I needed something better.

The things I need:

  1. Connects over SSH, works with an SFTP/SSH server
  2. Transmits only file differences
  3. Reliably determines which copy of the pair is the newest
  4. Takes any required actions to make the older copy be identical to the newer one
  5. Works on GNU/Linux and Windows, since some games still aren’t available on Linux
  6. Should be able to save sets of pre-defined sources and targets so I can sync with one click or command

If you have similar needs, I might have a recommendation for you: Unison. Unison is not new, but directory synchronization (especially cross-platform) is not a trivial problem, and Unison has had a solution since 1998 and has only improved since then.

I have tried WinSCP’s synchronize mode on Windows, which worked quite well, but with Unison I now have a solution for Linux and Windows.

I didn’t actually need to do anything to get Unison to work. I have an SSH account on a server, and I just had to make sure that server has Unison installed as well (apt-get install unison-all on Debian GNU/Linux). Then I simply create one profile per game. A profile contains the source directory (where the savegames are stored on the local machine) and one target (the SSH server and subdirectory for that particular game).

With this set up, I can double-click a profile, it shows me the changes it needs to sync, and I just have to send them off with the “Go” button. Trivial!

So far, Unison has made no mistakes whereas Steam Cloud made two, twice assuming that the remote files were newer when in fact they were older. Maybe it’s a bug with Steam’s timezone conversion vs. DST? Anyhow, it feels nice to have my save game syncs under my own control, so if I fuck up the timezone, Unison will look at the file’s actual contents and do the right thing.

Building your own Sublime out of free components with vim

I recently discovered Sublime Text and bought a license even though I rarely use proprietary software for work. That’s how good it felt to me.

But if you’re comfortable with an editor like vim, you can make vim feel almost like Sublime, using only free and open source software (FOSS). vim (and emacs) have had many of the features that Sublime has, in some cases for decades. Here’s a very small and simple guide for making vim look and behave a little like Sublime.

First, install the required components into your .vim directory:

  1. vim-plug instead of pathogen or Vundle to manage plugins automatically.
  2. NERDtree gives you a separate directory tree to browse and open files from, like the left-hand side tree in Sublime.
  3. A terribly nice 256 color color scheme (works in the terminal and in the vim GUI version): PaperColor. Make sure your terminal supports 256 colors, set your TERM variable to something like xterm-256color. If you use a terminal multiplexer like screen or tmux, set it to screen-256color to make sure your background colors work properly.
  4. To replicate the Control-P/Command-P (Go To File) behavior found in Sublime, use the ctrlp.vim plugin.
  5. Set up your vimrc to load most of this stuff. See below for the relevant section of mine.

Example .vimrc:

call plug#begin('~/.vim/plugged')

Plug 'scrooloose/nerdtree'
Plug 'kien/ctrlp.vim'
Plug 'NLKNguyen/papercolor-theme'

call plug#end()

syntax on
filetype plugin indent on
set number

colorscheme PaperColor
set background=dark

Restart vim and do a :PluginInstall.

After another restart of vim you can execute :NERDtree to get a nice tree on the left-hand side, then open a file and perhaps split your window (Ctrl-W n) and load each of the files in separate split frames. Navigate back and forth between frames using Ctrl-w Ctrl-w (or Ctrl-w followed by the arrow key of the direction you want to move in). You can also split each of the frames into tabs, just like in Sublime.

And to finish off, one of the features I use most frequently in Sublime is the “find inside files in a folder” search (Ctrl-Shift-F). In vim, you can accomplish the same using e.g. vimgrep. First, grep for something in the files you want: :vimgrep meta_datum **/*.rb. Then bring up the quicklist to see your results: :cw. This way, you can navigate almost in the same way as in Sublime.

Two screenshots of all this combined below. Now go on and customize vim to fit your needs exactly!

Super-legible Heiti-style Chinese font for Debian GNU/Linux

I’m struggling with those crazy Chinese fonts in Mingti and Kaiti style and couldn’t find a goot Heiti font, but now this page:

Mentions that there is a perfectly fine Heiti-style font available in Debian: ttf-wqy-zenhei. So do this and be happy: apt-get install ttf-wqy-zenhei

Here’s a sort of preview from Wikipedia’s multilingual support page:

The samples above are in Heiti, the ones below in Mingti. Putting it into a screenshot makes it look shitty, so you’ll have to install the font (don’t install any other Chinese fonts!) and look at the page zoomed in to see it really well.

Avoding "Invalid byte sequence in UTF-8" with Ruby and CSV files

If you’re running into a ton of problems reading e.g. an ISO-8859-1 encoded CSV file into your (probably UTF-8) Ruby or Rails application, and if the error you get is “Invalid byte sequence in UTF-8” even though you’re giving the correct encoding options, here’s a solution.

The example CSV file is a tab-separated, ISO-8859-1 encoded file with CRLF line endings. You’d expect the following to work:, "r:ISO-8859-15:UTF-8", {:col_sep => "t", :headers => :first_row})

But it fails mysteriously! Even though the conversion to UTF-8 goes without problems, you get an ArgumentError complaining about some illegal byte sequence. If you analyze deeper, you might find (in this case) a complaint about rn. The solution is very, very non-obvious: You need to specify the row separator in addition to your encodings!

mjtko from the #rubyonrails channel on Freenode discovered this. If we change the line to the following:, "r:ISO-8859-15:UTF-8", {:col_sep => "t", :row_sep => "\n", :headers => :first_row})

Boom, there’s your working CSV object, with working encodings.

Watch TV on your PC, no ads, no Flash

Roman Haefeli strikes again: Watchteleboy makes it possible to watch dozens of live TV channels using mplayer in your very own machine, without the need for Flash, a web browser or any other such nonsense.

Here’s the source code:

Here are Ubuntu packages he maintains:

Caveat: This only works if you’re located in Switzerland or in some other place that Teleboy’s geotargetting likes (such as Italy).

Moving from Google Reader to Tiny Tiny RSS

In my quest for more freedom from companies that don’t take privacy too seriously (such as Facebook or Google), I found a fantastic FOSS replacement for Google Reader: Tiny Tiny RSS. It does everything important that Google Reader does and even has its own little syncable native Android app called ttrss reader (available through the Google Android Market).

The only additional feature I’d appreciate is a Reader Play-style view for very important tasks, such as scrolling through large amounts of animated gifs or lolcats quickly. Tiny Tiny RSS’s code seems to be clean and concise, though, so it might not be that much work to make such a feature if I ever find the time (which won’t happen).