The second webcomic is Real Life Comics. The author, Greg Dean, is having people submit scripts each day this week for how to continue the week-long (and non-canon) adventure. Today, he deemed my script the best. I’ve been reading his comic for over six years now (I think), and to see my name at the bottom of the comic is more than a little bizarre. I’m half tempted to buy a print. Of course, I now have even more of an interest in how this storyline turns out!

A long-term goal of mine had been to build an arcade stick for a games console. It took around ten days of near-solid work, but it’s finally done and is a joy to use. It’s already getting a lot of use in Street Fighter IV (and I’ve only received a single piece of abuse on Live so far, which is less than I was expecting), and come Wednesday it’ll be getting a lot of love through the medium of Marvel vs. Capcom 2. The electronics was, ironically, the easy bit, despite the fact I was soldering to an existing PCB and setting up two 25 pin harnesses. It was all done in under three days and worked correctly first time. Cutting six pieces of wood out of a floorboard, drilling a few holes, and screwing them together? Took the remainder of the time. I hate working with wood. I’m very happy with how it turned out, though – certainly at the lowest point I wouldn’t have expected it to turn out this good. It ended up costing around the same as buying a Madcatz Street Fighter IV SE stick, but is at least future-proof – I can build other breakaway project boxes that will let it work on other consoles, if I am struck by the urge.

Much to my delight, there are enough people around without jobs just yet (mostly people who took a gap year and so have only just graduated along with me), so there’s more than enough support for fun times. Notable mentions so far are a black-tie dinner party, sailing on Ben’s boat (where we saw seals), the beach, and going out having dressed up in owl material (oooooooo indeed).

I also trekked into London with some excellent co-conspirators who were game enough to see Derren Brown’s Enigma show. We’re asked to ‘keep the mysteries mysterious, and the surprises surprising’, so in that spirit I won’t delve into too much detail. However, speaking loosely, I think I can safely say that we were all extremely impressed. We managed to deduce how some of the effects were done to our satisfaction, though many still eluded us. I believe I know loosely how the final effect was done (at least, I have a method that I would use if demanded to reproduce it at gunpoint), and I had a particular song stuck in my head for a couple of days after the show. What the most shocking thing about the show was just how impressive Derren putting people into a dissociative state is; there was one girl he kept putting under and pulling back, and she just went completely limp instantly, each time.

I’m conscious of the fact that compared to a lot of people I know, I’m not doing anything too extravagant with my prolonged time off. Phil’s off trekking South America, and I know a few other people who are travelling to similar extents. While doing something along those lines would be fun, I’m not pining for it to any great extent. I’m away for a week towards the end of next month, but it’s primarily Ph.D. related rather than outright fun – though I’m anticipating some of that as well; after all, all work and no play…

For a competition I wrote a program to ’solve’ any Countdown numbers game (assuming a solution exists). The source is here, though it is very slow (a couple of minutes to exhaustively search a tree). Poking around the internet after I wrote it shows that there are much faster implementations of what is essentially the same algorithm, just with more shortcuts taken and a stricter adherence to the rules (it transpires that intermediate fractions are not allowed in Countdown, so by checking a % b before dividing you can save yourself a lot of time). It took me most of the day, but only because I don’t use Perl on a regular basis; I was initially unfamiliar with how it displays numbers and truncates their representations when asked to display them if there are a sufficient number of 0s after the decimal point. Frustrating, but good to know.

In english: I wondered just how much information you could send in a single tweet, and decided to find out.

Armed with the knowledge from an Information Theory course given by the singular David MacKay, I decided to see. So, it’s common knowledge that Arithmetic Coding is the best way to compress information, getting you to within a couple of bits of the Shannon Limit so long as you have an accurate modelling system. The ‘best’ bit of the compression comes from having suitably subtle and ingenious models of the text that you’re encoding; without such a model, you’re probably not going to see the benefits of Arithmetic Coding. However, I’m always up for a challenge – especially one set by someone with an Erdos Number of two.

Admittedly, I’d attempted to make an arithmetic encoder for a homework, which I almost achieved. However, I cheated by using the arbitrary precision maths module in PHP which would break after a suitably long string. As I presented it to him, I felt a pang of shame even before I explained… so reinvigorated with the fire of project procrastination, I thought I’d try and get one working properly. Admittedly, my first instinct was to just find one written in PHP, to get onto the meat of this project. But there didn’t seem to be one on the internet – until now! And it really does work properly, much to my amazement. Technically, it’s a range encoder, but the two are mathematically identical. As an added bonus, a range encoder isn’t protected by IBM patents, whereas an arithmetic coder is.

So, how do we transmit our bitstream over Twitter? Happily, Twitter supports UTF-8 characters, which can occupy up to four bytes each. For a four byte character, 21 are under control of the user. If we were to use just uppercase characters and space, for English this corresponds to about four bits per character we wish to send – so we could send up to 700 characters in such a case! We like punctuation though, so this means we can send fewer total characters, but it’s probably worth it. And so began my trial with Unicode.

1. First attempt: split the binary string into 21 bit blocks, pad the last one and send the corresponding UTF-8 characters. Unfortunately, not all 21 bit long binary strings map to valid characters – those above 0×10ffff are invalid. So drop down to 20 bits per UTF-8 character, and have the first bit as 0 – we’re now definitely going to map onto a valid unicode character, right?
2. Turns out, no. The points 0xD800 to 0xDFFF are UTF-16 surrogate pairs. So look for these, and if we would be trying to send one of these, send a three byte character instead (which will only be 16 bytes, but still a bargain).
3. Unicode has canonical decompositions – so the character á is canonically identical to the characters representing a´ in sequence, for example. These are defined to be identical, and no application should function differently when presented by one sequence or the other. So now check to see that a character we are trying to send is fully decomposed – if not, drop to a three byte character if we were four, and a two byte character if we were three. I’m not convinced that there’s a PHP library that does a complete job of this anywhere, so this is currently only a ‘probably’ step of the process – if a message fails to send, it’s probably failed here.
4. Line Feed / Carriage Return. Different OSes introduce the other – or not – after seeing one. So if the character we’re trying to send is actually in the bottom 128 of characters, only take seven bits. Set the 32s bit to 1, so we’re not using the awkward part of the ASCII table.

I then set about trying to send some information over twitter. The first test worked well. I decided to push it to the limit, however, and encoded nearly 600 characters as a string of 138 UTF-8 characters. Dropped my clipboard into Tweetie, and was informed that I was over the limit. Perplexed, I did some poking around and found that, in fact, Twitter isn’t sure what they mean when they say 140 characters. They think they probably mean bytes, not characters, but there is evidence to the contrary. No further updates as yet that I can see, but problems seem to be in at least partly due to how Ruby counts characters. Until this is fixed, however, the amount of information able to be sent using a single tweet is going to be limited, somewhat. They’re looking into it.

The upshot is that there is now a proof of concept of the compressor up along with the source. I’ve included the UTF-8 libraries with it which are from a couple of sources, but mostly MediaWiki. I based the range encoder on the pseudocode over at the wiki article, changing it to use binary rather than base 10. It’s entirely for use at your own risk.

Lastly, it would seem that I am not the first to have an idea along these lines, and some have even gone much, much further. Arguably too far. Truly, the internet is a wonderful enabler for people who want to do things simply to see if they can be done. I find it amusing how all of these seem to have sprung up in the last month!