Meaningful highlighting of failed tests for test/unit in Ruby

People are often confused by what tests actually do - their role is not merely indicating that something is wrong.

If somehow Flying Spaghetti Monster bestowed upon you a divine gift of a Perfect Test Server, which would literally read your mind, and tested any of your programs instantaneously indicating by either green or red light if the tested program Does What You Mean - such a gift would be pretty much worthless, except for some extremely narrow domains like crypto protocols (which actually tend to fail because your threat model was insufficiently imaginative...).

First, it would universally show the red fail light on any program, because with sufficiently strong testing some imperfection will be eventually found, and second - without some kind of indication what's the nature of the problem there's pretty much nothing useful you can do other than stare at code and hope for sudden burst enlightenment.

This by the way is another reason why static typing and proving properties of code are a total waste of time - try making a typo in any nontrivial C++ STL program, and see for yourself how much compiler's perfectly correct information that "something's wrong with your program" is going to help you.

The primary function of a test suite is helping you locate the nature of a problem, and which code is likely to be responsible for the problem. That's why tests produce meaningful messages on what differed between expectation and actual result, why we have different kinds of tests (with very inconsistently applied names like "unit", "functional", "integration", "regression" and so on), why continuous integration servers test every commit to correlate code change with test failure and so on.

Piles of microassertions anti-pattern

One fairly common testing anti-patterns which annoys me a lot are tests which have ton of microassertions, which tend to pass together or all fail together, moral equivalent of this:

  assert_equal "Hello", hw.message
  assert_equal ", ", hw.separator
  assert_equal "world", hw.target
  assert_equal "!", hw.punctuation

Now imagine that hello world package accidentally changed to German somehow, or UTF-16BE, or some other crazy thing - every single assertion will fail simultaneously. Unfortunately you will never get any information about what actually happened with any assertion other than first - and debug prints will come.

This can be improved somewhat to not terribly pretty but much more useful:

  assert_equal ["Hello", ", ", "world", "!"], [hw.message, hw.separator, hw.target, hw.punctuation]

If they fail together, you'll be given full information on what precisely happened.

This kind of structure-vs-structure comparison is much more useful - especially for regression testing where you test against saved known complex output, and integration testing where you test outputs of individual component against outputs of entire subsystem. Probably less so in low level unit tests where you'd actually have to type expected value manually.

Unfortunately we immediately run into second problem - when such comparison fails, we get a massive message in which it might be hard to localize which parts are the same and which differ.

Highlighting

This is where my small library comes into play. It overrides assert_equal, and if expected and actual value aren't equal, it calls #inspect on them, tokenizes them with a simple regular expression, uses diff/lcs library to compute diffs, and then outputs the same message as plain old test/unit's assert_equal except with added and deleted parts highlighted using ANSI color codes which should work on just about any kind of terminal.

It also includes a small hack to make TextMate's test runner window display this highlighting.

Now this library has just been extracted from old-style Rails plugin, it's a bit messy, but it doesn't depend on Rails anymore.

It shouldn't be too hard to adapt to other testing libraries if you need to do so.

My Christmas wishlist - redcar and desktop Linux

Who would have thought my blog would be so high on OSX vs Linux. Well, this subject never gets old, so here's another instalment.

For personal reasons which Facebook aptly calls "It's complicated" I haven't had much time to update this blog recently.

I have so many things I'd like to write about - but marginal utility of time keeps getting in a way - with a lot of my previously free time no longer available, I have to make decisions which of too many things I'd like to do I should drop or postpone, hopefully starting from the least important of them.

And just like nearly everybody else, I keep confusing important with urgent - this blog is never really urgent - no cat will die if I leave writing for another day or week - but it surely is not the least important of all things I could be doing, or have been doing in the last month or so. So quadrant two, here I come!

Macs suck

GPU in my MacBookPro failed, so I'm temporarily back on Linux, on a desktop box which I was using mostly for gaming.

I will probably stay on Linux for quite some time, even after I get my laptop back. And they're really slow with repairs - I will forgive them only if they keep my hard drive undisturbed like they promised - my backups were not terribly up to date (about a month old) and what's worse - didn't include everything (just imagine - I don't have kitten picture downloader compatible with the most recent flickr API and I have to download kittens manually!). Now I backup everything to two independent external USB disks just to be sure, so I was prepared for this kind of backup failure, but I don't backup as often as I should - mostly because dual backup is such a pain.

Anyway, all cost of switching from one system to another must be paid up front - all the time necessary to setup and upgrade the new system, to change my habits and so on. Once the cost is paid (and with this post it will mostly be so), I can as well continue using it - so I might very well wipe out OSX and install Ubuntu on the laptop once I get it - I haven't decided it yet.

Well, first here's the list of the most important things how Linux (kUbuntu, old install upgraded to 10.4) is way better than OSX:

• Polish Dvorak keyboard, thanks to divide. I kept meaning to write one for OSX, but never quite did it. You can write keyboard layouts for OSX as XMLs these days, but sadly there seems to be no way to get XMLs of existing layouts as starting point.
• apt-get. Seriously, MacPorts is such a pile of fail compared to apt-get, it's just ridiculous. Imagine that - with apt-get you can upgrade system without breaking anything. It just works! I know, it's crazy!
• I mostly develop for EC2, so quite a few things are easier on Linux than on OSX.
• konsole and virtual desktops completely destroy Terminal.app and Spaces.

Sadly from this point it only gets worse.

Desktop Linux sucks

When you keep using the same system for long, you stop noticing some problems - you just get used to them, just like Congolese got used to malaria.


And this is actually a big fucking deal. People "in charge" of desktop Linux - be in GNOME, KDE, or whatever - are mostly old-timers. They got so used to all the fail present in desktop Linux, it doesn't seem like a big deal to them.

The last non-Linux system they used regularly is probably Windows XP or something even older - they have only very faint idea what progress the closed source world have made with OSX and even Windows 7. Did I ever mention I sort of like Windows 7? Not for development of course, but you just don't see fail jumping at you all the time like it used to be on XP and Vista.

Anyway, the first impression I had once I switched from OSX to KDE was:

Are they fucking trying to reimplement Windows XP here?

I am deeply disturbed by what I've seen. KDE not only lacks all the awesome new stuff you get in OSX, it also lacks all the awesome old Unix stuff you get in OSX.

Let's start with the basics - a text editor. Control-A and Control-E don't work. A shortcut as convenient as Control-A instead does something as incredibly useless as selecting entire document. Control-E seems to do nothing. Just like Control-D. Control-K deletes entire line, which is broken, but at least only halfway so.

Fine, whatever, I thought - Windows noobs also use KDE, and if they want such broken shortcuts it's their problem - I'll just go to keyboard settings and switch to Emacs mode.

It's such a shame I haven't been recording myself, as the very moment I realized they removed Emacs mode I produced such an awasome stream of Polish and English swear-words that you could write a research paper about bilingual cursing based on it.


And amazingly, even mcedit lacks Emacs keybindings these days. How is it even possible that even the supposedly most noob-friendly operating system OSX has standard Emacs keybindings, but Linux doesn't even support them as an option any more? It doesn't even seem possible to set them up shortcut by shortcut.

At least they still work in konsole+bash, and in gtk-based programs if you add gtk-key-theme-name = "Emacs" to your ~/.gtkrc-2.0.

Unfortunately even that much barely works. In some contexts Control-F means forward, in some it means search. Control-W changes meaning between close tab and delete previous word - it's an unreliable mess.

They way OSX does it is simple - one modifier key (Control) is used for text editing, another (Command) for application control. I understand Linux cannot really follow this way, but would it really be that difficult to have a single global mapping between key presses and actions?

When you think about it most programs use the same shortcuts:

• text editing (control-A, control-E, ...)
• copy&paste (command-C, command-V, command-X)
• undo stack control (command-Z, command-Y)
• document/tab control - save (command-S), load (command-L), close document/tab/window (command-W), new (command-N) etc.
• switching between tabs - command-1/2/3/..., control-pageup/pagedown etc.
• application control - let's not get into application/window/sdi/mdi mess, there's time for everything and it isn't now - mostly quit (command-Q)
• window control - maximize, minimize etc.
• system control - like switching between windows/applications, making screenshots, locking screen, some dashboard, start menu, volume up/down/mute - the list is pretty long, but they're global by definition

There are very few applications that really need more than a few important shortcuts to this list. Some of them are important applications (Office, Photoshop, Emacs), but you'll be mostly using these anyway.

I would be surprised if GNOME/KDE lacked infrastructure for this (what's gtk-key-theme if not half of this?) - it just that nobody bothered doing this consistently. Or if they did, their efforts were blocked by some old-timer whose last recollections of life outside desktop Linux is Windows 3.11.

My Christmas wish: Good keybindings for Linux that won't make me miss OSX, or Linux from a decade ago.

It cannot be that hard, as Linux used to have those before losing them very recently.

Redcar

Let's continue the story. Kate and gedit were various levels of unusable due to keybindings, as was mcedit in Emacs mode which I mostly use over ssh instead of real Emacs. I could try the actual Emacs, but after a few years of TextMate it just isn't the same. Not even close.

Fortunately the totally awesome Daniel Lucraft wrote a TextMate-like editor in JRuby redcar. Every time I check it it gets more and more awesome, and it's safe to say it's already far better than any other editor you can get on Linux.

Of course I wouldn't be myself if I didn't start bitching. So here's my quick list of easy improvements to redcar.

Any of these will make a great Christmas gift for me:

• Context menu absolutely needs TextMate-like "Filter through command" features and all its options (mostly "input selection / output selection", "input selection / output create new document", and "input document / output create new document", but others are occasionally useful as well). This is easily one of top five most useful features of TextMate ever and it's really not that hard.
  
• Getting a lot of common bundles/plugins/snippets/whatever-else-it-is-using with a single command. Batteries included you know.
• Tab size control really needs to be visible by default. I'm sure this is one of the first thing requested by half of the redcar users, so why not just do it?
• Support for Control-A and Control-E, seriously. Patch on github.
• Syntax highlighting often breaks. Now making it unbreakable would require massive architectural charget, but can we at least get easy refresh shortcut, or refresh on switching tabs (if edited since last switch)? Most documents are not that huge, so it shouldn't be a major performance problem, right?
• Does it really need to create .redcar directories everywhere? It creates ~/.redcar/ already, won't that be enough?
• Unixy interface like mate command. Right now starting redcar ~/some/dir from console makes it stay there and keep producing java.lang.IllegalArgumentException stack traces. It's definitely a good thing for developing redcar, but would it be possible to get some command line switches to make redcar fork away like mate command does?
• TextMate-compatible Control-K - it should delete everything between cursor and end of line (right), or delete linebreak if cursor is already at end of line. Right now in redcar pressing Control-K multiple times doesn't do anything useful. Patch on github.
• Can we get some sort of 2D selections? Even Firefox has them for tables with control-mouse. I understand they might be too hard architecturally, but it would be pretty awesome. Ctrl+B does exactly that already.
• Control-T (search file) should ignore spaces like in TextMate. I know this behaviour is irregular, but it just makes so much more sense to me to search for "login ctr test" like in TextMate than for "loginctrtest" redcar wants. By the way history for Control-T is pretty awesome.Patch on github.
• Search and replace within selection.
• Not in TextMate but would be awesome: support for filtering by full path with Control-T, like login/index.

That was a lot of bitching - but then isn't this exactly what this blog is made for?

Yes, I'm aware of official redcar wishlist and I might convert some of this bitching to requests and maybe even - imagine that - patches (I've been doing quite a lot of JRuby lately, so I'm no longer scared of JVM). Just not now. Marginal utility of time, "it's complicated", delegate if possible and so on... You know, the usual excuses for not contributing.

EDIT: I fixed a few easy ones at github.

I'm not done yet

Silly me, I thought I would be able to include my Ruby standard library wishlish in this post - this will have to wait, there are two weeks left so I might find an evening for that as well - and that will be a ridiculously long list - and judging from reactions to my preliminary post about Ruby stdlib it will make everybody hate me even more than they hate me now.

So just a few minor additions to my Christmas wishlish, just in case robot Santa or spirit of Kwanzaa or your personification of gift giving of choice happens to be deluded enough to think I was nice enough that my wishlist so far is somehow not yet long enough:

• Is there anything like OmniFocus for Linux?
• Sound system on Linux is still broken in 2010. Skype mysteriously cannot find my microphone, even after spending far too much time trying various alsamixer and Skype options. Rebooting to Windows on same hardware - zero problems, works out of the box. Can we please get this?
• System settings are just dreadful. Why cannot we have everything in one place, instead of some options in General, some in Advanced, some inside various programs, and some who knows where. This has always been a big weakness of Linux, but total lack of progress is disturbing. Even something as simple as telling Linux to automatically mount an external disk is nontrivial - in dolphin there's nothing like it in context either on disk icon on sidebar, or disk mountpoint directory in /media. Searching "usb" or "external" in system settings also returned nothing. It turns out I need to switch to "Advanced" tab and search for "removable". I seem to recall this used to work far better than that, but maybe that was GNOME. Is anybody working to improve this situation?

By the way is there any major reason for using KDE versus GNOME these days? It used to be so that GNOME had much worse Windows-itis and hypernoobemia than KDE, but these days I'm getting the impression that noobness rankings changed from Mac > Windows > GNOME > KDE to Mac > Windows > KDE = GNOME > Mac. Yes - somehow Macs dominate both total noob and expert user part of the spectrum.

Or actually - the person who does the most to fulfil my wishlist gets to decide KDE vs GNOME for me.