There's something I don't know about software development, and I'd like to know. Everybody believes Test-Driven Development is the right way to build software, and it's very easy to develop libraries this way, and still reasonably easy to develop applications, but how does one even test glue code ?
The software which got me thinking about this is my iPod-last.fm bridge. It extracts song statistics from an iPod and submits them to last.fm.
It's a bit over 200 lines of code, and it had a few bugs:
- The magic constant to convert time was off by an hour. My iPod had DST settings wrong, and I miscalculated the constant so the result was right for my iPod.
- It assumed record size was 16-bytes instead of looking at the headers. Older iPods had 12-byte records.
- It expected upper-case MD5 in response from last.fm. last.fm switched to sending lower-case MD5s later.
- Documentation didn't state that Ruby version 1.8.3 or newer is required, and it won't work with Ruby 1.8.2.
The script is very simple glue code, and it glued things on which it was tested perfectly - Ruby 1.8.4, an iPod with 16-byte records and broken DST settings, whichever version of software last.fm used when I coded the script.
I don't see how it would be possible to test such script. Building simple mock iPod or mock last.fm server wouldn't find any of the bugs, and complex mocks would be far more complex than the script itself, and most likely would contain even more bugs.
I didn't think about testing it with Ruby 1.8.2, as from Linux (crontabbed apt-get upgrade) perspective this version was ancient, it requires effort to install multiple minor versions of Ruby on Ubuntu, and I didn't expect any problems there. After discovering a few more minor incompatibilities, I automatically run unit tests for all my Ruby code on 1.8.4, 1.8.5, 1.8.6, and 1.9. I don't think I need to support 1.8.2 or 1.8.3, but I'd like to be able to know what is supported and what isn't.
I somehow don't see code review finding any of these problems. If I was doing code review, I think I'd be unlikely to question 2082848400 time conversion magic, 16-byte record size, upper case assumption in MD5 rensponse format regexp, or to ask if it would work with very old versions of Ruby would.
Not a single one of this bug could be found by static typing, no matter how strict, or by avoiding side effects.
Releasing the code found 3 of 4 bugs, and they were fixed in a few minutes at most each. Nobody cared enough about time being off by 1 hour to report it, and I found it on my own while playing with iPod settings.
I think it's a typical case for glue code. Very simple code works with some complex systems, and makes assumptions about them, some of which may later prove unwarranted. Testing, code review, static typing and other standard recipes for reducing bug counts don't seem to be working. Most likely they would simply greatly increase amount of code and work. Releasing the code in the wild seems to be very effective (but it failed to find the time bug).
Is it really the best solution we know ?