taw's blog

The best kittens, technology, and video games blog in the world.

Thursday, April 21, 2016

Dealing with transient test failures due to database results order

Fox by kindl_jiri from flickr (CC-ND)
The most #mildlyinfuriating aspect of testing are tests which work most of the time, but fail occasionally. Even worse are tests which always work when you run them individually, but sometimes fail when ran as part of a test suite.

The most common category of transient test errors I've seen are Capybara Javascript testing, and I don't have a great solution for those, but there's another category of really obnoxious tests - tests which expect results from database to be in specific order.

They usually work, as databases do the laziest thing possible, so when you ask one for a bunch of records without specifying any particular ordering, it will usually return them in whichever order they are physically on the disk, which usually corresponds to order of their creation, which then usually corresponds to their serial or GUID primary key, so you get that implicit ORDER BY id, most of the time.

Which is just fine, except once in a while the database will reorder physical records to compact tables, causing physical order of record to no longer correspond to primary key order, and test to fail.

Then you rerun it individually over and over, and it works every time, as this kind of reordering is not going to happen mid-test, only once enough repeatedly created and deleted data accumulated in the table. Like once every 20 full test runs, each half an hour long. Very frustrating to debug.

What if you could force such tests to reveal themselves somehow? Most databases won't be of much help, and ORM is standing in the way of adding ORDER BY to every query which doesn't have one, but fortunately it's not too difficult to tell ActiveRecord to shuffle results of everything that didn't request specific order, by placing this kind of code in your spec/spec_helper.rb or equivalent:

ObjectSpace.each_object(Class) do |cls|
  if cls.ancestors[1..-1].include?(ActiveRecord::Base)
      cls.instance_eval do
        default_scope { order('rand()') }
      warn "Can't order #{cls}"

I wouldn't recommend running it like that every time, and especially not in production, as ORDER BY RAND() everywhere is going to have annoying performance impact, but enabling it temporarily just to debug already existing transient test failures might be just the right tool.

Wednesday, April 20, 2016

Patterns for testing command line scripts

Lab Mouse checkin out the camera by Rick Eh? from flickr (CC-NC-ND)
It's relatively easy to test code which lives in the same process as test suite, but a lot of the time you'll be writing standalone scripts, and it's a bit more complicated to test those. Let's talk about some patterns for testing them.

Examples in RSpec, but none of these patterns depend on test framework.

Manual Testing Only

That's actually perfectly legitimate. If your script is a few lines of straightforward code, you can just check that it works manually a few times, and then completely forget about it. Usefulness of automated tests in such case probably won't be very high.

I'd recommend not relying on just that for more complicated scripts.

STDOUT testing

A lot of scripts take arguments from command line or STDIN, and output results to STDOUT, possibly STDERR or exit code as well.

A bunch of expect(`script --input`).to eq("output\n") style tests are very easy to use and can go very far.

If you need to test a bit more complicated interactions - setting environmental variables, writing to STDIN, reading from both STDOUT and STDERR, checking error code etc. - IO.popen and Open3 module offer reasonably convenient APIs.

Of course only certain category of scripts can be reasonably tested this way, but it's a fairly big category.

Testing as library code

A fairly common pattern is to move most of the code from "script" file to a separate "library" file, which can be required by both. It's a bit awkward, as script no longer lives in one file.

It's not always obvious where to divide the library from the script - if you put everything in the library, it makes it pretty much useless for anything except program itself. If you keep things like parsing command line arguments separate, that results in possibly useful "library", but leaves more "script" code untested.

if __FILE__ == $0

It used to be a very common pattern which I don't see that often these days. What if we have a file which works as a library you can require, but it acts as a script if it's ran directly? Here's the typical code for such script:

class Script
  def initialize(*args)
  def run!

if __FILE__ == $0

Depending on how you feel you might do command line argument parsing either in initializer, or in if __FILE__ == $0 block.

Code written in this style generally doesn't intend to be used as a library, and this hook is there primarily for sake of testing.

Temporary directory

Frequently scripts interact with files. That's more complicated to setup. Don't try anything silly like using current directory or single tmp where leftovers from previous test runs might be left.

I'd recommend creating new temporary directory and going there. Add code like this to your test helpers:

def Pathname.in_temporary_directory(*args)
  Dir.mktmpdir(*args) do |dir|
    Dir.chdir(dir) do

Then you can then use Pathname.in_temporary_directory do |dir| ... end in your tests, and it will handle switching back to previous directory and removing temporary one automatically.

In every such block you can write files you want, run command, and check any generated files, without worrying about contaminating filesystem anywhere.

There's just a minor complication here - you'll be changing your working directory, so you'll need to call your script using absolute rather than relative path. Simply do something like:

let(:script) { Pathname(__dir__) + "../bin/script"  }

To get absolute path to your script and then use that.

Mocking network

All that covers most of possible scripts, but I recently figured out one really fun trick - how to test scripts which read from network?

Within our tests we have gems like webmock and vcr can fake network communication, but what if we want to run a script? Well, just save this file as mock_network.rb:

require "webmock"
require "vcr"

VCR.configure do |config|
  config.cassette_library_dir = Pathname(__dir__) + "vcr"
  config.hook_into :webmock

VCR.insert_cassette('network', :record => ENV["RECORD"] ? :new_episodes : :none)

END { VCR.eject_cassette }

And then run your script as system "ruby -r#{__dir__}/mock_network #{script} #{arguments}", possibly in conjunction with any other of the techniques presented here.

To record network traffic you can run your tests with RECORD=1 rspec, then once you're finished just run rspec normally and it will use recorded requests.

Mocking other programs

Previous pattern assumed the script was using some Ruby library like net/http or open-uri for network requests. But it's very common to use a program like curl or wget instead.

In such case:
  • write your mock curl, doing whatever you'd like it to do for such test
  • within test, change ENV["PATH"] to point to directory containing your mock curl as first element
  • run script under test
This works reasonably well, as almost all programs call each other via ENV["PATH"] search, not by absolute paths, and usually expect fairly simple interactions.

Like all heavy handed mocking, this can fail miserably if the program decides to pass slightly different options to curl etc., and unlike webmock this style of interaction doesn't block network access so you can miss something.

All these patterns leak

None of these pattern are perfect - they assume how script is going to interact, and they don't actually isolate script from network, filesystem (outside temporary directory you created), Unix utilities etc., so a buggy script can still rm -rf your home directory.

For testing very complicated interactions, you might need to use virtual machine, or some OS-specific isolation mechanism like chroot. Fortunately only relatively few scripts really need such techniques.

Tuesday, April 19, 2016

Automatically managing db/schema.rb

peering thru the bed rails watching coco play by damselfly58 from flickr (CC-NC-ND)
I've been living in schemaless NoSQL wonderland for quite a while, but I'm currently working on some mysql-based Rails applications, and managing db/schema.rb is a massive pain.

It's an automatically generated file, and such conventionally don't go into version control, but it also literally says "It's strongly recommended that you check this file into your version control system" in it. I'm still not completely convinced it's the right thing to do, but let's assume we follow the recommendations.

The problem is that you're probably not starting a fresh database for every branch - or carefully rolling back to master schema before you switch - you'll be switching branches a lot, and migrations will be applied in a different order than what they'd end up on master - so whenever you generate db/schema.rb you'll need to look at it manually to figure out what should be committed and what shouldn't. It's a very error prone process for something as frequent as writing migrations.

You could drop your database and recreate it from migrations every now and then, but you probably have some data you'd rather keep there.

Fortunately there's a solution! Oh, you can't just switch your team to a schemaless database? Well, in such case use this script:

Script regenerate_schema_rb:

#!/usr/bin/env ruby

require "fileutils"
require "pathname"

fake_database_yml = Pathname(__dir__) + "database.yml"
real_database_yml = Pathname("config/database.yml")

unless real_database_yml.exist?
  STDERR.puts "It doesn't seem like you're in a Rails application"
  exit 1

unless `git status` =~ /nothing to commit, working directory clean/
  STDERR.puts "Do not run this script unless git status says clean"
  exit 1

system "echo 'DROP DATABASE IF EXISTS schema_db_regeneration' | mysql -uroot"
system "echo 'CREATE DATABASE schema_db_regeneration' | mysql -uroot"

FileUtils.cp fake_database_yml, real_database_yml

system "rake db:migrate"
system "git checkout #{real_database_yml}"

Fake database.yml:

  adapter: mysql2
  encoding: utf8
  database: schema_db_regeneration
  pool: 5
  username: root

What it does is very straightforward - it keeps your existing database, and simply repoints database.yml at a fake one, and runs all the migrations against it. No manual edits necessary, and your next_tumblr_development database is safe.

Sunday, February 28, 2016

Let's Play Civilization 5 on Tamriel as the Khajiit - post-campaign retrospective

Biting the brother in the neck! by Tambako the Jaguar from flickr (CC-ND)

Unfortunately due to changing circumstances my Youtube channel is probably going to be somewhat less active for at least first half of the year.

My most recent and so far the only series on it was Let's Play Civilization 5 on Tamriel as the Khajiit, which is now slowly coming to a close, so it's time for some retrospective.

Balance between civilizations

I've chosen civilization based on a quick look at who seemed to have interesting gameplay potential - and also because I haven't played as cats before. I thought other civilizations are gonig to have comparable power level, but it's not ever close, and Khajiit are insanely overpowered. Here's what they do:
  • caravan range doubled
  • land military units (but not workers, settlers, great generals etc.) +1 movement and +1 sight
  • workshop replacement with +2 happiness, +1 culture, +2 science/jungle (stacking with University for +4 science/jungle), on top of the usual bonuses
  • caravansary replacement with +2 happiness, +1 culture, double trade range bonus, and the usual gold bonus
In other words:
  • ridiculously powerful happiness bonus (+4 per city), all available fairly early in game, making happiness maybe not quite irrelevant, but a secondary consideration even for highly expansionistic empires
  • ridiculously powerful military bonus (+1 movement and +1 sight is stronger than most UUs - and Khajiit get it for every land unit in every era.
    • To make it even more extreme, Tamriel map is almost all land, so lack of bonuses for ships matters little - and extra movement and sight is still quite effective at shooting ships with your missile units
    • Early game the Bosmer whom I was fighting built Great Wall - possibly the most obnoxious wonder in the game - that extra movement point more or less invalidated their wonder
  • fairly strong science bonus - requiring access to a lot of judge would make it somewhat situational, but guess what - that's precisely the kind of terrain Khajiit start with in every direction
  • fairly strong gold bonus - it's not that huge, mostly it lets you use caravans where others would use cargo ships, which on some maps would be fairly inconsequential, but Tamriel is basically Pangea, so it's especially strong on this map
  • modest culture bonus - +2 culture / city is very good value
  • and all of that requires very little effort - you start with military bonus and you'd be building workshops everywhere anyway - so now you just need to build caravansaries in places you wouldn't otherwise
This is all "way above Korea and Poland" tier.

It would be all right if everybody had comparable bonuses, but as far as I can tell, everybody else's are rather mediocre.

Back when I was playing Ravnica civs, half of them were really overpowered, but they matched each other reasonably well, and I don't think any of them were Khajiit tier.

This unfortunately means I'm not so inclined to give Tamriel another try. Other civs seem not only underpowered, but also rather boring.

Truce Breaking Bug

Civilization 5 won't let you break truces no matter how much you'd like to do so. However I ran into this bug:
  • I wanted to attack Orsimer, but they were a bit strong, so I asked Skyrim to attack them together
  • Skyrim said yes, but give us 10 turns
  • Well, I couldn't wait, so I attacked Orsimer on my own
  • They offered far less resistance than expected, so I took what I wanted and peaced them out
  • Next turn Skyrim came back to me with "10 turns is up, let's attack Orsimer together" dialog, and war started.
  • As far as I can tell, there were no negative diplomatic consequences of this truce breaking, because game didn't even consider truce breaking to be a possibility

Playing with 13 civilizations

The game had 13 civilizations on Standard sized maps (as opposed to the usual 8), so I expected a bloodbath fighting for what little land there was available. Three civilizations (Bretons, Orsimer, Hammerfell) had their capital so close I'd expect them to DoW each other before getting Composite Bowmen.

It turns out that number of tiles is Standard sized, but amount of land is much bigger, so there was plenty available for everyone - and map had a lot of strategically places mountains to slow down early warfare as well.

Unfortunately 13 civs on Immortal meant it was completely pointless to go after wonders, religions, or domination victory, while making research agreements and science victory a lot stronger.

I think I had 8 research agreements at one point, with +50% science from research agreement thanks to Porcelain Tower stacked on top of +25% from relevant Rationalism policy. That in addition to all my science bonuses, funded by my gold bonuses.

I think 13 civs is generally too many, and it would be better to play with a smaller subset.

Minor mods

I'm increasingly thinking that I should uninstall all promotions mods and just play with vanilla promotions. There's really very little value in it, and it replaces one optimal formula (accuracy/barrage 1-2-3, logistics, range) with another (unlock 1-3, range, unlock 4, logistics).

I'm less convinced that I should be playing with captureable settlers, as that makes things a bit easy with AI stupidity. I still enjoy capturing other great people a lot, so I'd need to customize that.

Other issues

Science victory is a bit boring, so I spent far too much time fighting completely pointless wars late game. I could have finished the campaign a lot faster if I just ignored everybody else and built space ship.

I only just realized I misspelled "Khajiit" as "Khajit". Oh well.

How to configure OSX 10.11 El Captain for software development

Cats by sokole oko from flickr (CC-NC-ND)

With every new Macbook, I'm updating the guide, previous version is here. A few thing changed - mostly some links which used to work have been taken over by malware. OSX is turning into Windows more and more.


  • Install some sensible browser like Chrome or Firefox.
  • Afterwards either sign up into your account on which you hopefully have your AdBlock setup, or install some. Most popular seems to be uBlock Origin these days, but pretty much any of them will do just fine.
  • Install whichever cloud sync service you're using like Dropbox etc. And start syncing your stuff.
  • Install iTerm2 for sensible terminal emulator.
  • Clean up all crap from dock. Other than Launchpad and System Settings, everything else should be gone. Add iTerm2, your browser, and your text editor, and any application you wish to install there instead of stock Apple crap.
  • It's also a good idea to disable Spotlight as soon as possible by running sudo mdutil -i off / - before it tries to index all of your dropbox and generally ruin performance of your machine.


Install some sensible text editor like Sublime Text (requires license key).

Whichever editor you choose, you'll definitely need to configure it to your liking.

Then symlink it so you can use it from command line

  mkdir -p ~/bin
 ln -s "/Applications/Sublime Text.app/Contents/SharedSupport/bin/subl" ~/bin/subl


Default Mac settings are totally awful, time to fix that.
  • Max brightness
  • Settings > Security > Allow apps downloaded from: > Anywhere
  • Settings > Mouse > Max out "Fast" setting on everything
  • Settings > Keyboard > Key Repeat > Fast
  • Settings > Keyboard > Delay Until Repeat > Short
  • Settings > Keyboard > Use all F1, F2, etc. keys as standard function key
  • Settings > Keyboard > Text > Disable "Correct spelling automatically"
  • Settings > Sound > Disable "Play user interface sound effects"
  • Settings > Sound > Alert volume > 0% (for Terminal ping)
  • Settings > Trackpad > Scroll & Zoom > Disable "Scroll direction: natural"
  • Settings > Energy Saver > Power Adapter > Display sleep > Never
  • Settings > Displays > Built-in Retina Display > Disable "Automatically adjust brightness"
  • Settings > Displays > Arrangement > drag and drop your external monitors into desired order
  • Settings > Dock > enable "Automatically hide and show the Dock"
  • Menu bar > Battery icon in task bar > Enable "Show Percentage"
  • iTerm > Preferences... > Profiles > Terminal > Unlimited Scrollback
Press Cmd-Up arrow, add a few desktops (or "spaces" as they were used to know), then go to Settings > Keyboard > Shortcuts > Mission Control - and enable their keyboard shortcuts Ctrl-1 to Ctrl-6 or however many you have there.


OSX already includes drivers for laptop itself, but you might need some for peripheral hardware.

If you need any keyboard drivers like for Microsoft Keyboard (otherwise Cmd key is in the wrong place) or just about any external keyboard, get necessary drivers.

If you need any special keyboard layouts, get them too.

Standard paths

OSX renames a lot of directories. While in theory scripts could just use env vars to find proper paths, it's more reliable to symlink all the things:

  sudo ln -s /Volumes /mnt
  sudo ln -s /Volumes /media
  sudo mv /home /home-old
  sudo ln -s /Users /home

Development tools

First, you'll need Xcode. Run xcode-select --install from command line to install it.

Now it's time for a package manager. They're all somewhat disappointing if you're used to apt-get. homebrew seems somewhat more popular than others these days, so you might just as well try that.

You'll also need X11 server like XQuartz.

Create new SSH key pair

Before you do that, name your computer something memorable with sudo scutil --set HostName your_host_name command.

Open Terminal and run ssh-keygen to create ~/.ssh/id_rsa, then upload the generated key to any place that needs to know about it like github, bitbucket, or whatever else you use.

Checkout your dotfiles

Hopefully you're storing your dotfiles somewhere. If it's a git repository, or your Dropbox account, get them now and symlink them all properly.

If there are any other repositories you might need, checkout them too.

Install homebrew packages

Your list might vary. Here's mine (fun story - order you install homebrew packages matters, every packaging system that's not apt-get sucks so hard):

brew install mongodb mysql postgresql rbenv ruby-build wget htop unrar mc mplayer coreutils libxml2 libxslt bash poppler redis qt youtube-dl trash rabbitmq pcre exiftool lame id3v2 sox jq git bash-completion p7zip imagemagick

Then enable all services you installed, unless you want to start them manually:

  ln -sfv /usr/local/opt/*/*.plist ~/Library/LaunchAgents/

And install non-system ruby, so you can install gems without sudo:

  rbenv install 2.4.0-dev
  rbenv global 2.4.0-dev

To make that actually work, you need to make sure ~/.rbenv/shims is in your $PATH.

Due to OSX limitations you'll need to run sudo htop if you want to use htop.

Sane bash and coreutils

bash version shipped with OSX is ancient and BSD utilities are all awful. In previous steps you installed proper versions, now you need to tell the system to use it.

Add homebrew version of bash as allowed shell by appending /usr/local/bin/bash at end of /etc/shells

Then set it as your shell: chsh -s /usr/local/bin/bash $USER

Then make sure to add GNU coreutils to your PATH:

  export PATH="/usr/local/opt/coreutils/libexec/gnubin:$PATH"
  export MANPATH="/usr/local/opt/coreutils/libexec/gnuman:$MANPATH"

You'll also probably want to touch ~/.hushlogin to prevent some worthless spam on every open terminal tab.

Install gems

Again, your list my vary. Here's mine:

gem install beeminder moneta octokit term-ansicolor pry-plus rak objectiveflickr hpricot color nokogiri bundler

All other software

Sadly OSX window manager is extremely dubious for keyboard use. Fortunately programs to make it usable exist. I recommend installing these two:
  • ShiftIt - for tiling by keyboard shortcuts.
  • HyperSwitch - for sane alt-tab window switching. (there's also Witch, but it has issues with 10.10)
You'll need to give them necessary access. To do so:
  • Settings > Security & Privacy > Privacy > Allow the apps below to control your computer > enable them both
You'll probably need these or similar programs:
Once upon a time Open Source Mac website contained links to a lot of useful software, but these days targets seem to be malware infested crap a lot. Don't go there.


Once you go through this list, and successfully get everything going, I'd recommend modifying it to your liking and reposting your version on your blog. Everybody's needs are different, so guide like this is just a starting point.

Saturday, February 13, 2016

Adventures with Raspberry Pi: RGB Led

I've done regular LEDs before, so I wanted to try RGB Led now.

RGB Led is like red, green, and blue RED in one package, sharing common ground (so 4 pins total), supposedly allowing any color.

Now your first idea might be to use analog output and set diode intensity to control red, blue, and green, and that would work with a regular lamp, more or less, but diodes are really just on or off, so we need to do pulse width modulation.

Unfortunately it turns out that Raspberry Pi, ruby and pi_piper gem are not fast enough for that. This code really ought to work:

class RGBLed
  def initialize
    @blue  = PiPiper::Pin.new(:pin => 17, :direction => :out)
    @red   = PiPiper::Pin.new(:pin => 22, :direction => :out)
    @green = PiPiper::Pin.new(:pin => 27, :direction => :out)

  def display(r,g,b)
    while true
      @red.send(if rand < r then :on else :off end)
      @green.send(if rand < g then :on else :off end)
      @blue.send(if rand < b then :on else :off end)

  def on

  def off

led = RGBLed.new

With iterations being done at a bit oven 1300 iterations per second, it should provide half intensity white light. Instead it's flashing all the colors randomly, so presumably something downstream from ruby is being really slow.

There's also minor problem of green LED component being much brighter than others (all use 220 hm resistors, and changing I/O ports or resistors around doesn't change that).

All that is a shame, as software PWM is a pretty useful building block for a lot of things, and if that doesn't work, I won't be able to do a ton of other thing. If anybody has any ideas, please tell me.

Fun and Balance mod for EU4 1.15.1

The Fox - Tranmautritam's Puppy by tranmautritam from flickr (CC-BY)

Fun and Balance is available updated for 1.15.1:

Thursday, February 11, 2016

Modern Times mod for Crusader Kings 2 - Conclave release

The Devil Himself by inspiring! from flickr (CC-NC) Modern Times mod is updated for 2.5.x and Conclave DLC. (Steam WorkshopDirect download).

Changes are mostly just compatibility, but also include:

  • taking advantage of new character selection screen
  • Kazakh and Uzbek cultures, to get Eastern Europe / Central Asia a few more steps towards sanity (it's not there yet)
  • if you're playing with Conclave enabled, administration and status of women laws will be automatically determined based on your religion (Muslim or not) and starting technology. I could probably tweak it a lot more, and maybe even add some new interesting laws.
  • bug fixes for de jure map and holy orders
All my CK2 minimods on Steam Workshop got also upgraded to 2.5.x.

Unfortunately it seems that Conclave doesn't allow naming kings "presidents", "fuhrers", "ayatollahs" etc. based on laws as I hoped it would. It only allows changing what laws tab describes the realms as (like "Hereditary Despotic Kingdom" and such), but that's of fairly little use. Maybe next time with China DLC.

Saturday, January 23, 2016

Let's Play Civilization 5 on Tamriel as the Khajit

It's new year and it's time to play some video games after six weeks break with nearly none.

For this campaign I grabbed Tamriel civilization pack and Tamriel map pack, with every civilization starting in its historical part of Tamriel, and unusually high number of 13 civilizations on standard-sized map which normally holds just 8. On the other hand the map has relatively little water, so number of land tiles is probably high enough to support 13.

It will still be harder to win with so many competitors.

Civilizations fighting over control of Tamriel are:
  • Cyrus (Hammerfell / Redguards)
  • Gortwog gro-Nagorm (Orsinium / Orsimer)
  • Haymon Camoran (Valenwood / Bosmer)
  • High King Emeric (High Rock / Bretons)
  • Hlaalu Helseth (Morrowind / Dunmer)
  • Keirgo (Elsweyr / Khajit)
  • King Dumac (Dwemereth / Dwemer)
  • Mehrunes Dagon (Oblivion)
  • Queen Ayrenn (Summerset Isles / Altmer)
  • The An-Xileel (Argonia / Argonians)
  • Tiber Septim (Cyrodiil / Imperials)
  • Umaril the Unfeathered (The Ayleids)
  • Ysgramor (Skyrim / Nords)
They're generally all fairly strong, even if somewhat less so than Ravnica civilizations.

Here's episode one. Episodes will be released one a day at same time on this playlist:

As Khajit we're getting double land caravan range, +1 movement and +1 sight to all military land units (not great generals, workers etc.), and two special buildings - caravansary replacement which lets us be trade-oriented civ, and workshop replacement, which allegedly produces some skooma.

Full list of mods used:

Thursday, January 21, 2016

Review of The Manga Guide to Biochemistry by Masaharu Takemura

Following highly entertaining The Manga Guide to Statistics, I decided to grab another book in the loose series - The Manga Guide to Biochemistry by Masaharu Takemura.

The book is taking its role as a textbook somewhat more seriously, and is packed with higher ratio of information to manga than the Statistics guide. They obviously had a lot to say, and I'd say they covered it fairly well without any obvious mistakes or oversights. A good amount of it feels poorly integrated into the plot, especially bits towards the end which feel almost like a disconnected appendix.

The plot puts a lot of effort on putting all that information in context, but it still manages to setup some manga romance action - pretty lady professor of biotechnology tries to manipulate high school girl and university student boy tutoring her into realizing their feelings for each other. Oops, spoilers.

Overall, I'd say it does its job pretty well, not just as a novelty item, but arguably even as a legitimate textbook.

tl;dr 4/5 stars