Using a DDR dance mat as a musical instrument

That's an idea I had for a very long time - connect a DDR dance mat to some music generation software so that people would control the music by dancing. Step first is getting data from a dance mat. Most dance mats these days are USB HID devices. Their data can be read from /dev/input/event<number> (generic HID interface) and /dev/input/js<number> (joystick interface).

I wanted to get some clue what the data might look like, so I catted a sample of it to sample file and then reverse engineered the format. I guess there's documentation somewhere, it's just often faster to look at the output and guess what it's supposed to mean.

key = Hash.new{|ht,k| k}.merge({
292 => "up",
288 => "down",
289 => "right",
291 => "left",
290 => "up-right",
293 => "up-left",
295 => "down-right",
294 => "down-left",
297 => "select",
296 => "start",
})

data = File.read("sample")
samples = data.size/16
samples.times{|i|
  sample = data[16*i, 16]
  sec, ms, type, code, value = sample.unpack("VVvvV")

  tm = Time.at(sec).strftime("%H:%M:%S")
  ms = sprintf("%06d", ms)
  # Types:
  # * 0 - SYN
  # * 1 - KEY
  # * 2 - REL (relative position)
  # * 3 - ABS (absolute position)

  next unless type == 1
  if value == 0
    dt = "PRESS #{key[code]}"
  elsif value == 1
    dt = "RELEASE #{key[code]}"
  else
    dt = sprintf("UNKNOWN #{type} #{code} #{value}")
  end
  print "#{tm}.#{ms} - #{dt}\n"
}

So the dance mat driver gives us nicely timestamped key presses and releases. No "simultaneous keypress" events for jumping on two arrows at once, so they would have to be emulated in software. That's about it as far as input is concerned. Now the output.

My first idea was looking for some sort of Ruby (or Perl or whatever) MIDI library. Unfortunately I could find nothing useful. After a long search, I finally got to ChucK, the "Strongly-timed, Concurrent, and On-the-fly
Audio Programming Language". Unfortunately Ubuntu package sucks - it doesn't tell you that you need to install jackd (in Recommended or Suggested or documentation), nor is it in a manual or FAQ or wherever I looked. I found that out by strace and guesswork. So to run ChucK in Ubuntu you need to sudo apt-get install chuck jackd.

Here's the highly uninformative error message you get when you run ChucK without a running jackd:

 $ chuck something.ck
[chuck]: (via rtaudio): no devices found for compiled audio APIs!
[chuck]: cannot initialize audio device (try using --silent/-s)

So open a terminal and start jackd:
$ jackd -d alsa

ChucK still won't work:

 $ chuck something.ck
[chuck]: (via rtaudio): no devices found for given stream parameters:
... RtApiJack: the requested sample rate (44100) is different than the JACK server rate (48000).

[chuck]: cannot initialize audio device (try using --silent/-s)

You need to tell ChucK to use sampling rate of 48000 (telling jackd to use sampling rate of 44100 somehow didn't work). The manual says:

--srateN  Set sampling rate (default to 48000 for jack, auto detected otherwise).

which is doubly untrue, as it used 44100 with jack, and apparently didn't do any autodetection.

After overcoming all the obstacles, finally we can get some sounds.

$ cd /usr/share/doc/chuck/examples/stk/
$ chuck --srate48000 clarinet2.ck
---
reed stiffness: 67.172840
noise gain: 29.734127
vibrato freq: 75.111492
vibrato gain: 72.994233
breath pressure: 121.968367
---
reed stiffness: 120.611360
noise gain: 49.769301
vibrato freq: 60.092423
vibrato gain: 60.048969
breath pressure: 126.965529
^C
[chuck]: cleaning up...

I strongly recommend at this point to try musical version of Towers of Hanoi:

$ cd /usr/share/doc/chuck/examples/
$ chuck --srate48000 hanoi++.ck
move disk from peg 1  ->  peg 2
move disk from peg 1  ->  peg 3
move disk from peg 2  ->  peg 3
move disk from peg 1  ->  peg 2
move disk from peg 3  ->  peg 1
move disk from peg 3  ->  peg 2
move disk from peg 1  ->  peg 2
move disk from peg 1  ->  peg 3
move disk from peg 2  ->  peg 3
move disk from peg 2  ->  peg 1
move disk from peg 3  ->  peg 1
...

ChucK has C-like syntax and should be reasonably simple to understand. It already has HID examples in /usr/share/doc/chuck/examples/hid/, including some for joystick. After some experimentation, I was able to get them to make DDR dance mat to be a percussion instrument.
Button numbers may be different for your dance mat, but in mine up-left, left, and down-left control three kinds of beats in the left audio channel, and up-right, right, and down-right control the same kinds of beats in the right audio channel. Samples are copied from /usr/share/doc/chuck/examples/data/

// make HidIn and HidMsg
HidIn hi;
HidMsg msg;

// open joystick 0, exit on fail
if( !hi.openJoystick( 0 ) ) me.exit();

<<< "joystick ready", "" >>>;

SndBuf key_snd[8];

// load files
// Left
"data/snare-chili.wav" => key_snd[5].read;
"data/kick.wav" => key_snd[3].read;
"data/snare-hop.wav" => key_snd[6].read;

// Right
"data/snare-chili.wav" => key_snd[2].read;
"data/kick.wav" => key_snd[1].read;
"data/snare-hop.wav" => key_snd[7].read;

key_snd[5] => dac.left;
key_snd[3] => dac.left;
key_snd[6] => dac.left;

key_snd[2] => dac.right;
key_snd[1] => dac.right;
key_snd[7] => dac.right;

// infinite event loop
while( true )
{
  // wait on HidIn as event
  hi => now;

  // messages received
  while( hi.recv( msg ) )
  {
    if(msg.type == 1)
      <<< "Press", msg.which >>>;
    else if(msg.type == 2)
      <<< "Release", msg.which >>>;
    if(msg.type == 1)
    {
      int key;

      msg.which => key;
      if(key == 1 || key == 2 || key == 3 || key == 5 || key == 6 || key == 7)
      {
        0 => key_snd[key].pos;
        // gain
        Math.rand2f( 2.0, 6.0 ) => key_snd[key].gain;
      }
    }
  }
}

Some explaining. It uses joystick-specific interface (/dev/input/js<number>) not generic HID interface (/dev/input/events<number>), so data format and key numbers are a bit different.

source => dest is a connection operator. dac is a sound card, so obviously dac.left and dac.right are its left and right channels. Code like "data/snare-chili.wav" => key_snd[5].read; reads sound files to a buffer, and key_snd[5] => dac.left; plays them (so you get a loud beat when you stard the script). 0 => key_snd[key].pos; resets the position to 0, in effect playing a given beat again. Math.rand2f( 2.0, 6.0 ) => key_snd[key].gain; sets gain (volume) of a particular beat to random number between 2 and 6. <<< ... >>> means print.

The script is just a proof of concept, so I'm not going to upload an mp3 (or a YouTube movie). I think it should be possible to generate some really interesting music by dancing, if the script became smarter. I'd like to hear from anyone who tried to code such a thing.

Problems I had with ChucK were reported to Ubuntu: bug #114161, bug #114162