The best kittens, technology, and video games blog in the world.

Saturday, May 12, 2007

Using a DDR dance mat as a musical instrument

The Other Side... by Tim Morgan from flickr (CC-NC-SA)That's an idea I had for a very long time - connect a DDR dance mat to some music generation software so that people would control the music by dancing. Step first is getting data from a dance mat. Most dance mats these days are USB HID devices. Their data can be read from /dev/input/event<number> (generic HID interface) and /dev/input/js<number> (joystick interface).

I wanted to get some clue what the data might look like, so I catted a sample of it to sample file and then reverse engineered the format. I guess there's documentation somewhere, it's just often faster to look at the output and guess what it's supposed to mean.

key = Hash.new{|ht,k| k}.merge({
292 => "up",
288 => "down",
289 => "right",
291 => "left",
290 => "up-right",
293 => "up-left",
295 => "down-right",
294 => "down-left",
297 => "select",
296 => "start",
})

data = File.read("sample")
samples = data.size/16
samples.times{|i|
sample = data[16*i, 16]
sec, ms, type, code, value = sample.unpack("VVvvV")

tm = Time.at(sec).strftime("%H:%M:%S")
ms = sprintf("%06d", ms)
# Types:
# * 0 - SYN
# * 1 - KEY
# * 2 - REL (relative position)
# * 3 - ABS (absolute position)

next unless type == 1
if value == 0
dt = "PRESS #{key[code]}"
elsif value == 1
dt = "RELEASE #{key[code]}"
else
dt = sprintf("UNKNOWN #{type} #{code} #{value}")
end
print "#{tm}.#{ms} - #{dt}\n"
}
So the dance mat driver gives us nicely timestamped key presses and releases. No "simultaneous keypress" events for jumping on two arrows at once, so they would have to be emulated in software. That's about it as far as input is concerned. Now the output.

My first idea was looking for some sort of Ruby (or Perl or whatever) MIDI library. Unfortunately I could find nothing useful. After a long search, I finally got to ChucK, the "Strongly-timed, Concurrent, and On-the-fly
Audio Programming Language". Unfortunately Ubuntu package sucks - it doesn't tell you that you need to install jackd (in Recommended or Suggested or documentation), nor is it in a manual or FAQ or wherever I looked. I found that out by strace and guesswork. So to run ChucK in Ubuntu you need to sudo apt-get install chuck jackd.

Here's the highly uninformative error message you get when you run ChucK without a running jackd:
 $ chuck something.ck
[chuck]: (via rtaudio): no devices found for compiled audio APIs!
[chuck]: cannot initialize audio device (try using --silent/-s)


So open a terminal and start jackd:
$ jackd -d alsa

ChucK still won't work:
 $ chuck something.ck
[chuck]: (via rtaudio): no devices found for given stream parameters:
... RtApiJack: the requested sample rate (44100) is different than the JACK server rate (48000).

[chuck]: cannot initialize audio device (try using --silent/-s)

You need to tell ChucK to use sampling rate of 48000 (telling jackd to use sampling rate of 44100 somehow didn't work). The manual says:
--srateN  Set sampling rate (default to 48000 for jack, auto detected otherwise).

which is doubly untrue, as it used 44100 with jack, and apparently didn't do any autodetection.

After overcoming all the obstacles, finally we can get some sounds.
$ cd /usr/share/doc/chuck/examples/stk/
$ chuck --srate48000 clarinet2.ck
---
reed stiffness: 67.172840
noise gain: 29.734127
vibrato freq: 75.111492
vibrato gain: 72.994233
breath pressure: 121.968367
---
reed stiffness: 120.611360
noise gain: 49.769301
vibrato freq: 60.092423
vibrato gain: 60.048969
breath pressure: 126.965529
^C
[chuck]: cleaning up...


I strongly recommend at this point to try musical version of Towers of Hanoi:
$ cd /usr/share/doc/chuck/examples/
$ chuck --srate48000 hanoi++.ck
move disk from peg 1 -> peg 2
move disk from peg 1 -> peg 3
move disk from peg 2 -> peg 3
move disk from peg 1 -> peg 2
move disk from peg 3 -> peg 1
move disk from peg 3 -> peg 2
move disk from peg 1 -> peg 2
move disk from peg 1 -> peg 3
move disk from peg 2 -> peg 3
move disk from peg 2 -> peg 1
move disk from peg 3 -> peg 1
...

ChucK has C-like syntax and should be reasonably simple to understand. It already has HID examples in /usr/share/doc/chuck/examples/hid/, including some for joystick. After some experimentation, I was able to get them to make DDR dance mat to be a percussion instrument.
Button numbers may be different for your dance mat, but in mine up-left, left, and down-left control three kinds of beats in the left audio channel, and up-right, right, and down-right control the same kinds of beats in the right audio channel. Samples are copied from /usr/share/doc/chuck/examples/data/

// make HidIn and HidMsg
HidIn hi;
HidMsg msg;

// open joystick 0, exit on fail
if( !hi.openJoystick( 0 ) ) me.exit();

<<< "joystick ready", "" >>>;

SndBuf key_snd[8];

// load files
// Left
"data/snare-chili.wav" => key_snd[5].read;
"data/kick.wav" => key_snd[3].read;
"data/snare-hop.wav" => key_snd[6].read;

// Right
"data/snare-chili.wav" => key_snd[2].read;
"data/kick.wav" => key_snd[1].read;
"data/snare-hop.wav" => key_snd[7].read;

key_snd[5] => dac.left;
key_snd[3] => dac.left;
key_snd[6] => dac.left;

key_snd[2] => dac.right;
key_snd[1] => dac.right;
key_snd[7] => dac.right;

// infinite event loop
while( true )
{
// wait on HidIn as event
hi => now;

// messages received
while( hi.recv( msg ) )
{
if(msg.type == 1)
<<< "Press", msg.which >>>;
else if(msg.type == 2)
<<< "Release", msg.which >>>;
if(msg.type == 1)
{
int key;

msg.which => key;
if(key == 1 || key == 2 || key == 3 || key == 5 || key == 6 || key == 7)
{
0 => key_snd[key].pos;
// gain
Math.rand2f( 2.0, 6.0 ) => key_snd[key].gain;
}
}
}
}
Some explaining. It uses joystick-specific interface (/dev/input/js<number>) not generic HID interface (/dev/input/events<number>), so data format and key numbers are a bit different.

source => dest is a connection operator. dac is a sound card, so obviously dac.left and dac.right are its left and right channels. Code like "data/snare-chili.wav" => key_snd[5].read; reads sound files to a buffer, and key_snd[5] => dac.left; plays them (so you get a loud beat when you stard the script). 0 => key_snd[key].pos; resets the position to 0, in effect playing a given beat again. Math.rand2f( 2.0, 6.0 ) => key_snd[key].gain; sets gain (volume) of a particular beat to random number between 2 and 6. <<< ... >>> means print.

The script is just a proof of concept, so I'm not going to upload an mp3 (or a YouTube movie). I think it should be possible to generate some really interesting music by dancing, if the script became smarter. I'd like to hear from anyone who tried to code such a thing.

Problems I had with ChucK were reported to Ubuntu: bug #114161, bug #114162

6 comments:

Chris Carter said...

Hi!

I am working on some MIDI/ruby audio magic with JRuby right now, but it should work fine with RJB too. If you want, I can send you a copy of the code, and we could try to work on getting a dance pad input system. I don't own a dance pad, but this post has inspired me!

cdcarter at gmail (ruby message send glyph)(.) com

taw said...

Chris Carter: Sure, just send me something that works with keyboard or some other input device and I'll port it to dance mat.

Chris Carter said...

http://pastie.caboo.se/61246

Right now its "keyboard input" is sending it note messages through irb. It is also set up for easy DRb use, so the easiest way to do the client would probably just open a DRb connection, and send note messages. Drop me a line if something confuses you.

Chris Carter said...

http://pastie.caboo.se/61277
There is a client that will take keyboard input.

taw said...

Chris Carter: I changed your keyboard client to use DDR dance mat instead. Unlike the ChucK-based drummer, this one enqueues the notes instead of playing them in real time, so most of the fun is lost :-/

http://pastie.caboo.se/61345

josh said...

Hey,

I have the same error as you with the chuck command, but did you notice there is a chuck.alsa command? I don't need the jackd package installed for that, nor do I have to use any crazy parameters to the command.

Give it a try.