RLisp code formatter

Just after the highlighter, here's another important piece of language infrastructure - code formatter. For now it only re-adjusts amount of indentation. It doesn't move )s around, change other whitespace, or modify any code.

Here's an example. Before:

(defmacro cmd options
(let hash_index -1)
[options each_with_index &(fn (x i)
(cond
(and [hash_index == -1] [x is_a? Symbol] [[x to_s] =~ /:\Z/]) (set! hash_index i)
(and [hash_index == -1] [x == '=>]) (set! hash_index [i - 1]))
)]
(if (== hash_index -1)
(sendq options)
(do
(let normal_args [options get [0 ... hash_index]])
(let hash_args [options get [hash_index .. -1]])
`(sendq ,@normal_args (hash ,@hash_args))))
)

After:

(defmacro cmd options
  (let hash_index -1)
  [options each_with_index &(fn (x i)
    (cond
      (and [hash_index == -1] [x is_a? Symbol] [[x to_s] =~ /:\Z/]) (set! hash_index i)
      (and [hash_index == -1] [x == '=>]) (set! hash_index [i - 1]))
  )]
  (if (== hash_index -1)
    (sendq options)
    (do
      (let normal_args [options get [0 ... hash_index]])
      (let hash_args [options get [hash_index .. -1]])
      `(sendq ,@normal_args (hash ,@hash_args))))
)

RubyC programming language

There's a programming language in which Ruby and many Ruby libraries are written, but pretty much nothing else. Let's call it RubyC. It looks a lot like C, and can even be compiled by C compilers, but it's different in many important ways:

• It is garbage-collected
• All functions return one VALUE, and all their arguments are VALUEs
• It supports many features of Ruby, usually with just some syntactic overhead. In particular Ruby standard library and object-oriented model are available

Here are a few RubyC snippets (slightly reformatted for better readability). This one implements Array#sort:

VALUE rb_ary_sort(VALUE ary)
{
 ary = rb_ary_dup(ary);
 rb_ary_sort_bang(ary);
 return ary;
}

Notice that both the argument (from Ruby point of view ary is self), and the return value are VALUEs, and that it completely ignores memory management - freshly allocated result of rb_ary_dup is going to manage itself, somehow.

Here are a few more snippets. This one sets up Range class:

void Init_Range()
{
 rb_cRange = rb_define_class("Range", rb_cObject);
 rb_include_module(rb_cRange, rb_mEnumerable);
 rb_define_method(rb_cRange, "initialize", range_initialize, -1);
 rb_define_method(rb_cRange, "==", range_eq, 1);
 rb_define_method(rb_cRange, "===", range_include, 1);
 rb_define_method(rb_cRange, "eql?", range_eql, 1);
 rb_define_method(rb_cRange, "hash", range_hash, 0);
 rb_define_method(rb_cRange, "each", range_each, 0);
 rb_define_method(rb_cRange, "step", range_step, -1);
 rb_define_method(rb_cRange, "first", range_first, 0);
 rb_define_method(rb_cRange, "last", range_last, 0);
 rb_define_method(rb_cRange, "begin", range_first, 0);
 rb_define_method(rb_cRange, "end", range_last, 0);
 rb_define_method(rb_cRange, "to_s", range_to_s, 0);
 rb_define_method(rb_cRange, "inspect", range_inspect, 0);

 rb_define_method(rb_cRange, "exclude_end?", range_exclude_end_p, 0);

 rb_define_method(rb_cRange, "member?", range_include, 1);
 rb_define_method(rb_cRange, "include?", range_include, 1);

 id_cmp = rb_intern("<=>");
 id_succ = rb_intern("succ");
 id_beg = rb_intern("begin");
 id_end = rb_intern("end");
 id_excl = rb_intern("excl");
}

And this one implements Float#<=.

static VALUE flo_le(VALUE x, VALUE y)
{
 double a, b;

 a = RFLOAT(x)->value;
 switch (TYPE(y)) {
   case T_FIXNUM:
     b = (double)FIX2LONG(y);
     break;

   case T_BIGNUM:
     b = rb_big2dbl(y);
     break;

   case T_FLOAT:
     b = RFLOAT(y)->value;
     if (isnan(b)) return Qfalse;
     break;

   default:
     return rb_num_coerce_relop(x, y);
 }
 if (isnan(a)) return Qfalse;
 return (a <= b)?Qtrue:Qfalse;
}

I'm not going to describe it in detail, README.EXT in Ruby source distribution already does a pretty good job. I just think it's a pretty good language:

• It's almost as fast as pure C
• It is far easier to code than pure C, thanks to garbage collection, decent stdlib, and real OO (where real means something based on message-passing, no C++ crap)
• It is far nicer than most similar languages like Perl's XS.
  

One of the few downsides is its threads-unfriendliness. I'm thinking about RubyC back-end for RLisp compiler. Compilation with gcc would be somewhat slow, but it would generate much faster code than now.

RubyC also offers pretty good performance/expressiveness trade-off - one can code performance-critical sections in C, the glue in RubyC, the high-level stuff in Ruby, and it would all work together well. At least it beats Java.

Testing programs with multiple versions of Ruby

Long time ago when I was young and naive, I thought it's very unusual for a language to significantly change between minor versions. Later I learned that things break quite often, but I was still too lazy to routinely test them against all versions of Ruby. I'm still pretty lazy, so I automated it. Here are two snippets which might help you with multiversion testing. Instalation:

desc "Install many versions of Ruby"
task :install_rubies => [:install_ruby_1_8_4, :install_ruby_1_8_5, :install_ruby_1_8_6, :install_ruby_1_9]

desc "Install Ruby 1.8.4"
task :install_ruby_1_8_4 do
 sh "wget -c ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.4.tar.gz"
 sh "tar -xvzf ruby-1.8.4.tar.gz"
 Dir.chdir("ruby-1.8.4") {
   sh "./configure --prefix=/home/taw/local/ruby-1.8.4"
   sh "make"
   sh "make install"
 }
end

desc "Install Ruby 1.8.5"
task :install_ruby_1_8_5 do
 sh "wget -c ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.5.tar.gz"
 sh "tar -xvzf ruby-1.8.5.tar.gz"
 Dir.chdir("ruby-1.8.5") {
   sh "./configure --prefix=/home/taw/local/ruby-1.8.5"
   sh "make"
   sh "make install"
 }
end

desc "Install Ruby 1.8.6"
task :install_ruby_1_8_6 do
 sh "wget -c ftp://ftp.ruby-lang.org/pub/ruby/ruby-1.8.6.tar.gz"
 sh "tar -xvzf ruby-1.8.6.tar.gz"
 Dir.chdir("ruby-1.8.6") {
   sh "./configure --prefix=/home/taw/local/ruby-1.8.6"
   sh "make"
   sh "make install"
 }
end

desc "Install Ruby 1.9"
task :install_ruby_1_9 do
 sh "sudo apt-get install ruby1.9"
end

And testing:

desc "Run all tests with Ruby 1.8.6/1.8.5/1.8.4/1.9"
task :test_all_rubies => [:tests_1_8_6, :tests_1_8_5, :tests_1_8_4, :tests_1_9]

desc "Run tests with Ruby 1.8.4"
task :tests_1_8_4 do
   ruby_bin = "/home/taw/local/ruby-1.8.4/bin/ruby"
   sh ruby_bin, "./tests.rb"
end

desc "Run tests with Ruby 1.8.5"
task :tests_1_8_5 do
   ruby_bin = "/home/taw/local/ruby-1.8.5/bin/ruby"
   sh ruby_bin, "./tests.rb"
end

desc "Run tests with Ruby 1.8.6"
task :tests_1_8_6 do
   ruby_bin = "/home/taw/local/ruby-1.8.6/bin/ruby"
   sh ruby_bin, "./tests.rb"
end

desc "Run tests with Ruby 1.9"
task :tests_1_9 do
   ruby_bin = "/usr/bin/ruby1.9"
   sh ruby_bin, "./tests.rb"
end

Active Record for RLisp

Every language needs a web framework. RLisp is in lucky position as it can reuse major parts of Ruby on Rails. So far Brad Ediger created RLisp plugin for Ruby on Rails, but that's just part of the story. Today I'm going to talk about a different part - the Object-Relational Mapper library Active Record.

Real databases are complex, ugly, and take hundreds of megabytes or more, so as an example I used a toy one by Mike Wilson, which describes music albums.

The original Ruby version looks like this, and we want to recreate it in possibly the nicest RLisp we can.

require 'active_record'

ActiveRecord::Base.logger = Logger.new(STDERR)
ActiveRecord::Base.colorize_logging = false

ActiveRecord::Base.establish_connection(
 :adapter => "sqlite3",
 :dbfile  => ":memory:"
)

ActiveRecord::Schema.define do
 create_table :albums do |table|
   table.column :title, :string
   table.column :performer, :string
 end

 create_table :tracks do |table|
   table.column :album_id, :integer
   table.column :track_number, :integer
   table.column :title, :string
 end
end

class Album < album =" Album.create(:title"> 'Black and Blue', :performer => 'The Rolling Stones')
album.tracks.create(:track_number => 1, :title => 'Hot Stuff')
album.tracks.create(:track_number => 2, :title => 'Hand Of Fate')
album.tracks.create(:track_number => 3, :title => 'Cherry Oh Baby ')
album.tracks.create(:track_number => 4, :title => 'Memory Motel ')
album.tracks.create(:track_number => 5, :title => 'Hey Negrita')
album.tracks.create(:track_number => 6, :title => 'Fool To Cry')
album.tracks.create(:track_number => 7, :title => 'Crazy Mama')
album.tracks.create(:track_number => 8, :title => 'Melody (Inspiration By Billy Preston)')

album = Album.create(:title => 'Sticky Fingers',  :performer => 'The Rolling Stones')
album.tracks.create(:track_number => 1, :title => 'Brown Sugar')
album.tracks.create(:track_number => 2, :title => 'Sway')
album.tracks.create(:track_number => 3, :title => 'Wild Horses')
album.tracks.create(:track_number => 4, :title => 'Can\'t You Hear Me Knocking')
album.tracks.create(:track_number => 5, :title => 'You Gotta Move')
album.tracks.create(:track_number => 6, :title => 'Bitch')
album.tracks.create(:track_number => 7, :title => 'I Got The Blues')
album.tracks.create(:track_number => 8, :title => 'Sister Morphine')
album.tracks.create(:track_number => 9, :title => 'Dead Flowers')
album.tracks.create(:track_number => 10, :title => 'Moonlight Mile')

puts Album.find(1).tracks.length
puts Album.find(2).tracks.length

puts Album.find_by_title('Sticky Fingers').title
puts Track.find_by_title('Fool To Cry').album_id

It's a very straightforward database schema with just two tables. There's some duplication in code, especially all the table.column, and album.tracks.create fragments, but overall it's a pretty nice code. Its direct translation to RLisp however leaves a lot to be desired.

(ruby-require "active_record")

[ActiveRecord::Base logger= [Logger new STDERR]]
[ActiveRecord::Base colorize_logging= false]

[ActiveRecord::Base establish_connection (hash adapter: "sqlite3" dbfile: ":memory:")]

[ActiveRecord::Schema define &(fn()
[self create_table 'albums &(fn (table)
  [table column 'title 'string]
  [table column 'performer 'string]
)]

[self create_table 'tracks &(fn (table)
  [table column 'album_id 'integer]
  [table column 'track_number 'integer]
  [table column 'title 'string]
)]
)]

(let Album [Class new ActiveRecord::Base])
(class Album
[self has_many 'tracks]
)

(let Track [Class new ActiveRecord::Base])
(class Track
[self belongs_to 'album]
)

(let album [Album create (hash title: "Black and Blue" performer: "The Rolling Stones")])

[[album tracks] create (hash track_number: 1 title: "Hot Stuff")]
[[album tracks] create (hash track_number: 2 title: "Hand Of Fate")]
[[album tracks] create (hash track_number: 3 title: "Cherry Oh Baby ")]
[[album tracks] create (hash track_number: 4 title: "Memory Motel ")]
[[album tracks] create (hash track_number: 5 title: "Hey Negrita")]
[[album tracks] create (hash track_number: 6 title: "Fool To Cry")]
[[album tracks] create (hash track_number: 7 title: "Crazy Mama")]
[[album tracks] create (hash track_number: 8 title: "Melody (Inspiration By Billy Preston)")]

(let album [Album create (hash title: "Sticky Fingers"  performer: "The Rolling Stones")])
[[album tracks] create (hash track_number: 1 title: "Brown Sugar")]
[[album tracks] create (hash track_number: 2 title: "Sway")]
[[album tracks] create (hash track_number: 3 title: "Wild Horses")]
[[album tracks] create (hash track_number: 4 title: "Can't You Hear Me Knocking")]
[[album tracks] create (hash track_number: 5 title: "You Gotta Move")]
[[album tracks] create (hash track_number: 6 title: "Bitch")]
[[album tracks] create (hash track_number: 7 title: "I Got The Blues")]
[[album tracks] create (hash track_number: 8 title: "Sister Morphine")]
[[album tracks] create (hash track_number: 9 title: "Dead Flowers")]
[[album tracks] create (hash track_number: 10 title: "Moonlight Mile")]

(print [[[Album find 1] tracks] length])
(print [[[Album find 2] tracks] length])

(print [[Album find_by_title "Sticky Fingers"] title])
(print [[Track find_by_title "Fool To Cry"] album_id])

We can make it a lot better. One feature which Ruby DSLs often take advantage of is Ruby syntax for keyword arguments - obj.foo(x, y, {:a => b, :c => d}) can be abbreviated to much better-looking obj.foo x, y, :a => b, :c => d. RLisp doesn't support such abbreviations directly, but it can do something similar with macros. Macro (cmd ...) defined below will support similar syntax - (cmd obj foo x y a: b c: d) expanding to [obj foo x y (hash a: b c: d)]. We can also provide (cmds obj (command one) (command two) ...) macro for common pattern of issuing multiple commands to the same objects, something like Smalltalk's ;.

(defmacro sendq (obj meth . args)
`(send ,obj ',meth ,@args))

(defmacro cmd options
(let hash_index -1)
[options each_with_index &(fn (x i)
  (cond
    (and [hash_index == -1] [x is_a? Symbol] [[x to_s] =~ /:\Z/]) (set! hash_index i)
    (and [hash_index == -1] [x == '=>]) (set! hash_index [i - 1]))
)]
(if (== hash_index -1)
  (sendq options)
  (do
    (let normal_args [options get [0 ... hash_index]])
    (let hash_args [options get [hash_index .. -1]])
    `(sendq ,@normal_args (hash ,@hash_args))))
)

(defmacro cmds (recv . args)
(let tmp (gensym))
(let args-expanded [args map &(fn (a)
  `(cmd ,recv ,@a)
)])
`(do ,@args-expanded)
)

These macros were pretty generic. Now let's build a simple wrapper for Active Record.

(ruby-require "active_record")

(defmacro active-record-establish-connection options
`(cmd ActiveRecord::Base establish_connection ,@options))

(defmacro active-record-schema-define defs
(let expanded-defs [defs map & (fn (d) (match d
  ('create-table . args) `(schema-define-create-table ,@args)
  (raise "Unrecognized statement in schema definition: #{d}")
))])
`[ActiveRecord::Schema define &(fn() ,@expanded-defs)]
)

(defmacro schema-define-create-table (table-name . entries)
(let tmp (gensym))
(let expanded-entries [entries map &(fn (entry)
  (let name [entry get 0])
  (let type [entry get 1])
  `[,tmp column ',name ',type])])
`[self create_table ',table-name &(fn (,tmp) ,@expanded-entries)]
)

(defmacro define-active-record-class (class-name . defs)
(let expanded-defs [defs map &(fn (d) (match d
  (has_many . args)    `[self has_many ,@args]
  (has_one . args)     `[self has_one ,@args]
  (belongs_to . args)  `[self belongs_to ,@args]
  (has_and_belongs_to_many . args)  `[self has_and_belongs_to_many ,@args]
  _ d
))])
`(do
    (let ,class-name [Class new ActiveRecord::Base])
    (class ,class-name
      ,@expanded-defs
    )
  )
)

Finally Active Record can be used in a nice manner. I think it looks even nicer than Ruby version now.

(require "active_record.rl")

[ActiveRecord::Base logger= [Logger new STDERR]]
[ActiveRecord::Base colorize_logging= false]

(active-record-establish-connection adapter: "sqlite3" dbfile: ":memory:")

(active-record-schema-define
(create-table albums
  (title string)
  (performer string))
(create-table tracks
  (album_id integer)
  (track_number integer)
  (title string))
)

(define-active-record-class Album
(has_many 'tracks)
)

(define-active-record-class Track
(belongs_to 'album)
)

(let album (cmd Album create title: "Black and Blue" performer: "The Rolling Stones"))

(cmds [album tracks]
(create track_number: 1 title: "Hot Stuff")
(create track_number: 2 title: "Hand Of Fate")
(create track_number: 3 title: "Cherry Oh Baby ")
(create track_number: 4 title: "Memory Motel ")
(create track_number: 5 title: "Hey Negrita")
(create track_number: 6 title: "Fool To Cry")
(create track_number: 7 title: "Crazy Mama")
(create track_number: 8 title: "Melody (Inspiration By Billy Preston)"))

(let album (cmd Album create title: "Sticky Fingers"  performer: "The Rolling Stones"))
(cmds [album tracks]
(create track_number: 1 title: "Brown Sugar")
(create track_number: 2 title: "Sway")
(create track_number: 3 title: "Wild Horses")
(create track_number: 4 title: "Can't You Hear Me Knocking")
(create track_number: 5 title: "You Gotta Move")
(create track_number: 6 title: "Bitch")
(create track_number: 7 title: "I Got The Blues")
(create track_number: 8 title: "Sister Morphine")
(create track_number: 9 title: "Dead Flowers")
(create track_number: 10 title: "Moonlight Mile"))

(print [[[Album find 1] tracks] length])
(print [[[Album find 2] tracks] length])

(print [[Album find_by_title "Sticky Fingers"] title])
(print [[Track find_by_title "Fool To Cry"] album_id])

One more thing - as you can see from code examples (which hopefully weren't excessively mutilated by Blogger), the RLisp syntax highlighter got a lot smarter, and quasiquotes should be much easier to read.

Why RLisp will not support Ruby class variables

RLisp tries to be as Ruby-like as it reasonable can. For example it supports Global::Constants, @instance_variables, $global_variables, and self. It does not support semantically ugly features like Constants_Without_Fully_Specified_Scope, public/private/protected distinction, and Perlish magic variables of $~, $1 and family, trying to use cleaner solutions instead.

I'm adding another to the list of officially not supported features - @@class_variables. They simply lack any sensible semantics. The following code:

class Foo
 @@xyzzy = 6
 def hello
   @@xyzzy
 end
end

class Bar < Foo
 @@xyzzy = 42
end

p Foo.new.hello
p Bar.new.hello
p Foo.new.instance_eval { @@xyzzy }

in Ruby 1.8 produces:

42
42
semantics_of_cvars.rb:16: uninitialized class variable @@xyzzy in Object (NameError)
      from semantics_of_cvars.rb:16:in `instance_eval'
      from semantics_of_cvars.rb:16

On the other hand in Ruby 1.9:

6
6
semantics_of_cvars.rb:16: warning: class variable access from toplevel singleton method
semantics_of_cvars.rb:16: uninitialized class variable @@xyzzy in Object (NameError)
      from semantics_of_cvars.rb:16

I don't remember ever using class variables, but I had a vague feeling @@xyzzy would mean something like self.class.class_variable_get(:@@xyzzy). Apparently the meaning is much uglier and frankly I cannot see any situation in which it could possibly be useful in Ruby and even less so in RLisp - unlike Ruby def, RLisp fn/defun/method/... do not wipe out surrounding context, so we can use plain local variables.

(class Object
(let xyzzy 42)
(method hello () (print xyzzy))
; xyzzy is a normal lexical variable visible until we exit class definition
)
; not visible any more
[self hello] # => 42

Token-based syntax highlighter for RLisp

One of the most important parts of language infrastructure is a good syntax highlighter. I'm not claiming the highlighter I just committed to the repository is any good - it simply reuses the lexer, and outputs tokens in appropriate <span style='color: #00FF00'>...</span>. Making it use CSS classes would be nicer, but then I wouldn't be able to use it on my blog :-)

A more serious syntax highlighter should at least recognize what's quoted and what's not, which symbols are standard macros and control structures, and so on. Still, it's much better to have some syntax highlighting than none.

#!/home/taw/everything/rlisp/trunk/rlisp

(require "rlunit.rl")

; It's much easier to temporarily replace (gensym)
; with a mock one than to design an assertion system
; which can handle the real one.
(defun start-readable-gensym ()
 (let sid 0)
 (set! gensym
   (fn ()
      (let sym [(+ "tmp-" [sid to_s]) to_sym])
      (set! sid (+ 1 sid))
      sym
   )
 )
)
(defun restore-correct-gensym ()
 (set! gensym (fn () (ruby-eval "$gensym||=0; $gensym+=1; ('#:G' + $gensym.to_s).to_sym")))
)

(test-suite Readable_Gensym
 (test gensym
   (start-readable-gensym)
   (assert 'tmp-0 == (gensym))
   (assert 'tmp-1 == (gensym))
   (assert 'tmp-2 == (gensym))
   (restore-correct-gensym)
   (assert [(gensym) to_s] =~ /\A#:G\d+\Z/)
   (start-readable-gensym)
   (assert 'tmp-0 == (gensym))
   (assert 'tmp-1 == (gensym))
   (start-readable-gensym)
   (assert 'tmp-0 == (gensym))
   (restore-correct-gensym)
   (assert [(gensym) to_s] =~ /\A#:G\d+\Z/)
   (assert (gensym) != (gensym))
 )
 (method teardown () (restore-correct-gensym))
)

Regular expression matching in RLisp

Ruby copies Perl regular expression matching semantics:

• If a_string matches /a_rx/, return true and set $~ to match data
• Otherwise return false and set $~ to nil
• $~ is "Do-What-I-Mean-scoped". It is not local or global variable. "Do-What-I-Mean-scoping" is different in Perl and Ruby.
• $1, $2, $`, and related other "variables" are not variables at all - they're simply parser-level macros which expand into $~.do_something.
• Very often $1 etc. need to be converted to numbers. In Perl strings and numbers are unified and explicit conversion is not needed. In Ruby you often need to follow the match by something like a, b, c = $1.to_i, $2, $3.to_f.
  

I don't think it would be a good idea to copy all that to RLisp - semantically it's unbelievably ugly. It's also too damn convenient compared to the lengths one needs to go to parse something in Python, so something equally convenient but cleaner is needed instead.

Without any special text processing macros RLisp code to do something as simple as match IP numbers looks ugly, but at least it works:

(defun parse-ip (s)
 (let m [s match /\A(\d+)\.(\d+)\.(\d+)\.(\d+)\Z/])
 (if m
   (do
     (let a [[m get 1] to_i])
     (let b [[m get 2] to_i])
     (let c [[m get 3] to_i])
     (let d [[m get 4] to_i])
     (list a b c d))
   (raise "Cannot parse IP"))
)

(test-suite Text_Processing
 (test parse_ip
   (assert (parse-ip "1.2.3.4") == '(1 2 3 4))
   (assert (parse-ip "64.233.183.104") == '(64 233 183 104)))
)

The solution - macros (who might have guessed that ...). Very simple pair of macros lets us use (rx-match a_string /a_rx/ a b c d) just like a_string =~ /a_rx/; a, b, c, d = $1, $2, $3, $4 would be used in Ruby. It returns true or false just like Ruby =~.

(defmacro rx-results (m . args)
 (let res '(do))
 [args each_with_index & (fn (x i)
   [res push `(let ,x [,m get ,[i + 1]])]
 )]
 [res push 'true]
 res
)

(defmacro rx-match (s rx . args)
 (let tmp (gensym))
`(do
   (let ,tmp [,s match ,rx])
   (if ,tmp
     (rx-results ,tmp ,@args)
     false)
 )
)

(defun parse-ip-2 (s)
 (if (rx-match s /\A(\d+)\.(\d+)\.(\d+)\.(\d+)\Z/ a b c d)
   (list [a to_i] [b to_i] [c to_i] [d to_i])
   (raise "Cannot parse IP"))
)

That's pretty much as good as Ruby already, but we can do a bit better, integrating regexp matching and conversion to numbers. (i varname) and (f varname) mean convert the variable, while plain varname leaves it as a string.

(defmacro rx-results (m . args)
 (let res '(do))
 [args each_with_index & (fn (x i)
   (match x
     ('f v) [res push `(let ,v [[,m get ,[i + 1]] to_f])]
     ('i v) [res push `(let ,v [[,m get ,[i + 1]] to_i])]
     [res push `(let ,x [,m get ,[i + 1]])])
 )]
 [res push 'true]
 res
)

(defun parse-ip-3 (s)
 (if (rx-match s /\A(\d+)\.(\d+)\.(\d+)\.(\d+)\Z/ (i a) (i b) (i c) (i d))
   (list a b c d)
   (raise "Cannot parse IP"))
)

By the way it would be very difficult to build such macros in Scheme or Common Lisp, not only due to their lack of builtin Perl-compatible regular expressions (a library can solve that problem), but also due to (let ...) in these Lisps being much more restrictive.

(let variable value) in RLisp sets variable in scope of current function. (let ((variable value)) code) in Scheme sets variable only in scope of code. It makes RLisp code clearer and reduces number of nested parentheses, but it also does something far more important - it makes writing RLisp macros a lot easier than Scheme/Common Lisp macros.

Using home directory as GTD inbox

Here's a bit about my GTD system. It's mostly paper-based, with 85mm x 85mm paper cards, physical file folders and so on, but a lot of information comes in electronic form, and it needs to be managed.

So firstmost I have big directories for all kinds of information:

• /home/taw/reference/* - electronic equivalent of the general reference system, with thing being in direct subdirectories, one "item" per directory
• /home/taw/everything/ - checkout of svk (formerly svn) repository of all my software
• A few directories like /home/taw/movies/* for The Colbert Report etc.
• Everything else in the home directory or Desktop which is not a dotfile is "stuff" and needs to be processed.

There's a long way between knowing something needs to be processed and actually processing it in timely manner, so I want to be reminded about stuff I leave for too long. First I wrote a simple Ruby script that can be used to notify me about various events. It uses KDE DCOP service KNotify to actually display notifications.

header = "Notification"
msg = ARGV.join("\n")

system 'dcop', 'knotify', 'Notify', 'notify', 'notify',
      header, msg, 'nosound', 'nofile', '16', '0'

And now the script which runs on detached screen or something like that to bug me about unprocessed things.

items_whitelist = %w[
 /home/taw/Desktop
 /home/taw/everything
 /home/taw/ipoddb
 /home/taw/music
 /home/taw/movies
 /home/taw/local
 /home/taw/ebooks
 /home/taw/img
 /home/taw/website
 /home/taw/website_snapshot
]

max_displayed = 10

big_timer = 5
old_items = []

while true
items = (Dir["/home/taw/*"] + Dir["/home/taw/Desktop/*"] - items_whitelist).sort
items = items.map{|x|x.sub(%r[\A/home/taw/],"* ")}.sort
next if items == []

if items == old_items
 big_timer -= 1
 sleep 60
 next unless big_timer == 0
end
big_timer = 5

sz = items.size

if items.size > max_displayed
 displayed_items = items.sort_by{rand}[0, max_displayed].sort + ["* ..."]
else
 displayed_items = items
end
system "notify", "Inbox is not processed", "#{sz} items in your inbox:", *displayed_items

sleep 60
old_items = items
end

One more thing - I use Download Statusbar extension for Firefox and download everything to /home/taw/Desktop without asking any questions. That frees me from thinking about stuff I download - it's all going to the "home directory inbox".

How py2exe made me stop coding jrpg

That's a simple story but it illustrates something important. jrpg is an SNES RPG style game which helps the player learn Japanese writing system. It is written in Python with PyGame. Well, data preprocessing is done mostly in Perl, and the build system is in Ruby rake, but the game code is pure Python. Unfortunately that means users needed Python and PyGame installed to play jrpg. No problem for Linuxers, but Windows users were really confused by that.

Then I found py2exe - an awesome tool which turned Python programs into standalone Windows executables. Thanks to py2exe and some "marketing" jrpg downloads went into thousands. There was only one problem - py2exe doesn't run on Linux.

That turned rake package; rake update_website into a long and painful task. If I changed anything in jrpg, I had to copy jrpg package onto a pendrive, go to some Windows box (hopefully with Python, PyGame, and py2exe already installed), build the Windows package, copy it back to the pendrive, go back to the Linux box, put it in the right directory, and only then I could upload the package. Knowing that I have to go to such lengths every time I change one line of code in jrpg completely killed the joy of coding.

What could I do ? I couldn't make myself stop updating the Windows package and loss 80% of the users. I tried running py2exe under wine and Cedega (it failed), porting it to Linux (unfortunately py2exe uses too many Windows-specific APIs I don't have a slightiest idea about, and low-level C coding is not a fun thing to do), even hacking the produced exe file. Nothing worked.

I guess I might get back to coding jrpg. Maybe someone will hack a crosscompiler onto py2exe, maybe someone will create easily installable dosbox-alike with Windows on which I'll be able to run py2exe, maybe my desire to hack it will temporarily overcome my disgust.

The moral of the story - hacking must be fun, or people will go hack something else.

Paradox of the poor ignoring their self-interest in democratic countries

Yesterday I watched a movie "The Revolution Will Not Be Televised" on Google Video. It was about democratically elected president of Venezuela Hugo Chávez and 2002 failed coup attempt against him.

Here's a summary with a some background - Latin American countries like Venezuela have huge income inequalities, but are at least somewhat democratic, so every now and then masses of angry poor voters elect someone who promises to reduce income inequality somewhat. Most of the time nothing changes, but every now and then they elect someone who actually tries to do what they promised, what really annoys the local rich and the USA, so they try to get rid of them as soon as possible by any means necessary, often succeeding like in Chile in 1973. This time they failed.

Like everyone else I'm tired of the leftist bullshit about the poor, and low quality of discussion about politics in general. What we need is hard data, starting from measures of income inequality. We need reliable single number summaries, otherwise everyone will cherry-pick any data to support their chosen in advance conclusion. Some of the standard and widely available statistics of income inequality are:

• Gini coefficient of income inequality
• Ratio of income of 20% richest to 20% poorest

And of general well-being:

• Life expectancy
• Infant mortality
• Human Development Index

Gini coefficient is the most popular measure of income equality. 0 means perfectly equal, and 100 means perfectly inequal. For vast majority of the civilized world it is somewhere between 25 and 40. Inequality in poor third world countries is usually far greater, sometimes over 60. Here are pictures from Wikipedia.



Funny thing about many of these extremely inequal countries, especially in the Latin America, is that they are at least ostensibly democratic. The poor have majority, and could easily make the state implement policies reducing the inequality. Nothing "radical" would be necessary, simple thing like:

• free healthcare
• free education
• raising taxes on natural resources extraction
• raising minimum wage

would drastically improve situation of the poor without affecting the economy much. For example in Bolivia with income inequality of 60.1 doubling incomes of the poorest 20% would require reducing incomes of the richest 20% (who earn 42.3x as much) by just 2.3%. That's less than yearly GDP growth anyway. For comparison to do such thing in Germany (Gini index 28.3, quintile ratio 4.3:1), the richest 20% would need to loss 23% of their income - a very difficult thing to do without gravely disrupting the economy.

So what really happened to Venezuela under Chávez ?

• Statistics from the Venezuelan government. Human Development Index 1998-2005 - a huge jump from 0.6917 to 0.8144.
• Data from USA Department of State 1998 to 2006 - infant mortality 28.5 to 15.5 - a huge improvement again.
• From the same source, life expectancy - 73.31 to 74.8, a more modest but solid improvement.
• I wasn't able to find any source of Gini coefficients for the relevant time period. Unfortunately different sources use different methodologies, so gathering point data wouldn't be reliable.
• There's a bit of data on income distribution. Income of the "class E", defined as poorest 58% of the society, grew by 53% nominal (30% real) in 2004 and by 32% nominal (16% real) in 2005. Income growths of higher strata of the society seem to increase slower than inflation. That means the poor are getting a lot richer and the rich are getting somewhat poorer (data unfortunately only for two years).
• There are also some poverty statistics showing major decrease in poverty, but I'm not a big fan of poverty statistics, as the definition of poverty cutoff is quite arbitrary.

So basically in this case the angry poor managed to elect the right (for them) guy, and there is significant transfer of income from the increasing oil prices and from the rich to the poor.

The big question - why doesn't it happen more often ? There are so many countries with extreme income inequality, with a lot of money from natural resources, and with the poor being able to vote and affect income distribution, but somehow they very rarely use that power. The paradox is:

Why in so many cases where the poor could significantly improve their economic situation by voting, they aren't doing so ?

A few possible reasons:

• If the wrong guy is elected, the ruling elites are able to stage a coup, like in Chile 1973, and like they tried to do in Venezuela in 2002.
• The poor often vote for someone who promises a lot, but doesn't change anything.
• The poor can be convinced to accept the status quo. The media, educational system and so on, all frequently side with the ruling elite, and there are no strong organized opposition groups.
• In particular the poor can be convinced that inequality is required for economic growth. The data isn't conclusive within the civilized range of inequality (Gini index 25 to 40), but it is clear that Latin American style extreme inequality hampers growth.
• The voters can be sidetracked by another issue. It isn't that difficult to polarize the society, and in two-party and presidential systems it is very difficult for third parties to operate. A good example is USA, where the Democratic and Republican parties have very similar economical policies, and differ mostly on minor ideologal issues like gay marriage.

It would be nice to quantify these points some day.