The best kittens, technology, and video games blog in the world.
Sunday, August 05, 2012
RLisp is on github now
Continuing cleanup of my old software, I uploaded RLisp to github, cleaned it up a bit and made it compatible with both Ruby 1.8 and 1.9.
Stack backtraces with 1.9 seem somewhat less informative now, but they were pretty mediocre to begin with. Other than that and an small example involving complex numbers (1.8 Complex#new is 1.9 Complex#rect), it all works just the same.
Rakefile to build Debian/RPM packages is most likely broken, but if anybody cares it's probably just a matter of fixing a few paths, so send me a pull request if you do.
I would really love to build a gem for that, but I don't know how to build binary gems, and RLisp includes a small C program to make #! work correctly (99% of functionality works fine without it).
Another thing - how the hell should RLisp / Rlisp / rlisp be capitalized? I wrote it years ago and I still have no idea.
Tuesday, June 26, 2007
Toy C backend for RLisp compiler

I wrote a toy RLisp C "backend". It's not yet connected to the real compiler - the only thing it compiles is Ackermann function - of course you can make it compile something else by issuing right opcodes by hand.
First some support code. Symbol#<=> is already in rlisp_support.rb, String#ord is supposed to be a wrapper around different behaviour of Strings in 1.8 and 1.9. Symbol#mangle_c and Symbol#stringify_c convert Ruby symbols (which may contain funny characters like :"==\nblah!@#") to C variable names and strings. For brevity I'm not pasting the tests here.
class Symbol
include Comparable
def <=>(other)
to_s <=> other.to_s
end
protected :<=>
end
class String
# FIXME: 1.8 specific, add support for 1.9
def ord
self[0]
end
end
class Symbol
def mangle_c
to_s.gsub(/([^a-zA-Z0-9])/) { sprintf "_%02x", $1.ord }
end
def stringify_c
'"' + to_s.gsub(/([\\"])/) { "\\#{$1}" } + '"'
end
endHere's the code which calls the code generator, and then the
ack function. It uses Ruby->C trampoline to support closure variables. The interface used by code generator is similar to one used by normal Ruby-backed RLisp code generator, but not identical. I'm going to refactor them both to match, so RLisp frontend can talk with either backend. I think C-compiling at least RLisp stdlib is a good idea. C-compiling REPL less so. Code generators are so simple that I hope to be able to maintain both without too much work.class Stuff
cg = RLispCodegenC.new(:ack, [:m, :n])
t2, t3, t4, t5, t6, t7, t8, t9, t10, t11, t12, t13, t14, t15 = cg.tmps(14)
cg.funcall(t2, :m, :"==", 0)
cg.x_if(t2) {
cg.funcall(t4, :n, :+, 1);
cg.asg(t3, t4)
}
cg.x_else {
cg.funcall(t5, :n, :==, 0);
cg.x_if(t5) {
cg.global_get(t7, :ack)
cg.funcall(t8, :m, :-, 1);
cg.funcall(t9, t7, :call, t8, 1);
cg.asg(t6, t9)
}
cg.x_else {
cg.global_get(t10, :ack)
cg.funcall(t11, :m, :-, 1)
cg.global_get(t12, :ack)
cg.funcall(t13, :n, :-, 1)
cg.funcall(t14, t12, :call, :m, t13)
cg.funcall(t15, t10, :call, t11, t14)
cg.asg(t6, t15)
}
cg.asg(t3, t6)
}
cg.ret(t3)
inline do |builder|
result = builder.c cg.build
end
end
ack_m = Stuff.new.method(:ack)
globals = {}
closure = { :globals => globals }
globals[:ack] = lambda{|*args| ack_m.call(closure, args) }
print "retval = ", globals[:ack].call(3, 3), "\n"RLispCodegenC builds one function at time. To use it you need to create a new RLispCodegenC instance, call some opcode methods, call build on the RLispCodegenC object, and compile it using inline. inline caches .so objects, so C compiler is called only when something changed.require 'inline'
class RLispCodegenC
def initialize(name, args)
@name = name
@args = args
@syms = {:globals => true}
@ids = {}
@temps = []
@code = ""
end
def build
temps = [:globals] + @args + @temps
src = <<EOF
VALUE #{@name}(VALUE closure, VALUE args) {
VALUE #{temps.join(", ")};
#{static_init}
#{arg_init}
#{@code}
}
EOF
src
end
def arg_init
res = "globals = rb_hash_aref(closure, SYM_globals);\n"
@args.each_with_index{|arg, i|
res << "#{arg.mangle_c} = rb_ary_entry(args, #{i});\n"
}
res
end
# This code should be executed just once in Init_*, not every time the method is called
def static_init
ids = @ids.keys.sort
syms = @syms.keys.sort
res = ""
res << %Q[int #{ids.map{|i| "ID_#{i.mangle_c}"}.join(", ")};\n] unless ids.empty?
res << %Q[VALUE #{syms.map{|s| "SYM_#{s.mangle_c}"}.join(", ")};\n] unless syms.empty?
ids.each{|i| res << "ID_#{i.mangle_c} = rb_intern(#{i.stringify_c});\n" }
syms.each{|s| res << "SYM_#{s.mangle_c} = ID2SYM(rb_intern(#{s.stringify_c}));\n" }
res
end
def tmps(sz)
res = []
sz.times{
t = :"t#{@temps.size}"
res << t
@temps << t
}
res
end
def to_c(x)
if x.is_a? Fixnum
"INT2FIX(#{x})"
# It means a C temporary, ***NOT*** Ruby symbol
# FIXME: Mangling ?
elsif x.is_a? Symbol
x.mangle_c
else
raise "Don't know how to convert #{x.class}: #{x.inspect}"
end
end
def funcall(asg_to, recv, mid, *args)
@ids[mid] = true
mid_s = "ID_#{mid.mangle_c}"
args_m = args.map{|a| ", " + to_c(a)}.join
@code << "#{to_c(asg_to)} = rb_funcall(#{to_c(recv)}, #{mid_s}, #{args.size}#{args_m});\n"
end
def x_if(tmp)
@code << "if(RTEST(#{tmp})) {\n"
yield
end
def x_else
@code << "} else {\n"
yield
@code << "}\n"
end
def asg(to, from)
@code << "#{to_c(to)} = #{to_c(from)};\n"
end
def global_get(to, sym)
@syms[sym] = true
sym_s = "SYM_#{sym.mangle_c}"
@code << "#{to} = rb_hash_aref(globals, #{sym_s});\n"
end
def ret(val)
@code << "return #{to_c(val)};\n"
end
def debug(msg, val=nil)
@code << %Q[rb_funcall(self, rb_intern("print"), 1, rb_str_new2(#{msg.inspect})); /* DEBUG */\n]
@code << %Q[rb_funcall(self, rb_intern("p"), 1, #{val}); /* DEBUG */\n] unless val.nil?
end
end
Wednesday, June 20, 2007
Object-Oriented dialects of Lisp

Lisp is much older than object oriented programming. When OOP got popular people wanted to do OOP in Lisp too, but it wasn't obvious how to retrofit Lisp to make it object oriented. CLOS for Common Lisp became somewhat popular, but its quite far from what Smalltalkers would call "object oriented", and some people feel it wasn't very Lispy either. Many Lispers like Paul Graham simply gave up on OOP. Others like me decided to code their own object-oriented dialects of Lisp. Solutions they came up with are very different. I checked the following Object-Oriented dialects of Lisp:
- RLisp - Lisp integrated with Ruby
- e7 - Lisp integrated with Python
- goo - Object-Oriented Lisp inspired by Scheme
- CLOS - Object-Oriented system built on top of Common Lisp
- I also wanted to check Coke, but it segfaulted at me, and the last thing I felt like doing was debugging broken C code.
- Define class
Pointrepresenting 2D points of vectors - Define initializer for this class, which takes 2 arguments
xandyand returns aPointinstance - Define a method for stringifying
Points, like Ruby'sto_sand Python's__str__. If possible, I wanted it to be automatically called by REPL and(print a_point)or its equivalent. - Define a method for adding two
Points. I wanted to call it+if possible. - Create two points, add them, and print the result.
(let Point [Class new])
(class Point
(attr-reader 'x)
(attr-reader 'y)
(method initialize (x y)
(set! @x x)
(set! @y y))
(method to_s ()
"<#{@x},#{@y}>")
(method + (other)
[Point new (+ [self x] [other x]) (+ [self x] [other y])]))
(let a [Point new 1 5])
(let b [Point new -2 9])
(print [a + b])e7 code:
(class Point ()
(def (init self x y)
(set-self x x y y)))
(defmethod (+ (self Point) (other Point))
(Point
(+ self.x other.x)
(+ self.y other.y)))
(defmethod (print (self Point) f)
(fwrite f (format "<%s,%s>" self.x self.y)))
(def a (Point 1 5))
(def b (Point -2 9))
(print (+ a b))goo code:
(dc <point> (<any>))
(dp point-x (<point> => <num>))
(dp point-y (<point> => <num>))
(dm point-new (x|<num> y|<num>)
(new point-x x point-y y))
(dm point-add (p1|<point> p2|<point> => <point>)
(point-new
(+ (point-x p1) (point-x p2))
(+ (point-y p1) (point-y p2))))
(dm write-point (port|<out-port> x|<point>)
(msg port "<%s,%s>" (point-x x) (point-y x)))
(dv a (point-new 1 5))
(dv b (point-new -2 9))
(write-point out (point-add a b))
(newline out) CLOS code:
(defclass point ()
((x :reader point-x :initarg :x)
(y :reader point-y :initarg :y)))
(defmethod make-point (x y)
(make-instance 'point :x x :y y))
(defmethod point-add (a b)
(make-point
(+ (point-x a) (point-x b))
(+ (point-y a) (point-y b))))
(defmethod point-print ((p point))
(format t
"<~s,~s>" (point-x p) (point-y p)))
(setf a (make-point 1 5))
(setf b (make-point -2 9))
(point-print (point-add a b))The first thing I noticed is how different these snippets look in spite of doing pretty much the same thing. The second is that RLisp and e7 are much more concise than goo and CLOS. RLisp and e7 have more syntactic extensions for OO. RLisp supports
[ ] syntax for method calls, self for message receiver, and @ivar for instance variables. e7 doesn't go that far and limits itself to obj.attr syntax for attribute access and method calls. In e7 like in Python all attributes are public. In RLisp like in Ruby all attributes are private, and (attr-reader 'x) must be explicitly called.In RLisp and e7 attributes are dynamic and aren't predeclared anywhere. In goo and CLOS list of attributes is part of class definition. goo even requires attribute types, but will happily accept
<any>. Only RLisp seemed to provide a clear way of stringifying objects. recurring-write in goo converts standard objects to strings, but write for some reason ignored extensions of it. In e7 the standard way is overloading print. It seems more limited than stringification method, but it should be possible to print to string buffer instead of a real file. I have no idea what CLOS uses to stringify objects.The most important ideological difference is treatment of method calls. RLisp makes message sends and function calls separate operations much like Smalltalk-style OOP languages. All others define methods as some kind of generic functions. This means weaker encapsulation and some deep philosophical differences. Coke also separates function calls and message sends, but like I said it was segfaulting, so I was unable to take a closer look at it.
Only in RLisp creating a class and defining stuff inside it are separate operations. Ruby is ambiguous as
class Foo can either define a new class or reopen existing class, and RLisp tried to avoid this ambiguity. Of course it's possible to do both with a simple macro. It increased verbosity of code somewhat.In e7 class is also a constructor. In RLisp it's just an object. CLOS and goo seem to treat classes differently from other objects.
goo and CL provided only REPL environment by default, and didn't like running scripts. RLisp and e7 supported REPL and script mode without any extra hassle.
From a purely subjective point of view, I liked RLisp way most, what's not particularly surprising coming from RLisp's author ;-). Coding in e7 also felt good. On the other hand goo and especially CLOS felt rather unelegant and unpleasant.
Tuesday, June 05, 2007
Mock C backend for RLisp and its performance

When RLisp was first created it was purely interpreted, with an interpreter written in Ruby. It was about 50x slower than Ruby. Then it went through unsuccessful stage of mixed interpreted/compiled-to-Ruby mode, finally getting a real compiler with Ruby back-end. That's where it is now, and it is about 5x slower than Ruby.
The code generated by RLisp compiler doesn't need Ruby power, it just needs a few simple operations and access to Ruby runtime. That is - it could be RubyC code for matz's Ruby, or Java code for JRuby. If using C/Java compiler was too slow, the RLisp compiler could generate LLVM, x86 asm, or JVM code directly. It would be a little bit slower than going through C/Java, as we wouldn't run the optimizations, but the overhead is in Ruby runtime anyway, and I doubt the difference would be more than a few percent.
Before implementing the RubyC backend I wanted to measure how fast would it be and what would it look like. For that I used the Ackermann benchmark from the Great Programming Language Shootout, which Ruby by the way drastically loses. Ruby and RLisp implementations looks like that:
def ack(m, n)
if m == 0 then
n + 1
elsif n == 0 then
ack(m - 1, 1)
else
ack(m - 1, ack(m, n - 1))
end
end
NUM = Integer(ARGV.shift || 1)
print "Ack(3,", NUM, "): ", ack(3, NUM), "\n"(defun ack (m n)
(cond (eq? m 0) (+ n 1)
(eq? n 0) (ack (- m 1) 1)
(ack (- m 1) (ack m (- n 1)))
)
)
(let num [[ARGV shift] to_i])
(print "Ack(3," num "): " (ack 3 num))Compiled RLisp code, after some reformatting or readability:
module ::RLispC7054184
def self.ack(globals)
Proc.new do |*args|
m, n, = *args
t2 = m.send(:==, 0)
if t2
t4 = n.send(:+, 1)
t3 = t4
else
t5 = n.send(:==, 0)
if t5
t7 = globals[:ack]
t8 = m.send(:-, 1)
t9 = t7.call(t8, 1)
t6 = t9
else
t10 = globals[:ack]
t11 = m.send(:-, 1)
t12 = globals[:ack]
t13 = n.send(:-, 1)
t14 = t12.call(m, t13)
t15 = t10.call(t11, t14)
t6 = t15
end
t3 = t6
end
t3
end
end
end
t16 = ::RLispC7054184::ack(globals)
globals[:ack] = t16
t17 = ::ARGV.send(:shift)
t18 = t17.send(:to_i)
globals[:num] = t18
t19 = globals[:print]
t20 = globals[:num]
t21 = globals[:ack]
t22 = globals[:num]
t23 = t21.call(3, t22)
t24 = t19.call("Ack(3,", t20, "): ", t23)It should be pretty straightforward
globals is a Hash containing RLisp global variables, the three-level module/def/Proc.new procedure creation is used to- Make sure procedures with the same name won't accidentally overwrite each other.
- Make sure stack backtraces contain name
ack(instead ofack_CHECKSUM. - To build a closure if necessary. As
ackis global the only captured variable isglobals.
I compiled the
ack function to RubyC by hand. It follows generated Ruby code statement-by-statement, except that I didn't bother compiling things outside the ack function, which are executed only once and don't affect performance anyway. So the code consists of two files, Ruby file ackermann_driver.rb:require 'ackermann'
globals = {}
closure_ack = RLispCompiledCode.build_closure_ack_CHECKSUM(globals)
globals[:ack] = lambda{|*args| RLispCompiledCode.ack_CHECKSUM(closure_ack, self, args, nil)}
num = (ARGV.shift || 1).to_i
res = globals[:ack].call(3, num)
print "Ack(3,", num, "): ", res, "\n"and RubyC file
ackermann.c:#include <ruby.h>
#include <intern.h>
int ID_EQEQ, ID_PLUS, ID_MINUS, ID_call;
VALUE SYM_ack, SYM_globals;
static VALUE ack_CHECKSUM(VALUE rlisp_compiled_module, VALUE closure, VALUE self, VALUE args, VALUE block_arg)
{
VALUE globals, m, n, t2, t3, t4, t5, t6, t7, t8, t9, t10, t11, t12, t13, t14, t15;
globals = rb_hash_aref(closure, SYM_globals);
m = rb_ary_entry(args, 0);
n = rb_ary_entry(args, 1);
t2 = rb_funcall(m, ID_EQEQ, 1, INT2FIX(0));
if(t2) {
t4 = rb_funcall(n, ID_PLUS, 1, INT2FIX(1));
t3 = t4;
} else {
t5 = rb_funcall(n, ID_EQEQ, 1, INT2FIX(0));
if(t5) {
t7 = rb_hash_aref(globals, SYM_ack);
t8 = rb_funcall(m, ID_MINUS, 1, INT2FIX(1));
t9 = rb_funcall(t7, ID_call, 2, t8, INT2FIX(1));
t6 = t9;
} else {
t10 = rb_hash_aref(globals, SYM_ack);
t11 = rb_funcall(m, ID_MINUS, 1, INT2FIX(1));
t12 = rb_hash_aref(globals, SYM_ack);
t13 = rb_funcall(n, ID_MINUS, 1, INT2FIX(1));
t14 = rb_funcall(t12, ID_call, 2, m, t13);
t15 = rb_funcall(t10, ID_call, 2, t11, t14);
t6 = t15;
}
t3 = t6;
}
return t3;
}
static VALUE build_closure_ack_CHECKSUM(VALUE rlisp_compiled_module, VALUE globals)
{
VALUE closure = rb_hash_new();
rb_hash_aset(closure, SYM_globals, globals);
return closure;
}
void Init_ackermann()
{
VALUE mRLispCompiledCode;
ID_EQEQ = rb_intern("==");
ID_PLUS = rb_intern("+");
ID_MINUS = rb_intern("-");
ID_call = rb_intern("call");
SYM_ack = ID2SYM(rb_intern("ack"));
SYM_globals = ID2SYM(rb_intern("globals"));
mRLispCompiledCode = rb_define_module("RLispCompiledCode");
rb_define_module_function(mRLispCompiledCode, "ack_CHECKSUM", ack_CHECKSUM, 4);
rb_define_module_function(mRLispCompiledCode, "build_closure_ack_CHECKSUM", build_closure_ack_CHECKSUM, 1);
}The code is so scary that I don't even know where to start with explanations. The huge problem is that the compiled function
ack_CHECKSUM needs to take closure argument somehow, but Ruby is not going to provide it. So we need an ugly wrapper globals[:ack] = lambda{...} in the driver.As it turns out the performance is horrible, still over 2x slower than Ruby:
Ruby: 1.18s
RLisp: 5.93s (x5.0)
Mock binary RLisp: 2.48s (x2.1)Many optimizations are possible, but I thought the guilty party is the ugly wrapper lambda. This or some other wrapper needs to be called every time we call RLisp function from Ruby, but what if we somehow optimized compile RLisp to compiled RLisp calls ? To make it really work we would probably subclass
Proc to CompiledRLispProc and in the compiled code check wheather a Proc is of this particular kind, and if so, call it directly instead of using a wrapper. In this code we simply do that for calls do ack and nothing else. Here's the new driver ackermann_driver2.rb:require 'ackermann2'
globals = {}
closure_ack = RLispCompiledCode.build_closure_ack_CHECKSUM(globals)
globals[:ack] = lambda{|*args| RLispCompiledCode.ack_CHECKSUM(closure_ack, self, args, nil)}
num = (ARGV.shift || 1).to_i
res = globals[:ack].call(3, num)
print "Ack(3,", num, "): ", res, "\n"and the new hand-compiled code:
#include <ruby.h>
#include <intern.h>
int ID_EQEQ, ID_PLUS, ID_MINUS, ID_call;
VALUE SYM_ack, SYM_globals;
static VALUE ack_CHECKSUM(VALUE rlisp_compiled_module, VALUE closure, VALUE self, VALUE args, VALUE block_arg)
{
VALUE globals, m, n, t2, t3, t4, t5, t6, t8, t9, t11, t13, t14, t15;
VALUE new_args;
globals = rb_hash_aref(closure, SYM_globals);
m = rb_ary_entry(args, 0);
n = rb_ary_entry(args, 1);
t2 = rb_funcall(m, ID_EQEQ, 1, INT2FIX(0));
if(t2) {
t4 = rb_funcall(n, ID_PLUS, 1, INT2FIX(1));
t3 = t4;
} else {
t5 = rb_funcall(n, ID_EQEQ, 1, INT2FIX(0));
if(t5) {
t8 = rb_funcall(m, ID_MINUS, 1, INT2FIX(1));
new_args = rb_ary_new();
rb_ary_push(new_args, t8);
rb_ary_push(new_args, INT2FIX(1));
t9 = ack_CHECKSUM(rlisp_compiled_module, closure, self, new_args, Qnil);
t6 = t9;
} else {
t11 = rb_funcall(m, ID_MINUS, 1, INT2FIX(1));
t13 = rb_funcall(n, ID_MINUS, 1, INT2FIX(1));
new_args = rb_ary_new();
rb_ary_push(new_args, m);
rb_ary_push(new_args, t13);
t14 = ack_CHECKSUM(rlisp_compiled_module, closure, self, new_args, Qnil);
new_args = rb_ary_new();
rb_ary_push(new_args, t11);
rb_ary_push(new_args, t14);
t15 = ack_CHECKSUM(rlisp_compiled_module, closure, self, new_args, Qnil);;
t6 = t15;
}
t3 = t6;
}
return t3;
}
static VALUE build_closure_ack_CHECKSUM(VALUE rlisp_compiled_module, VALUE globals)
{
VALUE closure = rb_hash_new();
rb_hash_aset(closure, SYM_globals, globals);
return closure;
}
void Init_ackermann2()
{
VALUE mRLispCompiledCode;
ID_EQEQ = rb_intern("==");
ID_PLUS = rb_intern("+");
ID_MINUS = rb_intern("-");
ID_call = rb_intern("call");
SYM_ack = ID2SYM(rb_intern("ack"));
SYM_globals = ID2SYM(rb_intern("globals"));
mRLispCompiledCode = rb_define_module("RLispCompiledCode");
rb_define_module_function(mRLispCompiledCode, "ack_CHECKSUM", ack_CHECKSUM, 4);
rb_define_module_function(mRLispCompiledCode, "build_closure_ack_CHECKSUM", build_closure_ack_CHECKSUM, 1);
}It's over 15 times faster than current RLisp and over 3 times faster than Ruby:
Ruby: 1.18s
RLisp: 5.93s (x5.0)
Mock binary RLisp: 2.48s (x2.1)
Mock binary RLisp 2: 0.38s (x0.3)The road from a mock RubyC code to real RubyC back-end is a long one, but basically it proves what I always knew - RLisp can be much faster than Ruby.
Oh and the
Rakefile:task :default => ["ackermann.so", "ackermann2.so"]
def time_cmd(cmd)
cmd = "/usr/bin/time -f '%U+%S' #{cmd} 2>&1 >/dev/null"
raw_time = `#{cmd}`.chomp
raw_time =~ /\A(\d+\.\d*)\+(\d+\.\d*)\Z/ or raise "Cannot parse time: #{raw_time}"
$1.to_f + $2.to_f
end
desc "Run benchmarks"
task :benchmark do
ruby_time = time_cmd "ruby ./ackermann.ruby 6"
rlisp_time = time_cmd "./rlisp.rb ./ackermann.rl 6"
binrl_time = time_cmd "./ackermann_driver.rb 6"
binrl2_time = time_cmd "./ackermann_driver2.rb 6"
print "Ruby: #{ruby_time}s\n"
print "RLisp: #{rlisp_time}s (x#{sprintf "%0.1f", (rlisp_time.to_f/ruby_time)})\n"
print "Mock binary RLisp: #{binrl_time}s (x#{sprintf "%0.1f", (binrl_time.to_f/ruby_time)})\n"
print "Mock binary RLisp 2: #{binrl2_time}s (x#{sprintf "%0.1f", (binrl2_time.to_f/ruby_time)})\n"
print "\n"
end
desc "Compile ackermann.c"
file "ackermann.so" => "ackermann.c" do
sh "gcc -Wall -W -O6 -g -I/usr/lib/ruby/1.8/i486-linux/ -fPIC -shared ackermann.c -o ackermann.so"
end
desc "Compile ackermann2.c"
file "ackermann2.so" => "ackermann2.c" do
sh "gcc -Wall -W -O6 -g -I/usr/lib/ruby/1.8/i486-linux/ -fPIC -shared ackermann2.c -o ackermann2.so"
end
Monday, June 04, 2007
Some thoughts on RLisp style

Established languages tend to have more or less official style guides. Here are links to a few random style guides - Ruby, Python, Perl, Common Lisp, Smalltalk [PDF]. Style guides would be pointless for languages like C and Java, as coding in them requires utter disregard for aesthetics.
RLisp is still too young for a definite style guide, but I have a few tips. They're not "binding" in any way, not even for patch submitters for RLisp, and I am likely change my mind many times before 1.0 (if there will ever be 1.0).
- Use
fn/hd/tlinstead oflambda/car/cdr - Use 2-space indentation. Or maybe 4-space indentation. In any case don't use tabs.
- Function and variable naming should mostly follow Ruby and Scheme style:
- Functions which return boolean values should end with
?, like(empty? obj). - "Dangerous" functions should end with
!. Likesortfor returning sorted container versussort!for sorting it in-place.
- Functions which return boolean values should end with
- Message passing should usually be done with
[obj method arguments]instead of(send obj 'method arguments). In cases where it's possible to use either a message send[a == b]or a function call(eq? a b), pick your choice. - I consider strings of
car/cdrs a bad style. Usually you can pattern match, or use something like(nth a_list index), or(ntl a_list index), or[a_list get index], or[a_list get [index .. -1]]. I think any of these look better than a string ofcars andcdrs. - RLisp doesn't support non-ASCII (that is UTF-8, other character sets would definitely not be supported) characters in identifiers. I'm not decided yet if it would be a good idea to support them or not.
- There's no need to use excessive indentation. Because of different
(let ...)semantics RLisp programs can often be indented a lot less than Scheme/Common Lisp programs. - RLisp doesn't do tail call optimization yet, and Ruby stack isn't that big. That means Scheme-style excessive recursion is not a good idea. It should be fixed in the future.
- There's a simple code formatter for RLisp, but I don't think you should trust it too much.
- I sometimes place
)s and]s on their own lines, and sometimes together with the last expression line. A rough guide would be - place)on its own line if you'd useendin Ruby, and place it together with expression line if in Ruby you'd use}. That's not very helpful, asendvs}is a matter of personal style. Here's an example from the RLisp test suite, which might make it a bit more clear. Only)for(test-suite ...got its own line, and I think it would look weird if it didn't. Other)s don't really need own lines.
(test-suite Text_Processing
(test parse_ip
(assert (parse-ip "1.2.3.4") == '(1 2 3 4))
(assert (parse-ip "64.233.183.104") == '(64 233 183 104)))
(test parse_ip_2
(assert (parse-ip-2 "1.2.3.4") == '(1 2 3 4))
(assert (parse-ip-2 "64.233.183.104") == '(64 233 183 104)))
(test parse_ip_3
(assert (parse-ip-3 "1.2.3.4") == '(1 2 3 4))
(assert (parse-ip-3 "64.233.183.104") == '(64 233 183 104)))
)
A more definite RLisp style will only develop if a lot more RLisp code is written, preferably by a lot of people other than me.
Sunday, June 03, 2007
RLisp code formatter

Just after the highlighter, here's another important piece of language infrastructure - code formatter. For now it only re-adjusts amount of indentation. It doesn't move )s around, change other whitespace, or modify any code.
Here's an example. Before:
(defmacro cmd options
(let hash_index -1)
[options each_with_index &(fn (x i)
(cond
(and [hash_index == -1] [x is_a? Symbol] [[x to_s] =~ /:\Z/]) (set! hash_index i)
(and [hash_index == -1] [x == '=>]) (set! hash_index [i - 1]))
)]
(if (== hash_index -1)
(sendq options)
(do
(let normal_args [options get [0 ... hash_index]])
(let hash_args [options get [hash_index .. -1]])
`(sendq ,@normal_args (hash ,@hash_args))))
)After:
(defmacro cmd options
(let hash_index -1)
[options each_with_index &(fn (x i)
(cond
(and [hash_index == -1] [x is_a? Symbol] [[x to_s] =~ /:\Z/]) (set! hash_index i)
(and [hash_index == -1] [x == '=>]) (set! hash_index [i - 1]))
)]
(if (== hash_index -1)
(sendq options)
(do
(let normal_args [options get [0 ... hash_index]])
(let hash_args [options get [hash_index .. -1]])
`(sendq ,@normal_args (hash ,@hash_args))))
)
Friday, June 01, 2007
Active Record for RLisp

Every language needs a web framework. RLisp is in lucky position as it can reuse major parts of Ruby on Rails. So far Brad Ediger created RLisp plugin for Ruby on Rails, but that's just part of the story. Today I'm going to talk about a different part - the Object-Relational Mapper library Active Record.
Real databases are complex, ugly, and take hundreds of megabytes or more, so as an example I used a toy one by Mike Wilson, which describes music albums.
The original Ruby version looks like this, and we want to recreate it in possibly the nicest RLisp we can.
require 'active_record'
ActiveRecord::Base.logger = Logger.new(STDERR)
ActiveRecord::Base.colorize_logging = false
ActiveRecord::Base.establish_connection(
:adapter => "sqlite3",
:dbfile => ":memory:"
)
ActiveRecord::Schema.define do
create_table :albums do |table|
table.column :title, :string
table.column :performer, :string
end
create_table :tracks do |table|
table.column :album_id, :integer
table.column :track_number, :integer
table.column :title, :string
end
end
class Album < album =" Album.create(:title"> 'Black and Blue', :performer => 'The Rolling Stones')
album.tracks.create(:track_number => 1, :title => 'Hot Stuff')
album.tracks.create(:track_number => 2, :title => 'Hand Of Fate')
album.tracks.create(:track_number => 3, :title => 'Cherry Oh Baby ')
album.tracks.create(:track_number => 4, :title => 'Memory Motel ')
album.tracks.create(:track_number => 5, :title => 'Hey Negrita')
album.tracks.create(:track_number => 6, :title => 'Fool To Cry')
album.tracks.create(:track_number => 7, :title => 'Crazy Mama')
album.tracks.create(:track_number => 8, :title => 'Melody (Inspiration By Billy Preston)')
album = Album.create(:title => 'Sticky Fingers', :performer => 'The Rolling Stones')
album.tracks.create(:track_number => 1, :title => 'Brown Sugar')
album.tracks.create(:track_number => 2, :title => 'Sway')
album.tracks.create(:track_number => 3, :title => 'Wild Horses')
album.tracks.create(:track_number => 4, :title => 'Can\'t You Hear Me Knocking')
album.tracks.create(:track_number => 5, :title => 'You Gotta Move')
album.tracks.create(:track_number => 6, :title => 'Bitch')
album.tracks.create(:track_number => 7, :title => 'I Got The Blues')
album.tracks.create(:track_number => 8, :title => 'Sister Morphine')
album.tracks.create(:track_number => 9, :title => 'Dead Flowers')
album.tracks.create(:track_number => 10, :title => 'Moonlight Mile')
puts Album.find(1).tracks.length
puts Album.find(2).tracks.length
puts Album.find_by_title('Sticky Fingers').title
puts Track.find_by_title('Fool To Cry').album_id
It's a very straightforward database schema with just two tables. There's some duplication in code, especially all the
table.column, and album.tracks.create fragments, but overall it's a pretty nice code. Its direct translation to RLisp however leaves a lot to be desired.(ruby-require "active_record")
[ActiveRecord::Base logger= [Logger new STDERR]]
[ActiveRecord::Base colorize_logging= false]
[ActiveRecord::Base establish_connection (hash adapter: "sqlite3" dbfile: ":memory:")]
[ActiveRecord::Schema define &(fn()
[self create_table 'albums &(fn (table)
[table column 'title 'string]
[table column 'performer 'string]
)]
[self create_table 'tracks &(fn (table)
[table column 'album_id 'integer]
[table column 'track_number 'integer]
[table column 'title 'string]
)]
)]
(let Album [Class new ActiveRecord::Base])
(class Album
[self has_many 'tracks]
)
(let Track [Class new ActiveRecord::Base])
(class Track
[self belongs_to 'album]
)
(let album [Album create (hash title: "Black and Blue" performer: "The Rolling Stones")])
[[album tracks] create (hash track_number: 1 title: "Hot Stuff")]
[[album tracks] create (hash track_number: 2 title: "Hand Of Fate")]
[[album tracks] create (hash track_number: 3 title: "Cherry Oh Baby ")]
[[album tracks] create (hash track_number: 4 title: "Memory Motel ")]
[[album tracks] create (hash track_number: 5 title: "Hey Negrita")]
[[album tracks] create (hash track_number: 6 title: "Fool To Cry")]
[[album tracks] create (hash track_number: 7 title: "Crazy Mama")]
[[album tracks] create (hash track_number: 8 title: "Melody (Inspiration By Billy Preston)")]
(let album [Album create (hash title: "Sticky Fingers" performer: "The Rolling Stones")])
[[album tracks] create (hash track_number: 1 title: "Brown Sugar")]
[[album tracks] create (hash track_number: 2 title: "Sway")]
[[album tracks] create (hash track_number: 3 title: "Wild Horses")]
[[album tracks] create (hash track_number: 4 title: "Can't You Hear Me Knocking")]
[[album tracks] create (hash track_number: 5 title: "You Gotta Move")]
[[album tracks] create (hash track_number: 6 title: "Bitch")]
[[album tracks] create (hash track_number: 7 title: "I Got The Blues")]
[[album tracks] create (hash track_number: 8 title: "Sister Morphine")]
[[album tracks] create (hash track_number: 9 title: "Dead Flowers")]
[[album tracks] create (hash track_number: 10 title: "Moonlight Mile")]
(print [[[Album find 1] tracks] length])
(print [[[Album find 2] tracks] length])
(print [[Album find_by_title "Sticky Fingers"] title])
(print [[Track find_by_title "Fool To Cry"] album_id])
We can make it a lot better. One feature which Ruby DSLs often take advantage of is Ruby syntax for keyword arguments -
obj.foo(x, y, {:a => b, :c => d}) can be abbreviated to much better-looking obj.foo x, y, :a => b, :c => d. RLisp doesn't support such abbreviations directly, but it can do something similar with macros. Macro (cmd ...) defined below will support similar syntax - (cmd obj foo x y a: b c: d) expanding to [obj foo x y (hash a: b c: d)]. We can also provide (cmds obj (command one) (command two) ...) macro for common pattern of issuing multiple commands to the same objects, something like Smalltalk's ;.(defmacro sendq (obj meth . args)
`(send ,obj ',meth ,@args))
(defmacro cmd options
(let hash_index -1)
[options each_with_index &(fn (x i)
(cond
(and [hash_index == -1] [x is_a? Symbol] [[x to_s] =~ /:\Z/]) (set! hash_index i)
(and [hash_index == -1] [x == '=>]) (set! hash_index [i - 1]))
)]
(if (== hash_index -1)
(sendq options)
(do
(let normal_args [options get [0 ... hash_index]])
(let hash_args [options get [hash_index .. -1]])
`(sendq ,@normal_args (hash ,@hash_args))))
)
(defmacro cmds (recv . args)
(let tmp (gensym))
(let args-expanded [args map &(fn (a)
`(cmd ,recv ,@a)
)])
`(do ,@args-expanded)
)
These macros were pretty generic. Now let's build a simple wrapper for Active Record.
(ruby-require "active_record")
(defmacro active-record-establish-connection options
`(cmd ActiveRecord::Base establish_connection ,@options))
(defmacro active-record-schema-define defs
(let expanded-defs [defs map & (fn (d) (match d
('create-table . args) `(schema-define-create-table ,@args)
(raise "Unrecognized statement in schema definition: #{d}")
))])
`[ActiveRecord::Schema define &(fn() ,@expanded-defs)]
)
(defmacro schema-define-create-table (table-name . entries)
(let tmp (gensym))
(let expanded-entries [entries map &(fn (entry)
(let name [entry get 0])
(let type [entry get 1])
`[,tmp column ',name ',type])])
`[self create_table ',table-name &(fn (,tmp) ,@expanded-entries)]
)
(defmacro define-active-record-class (class-name . defs)
(let expanded-defs [defs map &(fn (d) (match d
(has_many . args) `[self has_many ,@args]
(has_one . args) `[self has_one ,@args]
(belongs_to . args) `[self belongs_to ,@args]
(has_and_belongs_to_many . args) `[self has_and_belongs_to_many ,@args]
_ d
))])
`(do
(let ,class-name [Class new ActiveRecord::Base])
(class ,class-name
,@expanded-defs
)
)
)
Finally Active Record can be used in a nice manner. I think it looks even nicer than Ruby version now.
(require "active_record.rl")
[ActiveRecord::Base logger= [Logger new STDERR]]
[ActiveRecord::Base colorize_logging= false]
(active-record-establish-connection adapter: "sqlite3" dbfile: ":memory:")
(active-record-schema-define
(create-table albums
(title string)
(performer string))
(create-table tracks
(album_id integer)
(track_number integer)
(title string))
)
(define-active-record-class Album
(has_many 'tracks)
)
(define-active-record-class Track
(belongs_to 'album)
)
(let album (cmd Album create title: "Black and Blue" performer: "The Rolling Stones"))
(cmds [album tracks]
(create track_number: 1 title: "Hot Stuff")
(create track_number: 2 title: "Hand Of Fate")
(create track_number: 3 title: "Cherry Oh Baby ")
(create track_number: 4 title: "Memory Motel ")
(create track_number: 5 title: "Hey Negrita")
(create track_number: 6 title: "Fool To Cry")
(create track_number: 7 title: "Crazy Mama")
(create track_number: 8 title: "Melody (Inspiration By Billy Preston)"))
(let album (cmd Album create title: "Sticky Fingers" performer: "The Rolling Stones"))
(cmds [album tracks]
(create track_number: 1 title: "Brown Sugar")
(create track_number: 2 title: "Sway")
(create track_number: 3 title: "Wild Horses")
(create track_number: 4 title: "Can't You Hear Me Knocking")
(create track_number: 5 title: "You Gotta Move")
(create track_number: 6 title: "Bitch")
(create track_number: 7 title: "I Got The Blues")
(create track_number: 8 title: "Sister Morphine")
(create track_number: 9 title: "Dead Flowers")
(create track_number: 10 title: "Moonlight Mile"))
(print [[[Album find 1] tracks] length])
(print [[[Album find 2] tracks] length])
(print [[Album find_by_title "Sticky Fingers"] title])
(print [[Track find_by_title "Fool To Cry"] album_id])
One more thing - as you can see from code examples (which hopefully weren't excessively mutilated by Blogger), the RLisp syntax highlighter got a lot smarter, and quasiquotes should be much easier to read.
Wednesday, May 30, 2007
Why RLisp will not support Ruby class variables

RLisp tries to be as Ruby-like as it reasonable can. For example it supports Global::Constants, @instance_variables, $global_variables, and self. It does not support semantically ugly features like Constants_Without_Fully_Specified_Scope, public/private/protected distinction, and Perlish magic variables of $~, $1 and family, trying to use cleaner solutions instead.
I'm adding another to the list of officially not supported features - @@class_variables. They simply lack any sensible semantics. The following code:
class Foo
@@xyzzy = 6
def hello
@@xyzzy
end
end
class Bar < Foo
@@xyzzy = 42
end
p Foo.new.hello
p Bar.new.hello
p Foo.new.instance_eval { @@xyzzy }in Ruby 1.8 produces:
42
42
semantics_of_cvars.rb:16: uninitialized class variable @@xyzzy in Object (NameError)
from semantics_of_cvars.rb:16:in `instance_eval'
from semantics_of_cvars.rb:16On the other hand in Ruby 1.9:
6
6
semantics_of_cvars.rb:16: warning: class variable access from toplevel singleton method
semantics_of_cvars.rb:16: uninitialized class variable @@xyzzy in Object (NameError)
from semantics_of_cvars.rb:16I don't remember ever using class variables, but I had a vague feeling
@@xyzzy would mean something like self.class.class_variable_get(:@@xyzzy). Apparently the meaning is much uglier and frankly I cannot see any situation in which it could possibly be useful in Ruby and even less so in RLisp - unlike Ruby def, RLisp fn/defun/method/... do not wipe out surrounding context, so we can use plain local variables.(class Object
(let xyzzy 42)
(method hello () (print xyzzy))
; xyzzy is a normal lexical variable visible until we exit class definition
)
; not visible any more
[self hello] # => 42
Token-based syntax highlighter for RLisp

One of the most important parts of language infrastructure is a good syntax highlighter. I'm not claiming the highlighter I just committed to the repository is any good - it simply reuses the lexer, and outputs tokens in appropriate <span style='color: #00FF00'>...</span>. Making it use CSS classes would be nicer, but then I wouldn't be able to use it on my blog :-)
A more serious syntax highlighter should at least recognize what's quoted and what's not, which symbols are standard macros and control structures, and so on. Still, it's much better to have some syntax highlighting than none.
#!/home/taw/everything/rlisp/trunk/rlisp
(require "rlunit.rl")
; It's much easier to temporarily replace (gensym)
; with a mock one than to design an assertion system
; which can handle the real one.
(defun start-readable-gensym ()
(let sid 0)
(set! gensym
(fn ()
(let sym [(+ "tmp-" [sid to_s]) to_sym])
(set! sid (+ 1 sid))
sym
)
)
)
(defun restore-correct-gensym ()
(set! gensym (fn () (ruby-eval "$gensym||=0; $gensym+=1; ('#:G' + $gensym.to_s).to_sym")))
)
(test-suite Readable_Gensym
(test gensym
(start-readable-gensym)
(assert 'tmp-0 == (gensym))
(assert 'tmp-1 == (gensym))
(assert 'tmp-2 == (gensym))
(restore-correct-gensym)
(assert [(gensym) to_s] =~ /\A#:G\d+\Z/)
(start-readable-gensym)
(assert 'tmp-0 == (gensym))
(assert 'tmp-1 == (gensym))
(start-readable-gensym)
(assert 'tmp-0 == (gensym))
(restore-correct-gensym)
(assert [(gensym) to_s] =~ /\A#:G\d+\Z/)
(assert (gensym) != (gensym))
)
(method teardown () (restore-correct-gensym))
)


