The best kittens, technology, and video games blog in the world.

Thursday, June 12, 2008

Bolting Aspect Oriented Programming on top of Python

What's this? by Steffe from flickr (CC-NC-SA)
The bigest difference between native support and bolting things on top of a programming language is that you can only bolt so much before things start to collapse. In C++ even strings, arrays, and hashtables are bolted on - and while they work just fine any interoperability between different libraries using strings, arrays, and hashtables is almost impossible without massive amount of boilerplate code.

In Perl and Python these basic data structures are native and well supported, but the next step of supporting objects is bolted on. So the objects work reasonably well, but metaprogramming with them is very difficult and limited (in Python) or outright impossible in any sane way (in Perl).

Ruby takes a step further and real object-oriented programming is native, so people can bolt other things on top of it like aspect-oriented programming. AOP in Ruby (before_foo, after_bar, alias_method_chain, mixins, magical mixins, many method_missing hacks etc.) works reasonably well, but I wouldn't want to bolt anything on top of that, or the whole thing would fall apart.

This is the problem with bolting stuff on - bolting stuff on in a valid technique (just like design patterns, code generation and other band-aids), and bolted-on stuff like objects in Perl/Python or arrays/strings/hashtables in C++ do work, they're just infinitely less flexible than native types when it comes to further extending.

But I really miss AOP in Python. Multiple inheritance can kinda simulate very weak kind of mixins, but is rather cumbersome to use. I wanted to write a test suite using aspect-oriented mixins, but there were simply so many super(omg, who).made(up, this, syntax) calls that it looked as painful as Java inner classes. So I thought - would it already collapse if I added a very simple AOP support?

It turned out not to be so bad. Here's a distilled example. There's a bunch of classes inheriting from BaseTest. Their setup methods should be called from superclass down to subclass, while their teardown methods should be called from subclass up to superclass. If there are multiple AOP methods on the same level all should be called, in some consistent order (I do alphanumeric, order of definition would be better but Python metaprogramming isn't powerful enough to do that). It's also possible to override parent's AOP methods (you could even compose AOP method override using super if you really needed). Or you could override the whole setup/teardown method if you really needed - this is very flexible.

class BaseTest(object):
def setup(self):
aop_call_down(self, 'setup')
def teardown(self):
aop_call_up(self, 'teardown')

class WidgetMixin(object):
def setup_widget(self):
print "* Setup widget"
def teardown_widget(self):
print "* Teardown widget"

class Foo(BaseTest):
def setup_foo(self):
print "* Setup foo"
def teardown_foo(self):
print "* Teardown foo"

class Bar(WidgetMixin, Foo):
def setup_bar(self):
print "* Setup bar"
def teardown_bar(self):
print "* Teardown bar"

class Blah(Bar):
def setup_blah(self):
print "* Setup blah1"

def setup_widget(self):
print "* Setup widget differently"

def setup_blah2(self):
print "* Setup blah2"

def teardown_blah(self):
print "* Teardown blah1"

def teardown_blah2(self):
print "* Teardown blah2"

The output of a = Bar(); a.setup(); a.teardown() is exactly what we would expect:
* Setup foo
* Setup widget
* Setup bar
* Teardown bar
* Teardown widget
* Teardown foo

The more difficult case of b = Blah(); b.setup(); b.teardown() is also handled correctly - notice that setup of widget mixin was overriden:
* Setup foo
* Setup widget differently
* Setup bar
* Setup blah1
* Setup blah2
* Teardown blah2
* Teardown blah1
* Teardown bar
* Teardown widget
* Teardown foo

The code to call make it possible isn't strikingly beautiful but it's not any worse than some of my Django templatetags.

def aop_call_order(obj, prefix):
already_called = {}
for cls in reversed(obj.__class__.mro()):
for name in sorted(dir(cls)):
if name[0:len(prefix)+1] != prefix + '_':
if not already_called.has_key(name):
already_called[name] = True

def aop_call_up(obj, prefix):
for name in reversed(list(aop_call_order(obj, prefix))):
getattr(obj, name)()

def aop_call_down(obj, prefix):
for name in aop_call_order(obj, prefix):
getattr(obj, name)()

aop_call_order returns a list of methods with names like prefix_* defined in obj's ancestor classes in order of Python's multiple inheritance resolution, falling back to alphabetic if they're on the same layer. Overriding a method in subclass doesn't affect the order, making the "Setup widget differently" trick possible. aop_call_down and aop_call_up methods then call these methods in straight or reverse order.

Of course like all other multilayer bolted-on features, it's going to horribly collapse if you use it together with other metaprogramming feature. If you don't like that - switch to Ruby.

Coming up next - bolting closures on top of Fortran.


toidinamai said...

What do you mean by "the objects work reasonably well, but metaprogramming with them is very difficult and limited (in Python)"? I found metaprogramming in Python (using metaclasses for example) very powerful.

Also, sorting methods by definiton is possible (but not very pretty):

import types

class A(object):
....def method_a(self):

....def method_b(self):

....def method_c(self):

....def method_d(self):

print sorted(
........(f for f in vars(A).values() if isinstance(f, types.FunctionType)),
........key = lambda f: (f.func_code.co_filename, f.func_code.co_firstlineno))

Hans said...

"""In Perl and Python these basic data structures are native and well supported, but the next step of supporting objects is bolted on."""

I'm not sure what "bolted on" means here... if you are trying to say that Python originally had no OO (and therefore it was added later), that is incorrect. This seems to be a popular misconception, possibly due to the fact that Python has been a multi-paradigm language from the start, supporting procedural, functional (to some extent) and object-oriented programming, rather than being OO-only.

Kamil Dworakowski said...

I had a lot to do with AOP, and must say that I would be far from calling this aspect oriented. Here is a link to a paper that argues that there are two properties which a language must have in order to be aspect oriented, quantification and obliviousness.

taw said...

Kamil Dworakowski: I know I'm kinda overstating my case, and this is not a real full-blown AOP, but I still think it's a valuable AOP-ish feature, and it makes it easier to decompose your program into elements that deal with a single concern at time.

Goldfish said...

Spring Python offers an AOP solution for python. Due to python's dynamic nature, this was very easy to implement as a simple library.

Imagine you had written a python service called SampleService that looks like this:
class SampleService:
....def method(self, data):
........return "You sent me '%s'" % data

To use it:
service = SampleService()
print service.method("some data")

The results look like:
You sent me 'some data'

Later on, you want to develop an aspect that wraps the results with some special text. For Spring Python, all you have to do is extend the MethodInterceptor class and override the invoke method:
from springpython.aop import *
class BeforeAndAfter(MethodInterceptor):
....def invoke(self, invocation):
........results = "=BEFORE=" + invocation.proceed() + "=AFTER="
........return results

To wire this aspect to your service:
from springpython.aop import *
service = ProxyFactoryComponent(target = SampleService(), interceptors = [BeforeAndAfter()])

Now, the same code you called before:

...will return...
==BEFORE==You sent me 'Hello'==AFTER==

Any calls to the "service" object are routed through the interceptor you wrote first and then to the real SampleService.

You can put multiple interceptors in that list, making a chain of calls. Other support classes make it easy to use patterns in picking which methods get which advise applied.

Spring Python has already utilized this solution to implement things like security interceptors, @Transactional python decorators and so forth. You can use the pre-built aspects, or write your own.

gregturn said...

FYI: Spring Python's web pages have been updated. Go to, and you can navigate to reference documentation, forums, mailing lists, etc. from there.

Anonymous said...

haha, this is of course an old blog post.. and I hope the author is not as juvenile these days.