taw's blog

Friday Night Magic - Red Deck Wins

2011-05-01T11:26:00.000+02:00

On Friday (that would be 2011-04-29) I took part in my first Magic: The Gathering tournament ever, and I ended up winning it. I never drafted before so I made some notes afterwards and here they are.

Draft

The draft was one Mirrodin Besieged booster, and two Scars of Mirrodin boosters.

In case you haven't been following Magic, both sets are full of artifacts, almost purely monocolor, have moderate amount of infect, and use metalcraft mechanic, where various cards get significant bonuses if you control 3 or more artifacts.

In three boosters you'll see - 3 rares/mythics, 9 uncommons, 30 commons, plus as many basic lands as you want, so the deck of 40 cards you need to build will be based on mostly crap commons and crap uncommons. This is far worse than even 100% common Pauper format - there you at least get to choose any commons you want. In booster draft it's mostly commons others didn't want.

Power creep took place primarily on rare and mythic rare levels, so unlike with constructed, limited's power levels are very low. Majority of the cards will be stuff considered unplayable elsewhere, and the few "playable" cards will lack any consistency or synergy you'd normally expect if you could build the deck from larger set of cards.

The way draft works, everybody opens one booster, picks one card, then passes the rest to the person on the left, and it continues until all cards have been used. Then second booster goes to the right, and finally third again to the left.

This introduces a lot of luck into the process. Obviously people will take the best cards in their first few picks, so the first few cards are the most important. The only reason they would not take a great card if it was incompatible (most likely of wrong color) with the deck they got so far.

So if you're trying to build a deck very similar to what people next to you are making, they'll take all the good cards, and you're screwed. On the other hand if they build something in a completely different color, you'll get a lot of great
cards you can use. Where you sit decides a lot.

In Mirrodin Besieged booster draft you get to make three big decisions:

which colors to use
use a lot of artifacts and try to get metalcraft bonuses or not?
use infect or not?

My constructed experience of playing multicolor decks without dual lands and other solid mana fixing told me to never do that again. In constructed, multicolor requires expensive mana base, and that's the end of it.

I was quite strongly determined to go monocolor even with crap cards, or at worst mainly monocolor with just a splash of a second color but only if I really really have to. At least due to lack of any dual lands whatsoever it wouldn't matter if it's ally or enemy color.

Because I wanted monocolor and I couldn't really count on having enough cards of one color - artifact-heavy metalcraft deck was a totally obvious choice. I didn't want to use infect as infect is an all or nothing choice - and with red having so few decent infect cards that wasn't even much of a choice.

By the way after the tournament I found out I had the only monocolored deck of all players.

Deck

So here's my deck:

1x Memnite
1x Iron Myr
1x Blisterstick Shaman
1x Koth's Courier
2x Snapsail Glider
2x Blade-Tribe Berserkers
2x Ogre Resister
1x Oxidda Scrapmelter
1x Flameborn Hellion
1x Galvanic Blast
1x Rally the Forces
2x Assault Strobe
1x Metallic Mastery
1x Slagstorm
1x Golden Urn
1x Infiltration Lens
1x Panic Spellbomb
1x Golem's Heart
1x Trigon of Rage
1x Necrogen Censer
1x Semblance Anvil
15x Mountain

At first, I even wanted to go with 14 lands - 1/3 lands is entirely reasonable in constructed, right? Especially monocolor. Double especially monored.

But such low land counts rely purely on abundance of 1-drops and 2-drops, and here there are very few. Most people had 16-18 lands as far as I can tell and that's far more reasonable.

Very early I got rid of one card - Semblance Anvil - and replaced it with 16th mountain. Together with Iron Myr that increased count of mana sources to 17, which was totally reasonable.

I had no idea how bad Semblance Anvil would be. I had it three times in my hand, and every single time it was a dead card. It costs you two cards (a huge cost) for some mana savings, and it is only useful if vast majority of your spells have the same type. Mostly artifacts, or mostly creatures, or maybe some tribal tricks. This just never ever happens in draft. Even if I had two creatures and Semblance Anvil in my hand, I would have to exile one of them and waste a turn just to cast the second one more cheaply the next turn. In almost every case an extra Mountain instead would let me cast it turn earlier, and without wasting the other guy. Usually it was even worse.

I didn't do any other sideboarding - I simply had no red left, nor artifact cards that weren't total garbage.

Games

There were four rounds. My record was:

2-0 against Blue-Black
2-0 against Red-Green-Blue
2-1 against Green-White
2-0 against Red-White

I also played two more games outside the tournament with this deck, losing one and winning the other. I only had to mulligan in one game, down to 5 cards. I had some not terribly good cards, but I went ahead with them as draft doesn't need to worry about opponent winning on turn 3-4 the way constructed has.

Many of the games were very close.

Some of the decks I played against had much better cards, whet considered individually. Sword of Body and Mind and Thrun, the Last Troll have hit the board, but in both cases their timing was too late and they couldn't do much. Even many less powerful card were simply better than anything I had.

I think all decks I played against - and probably most of those I haven't - suffered from more or less serious consistency problems. Many times I won thanks to my opponent being color screwed. Quite a few times they were mana screwed or mana flooded, and their curve didn't work too well with what they got.

Many people have splashed some infect. I understand why they did it, as individually they were good cards, and there's only so much you get to work with, but most cards which are good in infect decks, are bad in decks which just splash infect.

In all games I wasn't countered even once, I had my cards removed in various ways quite often. Some of my opponents tried to get metalcraft as well, but they seemed to have harder time doing this, probably they simply had fewer artifacts.

At least that's my theory. It's entirely possible that I'm wrong and I simply got lucky this time ;-)

Cards

Now I'll talk about all cards individually. I'll order the cards from the best to the worst.

The best card was my only rare Slagstorm. It saved the game when I used it to clean the board when I was behind. Once I attacked with everything, the opponent let some damage through, and then I cleaned the board of everything. It won the game a few times - people defend their last point of life with everything they got but it's much easier to get them down to 3, and then it's over.

Oxidda Scrapmelter was really awesome in this deck, as pretty much everyone played with some artifacts. And 3/3 for 4 mana in this format is good.

Iron Myr generates mana, metalcrafts, and has 1/1 body which can be surprisingly useful.

Infiltration Lens is normally a really good card, but when it doubles as metalcraft pumper it's just amazing.

Blisterstick Shaman was usually 2-for-1, but at the very least it damaged the opponent one life.

Flameborn Hellion - 5/4 with haste and totally irrelevant downside for 6cmc - was my most common wincon. It often got 5 damage the turn it entered the battlefield, and then the opponent would get desperate, stop their attacks, and squandered their resources on chump blocking while my side of the board got bigger and bigger. When I managed to equip it with Infiltration Lens it was game over same turn.

Koth's Courier was just amazing as long as my opponent played green. 2 damage per turn means 10 turn clock, and it doesn't really matter on its own. But its presence on the board made every other threat at least twice as thretening, and the opponent would desperately throw all their resources at doing something at least about the other one - even at very unfavourable exchange, and that loss of resources would lose them the game.

Blade-Tribe Berserkers were really awesome as long as I had metalcraft - but more often than not I was out of luck here and I missed an artifact or two. With metalcraft it's essentially 4cmc 3/3, opponent sacrifices a creature or gets 6 damage. Almost as good as Oxidda Scrapmelter if it was only more consistent.

Golem's Heart was very good. As both me and the opponent had plenty of artifacts, it would usually get me enough life to win aggro race while pumping metalcraft. It was the target of opponent's removal spells more often than any other card, including my biggest threats.

Galvanic Blast was able to get rid of pretty big creatures if I had metalcraft. I had to use it without metalcraft once, but then the creature I wanted to get rid of was something small, so it didn't really matter.

Necrogen Censer is 4 damage to player for 3 mana. Normally it would be somewhat too slow, but here it helps with metalcraft.

Ogre Resister was a decent early fattie, unplayable in constructed but pretty good in draft.

Memnite mostly pumped metalcraft, I didn't have many other uses for it, and I rarely even wanted to exchange it for another 1/1 or I'd lose metalcraft bonus.

Snapsail Glider was only OK because it pumped its metalcraft, so its chances of gaining flying were higher than usual. In constructed it would be a crap card - even 2/2 unconditional flier for 3cmc would be highly unimpressive.

Metallic Mastery is an artifact version of Act of Treason, but I'd rather have Act of Treason. I only used it once to steal Myr Propagator for a turn and use it to make
myself a copy. For the rest of the game we kept creating and wasting Myr Propagators. There aren't really that many non-creature artifacts I could steal, and those I could for example sac for good effect would get prematurely sacced in response to my Metallic Mastery anyway, so it would be a waste.

Trigon of Rage was very hard to use well. Mostly metalcraft pumping.

Golden Urn was slower lifegain than Golem's Heart. Mostly metalcraft pumping.

Rally the Forces looks like a very nice card, but in all my games I never had a good opportunity to use it.

Panic Spellbomb only served to boost metalcraft, or as emergency card draw.

Assault Strobe was so often almost what I needed, except I couldn't use it because of its sorcery speed.

The single biggest disappointment was Semblance Anvil, as I mentioned before.

Future

Is my booster draft experience normal? I'd have expected a lot more monocolored decks, but I had the only one. We just had two sets with plenty of artifacts, now we'll get one with easily splashable phyrexian mana. Shouldn't more people try to build consistent monocolor decks?

My Christmas wishlist - redcar and desktop Linux

2010-12-10T10:16:00.002+01:00

Who would have thought my blog would be so high on OSX vs Linux. Well, this subject never gets old, so here's another instalment.

For personal reasons which Facebook aptly calls "It's complicated" I haven't had much time to update this blog recently.

I have so many things I'd like to write about - but marginal utility of time keeps getting in a way - with a lot of my previously free time no longer available, I have to make decisions which of too many things I'd like to do I should drop or postpone, hopefully starting from the least important of them.

And just like nearly everybody else, I keep confusing important with urgent - this blog is never really urgent - no cat will die if I leave writing for another day or week - but it surely is not the least important of all things I could be doing, or have been doing in the last month or so. So quadrant two, here I come!

Macs suck

GPU in my MacBookPro failed, so I'm temporarily back on Linux, on a desktop box which I was using mostly for gaming.

I will probably stay on Linux for quite some time, even after I get my laptop back. And they're really slow with repairs - I will forgive them only if they keep my hard drive undisturbed like they promised - my backups were not terribly up to date (about a month old) and what's worse - didn't include everything (just imagine - I don't have kitten picture downloader compatible with the most recent flickr API and I have to download kittens manually!). Now I backup everything to two independent external USB disks just to be sure, so I was prepared for this kind of backup failure, but I don't backup as often as I should - mostly because dual backup is such a pain.

Anyway, all cost of switching from one system to another must be paid up front - all the time necessary to setup and upgrade the new system, to change my habits and so on. Once the cost is paid (and with this post it will mostly be so), I can as well continue using it - so I might very well wipe out OSX and install Ubuntu on the laptop once I get it - I haven't decided it yet.

Well, first here's the list of the most important things how Linux (kUbuntu, old install upgraded to 10.4) is way better than OSX:

Polish Dvorak keyboard, thanks to divide. I kept meaning to write one for OSX, but never quite did it. You can write keyboard layouts for OSX as XMLs these days, but sadly there seems to be no way to get XMLs of existing layouts as starting point.
apt-get. Seriously, MacPorts is such a pile of fail compared to apt-get, it's just ridiculous. Imagine that - with apt-get you can upgrade system without breaking anything. It just works! I know, it's crazy!
I mostly develop for EC2, so quite a few things are easier on Linux than on OSX.
konsole and virtual desktops completely destroy Terminal.app and Spaces.

Sadly from this point it only gets worse.

Desktop Linux sucks

When you keep using the same system for long, you stop noticing some problems - you just get used to them, just like Congolese got used to malaria.

And this is actually a big fucking deal. People "in charge" of desktop Linux - be in GNOME, KDE, or whatever - are mostly old-timers. They got so used to all the fail present in desktop Linux, it doesn't seem like a big deal to them.

The last non-Linux system they used regularly is probably Windows XP or something even older - they have only very faint idea what progress the closed source world have made with OSX and even Windows 7. Did I ever mention I sort of like Windows 7? Not for development of course, but you just don't see fail jumping at you all the time like it used to be on XP and Vista.

Anyway, the first impression I had once I switched from OSX to KDE was:

Are they fucking trying to reimplement Windows XP here?

I am deeply disturbed by what I've seen. KDE not only lacks all the awesome new stuff you get in OSX, it also lacks all the awesome old Unix stuff you get in OSX.

Let's start with the basics - a text editor. Control-A and Control-E don't work. A shortcut as convenient as Control-A instead does something as incredibly useless as selecting entire document. Control-E seems to do nothing. Just like Control-D. Control-K deletes entire line, which is broken, but at least only halfway so.

Fine, whatever, I thought - Windows noobs also use KDE, and if they want such broken shortcuts it's their problem - I'll just go to keyboard settings and switch to Emacs mode.

It's such a shame I haven't been recording myself, as the very moment I realized they removed Emacs mode I produced such an awasome stream of Polish and English swear-words that you could write a research paper about bilingual cursing based on it.

And amazingly, even mcedit lacks Emacs keybindings these days. How is it even possible that even the supposedly most noob-friendly operating system OSX has standard Emacs keybindings, but Linux doesn't even support them as an option any more? It doesn't even seem possible to set them up shortcut by shortcut.

At least they still work in konsole+bash, and in gtk-based programs if you add gtk-key-theme-name = "Emacs" to your ~/.gtkrc-2.0.

Unfortunately even that much barely works. In some contexts Control-F means forward, in some it means search. Control-W changes meaning between close tab and delete previous word - it's an unreliable mess.

They way OSX does it is simple - one modifier key (Control) is used for text editing, another (Command) for application control. I understand Linux cannot really follow this way, but would it really be that difficult to have a single global mapping between key presses and actions?

When you think about it most programs use the same shortcuts:

text editing (control-A, control-E, ...)
copy&paste (command-C, command-V, command-X)
undo stack control (command-Z, command-Y)
document/tab control - save (command-S), load (command-L), close document/tab/window (command-W), new (command-N) etc.
switching between tabs - command-1/2/3/..., control-pageup/pagedown etc.
application control - let's not get into application/window/sdi/mdi mess, there's time for everything and it isn't now - mostly quit (command-Q)
window control - maximize, minimize etc.
system control - like switching between windows/applications, making screenshots, locking screen, some dashboard, start menu, volume up/down/mute - the list is pretty long, but they're global by definition

There are very few applications that really need more than a few important shortcuts to this list. Some of them are important applications (Office, Photoshop, Emacs), but you'll be mostly using these anyway.

I would be surprised if GNOME/KDE lacked infrastructure for this (what's gtk-key-theme if not half of this?) - it just that nobody bothered doing this consistently. Or if they did, their efforts were blocked by some old-timer whose last recollections of life outside desktop Linux is Windows 3.11.

My Christmas wish: Good keybindings for Linux that won't make me miss OSX, or Linux from a decade ago.

It cannot be that hard, as Linux used to have those before losing them very recently.

Redcar

Let's continue the story. Kate and gedit were various levels of unusable due to keybindings, as was mcedit in Emacs mode which I mostly use over ssh instead of real Emacs. I could try the actual Emacs, but after a few years of TextMate it just isn't the same. Not even close.

Fortunately the totally awesome Daniel Lucraft wrote a TextMate-like editor in JRuby redcar. Every time I check it it gets more and more awesome, and it's safe to say it's already far better than any other editor you can get on Linux.

Of course I wouldn't be myself if I didn't start bitching. So here's my quick list of easy improvements to redcar.

Any of these will make a great Christmas gift for me:

Context menu absolutely needs TextMate-like "Filter through command" features and all its options (mostly "input selection / output selection", "input selection / output create new document", and "input document / output create new document", but others are occasionally useful as well). This is easily one of top five most useful features of TextMate ever and it's really not that hard.
Getting a lot of common bundles/plugins/snippets/whatever-else-it-is-using with a single command. Batteries included you know.
Tab size control really needs to be visible by default. I'm sure this is one of the first thing requested by half of the redcar users, so why not just do it?
Support for Control-A and Control-E, seriously. Patch on github.
Syntax highlighting often breaks. Now making it unbreakable would require massive architectural charget, but can we at least get easy refresh shortcut, or refresh on switching tabs (if edited since last switch)? Most documents are not that huge, so it shouldn't be a major performance problem, right?
Does it really need to create .redcar directories everywhere? It creates ~/.redcar/ already, won't that be enough?
Unixy interface like mate command. Right now starting redcar ~/some/dir from console makes it stay there and keep producing java.lang.IllegalArgumentException stack traces. It's definitely a good thing for developing redcar, but would it be possible to get some command line switches to make redcar fork away like mate command does?
TextMate-compatible Control-K - it should delete everything between cursor and end of line (right), or delete linebreak if cursor is already at end of line. Right now in redcar pressing Control-K multiple times doesn't do anything useful. Patch on github.
Can we get some sort of 2D selections? Even Firefox has them for tables with control-mouse. I understand they might be too hard architecturally, but it would be pretty awesome. Ctrl+B does exactly that already.
Control-T (search file) should ignore spaces like in TextMate. I know this behaviour is irregular, but it just makes so much more sense to me to search for "login ctr test" like in TextMate than for "loginctrtest" redcar wants. By the way history for Control-T is pretty awesome.Patch on github.
Search and replace within selection.
Not in TextMate but would be awesome: support for filtering by full path with Control-T, like login/index.

That was a lot of bitching - but then isn't this exactly what this blog is made for?

Yes, I'm aware of official redcar wishlist and I might convert some of this bitching to requests and maybe even - imagine that - patches (I've been doing quite a lot of JRuby lately, so I'm no longer scared of JVM). Just not now. Marginal utility of time, "it's complicated", delegate if possible and so on... You know, the usual excuses for not contributing.

EDIT: I fixed a few easy ones at github.

I'm not done yet

Silly me, I thought I would be able to include my Ruby standard library wishlish in this post - this will have to wait, there are two weeks left so I might find an evening for that as well - and that will be a ridiculously long list - and judging from reactions to my preliminary post about Ruby stdlib it will make everybody hate me even more than they hate me now.

So just a few minor additions to my Christmas wishlish, just in case robot Santa or spirit of Kwanzaa or your personification of gift giving of choice happens to be deluded enough to think I was nice enough that my wishlist so far is somehow not yet long enough:

Is there anything like OmniFocus for Linux?
Sound system on Linux is still broken in 2010. Skype mysteriously cannot find my microphone, even after spending far too much time trying various alsamixer and Skype options. Rebooting to Windows on same hardware - zero problems, works out of the box. Can we please get this?
System settings are just dreadful. Why cannot we have everything in one place, instead of some options in General, some in Advanced, some inside various programs, and some who knows where. This has always been a big weakness of Linux, but total lack of progress is disturbing. Even something as simple as telling Linux to automatically mount an external disk is nontrivial - in dolphin there's nothing like it in context either on disk icon on sidebar, or disk mountpoint directory in /media. Searching "usb" or "external" in system settings also returned nothing. It turns out I need to switch to "Advanced" tab and search for "removable". I seem to recall this used to work far better than that, but maybe that was GNOME. Is anybody working to improve this situation?

By the way is there any major reason for using KDE versus GNOME these days? It used to be so that GNOME had much worse Windows-itis and hypernoobemia than KDE, but these days I'm getting the impression that noobness rankings changed from Mac > Windows > GNOME > KDE to Mac > Windows > KDE = GNOME > Mac. Yes - somehow Macs dominate both total noob and expert user part of the spectrum.

Or actually - the person who does the most to fulfil my wishlist gets to decide KDE vs GNOME for me.

Blogging about Robin Hanson is not about blogging about Robin Hanson

2010-10-12T22:48:00.002+02:00

This post is a semi-confession, and the difference between "I find X" statements and "X" statements is significant.

When I don't like something, I can of course come with some reasonable-sounding arguments specific to the particular subject, but usually "X is not really about X, it's just status seeking game gone awry" has no trouble getting onto such list. Sometimes high, sometimes not, but it's rarely missing.

When I like something, I tend to believe "X is not about X" type of arguments don't really apply - maybe there's some status seeking involved, but it surely cannot be that significant.

So here's my main point - "X is not about X" is not about X not being about X. It is a cheap and hard to refute shot at X to lower its status - essentially "X is not about X" argument is really about raising speaker's status relative to X.

Science is not about science

For example I find it intuitively obvious that academic science is mostly silly status game, with just tiny fraction of effort directed towards seeking important truth about reality.

How else can you explain:

persistent disregard for statistics, reliance on cargo cult statistics like p-values and PCA
vast majority of papers not getting even a single replication attempt
most research being about irrelevant minutiae
nobody bothered by research results being not publicly available online
review by "peers", not by outsiders
obsession about citations
obsession regarding where something gets published
routinely shelving results you don't like

This is exactly what we'd expect if it was just one big status seeking game! Truth seeking would involve:

everything being public from even before experiments start
all results being published online immediately, most likely on researcher's blog
results being routinely reviewed by outsiders with most clue about statistics and methodology at least as much as by insiders with domain knowledge
journals wouldn't exist at all
citations wouldn't matter much, hyperlinks work just as well - and nobody would bother with silliness like adding classics from 1980s to bibliography like it's still commonly done in many fields
most research focusing on the most important issues
vast majority of effort being put towards reproducing others' research - if it was worth studying, it's worth verifying; and if it's not worth verifying, why did anyone bothered with original research in the first place?
serious statistical training as part of curriculum in place of current sham in most disciplines

It's a miracle that anything ever gets discovered at all! It would be a very cheap shot, but it's really easy to finish this line of reasoning by attributing any genuine discovery as an attempt at acquiring funding from outsiders so the status game can continue. And isn't there a huge correlation between commercial funding of science and rate of discovery?

Medicine is definitely about health

And here's something I like - modern medicine. It's just obvious to me it's about health and I find claims to the contrary ridiculous.

Yes, plenty of medical interventions have only weak evidence behind them, but this is true about anything in the universe. Hard evidence is an exception (also see previous section) and among few things having hard evidence, medical interventions are unusually well represented.

And yes, a few studies show that medical spending might not be very beneficial on the margin - mostly in short term, mostly in United States, mostly on small samples, always without replication, and I could go on like this.

Anyway, modern medicine has enough defenders without me getting involved, so I'll just stop here. To me it just looks like what you'd get if it was about health, and it doesn't look like what you'd get if it was about status seeking, even if a tiny bit of it gets into it somehow.

But notice how the argument goes. Conclusions first, and you can always fit or refute status seeking in arguments later. This bothered me until I've seen even bigger point - it all works as well when you replace "seeking status" with "making money" or any other motivator!

Compare these semi-silly examples:

Writing books is not about content, it's about money
Writing blog posts is not about content, it's about status
Food industry is not really about food, it's about money
Organic food is not really about food, it's about status
Commercial software is not really about software, it's about money
Open Source is not really about software, it's about status

This is just universal and an argument like that can be applied to nearly everything. But in all these money and status are only mechanisms of compensation - people optimize for money or status because that's what people are good at, but the only reason there's a connection between money/status and given action or product is underlying connection with desirable outcome.

To pull an Inception - "X is about status, not X" is about status, not about X being about status, not X.

PS. Here's a paper claiming that 2/3 of highly cited papers had serious problems getting published at all because of peer review. How many breakthroughs didn't get published? How many got published and then ignored? How many people didn't even bother, limiting themselves to standard useless minutiae, or staying away from academic science entirely?

PPS. Here's a few more relevant links:

Basic statistics showing "Why Most Published Research Findings Are False".
Why replication matters and how little research survives it. Of over 600 published gene-disease associations claims, 6 survived replication, 160 failed replication, and for most nobody even bothered checking - not a big surprise with 1% success rate.
Scientists who spend time refuting false ideas are punished (paper gated).

Notice how except for medicine nobody seems bothered by this.

Neolithic Counter-Revolution in Diet

2010-09-25T03:31:00.001+02:00

All people have silly things to feel proud of, like their country or their soccer team - one unusual thing I feel very proud of myself is Neolithic Revolution. We fucking made it! We're the only beings in the whole damn universe who broke free from bondage of evolution, and that was the main act.

Like all irrationally proud people, I get easily irritated by all misguided criticism Neolithic Revolution has been getting - by amount of pure hatred it gets you'd think it was run by Justin Bieber and involved kitten sacrifice. Not that there would be any kittens without Neolithic Revolution - it's a true fact!

The most common thread of criticism is that somehow, with literally over 9000 years of lag, Neolithic food is making us unhealthy and overweight, and if only we could abandon civilization and go back to what was eaten in Paleolithic, we'd all be happy monkey once more.

This is plainly ridiculous. There's not a single group anywhere in the world even remotely living Upper Paleolithic life any more - even those that get most of their food hunting and gathering have long history of contact, trade, and interbreeding with agriculturalists and pastoralists, and every time you look more closely you'll see they don't really shy away from a bit of farming here and there themselves.

In other words - our knowledge of Paleolithic life and diet is about as good as our knowledge of mating habits of Hogwarts students - a lot of real fun speculation, and very little hard data.

And no matter where you look - vast majority of people have been unhealthy, with huge infant mortality, and all kinds of other severe health problems. True, cardiovascular problems and obesity are a fairly recent thing, but it is just ridiculous to focus exclusively on them and blame modern diet and lifestyle, while totally disregarding sheer count of much more debilitating diseases it saves us from.

It's a lot like complaining about Internet leading to more privacy violations, while ignoring how helpful it is in fixing a far more severe problem of not having fucking Internet access in the first place. Whiners gonna whine.

Isis as an Oroboros

Diet by era

Getting back on track, I took FAO data, and classified all food consumed into three big groups:

Paleolithic - vegetables, fruit (excluding wine), treenuts, meat, eggs, fish, seafood, other aquatic products, offal
Neolithic - cereals, pulses, alcoholic beverages, milk, butter, animal fats, starchy roots, spices, stimulants, miscellaneous
Industrial - sugar, sweeteners, sugarcrops, vegetable oils, oilcrops

For nitpickers who want to pick nits, here are exact definitions.

There are a few borderline cases I'll explain before proceeding:

Sugars and sweeteners (HFCS) are undoubtedly extremely recent introduction to diet.
A few vegetable oils like palm kernel oil and olive oil were common in traditional Neolithic, but they're a few percent of total, which is dominated by extremely recent soy, corn, sunflower etc., and they get far far more refining including partial oxidation than they ever used to. Traditional vegetable oils exist, but it would just complicate matters without changing conclusions.
Starchy roots category are almost all Neolithic crops like potatoes, sweet potatoes, and so on.
Minor categories without bold font are rather insignificant, and only included to make percents add up nicely.
Fans of Paleolithic will undoubtedly whine that modern vegetable, meat, etc. are not really Paleo due to all changes in modern agriculture, but by the same logic Neolithic foods are very rapidly disappearing as well. More seriously - in the grand scheme of things an egg is an egg, a pig is a pig, and a nut is a nut. Modern pig is far closer to wild pig than it is to a can of Diet Mountain Dew.

Still with me? Can you guess what changed in our diet between 1961 and 2007?

Our diet became a lot less Neolithic, a lot more Paleolithic, and a lot more Industrial.

Yes, we eat more Paleolithic, and we're more fatter than when we were eating Neolithic. How do you deal with that, Neolithic-haters?

Here's full list of countries for which data exists for both 1961 and 2007, percents are Paleolithic:Neolithic:Industrial by calories. Sadly there's no high quality data predating 1961, and for many countries like United States that was halfway through the Neolithic Counter-Revolution in diet.

Albania - 11.75%:80.75%:7.51% - 20.52%:65.36%:14.12%
Algeria - 9.75%:73.61%:16.65% - 11.11%:67.83%:21.06%
Angola - 7.89%:79.36%:12.75% - 9.32%:69.61%:21.07%
Antigua and Barbuda - 15.81%:57.22%:26.96% - 31.47%:46.85%:21.67%
Argentina - 27.36%:54.76%:17.88% - 24.13%:47.73%:28.13%
Australia - 24.36%:54.1%:21.54% - 25.47%:45.74%:28.79%
Austria - 18.21%:63.22%:18.57% - 22.02%:50.83%:27.15%
Bahamas - 23.15%:61.93%:14.93% - 28.53%:50.39%:21.08%
Bangladesh - 3.85%:89.29%:6.86% - 4.28%:84.97%:10.74%
Barbados - 13.98%:60.39%:25.63% - 23.63%:46.47%:29.9%
Belize - 11.35%:74.1%:14.55% - 19.45%:59.28%:21.26%
Benin - 6.94%:81.69%:11.37% - 5.59%:79.67%:14.74%
Bermuda - 29.55%:49.89%:20.56% - 26.64%:47.87%:25.49%
Bolivia - 16.84%:69.13%:14.03% - 20.5%:63.21%:16.29%
Botswana - 8.12%:81.73%:10.15% - 8.33%:68.47%:23.2%
Brazil - 10.84%:64.72%:24.44% - 18.5%:52.16%:29.34%
Brunei Darussalam - 11.3%:60.84%:27.86% - 18.18%:58.17%:23.65%
Bulgaria - 12.09%:73.62%:14.29% - 14.32%:59.42%:26.27%
Burkina Faso - 5.7%:82.29%:12.0% - 4.88%:82.2%:12.92%
Burundi - 12.63%:86.26%:1.12% - 18.25%:76.82%:4.93%
Cambodia - 6.9%:85.41%:7.69% - 10.36%:78.72%:10.92%
Cameroon - 13.2%:77.79%:9.02% - 14.16%:69.59%:16.25%
Canada - 19.53%:56.58%:23.89% - 20.26%:47.84%:31.9%
Cape Verde - 4.7%:81.66%:13.64% - 15.5%:66.54%:17.96%
Central African Republic - 7.06%:78.32%:14.62% - 12.51%:62.86%:24.63%
Chad - 5.5%:76.12%:18.37% - 4.72%:74.52%:20.76%
Chile - 14.27%:69.53%:16.2% - 22.35%:55.36%:22.29%
China - 7.84%:85.5%:6.66% - 26.97%:60.24%:12.8%
Colombia - 20.28%:58.13%:21.59% - 17.96%:54.0%:28.04%
Comoros - 10.92%:69.45%:19.63% - 12.41%:63.33%:24.26%
Congo - 9.8%:78.31%:11.89% - 11.48%:65.33%:23.19%
Costa Rica - 12.49%:58.25%:29.25% - 13.21%:54.1%:32.69%
Cuba - 12.87%:56.72%:30.41% - 17.15%:59.63%:23.22%
Cyprus - 18.96%:55.05%:25.99% - 25.71%:44.5%:29.79%
Côte d'Ivoire - 22.44%:67.91%:9.65% - 12.63%:71.54%:15.84%
Democratic People's Republic of Korea - 10.04%:82.17%:7.78% - 14.28%:76.58%:9.14%
Democratic Republic of the Congo - 11.51%:77.38%:11.11% - 5.62%:80.92%:13.46%
Denmark - 12.89%:62.25%:24.85% - 22.85%:56.69%:20.46%
Djibouti - 10.02%:64.9%:25.08% - 9.22%:64.56%:26.22%
Dominica - 17.34%:54.38%:28.27% - 25.55%:53.08%:21.37%
Dominican Republic - 30.24%:50.15%:19.62% - 21.13%:45.9%:32.97%
Ecuador - 19.72%:57.59%:22.69% - 23.44%:50.79%:25.77%
Egypt - 9.99%:76.98%:13.04% - 13.53%:72.27%:14.2%
El Salvador - 9.57%:71.64%:18.79% - 12.95%:63.52%:23.53%
Fiji - 5.56%:70.69%:23.75% - 13.74%:59.49%:26.77%
Finland - 12.06%:71.48%:16.46% - 24.17%:57.14%:18.7%
France - 21.62%:63.01%:15.37% - 22.55%:52.99%:24.46%
French Polynesia - 14.2%:63.64%:22.16% - 25.66%:52.12%:22.22%
Gabon - 32.81%:60.46%:6.72% - 24.44%:58.77%:16.79%
Gambia - 4.72%:71.77%:23.51% - 5.43%:62.54%:32.02%
Germany - 17.41%:61.61%:20.98% - 19.11%:54.34%:26.55%
Ghana - 14.96%:70.04%:14.99% - 15.66%:68.45%:15.88%
Greece - 16.44%:62.54%:21.02% - 20.88%:52.17%:26.95%
Grenada - 19.96%:51.5%:28.54% - 22.75%:44.22%:33.03%
Guatemala - 7.94%:78.14%:13.92% - 11.88%:62.1%:26.03%
Guinea - 19.42%:66.22%:14.36% - 12.7%:65.3%:22.0%
Guinea-Bissau - 13.84%:66.61%:19.55% - 8.88%:72.67%:18.45%
Guyana - 9.91%:63.19%:26.9% - 13.37%:61.33%:25.29%
Haiti - 12.78%:71.19%:16.03% - 11.49%:67.65%:20.86%
Honduras - 14.6%:70.33%:15.06% - 14.01%:60.13%:25.86%
Hungary - 15.99%:73.08%:10.93% - 18.2%:54.28%:27.52%
Iceland - 23.18%:52.58%:24.24% - 29.86%:50.0%:20.14%
India - 3.91%:79.42%:16.67% - 6.05%:75.19%:18.76%
Indonesia - 5.01%:78.85%:16.14% - 10.06%:70.67%:19.28%
Iran - 13.05%:71.72%:15.22% - 19.0%:64.28%:16.72%
Ireland - 13.96%:68.21%:17.83% - 18.92%:57.09%:23.99%
Israel - 18.05%:54.71%:27.25% - 24.11%:46.88%:29.01%
Italy - 15.24%:65.83%:18.93% - 22.54%:50.52%:26.95%
Jamaica - 17.55%:52.07%:30.37% - 19.78%:50.2%:30.02%
Japan - 10.96%:73.67%:15.37% - 20.18%:52.37%:27.45%
Jordan - 15.15%:61.34%:23.52% - 11.56%:56.94%:31.51%
Kenya - 10.21%:81.37%:8.42% - 10.96%:71.23%:17.8%
Kiribati - 15.97%:36.76%:47.27% - 17.87%:41.66%:40.47%
Kuwait - 20.35%:53.3%:26.35% - 20.59%:55.01%:24.4%
Lao People's Democratic Republic - 5.7%:92.49%:1.81% - 12.54%:80.1%:7.36%
Lebanon - 18.25%:63.08%:18.67% - 19.77%:51.64%:28.59%
Lesotho - 6.41%:87.67%:5.93% - 5.42%:86.71%:7.87%
Liberia - 11.77%:78.59%:9.64% - 7.39%:70.14%:22.47%
Libyan Arab Jamahiriya - 12.64%:64.69%:22.67% - 14.19%:57.38%:28.43%
Madagascar - 10.93%:82.92%:6.15% - 8.59%:82.55%:8.87%
Malawi - 6.94%:80.3%:12.76% - 8.17%:80.37%:11.46%
Malaysia - 10.39%:66.13%:23.48% - 17.46%:55.23%:27.31%
Maldives - 11.62%:52.79%:35.59% - 31.66%:47.91%:20.43%
Mali - 7.59%:84.68%:7.73% - 7.61%:78.81%:13.58%
Malta - 11.15%:68.35%:20.5% - 20.93%:58.91%:20.16%
Mauritania - 12.88%:75.94%:11.18% - 8.49%:68.07%:23.43%
Mauritius - 3.87%:65.63%:30.5% - 11.67%:60.76%:27.58%
Mexico - 10.63%:71.76%:17.61% - 17.84%:57.94%:24.21%
Mongolia - 39.69%:58.38%:1.94% - 21.64%:66.94%:11.42%
Morocco - 7.48%:72.86%:19.66% - 11.14%:67.73%:21.13%
Mozambique - 4.49%:86.97%:8.54% - 4.86%:81.05%:14.09%
Myanmar - 7.85%:79.08%:13.07% - 13.94%:68.05%:18.01%
Namibia - 11.65%:72.43%:15.92% - 9.47%:73.61%:16.93%
Nepal - 2.92%:92.98%:4.1% - 6.57%:83.19%:10.24%
Netherlands - 15.62%:54.88%:29.5% - 23.71%:48.54%:27.75%
Netherlands Antilles - 22.11%:56.63%:21.26% - 19.48%:57.6%:22.92%
New Caledonia - 18.0%:59.27%:22.73% - 20.3%:55.11%:24.58%
New Zealand - 25.31%:57.12%:17.57% - 26.86%:46.42%:26.72%
Nicaragua - 10.28%:69.71%:20.01% - 7.62%:66.84%:25.54%
Niger - 6.39%:88.74%:4.87% - 8.12%:82.06%:9.82%
Nigeria - 10.03%:69.04%:20.93% - 8.41%:72.24%:19.35%
Norway - 19.78%:61.74%:18.48% - 21.85%:55.4%:22.75%
Pakistan - 4.75%:81.31%:13.94% - 6.8%:69.01%:24.19%
Panama - 15.72%:66.74%:17.54% - 16.01%:62.74%:21.24%
Paraguay - 23.41%:66.75%:9.83% - 16.12%:59.2%:24.68%
Peru - 12.89%:68.83%:18.28% - 15.12%:69.75%:15.13%
Philippines - 19.21%:66.67%:14.12% - 20.62%:65.71%:13.68%
Poland - 10.91%:76.75%:12.35% - 18.45%:60.73%:20.82%
Portugal - 16.91%:66.28%:16.81% - 23.23%:56.13%:20.64%
Republic of Korea - 5.32%:90.37%:4.32% - 22.31%:52.44%:25.25%
Romania - 9.45%:82.0%:8.55% - 15.07%:67.21%:17.73%
Rwanda - 25.5%:73.72%:0.78% - 19.72%:73.29%:6.99%
Saint Kitts and Nevis - 8.93%:50.93%:40.14% - 20.29%:48.07%:31.65%
Saint Lucia - 25.06%:50.34%:24.6% - 29.78%:51.14%:19.08%
Saint Vincent and the Grenadines - 9.56%:60.38%:30.06% - 21.06%:53.12%:25.82%
Samoa - 23.99%:36.67%:39.34% - 30.0%:36.99%:33.01%
Sao Tome and Principe - 8.04%:60.04%:31.92% - 16.54%:55.14%:28.31%
Saudi Arabia - 15.92%:75.97%:8.11% - 18.27%:59.75%:21.98%
Senegal - 7.95%:71.51%:20.54% - 8.12%:67.61%:24.28%
Seychelles - 8.56%:67.95%:23.49% - 19.51%:55.67%:24.82%
Sierra Leone - 8.22%:60.19%:31.59% - 7.24%:69.23%:23.53%
Solomon Islands - 11.34%:76.01%:12.65% - 8.8%:74.13%:17.07%
South Africa - 11.65%:68.29%:20.06% - 13.0%:65.02%:21.98%
Spain - 14.51%:65.32%:20.17% - 26.26%:44.51%:29.23%
Sri Lanka - 7.11%:66.81%:26.08% - 7.77%:65.04%:27.2%
Sudan - 12.25%:69.05%:18.7% - 8.99%:73.13%:17.88%
Suriname - 10.5%:62.34%:27.17% - 17.27%:50.4%:32.33%
Swaziland - 9.94%:72.8%:17.26% - 11.05%:66.46%:22.48%
Sweden - 15.71%:55.94%:28.35% - 21.33%:52.34%:26.33%
Switzerland - 18.89%:56.97%:24.14% - 22.51%:47.44%:30.06%
Syrian Arab Republic - 15.76%:64.68%:19.57% - 12.54%:59.61%:27.85%
Thailand - 13.35%:75.37%:11.28% - 16.74%:57.68%:25.57%
Timor-Leste - 24.68%:71.22%:4.09% - 9.99%:75.86%:14.15%
Togo - 4.07%:86.81%:9.12% - 4.06%:78.59%:17.35%
Trinidad and Tobago - 10.2%:63.93%:25.88% - 14.56%:53.62%:31.82%
Tunisia - 9.36%:70.51%:20.13% - 13.45%:62.22%:24.33%
Turkey - 15.4%:74.39%:10.2% - 13.6%:62.28%:24.12%
Uganda - 17.44%:68.31%:14.25% - 22.27%:61.07%:16.66%
United Arab Emirates - 17.52%:70.51%:11.98% - 23.45%:57.72%:18.82%
United Kingdom - 20.41%:56.05%:23.54% - 22.54%:53.49%:23.97%
United Republic of Tanzania - 11.34%:80.15%:8.51% - 10.55%:75.45%:14.0%
United States of America - 20.36%:50.72%:28.92% - 20.51%:43.43%:36.06%
Uruguay - 28.54%:52.68%:18.78% - 15.05%:62.14%:22.82%
Vanuatu - 19.77%:57.07%:23.16% - 17.79%:52.8%:29.41%
Venezuela - 18.93%:54.48%:26.59% - 17.04%:53.45%:29.51%
Viet Nam - 10.49%:85.45%:4.06% - 19.59%:70.94%:9.46%
Yemen - 7.35%:85.13%:7.53% - 8.83%:69.29%:21.88%
Zambia - 6.0%:85.56%:8.44% - 5.53%:80.41%:14.05%
Zimbabwe - 5.85%:83.99%:10.16% - 5.64%:69.28%:25.07%

Almost universally, good old Neolithic foods that served us so well for entire history of human civilization are abandoned. And two primary foods of industrial era - sugar and vegetable oil - become the new basis of diet. What people rarely mention is how at the same time Paleolithic food - meat, fish, fruit, vegetables and so on - doubled in popularity in so many countries. And some combination of Paleolithic and Industrial is destroying everyone's hearts, livers, thyroids, and attractiveness outside certain narrow niche.

To save you some eye strain, here's the list of Top Ten Least Neolithic Countries. I don't even need to mention how it correlates with obesity rankings:

Samoa - 23.99%:36.67%:39.34% - 30.0%:36.99%:33.01%
Kiribati - 15.97%:36.76%:47.27% - 17.87%:41.66%:40.47%
United States of America - 20.36%:50.72%:28.92% - 20.51%:43.43%:36.06%
Grenada - 19.96%:51.5%:28.54% - 22.75%:44.22%:33.03%
Cyprus - 18.96%:55.05%:25.99% - 25.71%:44.5%:29.79%
Spain - 14.51%:65.32%:20.17% - 26.26%:44.51%:29.23%
Australia - 24.36%:54.1%:21.54% - 25.47%:45.74%:28.79%
Dominican Republic - 30.24%:50.15%:19.62% - 21.13%:45.9%:32.97%
New Zealand - 25.31%:57.12%:17.57% - 26.86%:46.42%:26.72%
Barbados - 13.98%:60.39%:25.63% - 23.63%:46.47%:29.9%

But no worries - The Neolithic Counter-Revolution seems to be reaching everyone who can afford it, so soon the entire planet will consist of walking blobs of omega-6 PUFA.

Anonymous Heroes of Science

You might now think that I'm some ridiculous kind of ultra-Conservative, who would like to move back not just to the Founding Fathers or their local equivalent but all the way to ancient Sumer. Not at all.

Food of Industrial era is mostly horrible crap, but it's also very plentiful, very cheap, and some of it is actually quite tasty. I'm sure we'll figure it out eventually - a big more Omega-3 here, a bit less Bisphenol A there - and step by step it might get just as good or even better than what farming came out with. Food eaten by early farmers was also horrible and caused severe health problems, but it was plentiful, cheap, and it took only a few millennia to figure out matters of health.

The thing is - I just don't feel particular need to be the early adopter of Industrial Diet. Early adopters are essential, as our scientific ethics committees are too chickenshit to approve genuine scientific experiments on humans, so we're doing the second best things by pretending it's all perfectly safe and letting nature run the experiment. Which trial of trans fats could even hope compete with feeding ridiculous amounts of them to hundreds of millions of people for nearly a century now? Without even telling them, to avoid placebo effect! Now that's proper science on grand scale!

And so I'd like to thank all people who unlike me bravely volunteer to let food industry run scientific experiments on their bodies by purchasing industrial food, and not stopping even after suffering severe side effects including but not limited to disfiguration, heart failure, liver failure, depression, diabetes, cancer, osteoarthritis, sleep apnea, and many others.

Your noble sacrifice will not be forgotten. Actually it totally will, but keep going please? The science needs you!

We need syntax for talking about Ruby types

2010-07-20T13:14:00.001+02:00

All this is about discussing types in blog posts, documentation etc. None of that goes anywhere near actual code (except possibly in comments). Ruby never sees that.

Statically typed languages have all this covered, and we need it too. Not static typing of course - just an expressive way to talk about what types things are - as plain English fails here very quickly. As far as I know nothing like that exists yet, so here's my proposal.

This system of type descriptions is meant for humans, not machines. It focuses on the most important distinctions, and ignores details that are not important, or very difficult to keep track of. Type descriptions should only be as specific as necessary in given context. If it makes sense, there rules should be violated.

In advance I'll say I totally ignored all the covariance / contravariance / invariance business - it's far to complicated, and getting too deeply into such issues makes little sense in a language where everything can be redefined.

Basic types

Types of simple values can be described by their class name, or any of its superclasses or mixins. So some ways to describe type of 15 would be Fixnum (actual class), Integer (superclass), Comparable (mixin), or Object (superclass all the way up).

In context of describing types, everything is considered an Object, and existence of Kernel, BasicObject etc. is ignored.

So far, it should all be rather obvious. Examples:

42 - Integer
Time.now - Time
Dir.glob("*") - Enumerable
STDIN - IO

`nil` and other ignored issues

nil will be treated specially - as if it was of every possible type. nil means absence of value, and doesn't indicate what type the value would have if it was present. This is messy, but most explicitly typed languages follow this path.

Distinction between situations that allow nils and those that don't will be treated as all other value range restrictions (Integer must be posibile, IO must be open for writing etc.) - as something outside the type system.

For cases where nil means something magical, and not just absence of value, it should probably be mentioned.

Checked exceptions and related non-local exits in Ruby would be a hopeless thing to even attempt. There's syntax for exceptions and catches used as control structures if they're really necessary.

Booleans

We will also pretend that Boolean is a common superclass of TrueClass and FalseClass.

We will also normally ignore distinction between situations where real true/false are expected, and situations where any object goes, but acts identically to its boolean conversion. Any method that acts identically on x and !!x can be said to take Boolean.

On the other hand if some values are treated differently than their double negation, that's not really Boolean and it deserves a mention. Especially if nil and false are not equivalent - like in Rails's #in_groups_of (I don't think Ruby stdlib ever does thing like that).

Duck typing

If something quacks like a Duck convincingly enough, it can be said to be of type Duck, it being object's responsibility that its cover doesn't get blown.

In particular, Ruby uses certain methods for automatic type conversion. In many contexts objects implementing #to_str like Pathnames will be treated as Strings, objects implementing #to_ary as Arrays, #to_hash as Hashes, and to_proc as Procs - this can be used for some amazing things like Symbol#to_proc.

This leads to a big complication for us - C code implementing Ruby interpreter and many libraries is normally written in a way that calls these conversion functions automatically, so in such contexts Symbol really is a Proc, Pathname really is a String and so on. On the other hand, in Ruby code these methods are not magical, and such conversions will only happen if explicitly called - for them Pathname and String are completely unrelated types. Unless Ruby code calls C code, which then autoconverts.

Explicitly differentiating between contexts which expect a genuine String and those which expect either that or something with a valid #to_str method would be highly tedious, and I doubt anyone would get it exactly right.

My recommendation would be to treat everything that autoconverts to something as if it subclassed it. So we'll pretend Pathname is a subclass of String, even though it's not really. In some cases this will be wrong, but it's not really all that different from subclassing something and then introducing incompatible changes.

This all doesn't extend to #to_s, #to_a etc - nothing can be described as String just because it has to_s method - every object has to_s but most aren't really strings.

Technical explanation of `to_str` and friends

This section is unrelated to post's primary subject - skip if uninterested.

Ruby uses special memory layout for basic types like strings and arrays. Performance would be abysmal if string methods had to actually call Ruby code associated with whatever [] happened to be redefined to for every character - instead they ask for a certain C data structure, and access that directly (via some macros providing extra safety and convenience to be really exact).

By the way this is a great example of C being really slow - if Ruby was implemented on a platform with really good JIT, it could plausibly have every single string function implemented in term of calls to [], []=, size, and just a few others, with different subclasses of String providing different implementations, and JIT compiling inlining all that to make it really fast.

It would make it really simple to create class representing a text file, and =~ /regexp/ that directly without reading anything more than required to memory, or maybe even gsub! it in a way that would read it in small chunks, saving them to another file as soon as they're ready, and then renaming in one go. All that without regexp library knowing anything about it all. It's all just my fantasy, I'm not saying any such JIT actually exists.

Anyway, strings and such are implemented specially, but we still want these types to be real objects, not like what they've done in Java. To make it work, all C functions requiring access to underlying storage call a special macro which automatically calls a method like to_str or to_ary if necessary - so such objects can pretend to be strings very effectively. For example if you alias method to_str to path on File code like system File.open("/bin/hostname") will suddenly start working. It really makes sense only for things which are "essentially strings" like Pathname, URI, Unicode-enhanced strings, proxies for strings in third party libraries like Qt etc.

To complicate things further objects of all classes inheriting from String automatically use String's data representation - and C code will access that, never calling to_str. This leaves objects which duck type as Strings two choices:

Subclass String and every time anything changes update C string data. This can be difficult - if you implement an URI and keep query part as a hash instance variable - you need to somehow make sure that your update code gets run every time that hash changes - like by not exposing it at all and only allowing query updates via your direct methods, or wrapping it in a special object that calls you back.
Don't subclass String, define to_str the way you want. Everything works - except your class isn't technically a String so it's not terribly pretty OO design.

You probably won't be surprised that not subclassing is the more popular choice. As it's all due to technical limitations not design choices, it makes sense to treat such objects as if they were properly subclassed.

Collections

Back to the subject. For collections we often want to describe types of their elements. For simple collections yielding successive elements on #each, syntax for type description is CollectionType[MemberType]. Examples:

[42.0, 17.5] - Array[Float]
Set["foo","bar"] - Set[String]
5..10 - Range[Integer]

When we don't care about collection type, only about element types, descriptions like Enumerable[ElementType] will do.

Syntax for types of hashtables is Hash[KeyType, ValueType] - in general collections which yield multiple values to #each can be described as CollectionType[Type1, Type2, ..., TypeN].

For example {:foo => "bar"} is of type Hash[Symbol, String].

This is optional - type descriptions like Hash or Enumerable are perfectly valid - and often types are unrelated, or we don't care.

Not every Enumerable should be treated as collection of members like that - File might technically be File[String] but it's usually pointless to describe it this way. In 1.8 String is Enumerable, yielding successive lines when iterated - but String[String] make no sense (no longer a problem in 1.9).

Classes other than Enumerable like Delegator might need type parameters, and they should be specified with the same syntax. Their order and meaning depends on particular class, but usually should be obvious.

Literals and tuples

Ruby doesn't make distinction between Arrays and tuples. What I mean here is a kind of Array which shouldn't really be treated as a collection, and in which different members have unrelated type and meaning depending on their position.

Like method arguments. It really wouldn't be useful to say that every method takes Array[Object] (and an optional Proc) - types and meanings of elements in this array should be specified.

Syntax I want for this is [Type1, Type2, *TypeRest] - so for example Hash[Date, Integer]'s #select passes [Date, Integer] to the block, which should return a Boolean result, and then returns either Array[[Date, Integer]] (1.8) or Hash[Date, Integer] (1.9). Notice double [[]]s here - it's an Array of pairs. In many contexts Ruby automatically unpacks such tuples, so Array[[Date,Integer]] can often be treated as Array[Date,Integer] - but it doesn't go deeper than one level, and if you need this distinction it's available.

Extra arguments can be specified with *Type or ... which is treated here as *Object. If you want to specify some arguments as optional suffix their types with ? (the most obvious [] having too many uses already, and = not really fitting right).

In this syntax [*Foo] is pretty much equivalent to Array[Foo], or possibly Enumerable[Foo] (with some duck typing) - feel free to use that if it makes things clearer.

Basic literals like true, false, nil stand for themselves - and for entire TrueClass, FalseClass, NilClass classes too as they're their only members. Other literals such as symbols, strings, numbers etc. can be used too when needed.

To describe keyword arguments and hashes used in similar way, syntax is {Key1=>Type1, Key2=>Type2} - specifying exact key, and type of value like {:noop=>Boolean, :force=>Boolean}.

It should be assumed that keys other than those listed are ignored, cause exception, or are otherwise not supported. If they're meaningful it should be marked with ... like this {:query=>String, ...}. Subclasses often add extra keyword arguments, and this issue is ignored.

Functions

Everything so far was just a prelude to the most important part of any type system - types for functions. Syntax I'd propose it: ArgumentTypes -> ReturnType (=> being already used by hashes).

I cannot decide if blocks should be specified in Ruby-style notation or a function notation, so both & {|BlockArgumentTypes| BlockReturnType} and &(BlockArgumentTypes->BlockReturnType) are valid. & is necessary, as block are passed separately from normal arguments, however strong the temptation to reuse -> and let the context disambiguate might be.

Blocks that don't take any arguments or don't return anything can drop that part, leaving only something like &{|X|}, &{Y}, &{}, or in more functional notation &(X->), &(Y), &().

Because of all the [] unpacking, using [] around arguments, tuple return values etc. is optional - and just like in Ruby () can be used instead in such contexts.

If function doesn't take any arguments, or returns no values, these parts can be left - leaving perhaps as little as ->.

Examples:

In context of %w[Hello world !].group_by(&:size) method #group_by has type Array[String]&{|String| Integer}->Hash[Integer,String]
Time.at has type Numeric -> Time
String#tr has type [String, String] -> String
On a collection of Floats, #find would have type Float?&(Float->Boolean)->Float
Function which takes no arguments and returns no values has type []->nil

If you really need to specify exceptions and throws, you can add raises Type, or throws :kind after return value. Use only for control structure exceptions, not for actual errors exceptions. It might actually be useful if actual data gets passed around.

Find.find has type [String*]&(String->nil throws :prune)->nil

A standalone Proc can be described as (ArgumentsTypes->ReturnType) just as with notation for functions. There is no ambiguity between Proc arguments and block arguments, as blocks are always marked with |.

Type variable and everything else

In addition to names of real classes, any name starting with an uppercase letter should be consider a type. Unless it's specified otherwise in context, all such unknown names should be considered class variables with big forall quantifier in front of it all.

Examples:

Enumerable[A]#partition has type &(B->Boolean)->[Array[A], Array[A]]
Hash[A,B]#merge has type Hash[A,B]&(A,B,B->B)->Hash[A,B]
Array[A]#inject has either type B&(B,A->B)->B or &(A,A)->A. This isn't just a usual case of missing argument being substituted by nil - these are two completely different functions.

To specify that multiple types are allowed (usually implying that behaviour will be different, otherwise there should be a superclass somewhere, or we could treat it as common duck typing and ignore it) join them with |. If there's ambiguity between this use and block arguments, parenthesize. It binds more tightly than ,, so it only applies to one argument. Example:

String#index in 1.8 has type (String|Integer|Regexp, Integer?)->Integer (and notice how I ignored Fixnums here).

For functions that can be called in multiple unrelated ways, just list them separately - | and parentheses will work, but they are usually top level, and not needed anywhere deeper.

If you want to specify type of self, prefix function specification with Type#:

#sort has type like Enumerable[A]#()&(A,A->1|0|-1)->Array[A]

To specify that something takes range of values not really corresponding to a Ruby class, just define such extra names somewhere and then use like this:

File#chown has type (UnixUserId, UnixUserId)->0 - with UnixUserId being a pretend subclass of Integer, and 0 is literal value actually returned.

To specify that something needs a particular methods just make up a pretend mixin like Meowable for #meow.

Any obvious extensions to this notation can be used, like this:

Enumerable[A]#zip has type (Enumerable[B_1], *Enumerable[B_i])->Array[A, B_1, *B_i] - with intention that B_is will be different for each argument understood from context. (I don't think any static type system handles cases like this one reasonably - most require separate case for each supported tuple length, and you cannot use arrays if you mix types. Am I missing something?)

The End

Well, what I really wanted to do what talk about Ruby collection system, and how 1.9 doesn't go far enough in its attempts at fixing it. And without notation for types talking about high order functions that operate on collections quickly turns into a horrible mess. So I started with a brief explanation of notation I wanted to use, and then I figured out I can as well do it right and write something that will be reusable in other contexts too.

Most discussion of type systems concerns issues like safety and flexibility, which don't concern me at all, and limit themselves to type systems usable by machines.

I need types for something else - as statements about data flow. Type signature like Enumerable[A]#()&(A->B)->Hash[A,B] doesn't tell you exactly what such function does but narrows set of possibilities extremely quickly. What it describes is a function which iterates over collection in order while building a Hash to be returned, using collection's elements as keys, and values returned by the block as values. Can you guess the function I was thinking about here?

Now a type like that is not a complete specification - a function that returns an empty hash fits it. As does one which skips every 5th element. And one that only keeps entries with unique block results. And for that matter also one that sends your email password to NSA - at least assuming it returns that Hash afterwards.

It was still pretty useful. How about some of those?

Hash[A,B] -> Hash[B, Array[A]]
Hash[A,B] &(A,B->C) -> Hash[A,C]
Hash[A, Hash[B,C]] -> Hash[[A,B], C]
Hash[A,B] &(A,B->C) -> Hash[C, Hash[A,B]]
Enumerable[Hash[A,B]] &(A,B,B->B) -> Hash[A,B]
Hash[A,Set[B]] -> Hash[Set[A], Set[B]]

Even these short snippets should give a pretty good idea what these are all about.

That's it for now. Hopefully it won't be long until that promised 1.9 collections post.

If only Ruby had macros

2010-07-18T07:22:00.000+02:00

Blogger will most likely totally destroy code formatting again, sorry about that.

Ruby annoys me a lot - the code gets so close to being Just Right, with only that last little bit of wrongness that won't go away no matter what. With everything except Ruby at least I know it will be crap no matter what, so I never get this.

For example it's so easy to make a function generating successive values on each call:

def counter(v)

return counter(v, &:succ) unless block_given?
  proc{ v = yield(v) }
end

But you must give it value before the first - and sometimes such a thing doesn't exist, like with generating successive labels "a", "b", "c" ... A counter starting from the first value passed isn't exactly difficult, it just doesn't feel right:

def counter(v)

return counter(v, &:succ) unless block_given?
  proc{ old, v = v, yield(v); old }
end

Useless variables like old that only indicate control flow just annoy me. Not to mention lack of default block argument. I'm undecided if this tap makes things better or worse.

def counter(v)

return counter(v, &:succ) unless block_given?
  proc{v.tap{ v = yield(v) }}
end

Another example. This wrapper for Ruby executable makes rubygems and -r compatible. It's so close to being able to use Array#map, and yet so far away:

args = []
while arg = ARGV.shift
  if arg =~ /\A-r(.*)\z/
    lib = $1.empty? ? ARGV.shift : $1
    args << "-e" << "require 'rubygems'; require '#{lib}'"
  else
    args << arg
  end
end
exec "ruby", *args

Yes, these are tiny things, but it's frustrating to get almost there. By the way, -r should just call require, another thing which is almost right but no.

I could go on with these small examples, but I want to talk about something bigger. A very common pattern in all programming languages is something like this:

collection.each{|item|

  if item.test_1
    item.action_1 
  elsif item.test_2
    item.action_2
  elsif item.test_3
    item.action_3
  else
    item.otherwise
  end
}

Or a very similar:

collection.each{|item|

case item
  when pattern_1
    item.action_1 
  when pattern_2
    item.action_2
  when pattern_3
    item.action_3
  else
    item.otherwise

end
}

Tests and actions are all next to each other, where they belong. But what if instead of executing an action on a single item at a time, we wanted to do so on all matching items together?

If Ruby had proper macros it would be totally trivial - unfortunately Ruby forces us to choose one of bad options. First, the most straightforward:

yes1, no1 = collection.partition{|item| item.test_1}
yes2, no12 = no1.partition{|item| item.test_2}
yes3, no123 = no12.partition{|item| item.test_3}

yes_1.action_1
yes_2.action_2
yes_3.action_3
no123.otherwise

Rather awful. Or perhaps this?

groups = collection.group_by{|item|

if item.test_1 then 1
  elsif item.test_2 then 2
  elsif item.test_3 then 3
  else 4
  end
}
(groups[1]||[]).action_1
(groups[2]||[]).action_2
(groups[3]||[]).action_3
(groups[4]||[]).otherwise

By the way we cannot use a series of selects here - action_3 should apply only to items which pass test_3 but not test_1 or test_2.

We can imagine adding extra methods to Enumerable to get syntax like this:

collection.run_for_each_group(

proc{|item| item.test_1}, proc{|group| group.action_1},
  proc{|item| item.test_2}, proc{|group| group.action_2},
  proc{|item| item.test_3}, proc{|group| group.action_3},
                            proc{|group| group.otherwise})

Or maybe like this (looks even worse if you need to assign groups to a variable before performing the relevant action):

tmp = collection.dup

tmp.destructive_select!{|item| item.test_1}.action_1
tmp.destructive_select!{|item| item.test_2}.action_2
tmp.destructive_select!{|item| item.test_3}.action_3
tmp.otherwise

#destructive_select! being a method in style of Perl's splice - removing some items from collection, and returning removed values.

Possibly wrapping it in something like:

collection.filter{|item| item.test_1}.action{|group| group.action_1}.
          .filter{|item| item.test_2}.action{|group| group.action_2}.
          .filter{|item| item.test_3}.action{|group| group.action_3}.
                                     .action{|group| group.otherwise}

A few more bad ideas (David Allen says the way you can tell a highly creative person is that they generate bad ideas faster than anyone else). With instance_eval we could do something like this, with item and group being appropriate method calls.

collection.run_for_each_group{

  rule{ item.test_1 }
  action{ group.action_1 }

  rule{ item.test_2 }
  action{ group.action_2 }

  rule{ item.test_3 }
  action{ group.action_3 }

  action{ group.otherwise }
}

It would be pretty hard to do that while still being able to have inner blocks with your current object's context. By the way trying this out I found out that it's impossible to call a block specifying self, and call a block passing arguments at the same time - it's only one or the other - and no combination of the two makes it work. Those tiny limitations are just infuriating.

I also tried overriding ===. Now that would only work for a small subset of cases but was worth a try:

collection.run_for_each_group{|item, group|
  case item
  when pattern_1
    group.action_1
  when pattern_2
    group.action_2
  when pattern_3
    group.action_3
  else
    group.otherwise
  end
}

This item would actually be a special object, calling === on which would callcc, partition collection in two, and resume twice modifying group variable (initially set to the entire collection). That would be pretty cool - except Ruby doesn't use double dispatch, so === is not a CLOS style generic function - it's a method, set on pattern objects, and while adding new pattern types is easy, making old patterns match new kinds of objects is hard. It would require manually finding out every pattern, and manually overriding it to handle our magic item type - and then a lot of hackery to make Regexp#=== work, and then it would fail anyway, as Range#=== and such seem to be handled specially by Ruby.

There was a related possibility of not doing anything weird to item, but requiring special patterns:

collection.run_for_each_group{|item, group, all|
  case item
  when all[pattern_1]
    group.action_1
  when all[pattern_2]
    group.action_2
  when all[pattern_3]
    group.action_3
  else
    group.otherwise
  end
}

We're not actually using item here all, so we don't really need to pass it:

collection.run_for_each_group{|group, all|
  if all[pattern_1]
    group.action_1
  elsif all[pattern_2]
    group.action_2
  elsif all[pattern_3]
    group.action_3
  else
    group.otherwise
  end
}

Totally implementable, only somewhat ugly with all these all[]s. There are two good ways to implement it - all function would test all items, and if all returned the same value it would just return. Otherwise, it would divide the collection, and in one implementation use callcc, or in alternative implementation, throw something, and restart the whole block twice - this assumes tests are cheap and deterministic.

It looks good, but it doesn't make me happy, as I want all kinds of tests, not just pattern matches. And eventually I came up with this:

collection.run_for_each_group{|item, group, all|
  if all[item.test_1]
    group.action_1
  elsif all[item.test_2]
    group.action_2
  elsif all[item.test_3]
    group.action_3
  else
    group.otherwise
  end
}

This way, you can do any test on item you want - just pass the result to all[] before proceeding.

How is it implemented? I could callcc for every element, but unlike Scheme's, Ruby's callcc is rather expensive. And not every version of Ruby has it. So it's the naive throw-and-restart-twice instead. This means tests on each item can be rerun many times, so they better be cheap. Determinism is also advised, even though my implementation caches the first value returned to avoid troubles.

Well, first some usage example you can actually run:

require "pathname"

files = Pathname("/etc").children
files.run_for_each_group{|x,xs,all|
  if all[x.directory?]
    puts "Subdirectories: #{xs*' '}"
  elsif all[x.symlink?]
    puts "Symlinks: #{xs*' '}"
  elsif all[x.size > 2**16]
    puts "Big files: #{xs*' '}"
  else
    puts "The rest: #{xs.size} files"
  end
}

Doesn't it look a lot lot better than a long cascade of #partitions?

And now #run_for_in_group:

module Enumerable 
  def run_for_each_group(expected=[], &blk)
    return if empty?
    xst, xsf = [], []
    each{|it|
      answers = expected.dup
      catch :item_tested do
        yield(it, self, proc{|v|
          if answers.empty?
            (v ? xst : xsf) << it
            throw :item_tested

          end
          answers.pop
        })
        return
      end
    }
    xst.run_for_each_group([true, *expected], &blk)
    xsf.run_for_each_group([false, *expected], &blk)
  end
end

It shouldn't be that difficult to understand. expected tracks the list of expected test results for all items in current collection. Now we iterate, passing each element, the entire group, and all callback function.

The first few times all is called, it just returns recorded answers - they're the same for every element. If after all recorded answers all is called again - we record its result, throw out of the block, and rerun it twice with expanded expectations.

On the other hand if we didn't get any calls to all other than those already recorded, it means we reached the action - group it sees is every element with the same test history. This must only happen once for group, so we return from function.

Total number of block calls is - 1x for each action, 2x for directories, 3x for symlinks, 4x for big files, and also 4x for everything else. Avoiding these reruns would be totally possible with callcc - but it's rather ugly, and often these tests aren't an issue.

So problem solved? Not really. I keep finding myself in situations where a new control structure would make a big difference, and there just doesn't seem to be any way of making it work in Ruby without enough boilerplate code to make it not worthwhile.

I'll end this post with some snippets of code which are just not quite right. Any ideas for making them suck less?

urls = Hash[file.map{|line| id, url = line.split; [id.to_i, url]}]

each_event{|type, *args| 
  case type
  when :foo
    one, two = *args
    # ...
  when :bar
    one, = *args
    # ...
  end
}

if dir

Dir.chdir(dir){ yield(x) }
else
  yield(x)
end

Ruby is now more popular than C++

2010-07-17T21:44:00.000+02:00

There are many ways to measure popularity, and I'll take one proposed by creator of C++ Bjarne Stroustrup. According to him there are only two kinds of languages - those that nobody uses and those that everybody bitches about - so counting google results for "<language> sucks" is a perfectly good way of measuring popularity.

I did an identical experiment exactly 4 years ago, so it's interesting to see what changed since then.

D 147000 - mostly bogus matches for "3D sucks" etc.
Java 56300
C 48900 - possibly also many bogus matches
PHP 34500
Ruby 25900
Ruby on Rails 18100 - included for comparison only
Scheme 14900 - my blog is #1, also many bogus matches
C++ 14000
Visual Basic 11600
Python 8930
Perl 5450
Lisp 3510
C# 3310
Ada 1240
OCaml 1070
SML 784
Erlang 750
Cobol 641
Fortran 476
Haskell 416
Smalltalk 176
Prolog 161

Some things just don't change. Ignoring queries with too many false positives, the list is dominated by Java and PHP, just as four years ago - with Java lead now being a lot stronger. Most niche languages like OCaml, Smalltalk, and Prolog are still niche languages - although many get a lot of bitching these days (like OCaml's 17x increase).

On the other hand some things changed. Perl used to be very high in the sucking charts - at about 15x as many sucks as Ruby and Python, but isn't anywhere close to the top now, losing almost half of the sucks in that time, as old ones die in link rot, and new ones stop being generated.

The second biggest success story is Python, which sucks 12x as much now, finally overtaking Perl.

But the biggest success is a spectacular explosion of popularity of Ruby. My first list was released only half a year after Rails 1.0, when many people were intrigued by Ruby, but few were actually using it. In those four years Ruby suckiness levels exploded 43x - not even counting Rails bitching, a lot of which is as much about Ruby as about Rails themselves.

Ruby is now a lot more popular than C++ - according to the very metric endorsed by C++ creator. What alternative explanation is there? That C++ is used a lot, it just happens to suck less? Come on.

People complaining about scientific validity of this post are sent here.

Another example of Ruby being awesome - %W

2010-07-17T20:22:00.000+02:00

And there I was thinking I knew everything about Ruby, at least as far as its syntax goes...

As you might have figured out from my previous posts, I'm totally obsessed about string escaping hygiene - I would never send "SELECT * FROM reasons_why_mysql_sucks WHERE reason_id = #{id}" to an sql server even if I was absolutely totally certain that id is a valid integer and nothing can possibly go wrong here. Sure, I might be right 99% of time, but it only takes a single such mistake to screw up the system. And not only with SQL - it's the same with generated HTML, generated shell commands and so on.

And speaking of shell commands - system function accepts either a string which it then evaluates according to shell rules (big red flag), or a list of arguments which it uses to fork+exec right away. Of course we want to do that - except it's really goddamn ugly. Faced with a choice between this insecure but reasonably looking way of starting MongoDB shard servers:

system "mongod --shardsvr --port '#{port}' --fork --dbpath '#{data_dir}' \

--logappend --logpath '#{logpath}' --directoryperdb"

And this secure but godawful (to_s is necessary as port is an integer, and system won't take that):

system *["mongod", "--shardsvr", "--port", port, "--fork",

"--dbpath", data_dir, "--logappend",

"--logpath", logpath, "--directoryperdb"].map(&:to_s)

Even I have my doubts.

And then I found something really cool in Ruby syntax that totally solves the problem. Now I was totally aware of %w[foo bar] syntax Ruby copied from Perl's qw[foo bar], and while useful occasionally, is really little more than constructing a string, and then calling #split on that.

And I though I was also aware of %W - which obviously would work just like %w except evaluating code inside. Except that's not what it does! %W[foo #{bar}] is not "foo #{bar}".split - it's ["foo", "#{bar}"]! And using a real parser of course, so you can use as many spaces inside that code block as you want.

system *%W[mongod --shardsvr --port #{port} --fork --dbpath #{data_dir}

--logappend --logpath #{logpath} --directoryperdb]

There's nothing in Perl able to do that. Not only it's totally secure, it looks every better than the original insecure version as you don't need to insert all those 's around arguments (which only half-protected them anyway, but were better than nothing), and you can break it into multiple lines without \s.

%W always does the right thing - %W[scp #{local_path} #{user}@#{host}:#{remote_path}] will keep the whole remote address together - and if the code block returns an empty string or nil, you'll get an empty string there in the resulting array. I sort of wish there was some way of adding extra arguments with *args-like syntax like in other contexts, but %W[...] + args does exactly that, so it's not a big deal.

By the way, it seems to me that all % constructors undeservingly get a really bad reputation as some sort of ugly Perl leftover in Ruby community. This is so wrong - what's ugly is excessive escaping with \ which they help avoid. Which regexp for Ruby executables looks less bad, the one with way too many \/s - /\A(\/usr|\/usr\/local|\/opt|)\/bin\/j?ruby[\d.]*\z/, or one which avoids them all thanks to %r - %r[\A(/usr|/usr/local|/opt|)/bin/j?ruby[\d.]*\z]?

By the way - yes I used []s inside even though they were the big demarcator. That's another great beauty of % constructions - if you demarcate with some sort of braces like [], (), <>, or {} - it will only close once every matched pair inside is closed - so unlike traditional singly and doubly quoted strings % can be nested infinitely deep without a single escape character! (Perl could do that one as well)

And speaking of things that Ruby copied from Perl, and then made them much more awesome, here's a one-liner to truncate a bunch of files after 10 lines, with optional backups. Which language gets even close to matching that? ($. in both Perl and Ruby will keep increasing from file to file, so you cannot use that)

ruby -i.bak -ple 'ARGF.skip if ARGF.file.lineno > 10' files*.txt

Arrays are not integer-indexed Hashes

2010-07-16T07:44:00.000+02:00

We use a separate Array type even though Ruby Hashes can be indexed by integers perfectly well (unlike Perl hashes which implicitly convert all hash keys to strings, and array keys to integers). Hypothetically, we could get rid of them altogether and treat ["foo", "bar"] as syntactic sugar for {0=>"foo", 1=>"bar"}.

Now there are obviously some performance reasons for this - these are mostly fixable and a single data structure can perform well in both roles. And it would break backwards compatibility rather drastically, but let's ignore all that and imagine we're designing a completely fresh language which simply looks a lot like Ruby.

What would work

First, a lot of things work right away like [], []=, ==, size, clear, replace, and zip.

The first incompatibility is with each - for hashes it yields both keys and values, for arrays only values, and we'd need to decide one way or the other - I think yielding both makes more sense, but then there are all those non-indexable enumerables which won't be able to follow this change, so there are good reasons to only yield values as well. In any case, each_pair, each_key, and each_value would be available.

Either way, one more change would be necessary here - each and everything else would need to yield elements sorted by key. There are performance implications, but they're not so bad, and it would be nicer API.

Hash's methods keys, values, invert, and update all make perfect sense for Arrays. With keys sorted, first, last, and pop would work quite well. push/<< would be slightly nontrivial - but making it add #succ of the last key (or 0 for empty hashes) would work well enough.

Collection tests like any?, all?, one?, none? are obvious once we decide each, and so is count. map/collect adapts to hashes well enough (yielding both key and value, and returning new value).

Array methods like shuffle, sort, sample, uniq, and flatten which ignore indexes (but not their relative positions) would do likewise for hashes, so flattening {"a"=>[10,20], "b"=>30} would result in [10,20,30] ("a" yields before "b").

Enumerable methods like min/max/min_by/max_by, find, find_index, inject would do likewise.

include? checks values for Arrays and keys for hashes - we can throw that one out (or decide one way or the other, values make more sense to me), and use has_key?/has_value? when it matters.

reverse should just return values, but reverse_each should yield real keys.

I could go on like this. My point is - a lot of this stuff can be made to work really well. Usually there's a single behavior sensible for both Arrays, and Hashes, and if you really need something different then keys, values, or pairs would usually be a suitable solution.

What doesn't work

Unfortunately some things cannot be made to work. Consider this - what should be the return value of {0 => "zero", 1 => "one"}.select{|k,v| v == "one"}?

If we treat it as a hash - let's say a mapping of numbers to their English names, there is only one correct answer, and everything else is completely wrong - {1=>"one"}.

On the other hand if we treat it as an array - just an ordered list of words - there is also only one correct answer, and everything else is completely wrong - {0=>"one"}.

These two are of course totally incompatible. And an identical problem affects a lot of essential methods. Deleting an element renumbers items for an array, but not for a hash. shift/unshift/drop/insert/slice make so sense for hashes, and methods like group_by and partition have two valid and conflicting interpretations. It is, pretty much, unfixable.

So what went wrong? Thinking that Arrays are indexed by integers was wrong!

In {0=>"zero",1=>"one"} association between keys and values is extremely strong - key 0 is associated with value "zero", and key 1 with value "one". They exist as a pair and everything that happens to the hash happens to pairs, not to keys or values separately - there are no operations like insert_value, delete_value which would just shift remaining values around from one key to another. This is the nature of hashes.

Arrays are not at all like that. In ["zero", "one"] association between 0 and "zero" is very weak. The real keys are not 0, and 1 - they're two objects devoid of any external meaning, whose only property is their relative partial order.

To implement array semantics on top of hashes, we need a class like Index.new(greater_that=nil, less_than=nil). Then a construction like this would have semantics we desire.

arr = {}

arr[Index.new(arr.last_key, nil)] = "zero"

arr[Index.new(arr.last_key, nil)] = "one"

If we use these instead of integers, hashes can perform all array operations correctly.

# shift

arr.delete(arr.first_key)

# unshift

arr[Index.new(nil, arr.first_key)] = "minus one"

# select - indexes for "zero" and "two" in result have correct order

["zero", "one", "two"].select{|key, value| value != "one"}

# insert - nth_key only needs each

arr[Index.new(arr.nth_key(0), arr.nth_key(1))] = "one and half"

And so the theory is satisfied. We have a working solution, even if highly impractical one. Of course all these Index objects are rather hard to use, so the first thing we'd do is subclassing Hash so that arr[i] would really mean arr[arr.nth_key(i)] and so on, and there's really no point yielding them in #each and friends... oh wait, that's exactly where we started.

In other words, unification of arrays and hashes is impossible - at least unless you're willing to accept a monstrosity like PHP where numerical and non-numerical indexes are treated differently, and half of array functions accept a boolean flag asking if you'd rather have it behave like an array or like a hash.

Random sampling or processing data streams in Ruby

2010-07-16T03:23:00.000+02:00

It might sound like I'm tackling a long solved problem here - sort_by{rand}[0, n] is a well known idiom, and in more recent versions of Ruby you can use even simpler shuffle[0, n] or sample(n).

They all suffer from two problems. The minor one is that quite often I want elements in the sample to be in the same relative order as in the original collection (this in no way implies sorted) - what can be dealt with by a Schwartzian transform to [index, item] space, sampling that, sorting results, and transforming out to just item.

The major problem is far worse - for any of these to work, the entire collection must be loaded to memory, and if that was possible, why even bother with random sampling? More often than not, the collection I'm interested in sampling is something disk-based that I can iterate only once with #each (or twice if I really really have to), and I'm lucky if I even know its #size in advance.

By the way - this is totally unrelated, but I really hate #length method with passion - collections have sizes, not "lengths" - for a few kinds of collections we can imagine them arranged in a neat ordered line, and so their size is also length, but it's really lame to name a method after special case instead of far more general "size" - hashtables have sizes not lengths, sets have sizes not lengths, and so on - #length should die in fire!

When size is known

So we have a collection we can only iterate once - for now let's assume we're really lucky and we know exactly how many elements it has - this isn't all that common, but it happens every now and then. As we want n elements out of size, probability of each element being included is n/size, and so select{ n > rand(size) } will nearly do the trick - even keeping samples in the right order... except it will only return approximately n elements.

If we're sampling 1000 out of a billion we might not really care all that much, but it turns out it's not so difficult to do better than that. Sampling n elements out of [first, *rest] collection neatly reduces to: [first, *rest.sample(n-1)] with n/size probability, or rest.sample(n) otherwise. Except Ruby doesn't have decent tail-call optimization, so we'll use counters for it.

module Enumerable
  def random_sample_known_size(wanted, remaining=size)
    if block_given?
      each{|it|
        if wanted > rand(remaining) 
          yield(it)
          wanted -= 1
        end
        remaining -= 1
      }
    else
      rv = []
      random_sample_known_size(wanted, remaining){|it| rv.push(it) }
      rv
    end
  end
end

This way of sampling has an extra feature that it can yield samples one at a time and never needs to store any in memory - something you might appreciate if you want to take a couple million elements out of 10 billions or so, and you will not only avoid loading them to memory, you will be able to use the results immediately, instead of only when the entire input finishes.

This is only possible if collection size is known - if we don't know if there's 1 element ahead or 100 billion, there's really no way of deciding what to put in the sample.

If you cannot fit even the sample in memory at once, and don't know collection size in advice - it might be the easiest thing to iterate twice, first to compute the size, and then to yield random records one at a time (assuming collection size doesn't change between iterations at least). CPU and sequential I/O are cheap, memory and random I/O are expensive.

When size is unknown

Usually we don't know collection size in advance, so we need to keep a running sample - initialize it with the first n elements, and then for each element that arrives replace a random one from the sample with probability n / size_so_far.

The first idea would be something like this:

module Enumerable
  def random_sample(wanted)
    rv = []
    size_so_far = 0
    each{|it|
      size_so_far += 1
      j = rand(size_so_far)
      rv.delete_at(j) if wanted == rv.size and wanted > j
      rv.push(it) if wanted > rv.size
    }
    rv
  end
end

It suffers from a rather annoying performance problem - we're keeping the sample in a Ruby Array, and while they're optimized for adding and removing elements at both ends, deleting something from the middle is a O(size) memmove.

We could replace rv.delete_at(j); rv.push(it) with rv[j] = it to gain performance at cost of item order in the sample... or we could do that plus Schwarzian transform into [index, item] space to get correctly ordered results fast. This only matters once sample size reaches tens of thousands, before that brute memmove is simply faster than evaluating extra Ruby code.

module Enumerable
  def random_sample(wanted)
    rv = []
    size_so_far = 0
    each{|it|
      size_so_far += 1
      j = wanted > rv.size ? rv.size : rand(size_so_far)
      rv[j] = [size_so_far, it] if wanted > j
    }
    rv.sort.map{|idx, it| it}
  end
end

This isn't what stream processing looks like!

The algorithms are as good as they'll get, but API is really not what we want. When we actually do have an iterate-once collection, we usually want to do more than just collect a sample. So let's encapsulate such continuously updated sample into Sample class:

class Sample
  def initialize(wanted)
    @wanted = wanted
    @size_so_far = 0
    @sample = []
  end
  def add(it)
    @size_so_far += 1
    j = @wanted > @sample.size ? @sample.size : rand(@size_so_far)
    @sample[j] = [@size_so_far, it] if @wanted > j
  end
  def each
    @sample.sort.each{|idx, it| yield(it)}
  end
  def total_size
    @size_so_far
  end
  include Enumerable
end

It's a fully-featured Enumerable, so it should be really easy to use. #total_size will return count of all elements seen so far - calling that #size would conflict with the usual meaning of number of times #each yields. You can even nondestructively access the sample, and then keep updating it - usually you wouldn't want that, but it might be useful for scripts that run forever and periodically save partial results.

To see how it can be used, here's a very simple script, which reads a possibly extremely long list of URLs, and prints a sample of 3 by host. By the way notice autovivification of Samples inside the Hash - it's a really useful trick, and Ruby's autovivification can do a lot more than Perl's.

require "uri"
sites = Hash.new{|ht,k| ht[k] = Sample.new(3)}
STDIN.each{|url|
  url.chomp!
  host = URI.parse(url).host rescue next
  sites[host].add(url)
}
sites.sort.each{|host, url_sample|
  puts "#{host} - #{url_sample.total_size}:"
  url_sample.each{|u| puts "* #{u}"}
}

So enjoy your massive data streams.

Synchronized compressed logging the Unix way

2010-07-15T00:40:00.000+02:00

In good Unix tradition if a program generates some data, in general it should write it to STDOUT, and you'll redirect it to the right file yourself.

There are two problems with that, both easily solvable in separation:

If it's a lot of data, you want to store it compressed. It would be bad Unix to put compression directly in the program - the right way is to pipe its output through gzip with program | gzip >logfile.gz. gzip is really fast, and usually adequate.
You want to be able to see what were the last lines written out by the program at any time. Especially if it appears frozen. Sounds trivial, but thanks to a horrible misdesign of libc, and everything else based on it, data you write gets buffered before being actually written - a totally reasonable thing - and there are no limits whatsoever how long it can stay in buffers! Fortunately it is possible to turn this misfeature off with a single line of STDOUT.sync=true or equivalent in other languages.

Unfortunately while both fixes involve a single line of obvious code - there's no easy way to solve them together. Even if you flushed all data from the program to gzip, gzip can hold onto it indefinitely. Now unlike libc which is simply broken, gzip has a good reason - compression doesn't work on one byte at a time - it takes a big chunk, compresses it, and only then writes it all out.

Still, even if it has good reasons not to flush data as soon as possible, it can and very much should flush it every now and then - with flushing every few seconds reduction in compression ratio will be insignificant, and it will be possible to find out why the program frozen almost right away. The underlying zlib library totally has this feature - unfortunately command line gzip utility doesn't expose it.

So I wrote this:

#!/usr/bin/env ruby

require 'thread'require 'zlib'

def gzip_stream(io_in, io_out, flush_freq)
  fh = Zlib::GzipWriter.wrap(io_out)
  lock = Mutex.new
  Thread.new{
    while true
      lock.synchronize{
        return if fh.closed?
        fh.flush if fh.pos > 0
      }
      sleep flush_freq
    end
  }
  io_in.each{|line|
    lock.synchronize{
      fh.print(line)
    }
  }
  fh.close
end

gzip_stream(STDIN, STDOUT, 5)

It reads lines on stdin, writes them to stdout, and flushes every 5 seconds (or whatever you configure) in a separate Ruby thread. Ruby green threads are little more than a wrapper over select() in case you're wondering. The check that fh.pos is non-zero is required as flushing before you write something seems to result in invalid output.

Now you can program | gzip_stream >logfile.gz without worrying about data getting stuck on the way (if you flush in your program that is).

Palestinian problem fixed

2010-07-02T19:09:00.000+02:00

This solution is really really simple, has plenty of precedent in history, and yet I haven't heard it proposed seriously by anyone before. So here is goes:

Palestinians should convert to Judaism

According to Israeli laws, anybody who converts to Judaism can trivially get Isreali citizenship. Once sufficient number of Palestinians gets citizenship, not only will they no longer be persecuted as much (Israeli Arabs are still treated as second class citizen, but level of discrimination is much much less) - together with existing non-hawkish Israeli citizens they will constitute majority of voters who will ensure that Israel will become a country that's much more peaceful and friendly toward its neighbours. As a bonus on top of that such mass conversion would immensely piss off radicals on both sides.

Now you will very likely complain about freedom of religion and such stuff. But why are these people Muslim in the first place? Because their ancestors converted to religion of previous conquerors of these lands - some were forced to, others out of opportunism, perhaps some even out of genuine conviction - it doesn't matter. Is following religious choices of ancestors really worth all the death and suffering that exists in Palestine right now? I say no. Especially since the most painful part of such conversion - genital mutilation - they all underwent already.

This wouldn't be without precedent - not only religions like Islam and Christianity, but also languages like Latin, English, and Spanish, national identities, and cultures have all spread largely by members of the losing group adapting habits of members of the winning group. In fact it would be a lot more difficult to point to any major religion or nationality which didn't spread in such a way.

It's a topic for whole another post, but ignoring this is the primary reason why I hate most analysis of humans in terms of evolutionary biology so much - variance of human behaviour is almost entirely culture-driven, not gene-driven like they all pretend, and culture doesn't only spread from parents to children.

Anyway, it wouldn't be necessary for everyone to convert - as long as the portion of Palestinians who do is high enough - let's say half - it would completely alter current conflict dynamics enough to solve it. The die-hard few who care too much about Islam can then safety practice their faith in the emerging unified single state (with a huge Jewish majority, even if a lot of that would be rather unenthusiastic converts). As the law doesn't forbid it - children of Palestinian converts could choose whichever religion they wanted - Judaism, Islam, Catholicism, Baha'i, Spaghetti-Monsterism, or whatever. In practice vast majority would simply follow religion of their parents.

Alternatives

Everything else having been tried and having failed already - there is really only one alternative to such mass conversion - waiting a few decades for both Israel and Palestine (and if possible all other countries of the region) to undergo demographic transition from short lives and big families to long lives and small families. Countries after demographic transition are a lot less belligerent.

This is very much happening - but very slowly. According to CIA World Factbook total fertility (number of children per women) between 1990 and 2010 fell from 7.0 to 4.9 for Gaza Strip (still ridiculously high), from 5.0 to 3.12 for West Bank, and from 2.9 to 2.72 for Israel - moving them away from values typical for the worlds' war zones like Afghanistan's 5.50 and Democratic Republic of Congo's 6.11 and more like civilized world's 2.1 and less.

One day it will very likely happen - all will undergo demographic transition, and while they might still hate each other as much as they do now - they will express it like Japanese and Koreans - by flaming each other on the Internet - not in the good old way of bombs and bullets.

This will of course take many decades. My solution solves the whole problem overnight. Any takers?

Best sources of DHA omega-3 essential fatty acid

2010-06-09T10:07:00.002+02:00

This is surprisingly difficult to find out, so I decided to share the results with everyone. But first, background:

Animals need omega-3 and omega-6 essential fatty acids
The same enzymes are used for omega-3 and omega-6 processing, so too much of one will interfere with the other. People used to have diets with about 1:1 omega-3:omega-6. Today ratio is more like 1:20, and that little omega-3 we eat is mostly ALA.
Three most important omega-3 are ALA (18 carbons), EPA (20 carbons), and DHA (22 carbons).
Brains are made largely out of DHA.
Land plants produce no EPA / DHA. None whatsoever. Zero.
Some land plants produce adequate amounts of ALA, but even this is uncommon.
Algae produce quite a lot of EPA / DHA.
Animals including humans can convert ALA to EPA, and then DHA, but this is a painfully slow and inefficient process; and over-saturation of omega-6 and many conditions interfere with even that much.

Based on this some people believe it would be wise to try to increase amount of omega-3 fatty acids in diets. Hard evidence is rather lacking, this is however to be expected as hard evidence of anything about diet is essentially nil. It's almost only short term studies of crappy proxies, and there are millions of reasons why this is just wrong. Anyway, concerning supplementation:

All mixed omega-3/omega-6/omega-9 supplements are waste of money - you're eating too much omega-6 already, and you can make as much omega-9 as you wish yourself.
You don't want generic "omega-3" supplements - most of these are ALA, which is of very limited use. You want DHA. At worst EPA. ALA is little more than filler, it's not bad for you but it's less relevant, and much easier to get via normal diet anyway.
If you're surprised why it's so hard to get DHA, this is possibly highly relevant.

Hopefully now you see why I'm measuring DHA, not anything else. And to keep science proper what I'm interested in is "% of calories coming from DHA", not "grams of DHA per portion" or anything like it. Portions are whatever manufacturer says they are, "per 100g" measures mostly tell you how much water foods have, and only "% of calories from" measures the right thing.

Best Sources of DHA

I digged through USDA National Nutrient Database, and this is what I found.

The data only contains "food" not supplement pills and such. These will of course contain highest concentrations. Rarely eaten foods like dolphin meat are not in the database, so I have no idea how nutritious dolphin sashimi would be.
The best source of DHA is unsurprisingly - fish oil. Salmon oil is 18.2% DHA and 34.2% omega-3 altogether; other fish oils are pretty good too, but not so much. Other oils like cod liver, sardine, and menhaden are 8.5%-10.9% DHA, 18.8%-26.6% total omega-3. Herring oil is less impressive 4.2%/11.1%. Fish oil also contains most mercury and other poisoning, so enjoy that. Once ultra-refined you won't need to worry about poisoning but it's more supplementation than food.
The second best source is caviar/roe, with 7.7%-13.6% DHA, and 13.7%-24.2% total omega-3. Might be expensive to turn it into a major part of your diet.
The third best source is seal oil with 6.5-12% DHA, 14.0-27.7% omega-3. Let's see if you can buy some legally outside Canada. So far we're totally out of luck.
Finally something more useful. The fourth best source is salmon. There's wide range of nutritious value from 2%/4% to 8.9%/13.7% - depending on where they're caught and what's their diet. Fortunately there doesn't seem to be a big difference between wild and farmed salmon, so either will work. Unfortunately the same mercury poisoning problem applies as to fish oil - poisonous substances are stored in oil so the oilier (and more useful for us) the fish the more toxic, and there is no way to escape that.
Fifth best source is mackerel. Like salmon, it can have as little as 1.5%/2.8% or as much as 8.7%/14.7%.

USDA says the next best source is dried parsley leaves at 7.3%/9.9% - which is most likely a massive measurement error, as there are no other plants anywhere, and it really makes little sense.

Other than that, it's fish, fish, fish, mollusks, jellyfish, crustaceans, and more fish. My hopes for finding something that's not fish are getting slimmer and slimmer, so I'm just going to skip all of them now (you can probably see the pattern), and only focus on things which are not fish/seafood.

Brains. Beef/lamb/pork brains have 3%-5.4% DHA and 4.4%-7.7% total omega-3. Not surprisingly, as that's what animals primary need DHA for. And we simply throw away this most nutritious part.
Whale oil - 3.9%/8.3%. Whale meat on the other hand is pretty useless at 0.2%/0.5%. Not that you'll find much of either at the nearest supermarket. It's technically not a fish. Anyway, between brains and ocean creatures we pretty much ran out of good sources, the next source is:
Roasted squirrel - and that at mere 0.5%/0.6%
Chicken can be anywhere from 0.1%/0.2% to 0.5%/0.8%
Egg yolk - 0.3%-0.4% DHA, 0.4%-0.7% total.
Whole egg - 0.2%-0.3% DHA, 0.2%-0.8% total. Pretty much all of that in yolk.
We're long past useful concentrations anyway, so I won't be listing them. Next on the list are caribou, green turtles, turkeys, lamb kidneys, frog legs, guineahen, lamb hearts, squab/pigeon, pork livers, bear, raccoon, pork lung (and other beef/lamb/pork offal), and emu. Pork/beef/lamb meat doesn't register other than as rounding error.

So to summarize:

Get supplements;
Or eat a lot of fish and other seafood;
Or eat ridiculous amount of poultry, eggs, and game meat;
Or you're fucked.

There's no way to get enough DHA in anything resembling standard Western diet. Simply no way. Fruits and vegetables contain none, even "organic" ones. Actually fast food contains more as it often uses eggs and poultry as ingredients, but that's still not that useful.

One more long-term option would be to genetically engineer some common oily plants like soy or canola to produce some EPA/DHA - even if humans wouldn't eat them, if they're used as animal feed, we'd benefit indirectly quite a lot.

Actually someone already genetically engineered pigs to produce 4x omega-3 fatty acids, including 2x DHA, which sounds like the most urgent reminder that we need GMO now, and Luddites should not be in charge of policy.

EDIT: GM soybeans with more omega-3 (this most likely means ALA) and higher stability so they don't need partial hydrogenation has just been approved in US. If people moved from usual partially hydrogenated soybean oil to that it would be a massive health benefit. Of course our Euro-Luddites + CAP-paid farmer lobby coalition will probably ban it until long after "America vs Europe" picture get reversed. It's already much closer than you think.

What is Internet good for?

2010-06-02T07:21:00.000+02:00

Here a quick poll, pick the most fitting answer:

I spend too much time on the Internet, and I'm aware of this
I spend too much time on the Internet, I'm deluding myself about it

There's no need to bother with the third option, as people who don't use too much Internet are extremely unlikely to ever read this blog.

But in the spirit of "What have the Romans ever done for us?" - what good is Internet really for? Is it worth all the time we spend on it?

For the last two weeks in the spirit of Alicorn's luminosity - by the way definitely read that linked article, her first attempt was at that was a major tl;dr but these "seven shiny stories" are so short and insightful you're probably better off spending some time reading them than whatever else you're typically wasting your time on Internet, and it promoted her to my second favourite lesswrong writer after Eliezer.

So as I was saying before I interrupted myself, for the last two weeks I've been making a log of my daily activities and how much satisfaction I actually got from a given day.

Now many people who have learned economics 101 and are treating it too seriously think that whatever we're doing must by definition be the things we most enjoy, our claims to the contrary notwithstanding - so people who say the want to get thinner, but eat fuckloads of pizza actually prefer eating pizza and being fat to not eating pizza and being thin. This point of view is usually something worth considering - people usually say they want things they feel they're "supposed to want". On the other hand, the amount of evidence that we don't do what's best for us is ridiculously overwhelming.

A very very short list of such examples would include:

hyperbolic discounting - there's only one "mathematically consistent" way of treating values over time (exponential discounting), and the evidence is completely unambiguous that we're not doing so.
rodent experiments show also rather unambiguously that brains have separate systems for "wanting something" and "liking something". They are of course connected, so other things being equal if we like something more we will probably want it more - but this influence is far far less than total identity assumed by economics 101. Once you're aware that people might "want" things they don't really "like", and "not want" things they "like" (by the way - I'm using the words "want" and "like" rather vaguely - natural languages are spectacularly bad when analyzing humans - quite surprising actually as they seem to have been originally developed largely for social use) the entire utilitarian / consequentialist framework of analysis collapses.
Happiness research in spite of all its ambiguity at least shows that simple models of happiness are plain wrong. If you have time to waste on the Internet - TED is filled with good talks about happiness, not just the one linked.
Even disregarding these, you'd need to have ridiculously good information on yourself to decide what's the best thing to do. And it would be a massive understatement to say that we don't have it. And it's really really difficult to measure yourself. The idea behind "living luminously" is that increasing your self-awareness of your own mental state even somewhat might lead to highly positive results.
and many many more

Economics 101

By the way - and if you don't enjoy how I keep straying away from whatever is my main subject all the time, you probably shouldn't be reading this blog - I'm in no way disparaging "economics 101" thinking. I find it really sad that virtually every single person in the world pretty much belongs to these two classes:

People who don't understand Economics 101. They fail to get even such basic notions like comparative advantage or externalities.
People who get Economics 101 - and take it far too seriously. If you even briefly look at assumptions behind all its theorems, none of them is even approximately true in the real world. Very often you get lucky and this toolkit lets you predict things about the real world decently enough, but this is about as often not the case.

This is not to say Economics 101 should be thrown away. All models are wrong, some are useful. Or from a closely related perspective - all abstractions leak. If you don't perform sanity checks, and blindly trust everything the models tell you, they will lead you far astray (you could always argue that it's not models' fault, it's fault of the way you're applying them - but this is a purely theoretical distinction).

So for example in theory economics 101 models say that laissez-faire international trade policy should outperform any kind of intervention, but in practice countries which practice export subsidies by currency manipulation like the East Asia are better off than those with less laissez-faire trade policy like Europe, which in turn are better off than those that try to follow the route of import substitution via high tariffs like Latin America.

By the way if you find this curious, the standard answer to this puzzle is that benefits of economy of scale overwhelm loses due to comparative advantage - pumping subsidies into narrow range of related industries lets your country specialize in those, and import everything else - while limiting imports of wide range of products to protect diverse local industries means they will all be small and weak. As economics 101 completely ignores the dominant factor of scale advantages, focusing on an undoubtedly real but less important factor instead.

Similar misapplication of economics 101 says that minimum wage laws invariably increase unemployment. This was universally believed by nearly all economists a few decades ago according to some surveys cited by Wikipedia, and holding such belief became almost the canonical way of signaling that you're "economically savvy". And not surprisingly it turned out to be false, all research showing either no effect whatsoever, or effect that is really tiny compared to the huge increase in well-being of the working poor. The economists have finally figured that out, and they're more or less evenly divided on the question - even those standing against the minimum wage laws typically holding much more nuanced views - and yet some naive hard-liners still hold this as a measure of "economic enlightenment".

My logs

Getting back to the subject, for each day in a bit over the last two weeks I recorded my activities, and some measures of satisfaction. The logs weren't terribly detailed, and "recording one's satisfaction" is exactly the kind of thing which would never get published in any reputable peer-reviewed journal. That's not to say that it's useless - "the official way of doing science" has led to many important discoveries, but it's for many questions it's been rather impotent, and it's important that people try different ways of finding things out too, if for nothing else then to fill in the blind spots of the mainstream science.

So my logs, of however dubious methodology they are, seem to point to the following correlations. I won't even bother pretending to have any "statistical significance" in all that of course, even forgetting about small sample of just two weeks measuring "statistical significance" necessarily assumes that samples are essentially independent, and they're nothing like that. The list is:

Physical exercise of all kinds - definitely positive - this isn't really surprising, as this is something that's highly enjoyable once started, but it takes effort to begin, so the hyperbolic discounting excuse applies
With video games it's mixed. First person shooter games like online Modern Warfare 2 have positive correlations, but Total War games negatively correlate with my end-of-the-day satisfaction, even though they're not really frustrating or anything most of the time.
Cleaning up my GTD system - definitely positive
Being productive at work, and in general getting done things I want to get done, especially the long postponed ones - definitely positive
Reduction in caffeine consumption - mildly negative, but that doesn't really imply anything about long term effects of different levels of caffeine, and it wasn't even as bad as I expected
Watching TV series, and reading books - mildly negative; this might be a false result, or the effect might be real but minor, in any case - there's no reason to do much more of those
And the largest and rather surprising correlation - spending more time online has a huge negative correlation with my satisfaction levels

Now there standard economics 101 disclaimer applies - these measures are necessarily marginal - so finding out that I'm better off exercising more and Internetting less implies only that I would be better off exercising a bit more than now, and Internetting a bit less than now, and it's more likely than at some point increasing amount of exercise and decreasing Internet use will make me worse off.

It's an interesting find that I seem to actually enjoy my work - I should probably put that one in my CV for future reference. And it's nice to get some insight on what kinds of recreational activities work better for me than others (due to small sample size less repeatable events like those involving interaction with other people not included). But the big find is that Internet is bad for me, and let's focus on that.

What is Internet good for?

Do you remember what life was before the Internet? It was horrible! We had to copy games from friends on stacks of floppies instead of just bittorrenting them! But really, what good is Internet for?

Email and IM are always far superior means of communication than phone calls (I really hate those, they should all die in fire); and are so much faster and easier than driving all the way to meet someone in person that they usually win, even if the throughput is somewhat less.
There's shopping - at which Internet really excels, most of the time. At least when you know exactly what you want, otherwise not so much.
There is information - but I'm far from happy about it. For some kinds of thing that you want to find, if they follow "keyword keyword of keyword" pattern, you can usually google or bing it out in seconds. Otherwise, all search engines become nearly useless, even if this information is somewhere. And it requires a lot of knowledge to turn a problem into unique keywords - very often you know little more than "X doesn't work", or "I'm not happy about Y", and search engines won't help you with those at all. This assumes information is even online in the first place - as very often it's not - it's really sad how nearly all research papers have been successfully pay-walled. It seems that "information wants to be free" only when the information in question is something on the top 100 bestsellers list of one kind or another, and not to the long tail which contains most of the real value.
There's Google Maps and similar sites, which are far superior to paper maps.
There are some funny things online - but they're swamped by such amounts of unfunny repetitive material that I really doubt Internet is even good for that. I dare you, go to let's say /r/funny on reddit - which is supposedly about the most recent funny stuff online (or at least reddittors seem to believe they find stuff first, and everyone else copies stuff from them) - and how many things you'll find there that will make you laugh, and are not nearly ancient? And it's the same on nearly every other place which is supposedly filled with funny stuff. People keep going there because occasionally something good turns out, but it's so rare it's probably not worth it.
There are news - and again the flood problem applies. I'm yet to find any RSS feed with only important news. Everyone seems to believe the right way to do news online is to just throw 10+ trivia items a day - Obama said something, one minor celebrity divorced another minor celebrity, stock prices decreased somewhere, IDF shot a few unarmed civilians somewhere else - as if knowing things like these made you better off in any way. The choice is to either get ridiculous amount of political trivia, or just ignore the news altogether.
There are all kinds of social networking sites - and I'm increasingly doubting their value. I have a Facebook account (and accounts on some other sites) and I might even use it occasionally, but I don't see that my life would be that much worse if Facebook and the rest didn't exist.
There are blogs, wikis, and similar places where you can contribute your knowledge, which can give tremendous amount of satisfaction to the contributor. If you add them all up, they provide a lot of value for readers as long as search engines manage to find a relevant blog post or wiki article, which is always in doubt. On the other hand, I have serious doubts about reading everything on a blog, or unfocused browsing on wikis. I haven't yet seen a blog which had consistently good posts - much less consistently good and relevant to my interests.
There are online video games, for some things playing with people is more fun than playing against computer.

These seem to cover the main points. And what's obvious is that the most valuable online activities - email, highly focused search, shopping, maps - take rather little time. On the other hand, the ones that take a lot of time - like all the reddits, forums, social sites, wikis, blogs, etc. - don't provide that terribly much value per time spent.

Unless you actually measure how you use internet, it's really easy to overestimate how much value spending time on it gives you - as you're far more likely to remember the high points which didn't really take long - as opposed to relatively pointless activities which took most of your online time. Human memory just works like that.

This distinction would be pointless if it was impossible to make a distinction between the two - if reduction in the bad kind of internet use required essentially proportional reduction in the good kind. Fortunately it seems to me that this is fairly straightforward - good things like email (this of course assuming you have a working spam filter / and all mailing lists etc. go somewhere else than your inbox), shopping, maps, directed search - have different entry points than less useful things like social sites, wikis, news, and funnies etc. Sometime you'll look for something specific and in the process accidentally fall into a wiki trap, but this shouldn't be too common with some self-awareness. Much more often you waste a ridiculous amount of time by wanting to "quickly check if there's anything good on X" and having hyperbolic discounting ("just one more link") turn that into a disaster.

It doesn't mean reducing your lolcat consumption to zero - only that you should force yourself to make an up-front decision "I'm start looking at funnies now, even though I know well enough it will probably take the next few hours" and having a realistic idea how good this time will be; instead of fooling yourself it will be quick and only filled with the good stuff, as seems typical now.

tl;dr - using internet only when you have clear goal, and not for vague "maybe there's something good" is good for you.

Empire Total War mods - no walls, libertarians everywhere

2010-05-22T22:54:00.000+02:00

Rome Total War was highly moddable. Medieval 2 Total War somewhat less as all data files were in obfuscated packages - but once you unpacked them it was as good as Rome. My M2TW mod is generated by a bunch of simple regexps, with which I can experiment as much as I like, turning features on and off and changing their magnitude in seconds.

I knew Empire Total War won't be as easy, but nothing prepared me for the pain I suffered. But I'll complain as I go.

Libertarians everywhere

First to see how modding works I wrote a very small mod - turning off taxes adds +10 happiness. Not quite literally, as it seems impossible to have anything triggered by zero taxes, so I gave all non-zero taxes extra -10 happiness, and every government type gets extra +10 happiness - net result being what I wanted, except it looks a bit silly in game.

Now why did I do so? The answer is my favourite "less micromanagement". I don't particularly having to babysit conquered provinces, chasing rebels around, and counting how many units I can move and how many I need to leave. This is simply not fun. So I decided that every province has a sizable population of armed libertarians who will gladly shoot every protester for me as long as I set taxes to zero. When taxes are not zero they blog climate change denial or something, I don't care.

This is all a stop-gap measure for the first few turns after conquest - if you ever want to get any taxes out of the province, as you usually do, you'll need to deal with taxpayers' happiness eventually. By the way zero taxes essentially means provinces loses money every turn, as it increases administration cost of every other province in your empire.

Happiness +10 is enough most of the time, but when you conquer someone's capital and have to face -30, on top of all industrialization, religion etc., you might need to deal with rebellions anyway. And as such regions usually bring a lot of money, you probably want to set higher taxes anyway.

If you want to tweak this bonus, use DBEditor for it.

No walls

And now the big mod - removing all walls. I'm not ideologically opposed to settlement fortifications - in fact my M2TW mod makes settlements more difficult to take. However:

ETW sieges are broken due to stupid AI and stupid pathfinding
Because they're broken, all ETW sieges follow just two boring scripts:
- either: approach diagonally with infantry from 3 directions, bayonet charge everything;
- or: approach diagonally with howitzers with carcass shot and a few line infantry units, once you start bombarding them AI will charge you one by one and you win without loses
everything else fails as your units are too stupid to pathfind in more interesting strategies
After 1710 or so nearly every settlement outside Americas has city walls
And so after the first few years, 90% of battles is unbelievably boring

Field battles on the other hand are much more interesting. An unfortunate side effect of this mod is that while in vanilla town watch can use settlement fortifications to defend it from small enemy armies reasonably well, now it's completely useless. I can live with that.

Unfortunately, this has second order side effect of forcing you to be more aggressive. If before you'd be quite willing to have your armies further away from the front as town watch could serve as good enough first line of defense, now you need to have your armies closer to the borders, preferably on their side of it.

So how did I write this mod? It was truly painful. First, ETW has building features, and the fact that settlement fortifications building creates walls in the battle is supposedly encoded by such features. I ran into the first problem, as DBEditor is incapable of removing features - only adding or modifying them. To do such thing you need to go to PackFileManager, clone a table, make sure it's named exactly like the original one (so it will be overridden, not merged), and remove rows from that in DBEditor.

Except for some reason PackFileManager incorrectly mixes up slashes and backslashes, so I needed to fix the mod file from a hex editor.

Except it turned out in the end ETW ignores this feature, and simply hard-codes walls. So much for my effort.

It was plan B time. First I needed to remove all walls. And while db files are pain to edit it is nothing compared to the pain of editing startpos.esf which contains campaign information:

Start campaign, look at every single settlement on a map and write down which settlement has walls (as there's no search function in EsfEditor)
Manually find every such settlement in esf tree, and do all necessary changes (change True to False in one node, delete another node).

This is of course wholly incompatible with any other esf mod, like those which enable you to play minor factions.

After that I needed to make it impossible to build walls. I haven't figured out how to do that (I suspect if I remove walls completely the game will simply crash, and no other building in unbuildable) - so I made it require late technology of mass production, and take 999 turns to build. Sort of good enough.

And all this was possible only after a lot of effort of modding community members who created tools like DBEditor, EsfEditor, PackFileManager, documented what they could etc. This is borderline unmoddable, and I fully expect the next generation of Total War games after Napoleon (which is essentially extra scenario for ETW) to be completely impossible to mod. But hey, maybe they'll sell more DLC this way.

How to install

In case you want to install these mods, you need to:

Download starpos.esf and replace one in C:\Program Files (x86)\Steam\steamapps\common\empire total war\data\campaigns\main (or similar), first of course making a backup copy - this will remove starting walls. (small warning - it's a big file and the server I'm hosting it on is pretty slow; other files are tiny)
Download mod_no_walls.pack and put it in data directory together with all other packs - this will make walls unbuildable.
Optionally download mod_tax_break.pack and put it into data directory - this will give you +10 happiness bonus for no taxes.
Download ModManager and unpack it.
Start ModManager, select mods you want to use, and click Launch.

Greasemonkey and jQuery easier than ever

2010-05-22T21:23:00.002+02:00

Last year I wrote a short Greasemonkey tutorial, in which I explained how to use it with jQuery for some really simple scripts. Since then it became even easier, so here a few more useful scripts.

Since version 0.8 Greasemonkey has @require feature, in which your scripts can be made to depend on some external Javascript files - like jQuery. Unfortunately there are a few gotchas:

jQuery 1.4 doesn't work with Greasemonkey, you need to use jQuery 1.3 (or keep pestering them until it's fixed)
@require only works on installation time, you cannot use it with "New User Script..." feature, nor can you change @requires from the editor. If you simply use published scripts and don't write anything - you don't need to worry. If you write your own scripts you need to prepare something.user.js file somewhere, then open it from Firefox (copy&paste it's file path to Firefox URL bar, then click Install).

Show spoilers on tvtropes

Let's start with something really trivial. I don't care about spoilers. Maybe 1% of "spoilers" significantly diminish viewing pleasure, vast majority of them do not. I knew that Vader is Luke's father, that Snape killed Dumbledore, and that Titanic sunk before watching/reading, and it made it no less enjoyable.

On the other hand I'm quite annoyed that to read tvtropes I needed to constantly select "spoiler" text for everything to see it - but no more. This trivial script solves it entirely:

// ==UserScript==
// @name           Show all spoilers on tvtropes
// @namespace      http://t-a-w.blogspot.com/
// @include        http://tvtropes.org/*
// @require        http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js
// ==/UserScript==

$(".spoiler").removeClass('spoiler');

It's really simple - @name is unique name for the script, which should really be a very short description. We're not going to rely on @namespace at all, so put anything you feel like there.

@include is a pattern of URLs for which the script should be executed. @require is the jQuery library we're including. And thanks to jQuery, removing spoiler tags is really easy.

You can download this script here.

Always sort by seeders on The Pirate Bay

And extremely annoying thing about The Pirate Bay search engine is how it sorts results by its idea of "relevance" - usually giving you some dead ISO of three year old Ubuntu version when you obviously want the most recent one. And nearly always sorting by seeders is the right thing to do.

There is actually a GreaseMonkey script that claims to do exactly that, but it doesn't always work correctly, and the author decided to obfuscate the source for lulz. I'll have none of that, so I just wrote my own. Again, thanks to jQuery it's truly trivial.

// ==UserScript==
// @name           Always order by seed count
// @namespace      http://t-a-w.blogspot.com/
// @include        http://thepiratebay.org/*
// @require        http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js
// ==/UserScript==

$("input[name='orderby']").val(7);

You can download this script here.

Download Stewart and Colbert from bittorrent

Now a more complicated and perhaps more useful script. One of the recent developments on the Internet I hate the most are geographic restrictions - Americans can view whatever they like, for everyone else it's "Sorry, Videos are not currently available in your country" or other such nonsense.

Personally I don't care about licensing restrictions which led to this - I'm not going to accept development like that if I can help it. Fortunately in this case I can - all Stewart and Colbert is available on bittorrent.

The script does the following:

Find all dates on the page. They're not kind enough to use consistent class for those, I found at least 3, and perhaps I still missed a few.
Convert every date to from "Month DD, YYYY" to "YYYY MM DD" format, as this is what most torrents use. Half of the script is just doing that because Javascript doesn't have anything like Ruby's Date.parse.
Append link to bittorrent site next to the date. I'm using yourbittorrent here, as ThePirateBay is down due to overload far too often.

So every time someone on Reddit posts a "omg Colbert totally pwned Glenn Beck" link, I can actually see the pwnage now thanks to this script.

// ==UserScript==
// @name           View Stewart and Colbert on bittorrent
// @namespace      http://t-a-w.blogspot.com/
// @description    No more "Sorry, Videos are not currently available in your country"
// @include        http://www.thedailyshow.com/*
// @include        http://www.colbertnation.com/*
// @require        http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js
// ==/UserScript==

var month_names = {
  'January': '01',
  'February': '02',
  'March': '03',
  'April': '04',
  'May': '05',
  'June': '06',
  'July': '07',
  'August': '08',
  'September': '09',
  'October': '10',
  'November': '11',
  'December': '12'
};

$(".date, .airDate, .clipDate").each(function() {
  var txt = $(this).text();
  var m = txt.match(/(\S+)\s*(\d+),\s*(\d{4})/);
  var date = m[3] + "+" + month_names[m[1]] + "+" + m[2];
  var url;
  if(window.location.hostname == "www.thedailyshow.com") {
    url = 'http://www.yourbittorrent.com/?q=Daily+Show+' + date;
  } else {
    url = 'http://www.yourbittorrent.com/?q=Colbert+Report+' + date;
  }
  $(this).append("<div><a href='"+ url + "'>Download on bittorrent</a></div>");
});

You can download the script here.

See how easy it all was? Now start writing your own scripts and share them.

Everybody Draws Muhammad Day and lessons in cultural relativism

2010-05-20T07:04:00.001+02:00

May 20th is Everybody Draw Mohammed Day - a day when you too can join the defense of Free Speech by drawing a cartoon. Or if you really suck at drawing you can make a shop of Muhammad like me.

Prophet Muhammad, artist's conception
If you think I'm wrong, draw a better one

Lesson #1 - Principle of dissimilarity

Recently I have been convinced by some TTC audiobooks that Jesus might have probably existed as a historical person. You know what's the best argument for it? The criterion of dissimilarity.

It's not any pomo deconstructionism - just plain old historical analysis. The idea is that authors have an agenda, but life doesn't always agree with it. So every time the text admits to something that is clearly against the authors' agenda, it suggests it probably had some basis in reality - because they wouldn't make this up on purpose. A few examples from Jesus' life:

Crucifixion was embarrassing kind of death reserved for slaves, rebels, and other lowest status scum. If someone was making up the story they would have Jesus die in battle, or die some other "respectable kind of death". So actual Jesus was most likely actually crucified.
All the messy explanations how "everybody thought Jesus was born in middle-of-nowhere Nazareth but actually he was born in Bethlehem like King David" suggest that Jesus was probably born in Nazareth. If they were making this up, they'd skip the Nazareth story altogether.
Jesus was baptized by John the Baptist - being baptized by someone was signaling submission and inferiority to that person. This story is clearly embarrassing to Bible writers, they put words like "I should be baptized by you" in John the Baptist's mouth, and the last Gospel omits it altogether.

The sheer volume of such cases suggests that Bible is an "enhanced" story of some actual life, as opposed to being completely made up. There's no question that all the supernatural bits have been inserted, and it's widely believed that the real life stories were quite significantly massaged - but it does such a bad job at covering a lot of embarrassing parts like these that the hypothesis "Bible is vaguely based on a life of a real person, and these embarrassing stories were widely known so couldn't be ignored" is much more compatible with the evidence than "Bible is completely made up" - and as their Bayesian priors are not terribly different in the first place, historicity of Jesus can be reasonably believed in. The "Bible is real" story on the other hand is both against the priors and against the evidence, so let's not even go there.

Lesson #2 - Why is Muhammad widely considered a pedophile?

The criterion of dissimilarity must be truly infuriating to the true believers - it essentially says that every time they like some part it's probably false, and every time they dislike some part it's probably true. By the way this applies to all historical texts, not just religious ones - parts of Commentarii de Bello Gallico that are overly sympathetic to Julius Caesar should be looked at with more suspicion than parts which talk about his failures, and so on.

So, what does it all have to do with Muhammad being a pedophile? Muslim texts clearly say that he was married to Aisha when she was 6, and fucked her when she was 9 year old.

Now how likely is it that this particular bit was made up? If you had a prophet who didn't fuck children, would you casually make up a story that he actually did? That'd go against all rules of proper writing. And it's not one isolated verse somewhere - as Wikipedia says "references to Aisha's age by early historians are frequent", and nobody questioned that back then.

So regardless of our believes, we can be fairly certain that

Muhammed fucked a 9 year old girl

And now some time for cultural relativism - this was considered fairly unremarkable back then. Our modern culture is obsessed about young people's sexuality and as a civilization we essentially lost the ability to propagate the species, but historically it was entirely normal for much younger people to marry, fuck, and have babies - biologically late teens are the optimal time to have children. In completely typical Ancient Rome - girls typically married in their mid teens, very soon after puberty. The idea was simple:

Regardless of societal constraints, many people will get sexually active once they hit puberty
Of people who get sexually active, many will get pregnant
Nearly everywhere except for modern times, it's really difficult to either provide for your kids without a husband, or get a husband if you already have kids
To avoid risking that, it's better to marry your daughters sooner

This was the baseline normal case for girls.

Western pedophile obsession (where all sexuality of people under 20 or so is a big taboo - this has little to do with what psychology calls "pedophilia") is a highly atypical case. 7th century Arabic children marriage, and fucking of prepubescent 9 year old girls was also rather atypical.

Not that it matters what's typical and what's not - abject poverty, illiteracy, and lack of broadband Internet are historical norm, and yet I much rather prefer atypical modern situation.

Anyway, what we see here is a conflict of values - Muhammed fucking a 9 year old was acceptable within his culture, and is not considered acceptable within ours - not even for most modern Muslim countries. Aisha most likely didn't mind getting fucked, and nobody was weirded out by this, in spite of modern fiction of "age of consent". She was most likely not scarred for life or anything like that - these laws exist primarily to make adult bigots feel good, not to "protect" children, and evidence is clear that plenty of teenagers (and an occasional pre-teen) have sex and enjoy it.

Lesson #3 - Freedom of speech

But that's not all. On top of one value dissonance about sex with pre-teens we have a second value dissonance about free speech and religious respect.

In Western culture since the Enlightenment, we have been very strongly attached to the idea that there is nothing that cannot be criticized. Yes, laws of different countries prohibit different kinds of speech (including United States, Supreme Court's favourite activity is making exceptions to the First Amendment and Elena Kagan doesn't seem any different here) - but every time such law is applied the media get freaked out and people feel highly uneasy about it. Even people who want to limit freedom of speech considerably universally consider it to be the default case - exceptions to be few and made only when "necessary".

These believes are not shared by many, apparently including modern Muslim societies. It seems that they approach Muhammed cartoons from another direction - that people's religious believes should be respected, and their holy figures shouldn't be mocked - freedom of speech being far lower in their order of priorities than that.

They don't even see the cartoons as a freedom of speech issue, just like 7th century Arabs didn't see fucking a 9 year old girl as a child abuse issue, and we don't see the cartoons as a blasphemy issue. Different cultures have different perspectives.

This basic cultural relativism does not mean that all cultures are equally wrong, or equally right. It would be intellectually dishonest to think that your culture is uniquely correct about things, and its beliefs are some sort of human universals - history shows that there are hardly any true human universals, and we might even get rid of death and taxes one day. But you are still free to follow your own culture's value system.

If you think that freedom of speech is valuable, and fucking prepubescent girls is creepy, mocking that and not giving a shit about angry Pakistanis is all fine.

In other words - enjoy the Everybody Draws Muhammad Day.

Very simple parallelization with Ruby

2010-05-17T07:36:00.002+02:00

I hate waiting. And all too often when I tell computer to do stuff, the only reply I get is a progress bar with ETA far in the future.

Fortunately plenty of such things are highly parallelizable. For this post let's focus on a simple task of downloading every single Questionable Content strip ever released because QS has really simple URLs. It's real simple to do it serially but waiting time of about 23 minutes is killing me:

(1..1665).each{|i|
  system "wget", "-q", "http://www.questionablecontent.net/comics/#{i}.png"
}

Notice how I didn't even bother with shell, and went straight for a real programming language. Anyway, Ruby has very easy threading system - just use Thread.new{ ... } to spawn, Thread#join to join, avoid global mutable state, and you're done.

We can get it running in no time:

module Enumerable
  def in_parallel
    map{|x| Thread.new{ yield(x) } }.each{|t| t.join}
  endend
 
(1..10).in_parallel{|i|
  system "wget", "-q", "http://www.questionablecontent.net/comics/#{i}.png"
}

The only thing that changed was replacement a one-liner middle-ware method, and replacement of Enumerable#each with Enumerable#in_parallel - so far really good.

On the other hand, we don't want to spawn 1665 threads all at once - neither our network connection nor QS's servers would appreciate that much. Let's write some code which would create 10 threads and keep them all busy instead.

Exception handling interlude

A very common problem with Ruby multi-threaded programming is that we don't want the program (or even thread) to die just because of some error - usually we want to capture, print, and otherwise ignore exceptions. Here Ruby fails hard - Exception has no method to generate user-friendly message of the kind it prints when exceptions fall out of the main loop. There are some gems that handle that, but here I'm rather unconcerned with this - and instead I'll just use a really simple catch-print-ignore method - which does not even bother printing backtrace.

def Exception.ignoring_exceptions
  begin
    yield
  rescue Exception => e
    STDERR.puts e.message
  end
end

Execute N tasks in parallel

How do we communicate with N threads? A simple and wrong way would be to create N queues, and feed them in a round-robin manner. Unfortunately unless tasks always take nearly the same amount of time (this is almost never even close to true) - this results in a lot of threads waiting aimlessly.

We need exactly one todo queue. Now it would be better if this queue had a size limit - but SizedQueue was giving me troubles, so I did it with the plainest possible Queue.

The main thread is really simple:

require 'thread'

module Enumerable
  def in_parallel_n(n)
    todo = Queue.new # Create queue
    ts = (1..n).map{ # Start threads
      Thread.new{
        # Do stuff in threads
      }
    }
    each{|x| todo << x} # Push things into queue 
    ts.each{|t| t.join} # Wait until threads finish

  end
end

Now the main question is how do we tell threads there are more tasks. Unfortunately queues have only two states:

Queue not empty - dequeue one element, do work, go back for more
Queue empty - sleep and wait for the queue

While we need three:

Queue not empty - dequeue one element, do work, go back for more
Queue empty, producer not finished - sleep and wait for the queue
Queue empty, producer finished - finish thread

Now I could go for a more complicated signaling solution - but that would either introduce global mutable state (hell no!), or require too much complexity - so I simply went for the simplest possible "flood queue with nils once you're done" way.

And because I don't want the program to break when object we're iterating over contains legitimate nils I'm passing whatever we get in a one-element array (Ruby probably has equivalent of Ocaml's 'a option / Haskells's Maybe monad etc. - but for this the simplest solution will work just fine).

require 'thread'

module Enumerable
  def in_parallel_n(n)
    todo = Queue.new
    ts = (1..n).map{
      Thread.new{
        while x = todo.deq
          Exception.ignoring_exceptions{ yield(x[0]) } 
        end
      }
    }
    each{|x| todo << [x]}
    n.times{ todo << nil }
    ts.each{|t| t.join}
  end
end

And now we can happily get the comics. It took 4 minutes 11 seconds instead of 23 minutes - 8x speedup!

(1..1665).in_parallel_n(10){|i|
  system "wget", "-q", "http://www.questionablecontent.net/comics/#{i}.png"

What about other comics?

Now unfortunately far too many comics don't have pretty URLs like that. You can always (well, nearly always) resolve to wget --mirror or find a greasemonkey script which makes the comics' website bearable. Fortunately quite often we can handle even somewhat more complicated URL schemes.

First, for comicses like sinfest, which use dates in URLs, it's only marginally more difficult:

require 'date'

(Date.parse('2000-01-17')..Date.parse('2010-05-17')).in_parallel_n(10){|d|
  system "wget", "-q", d.strftime("http://www.sinfest.net/comikaze/comics/%Y-%m-%d.gif")
}

3536 files, 7 minutes 52 second.

Because Ruby dates are first class objects and Range class works very sanely with them - it took nearly no effort to code that, other than trying to remember wtf were all those strftime %-codes (or you could use Date#year etc. and plain string interpolation - it would probably be better for your sanity actually).

And wget will quietly ignore proper 404s, so you don't need to wonder if they're updating on weekends too, or only on weekdays, or what.

For other comicses like xkcd you might need some hpricoting (and put alt text in metadata, then teach image viewer to read them? Or use imagemagick to put them into the image? choices, oh all the choices...). Or you could use This awesome Webcomics Reader Greasemonkey script. Unfortunately it only knows about a handful of the most popular ones.

Now this technique is applicable for a lot of uses other than downloading funnies - like handling S3 backups, encoding large number of MP3s and so on. Everything where it makes sense to run multiple threads in parallel, but not too insanely many will be really easy to do now thanks to Enumerable#in_parallel_n or similar code.

It works with both 1.8, and 1.9 equally well, in spite of their completely different threading systems.

Enjoy.

Steam is ripping you off

2010-05-11T23:40:00.002+02:00

Online distribution of content should be cheap and easy. Compared to the complexities of making and moving around DVDs and such, moving bits is pretty much free. So it's a no-brained that games on Steam should be cheaper than the same games on DVDs, right? Of course that's not how it works.

These are top 10 Steam's best-selling games - this list is rather generous for Steam, as people would prefer to buy games on Steam which were better deal on Steam than on DVD, and would prefer to buy games on DVD which were better deal on DVD - so it's reasonable to expect comparing other games Steam would come out even worse.

As for DVD alternative, I wasn't looking too hard - just Amazon. I'm sure there are ways to get the games even more cheaply, but let's stick with the default online shop. Here are the prices, all Amazon's are with free shipping:

Call of Duty: Modern Warfare 2 - Amazon £29.65; Steam £39.99 - Steam costs 35% more
Battlefield: Bad Company 2 - Amazon £24.99; Steam £29.99 - Steam costs 20% more
Left 4 Dead 2 - Amazon £19.98; Steam £19.99 - essentially equal
Counter-Strike: Cource - Amazon £14.99; Steam £13.99 - Steam costs 7% less
Grand Theft Auto: Episodes from Liberty City - Amazon £14.99; Steam £19.99 - Steam costs 33% more
Civilization V (pre-order) - Amazon £30.99; Steam £29.99 - Steam costs 3% less
Mount & Blade: Warband - Amazon £16.97; Steam £24.99 - Steam costs 47% more
Borderlands - Amazon £11.78; Steam £19.99 - Steam costs 70% more
Just Cause 2 - Amazon £23.92; Steam £29.99 - Steam costs 25% more

In other words - median cost on Steam is about 25% higher than on Amazon. That's even though physical DVD costs quite some money to make, packaging, ship etc., while for Steam it's just pure profit.

And please don't even mention "Recommended Retail Price" - if you fall for this kind of blatant anchoring effect, there's no hope for you.

So to summarize: Steam games cost a lot more, and to make matters worse have no resell value, they're simply ripping you off.

PS. Amazon prices are for NEW games. These are not used titles. Anyone claiming otherwise has no idea what they're talking about.

PS2. Prices I gave are for UK (notice the £s). acct_rdt made a similar list for US prices, which shows exactly the same thing - US Amazon is cheaper than US Steam too, even though the difference is not as drastic (median 7%)

Call of Duty: Modern Warfare 2 - $37.80 new on Amazon, $59.99 on Steam
Bad Company 2 - $43.83 Amazon, $49.99 Steam
Left 4 Dead 2 - $29.99 (both)
Counter-Strike: Source - $19.99 (both)
Episodes from Liberty City - $27.99 Amazon, $29.99 Steam
Civilization V - $49.99 (both)
Mount & Blade: Warband - $24.53 Amazon, $29.99 Steam
Borderlands - $27.99 Amazon, $29.99 Steam
Just Cause 2 - $39.99 Amazon, $49.99 Steam

Greeks can pay, Greeks will pay

2010-05-11T17:42:00.000+02:00

Most normal blogs' favourite countries to rant about is US, or Israel, or China, or something like that. Not on this blog - my number one target, at least for the last month - is Greece. And I'm really shocked that some of the otherwise sane people I know buy into the whole "can't pay, won't pay" nonsense, so I wrote this short post explaining how it all really works.

Basics of money lending

Let's get back to basics. Why would anybody lend anybody else their money? The most common reason is they want to get even more money back. Now this is in no way the only reason - people routinely lend money to their family members and friends, IMF lends money to countries facing collapse, USA and Soviet Union lent money to support whichever dictator seemed more aligned with their interest, Europe lends money to shit-poor countries out of pity, AIG lent itself taxpayers' money by bribes and threats, and CIA lent money to Osama bin Laden because they're idiots - but while these are all very important cases, most money in the world is lent for profit motive.

This profit-motivated lending is not only largest in volume - it's also most reliable. Political will can evaporate overnight - ask South Vietnam or Cuba if you don't believe me - but speculators can be reasonably counted on to keep lending as long as it makes business sense. Even in the middle of the latest recession and the Great Depression solvent borrowers had no trouble getting loans - the problem was sudden decrease in solvency, not disappearance of the profit motive.

Political and charitable lending is highly complicated and varied form case to case, but for-profit lending is quite straightforward from game theoretic point of view:

Lender decides to give borrower money or not
Then borrower decides to repay or not
Because borrower would much rather get money, and not repay it - no lender would be willing to lend anything without some pretty hard assurance of seeing their money back
On the other hand, as repayment might be impossible due to objective circumstances, borrowers would be unwilling to provide too much assurance
Level of assurance is based on balance of these two factors.

And what forms of assurance are available? For private individuals there's debtors' prison - a barbaric institution which is still used in child support and tax cases - not coincidentally both being the kind of "debt" which is incurred unwillingly, where the "borrower" has no leverage whatsoever.

Another assurance is forceful confiscation of property, and various kinds of physical abuse - these were routinely used as international debt collection in 19th century, see Egypt, and Haiti for examples - but these rarely happen these days.

Yes, lenders could try to sue Greece or another unwilling country in foreign or domestic courts, but it would be mostly a PR stunt, and they wouldn't even get enough to cover their legal costs. If a country doesn't want to pay, lenders cannot do shit.

Hamster showing what debtors' prison looks like

The final assurance

So what keeps borrowers repaying and lenders borrowing? It's lenders not being stupid. They know very well that if someone didn't honour their debts in the past, they're unlikely to do so in the future - so if you tell the banks to go fuck themselves, you saved yourself a huge pile of money, but you have no chance of getting any money from them in the future.

When you think about it - it's a pretty weak assurance - it only works when the country is moderately-screwed-up:

If country's economic situation is truly screwed up - they won't be able to repay - so banks lose
If country's economic situation is bad but not horrible (like most countries now) - they cannot afford losing their credit lines - so they keep repaying - and banks win
If country's economic situation is some awesome they don't care about future loans - they can tell banks to go fuck themselves - and banks lose too!

Fortunately for banks, the last case is very difficult to achieve - not only you need zero deficit (before interest) now - something already very hard - you need to be certain of being able to keep this zero deficit essentially indefinitely. Recessions, commodity price fluctuations, wars, terrorist attacks, aging population, floods, tsunamis, volcanoes, epidemics, electing Conservatives, and plenty of other natural disasters can screw your budget essentially overnight - all sending you crawling back to the banks begging for money. And bankers will remember what happened to their old loans.

I cannot think of any country in the world which is financially healthy enough to pull that off. What makes it even less likely is that countries with most balanced budgets tend to have lowest debt levels and borrowing costs as a rule (debts are results of past performance, and past performance is the best predictor of future performance there is) - so for them repaying debts is nearly painless, and provides good insurance in case something screws their economies in the future.

Essentially the only case in which a country can tell the bankers to fuck off, and has debts high enough to make it worthwhile - is one which used to be horribly mismanaged for many decades, and then turned into one of the healthiest economies in the world essentially overnight.

Does it sound like Greece? If you believe so, I have some sovereign debt credit default swaps that you might be interested in.

Greeks simply cannot afford not paying. They can negotiate better terms of repayment, and get some help from EU and IMF, but essentially they will have to get their country in order - cut their bloated civil service and bloated military, cut - and repay their debts, because they simply cannot afford losing their credit lines. And so they will pay.

Audible review

2010-05-07T01:28:00.001+02:00

Before the posting frequency either reaches singularity or crashes like Nick Clegg's hopes, here's a brief review of Audible.

I have listened to ridiculous amounts of audiobooks (a term I'm going to use for essentially any spoken word - whether an original material or a spoken version of a paper book) for many years now - a very long time before they became anything even vaguely approaching mainstream. And obviously I got most of them from Linux ISO download sites, not counting a very large number which were officially free as podcasts and such.

Then, my cat threw my iPod into a bathtub full of water, I bought a new and much better MP3 player, and with it came free subscription to Audible, which I actually extended for a few more paid months more out of curiosity how this world of paid audiobooks operates than anything else.

DRM

First - DRM is as bad as you'd expect. Devices cannot be activate on anything except Windows (the stories about iTunes being able to activate MP3 players are a lie). Then after I formatted the MP3 player, it lost activation and it was impossible to get it back. There was no error message, nothing. Activated successfully, still doesn't work. I had to email Audible for them to fix it... it was really one big pile of ridiculousness, all horror stories about DRM basically came true.

Does it at least DRM the books? Not at all. First, they left CD burning as an option, so anyone who is desperate enough can get virtual CDs and then rip them. And you can download Audible-ripping software which will happily hack into Audible drivers on any activated PC, and convert all your DRMed files into MP3s, FLACs, or whatever. Essentially they fuck over paid customers, and the pirates will get their warez anyway.

This is all good enough reason for not using Audible even if they didn't suck in other ways, but suck they do...

Speech synthesis

But before I get to the main dish of Audible suckiness, a brief interlude for state of speech synthesis. 60 years ago, when computing started, people believed that in no time computers will be as smart as humans. You know - 10 years, 20 tops. Which as it turned out in computing world means essentially "never".

I'm just amazed at how filled with wrong are predictions of people who expect rapid arrival of AI - like most users of the ironically named LessWrong community blog. How the fuck are we going to see AI go "foom" if during the last 60 years there has been essentially zero progress at:

Speech recognition
Handwriting (and most importantly hand-written math/diagrams/mindmaps) recognition
Speech synthesis

Yes, they expect a big foom - but better be worried in case this foomy future Kindle 2020 or whatnot will turn out to be unfriendly - and let's say will read the latest Twilight sequel in a very ironic voice...

The truth is - automated speech synthesis of anywhere near human quality is not coming anytime soon. It just isn't. Amazon knows it as well, while building gimmicky crappy speech synthesis into Kindle, they also purchased Audible for $300M whose business model totally relies on speech synthesis being crappy for decades to come! How is this $300M for a prediction market? Are you shorting Amazon stocks already? I didn't think so.

And while I'm at it, remember how one of my favourite crackpots - Raymond Kurzweil boldly predicted that by 2009 "[people] interact with their computers primarily by voice and by pointing with a device that looks like a pencil. Keyboards still exist but most textual language is created by speaking". Obviously this crackpottery never came true, I'll keep my Microsoft Natural Ergonomic Keyboard 4000 thank you very much. I doubt even Apple would be stupid enough to go into the whole voice thing - but then with iPad they sort of have proven that they can sell absolutely anything, and hipsters will buy it...

Why Audible sucks

And now the main part - why Audible sucks. To explain this think for a moment what an Internet shop is. Yes, it needs to have some stuff to sell, and some way to deliver it, and some way to accept payment, but primarily:

An Internet shop is primarily a search engine

That's right! Everything else is just an insignificant attachment to the main functionality of letting people find the stuff they want to buy. Go to Amazon - the archetypal Internet shop. What's there - a huge search engine. With user reviews, recommendations based on your previous purchases, and purchases of other users, and a huge wealth of information which helps you find things you might want to buy. Why nobody has copied Amazon yet? Well, there are some economies of scales involved, but dis-economies of scale exist as well. This is not the answer.

The primary reason is that nobody else has this massive database of data which helps people find what they want. Even if you had all the products Amazon has, at identical prices and terms of delivery, without user reviews and contextual recommendations your shop would be hopeless. And without even as much as Amazon search system? Forget about it.

By the way, I'm probably biased as most of my commercial career was in one way or another related to building custom search engines, just saying for the sake of full disclosure...

Anyway, Audible fails so hard at search it's not even funny. You go to category - and it's sorted by release date. How is this sane? You can sort by customer review - but most of their books have at most one review and there's no Bayesian filtering applied, so everything on the top will be audiobooks with a single 5-star review. Yes, it's all great that ONE person liked it, but it has extremely low predictive power on how good it actually is. About the only useful way to sort anything is by top selling.

So what do we see - title, author, price, star rating, a picture (wtf? these are audiobooks, these pictures are essentially just some random stock photos of no relationship with the product) - and that's it. And max displayed at once is 30 - but don't worry, it will change back to the default useless value of 10 results as soon as you change anything.

This is not Google! You don't want to go from search term out of their site in less than a second - you want to figure out what's actually any good! The logic of having smaller number of results so they're faster doesn't apply at all!

So how do you figure out if the book is any good at all? Well, there are very rarely any customer reviews (just a few star ratings is typical) and the ones there are are typically as useless as "Took me a little while to get into this book, but when I did I absolutely loved it! Have just downloaded the second book and can't wait to begin!". How does this provide any information whatsoever?

There is usually a short publisher-provided blurb telling what the book is about - and you can probably guess what I think about such publishers' honesty.

The only thing that's of any use at all - and without which the entire Audible website would be utterly useless - is the audiobook sample. Unfortunately it doesn't work terribly well. One thing you can figure out is that quality of recording tends to be top notch - unlike some other sources which compress audiobooks as if people were still downloading them on dial-up modems, audible handles quality properly. And you can also find out if you like the narrator or not. Which might be very highly relevant for their "erotic" section at least... (seriously, go there for some major lulz and/or facepalming..., but then maybe I'm not the target audience...).

What you'll rarely find out is if the book is actually any good, which is what we want to establish. Random pages from the middle of the book rarely tell you that. Even worse, I've seen quite a few books for which the random sample was actually legal blurb + acknowledgments from the first few pages... that definitely is some fascinating book, right !?

Unfortunately that's all your search options. Audible won't search inside books to figure out what they're about (speech recognition obviously being crap, but on the other hand - they have access to written original most of the time, so that's a weak excuse), and there's no other information on their website.

If you want to find out a good audiobook, you need to check every audiobook's written equivalent's page on Amazon - for reviews there. Somebody should write a Greasemonkey script for that... (this problem also affects UK Amazon to some extent - often US version has 10x as many reviews, but there's no link and the only way to get to those reviews is Googling... it gets me raging every time...)

How difficult would it be to link it up with the rest of Amazon? Seriously now.

There's also another issue - the choice on Audible is simply less than on "Linux ISOs download sites" - so when you want an audiobook it's very likely you won't find it there, even if it exists. The opposite - Audible having something which pirates don't also happens, but usually for more obscure titles.

This is actually an opportunity for them - sure, many people will torrent Dan Brown, but there's long tail to explore - except it is exactly this long tail which is it direst need of good search functionality. And so Audible fails.

So maybe it's a good idea to short them after all...

Progress bar for Unix pipes

2010-05-06T23:01:00.001+02:00

This is getting quite ridiculous, as it's the third post I'm writing in just one day - that's more than I typically wrote monthly over the last two years. This is what you get for finally getting organized - a big pile of 90%-finished stuff which while useless on their own can be very quickly be turned into something of high utility. OmniFocus and GTD are truly awesome. (unfortunately this means there are now two apps for Mac I care about - TextMate and OmniFocus - so switching away from Mac will be even hander)

The script I want to show you today solves one of the most severe problems with Unix pipes - lack of progress indicators. Normally you'd start a pipe, and until it actually finished what it was doing, you'd have no way of finding out if it's 1% done or 99% done, and how fast it is progressing.

There are some nasty hacks. Many times I used strace or looked inside /proc to figure out how much progress has been done - but these are painful waste of effort for something that should be builtin. Feedback is the basic principle of good UI design.

Anyway, here's the final solution to progress bar question:

#!/usr/bin/env ruby

STDERR.sync = true

$bytes = true
$max = nil
$count = 0

until ARGV.empty?
  case (arg = ARGV.shift)
  when '-l'
    $bytes = false
  when '-b'
    $bytes = true
  when /\A(\d+)([kmg]?)\Z/
    units = {'k'=>2**10, 'm'=>2**20, 'g'=>2**10, ''=>1}
    $max = $1.to_i * units[$2]
  else
    raise "Unrecognized argument: `#{arg}'"
  end
end

$max = STDIN.stat.size if $bytes and STDIN.stat.file? and $max.nil?

Thread.new{
  last_count = nil
  while true
    if $count != last_count
      if $max
        STDERR.print "\r#{$count}/#{$max} [#{$count*100/$max}%]"
      else
        STDERR.print "\r#{$count}"
      end
      last_count = $count
    end
    sleep 1
  end  
}

begin
  while data = ($bytes ? STDIN.read(2**12) : STDIN.gets)
    STDOUT.print(data)
    $count += $bytes ? data.length : 1
  end
  STDERR.print "\n"
rescue Errno::EPIPE
end

Explanation time:

You can use this script at any point of the pipeline. foo | bar | progress | blah.
The script will take advantage of the fact that its STDERR is still linked with your terminal, and output progress information there. It will then clear and overwrite the same line every second with new information.
There is no support for multiple progressbars in one pipeline - his is not terribly difficult to do (one would be master, others would send info to it via socket based on tty's inode), but I never found a good use case for it, so I never bothered implementing it.
progress script works in two modes - by default it counts bytes (-b), but it can count lines (-l) as well.
You can specify what counts as 100% if you want percentage information - with progress -l 1234, progress -b 700m etc. If you specify wrong size of course you get garbage.
If you operate in the default byte mode and input is a file - the script will figure out file size automatically. This doesn't happen in line mode, as it would require a potentially expensive wc -l - it's easy to do it manually if you want.

Here's a "screenshot":

$ ./progress < kubuntu-10.04-beta2-desktop-amd64.iso | openssl md5
287244288/708704256 [40%]

Enjoy.

Brief history of human diet

2010-05-06T18:30:00.001+02:00

Previously in series: The myth of 5 a day - people never ate that much fruit and veg, Dietary consequences.

I've been listening to an audiobook about history of potato cultivation, which I purchased legitimately, and then proceeded to strip DRM of it in spite of Audible insisting in its FAQ that it's impossible. Are you getting WTF overload already?

Anyway... for people who are puzzled by the recent obesity/diabetes/metabolic syndrome clusterfuck, here's a quick reminder of history of human diet:

Humans did a lot of hunting and gathering with diet composition being highly varied depending on local circumstances. Normally hunting would bring most high quality protein, while gathering would bring most calories, and diversity of food sources would ensure good dietary balance and protect against bad seasons.
Then humans switched to agriculture. This happened at least 7 times, independently, almost at once, after millions of years during which it could have happened and didn't. Amount of evidence we have is just ridiculously high compared to all other human evolutionary transitions (like language, behavioral modernity, industrial revolution, and scientific revolution - all happening just once) that it's a shame that we don't have any convincing theories explaining why it happened when and where it happened.
Invariably, this switch resulted in severe health problems. Calories were more plentiful, but reliance on a small number of food sources virtually ensured micronutrient deficiencies, and regular famines due to bad weather. Farmers were shorter, died earlier, and generally suffered. This is well documented everywhere transition to agriculture happened.
Gradually, people figured out how to diversify their food sources, and in time farmers got as healthy as hunter-gatherers before them. There were more kinds of food, better agricultural practices, better preservation methods, better cooking methods, varieties of food grown got changed, and to a small extent even human biochemistry changed to adapt to agriculture.
There was still one huge problem - people were spending insane amount of time on food preparation. According to some statistics, even as recently as a century ago, average wife spent 45 hours a week on activities related to food preparation. Stay-at-home mums were not a cultural quirk - they were a necessity (unless you could afford servants).
A new revolution in food production happened in the last century. There was quantitative increase in productivity, but most interesting part of it was switch from highly home-labour-intensive food which required rigid gender role separation to semi-prepared food which freed women into general labour force.

Yes, it was not feminist nonsense which brought equality, or birth control, or anything like that - it was primarily food processing industry. You might protest that you don't eat that much processed food - but you're fooling yourself. How many of you have milked your own cows, churned your own butter, baked your own bread (milling flour has been industrialized ages ago - but at first it was also done at home), made your own pasta etc.? We have gone so far up the food processing ladder than we forgot how much processing goes into all those seemingly simple foods that form bulk of our diets. Actually, I do bake my own bread sometimes, but with a bread machine, so it hardly counts, and mostly because British bread is atrocious spongy shit.

Unfortunately food is a complex matter, and this transition went not without causing some serious problems, just like the original switch to agriculture. If you think about it - these problems aren't terribly severe. Yes - untreated obesity is live-threatening, and we haven't found good cure for it, but we have been really good at treating the symptoms. There are drugs for diabetes and hypertension, cars have replaced the need to even as much as walk longer distances, and loss of physical attractiveness can be largely covered by learning the art of myspace shots. People still live longer each year, and don't seem that terribly distressed.

Moreover, it seems that obesity rates have plateaued in States in the last decade - at a ridiculously high rate, but it's not unlikely that the worse is already over, and they'll only go down from there.

If the rates start indeed falling, then perhaps the transition to the new kind of food will have went surprisingly smoothly, compared to all such past transitions.

Case study - USA

Let's take a look at how eating patterns changed in one of the worst affected countries - USA. It is so commonly repeated that people eat more sugar, fat and animal products that it gets on my nerves. Here's the definite table of all major food groups (at least 1% share at either date), based on FAO data, as percentage of caloric intake, ordered by relative change. I'd love to have data for 1900, as 1961 was midway the shift from unprocessed to processed food, but we have to do with the data we have, instead of despairing for the data we wished to have.

Product	1961	2005
Rice	0.93%	2.46%
Poultry Meat	2.23%	5.48%
Cheese	1.67%	3.94%
Vegetable Oils	9.58%	18.58%
Groundnuts	1.1%	1.55%
Maize	2.05%	2.52%
Vegetal Products	64.93%	72.77%
Fruits - Excluding Wine	2.77%	3.03%
Beer	2.36%	2.57%
Meat	11.63%	12.0%
Alcoholic Beverages	3.77%	3.84%
Cereals - Excluding Beer	21.76%	21.44%
Sugar & Sweeteners	17.89%	16.92%
Pulses	1.25%	1.14%
Potatoes	2.69%	2.43%
Wheat	17.5%	15.5%
Vegetables	2.22%	1.93%
Starchy Roots	3.14%	2.57%
Animal Products	35.07%	27.23%
Other Alcoholic Beverages	1.19%	0.92%
Pigmeat	4.41%	3.36%
Milk - Excluding Butter	13.29%	9.9%
Beans	1.13%	0.79%
Bovine Meat	4.36%	3.02%
Eggs	2.33%	1.45%
Butter, Ghee	2.26%	1.05%
Animal Fats	6.92%	2.91%

That is:

Americans eat more vegetal, and less animal products - and within this category there's shift towards leaner meats
Americans eat about as much sugar & HFCS
Americans eat absolutely insane amounts of vegetable oil

And these are the biggest losers. I will slap around with a large healthy trout everyone who dares to suggest that HFCS is responsible for obesity. HFCS is damn sugar - statistics say it's vegetable oil you eat too much of.

"HFCS causes obesity" is the new "vaccines cause autism"

Case study - Germany

For a second case study, I took a less drastically affected country - Germany. Germany is taking the transition somewhat better than States, and your guess is as good as mine if this is a genuine success, or the worst is simply still ahead of them.

I wished to simply have EU15 average or something like that, but FAO lacks good aggregate data, so here's the next best thing.

Product	1961	2005
Maize	0.4%	2.44%
Rice	0.53%	1.34%
Poultry Meat	0.66%	1.53%
Wine	0.61%	1.32%
Nuts	0.55%	1.1%
Cheese	1.93%	3.76%
Other Alcoholic Beverages	0.81%	1.39%
Vegetables	1.08%	1.8%
Fish, Seafood	0.77%	1.17%
Vegetable Oils	8.81%	11.4%
Wheat	14.5%	18.41%
Sugar & Sweeteners	11.71%	14.22%
Alcoholic Beverages	5.71%	6.63%
Pigmeat	6.25%	6.96%
Milk - Excluding Butter	8.09%	8.89%
Meat	9.15%	9.79%
Vegetal Products	67.45%	69.96%
Cereals - Excluding Beer	24.94%	25.47%
Animal Products	32.55%	30.04%
Beer	4.3%	3.91%
Fruits - Excluding Wine	3.75%	3.28%
Stimulants	1.07%	0.91%
Eggs	1.7%	1.31%
Animal Fats	12.43%	8.76%
Butter, Ghee	6.56%	3.6%
Bovine Meat	1.99%	1.02%
Potatoes	8.81%	4.06%
Rye	8.2%	2.39%

Interesting points:

Changes are much less drastic than in States.
There's big shift away from local rye and potatoes to more international wheat, maize, and rice.
Just like in States, there's shift from butter to vegetable oil, and from fatter to leaner meat - but it's less significant.
Other than that, German food preferences are remarkably stable over such a long period of time, and such drastic changes in actual agricultural practice.

Case study - Japan

And finally the country which takes transition remarkably well - they are essentially free of Western metabolic syndrome troubles. You'd think they sticked to their traditional food? No such luck.

Product	1961	2005
Maize	0.16%	2.74%
Poultry Meat	0.18%	2.03%
Pigmeat	0.36%	3.37%
Meat	1.07%	6.38%
Vegetable Oils	3.26%	12.21%
Milk - Excluding Butter	1.42%	3.78%
Animal Products	9.64%	20.58%
Eggs	1.45%	2.75%
Beer	0.74%	1.3%
Other Alcoholic Beverages	1.51%	2.54%
Demersal Fish	1.19%	1.83%
Fruits - Excluding Wine	1.36%	2.0%
Sugar & Sweeteners	7.01%	10.01%
Wheat	9.73%	13.32%
Animal Fats	1.05%	1.38%
Fish, Seafood	4.52%	5.93%
Alcoholic Beverages	3.56%	4.65%
Vegetables	2.4%	2.82%
Pelagic Fish	2.5%	2.21%
Vegetal Products	90.36%	79.42%
Soyabeans	4.29%	3.52%
Potatoes	2.09%	1.5%
Cereals - Excluding Beer	60.34%	38.68%
Fermented Beverages	1.3%	0.68%
Rice	46.6%	22.09%
Beans	1.24%	0.54%
Starchy Roots	5.87%	2.45%
Pulses	1.44%	0.6%
Sweet Potatoes	3.31%	0.64%
Barley	3.6%	0.17%

Or:

Changes are even more drastic than in other case studies.
There has been huge reduction in consumption of rice, barley, sweet potatoes.
There has been huge increase in consumption of animal products, and vegetable oil, and big increase in consumption of sugar.
Japanese drink far more milk than they used to - and fuck lactose intolerance.

So to sum it up... I makes little sense. There's no going back. Stay-at-home mums are as much a thing of the past as bands of hunters chasing mammoths across tundra. You have to switch. The question remains - what to switch to.

The optimistic answer to this is simply follow the crowd, and hope they'll eventually figure out what works. Even if it ends up as badly as the idea of replacing animal fats with partially hydrogenated vegetable oils and simple sugars (yes, these were the original "healthy foods") - someone will come with a pretty good drugs for diabetes by the time you need them. Maybe we'll even stem-cell diabetes away entirely?

The pessimistic answer is to just copy the Japanese.

Enjoy your whale sushi.

The unified theory of online dating and porn

2010-05-06T12:29:00.000+02:00

Most information in this post is based on absolutely no citable sources - it's either private original research or outright ass-pull, and I won't even tell you which is which to protect anonymity of certain sources. Wikipedia protesting is futile. Now the subject of the post is:

There is huge gender asymmetry in online dating behaviour

In case you haven't tried online dating, and/or haven't set up large number of research honey pots it looks like this:

Everyone writes profiles with basic data, a few pictures, and a few fairly generic paragraphs about themselves.
Men message every woman with whom they see even a remote possibility of attraction developing - which is to say every woman with average-or-better picture (this implies that men who type faster get laid more - as if there weren't enough reasons to learn to type fast already). Unless there's something immediately disqualifying in profile like text-speak, in which case even pretty pictures won't help.
Women get swamped by these messages. They are invariably rather generic like they profiles, and few get a reply. Fortunately the criteria seems to include proper spelling, for another easy win.
Out of this maybe some conversation develops, and maybe some actual dating happens.

So what causes this asymmetry in messaging behaviour? First, we can reject a few ideas outright:

While this looks like a self-reinforcing equilibrium (men who don't spam don't get any replies, women don't have any reason to spam if they're getting spammed anyway) - this is not one of two possible self-reinforcing equilibria - the same result is reported on almost all dating sites.
Alexa insists that ratio of men to women on most dating sites in quite close to 50:50, and so do all dating sites. So it's not one gender being more numerous.
Another obvious theory would different age-dependent attractiveness curves for men and women - nearly every man, regardless of age, is attracted to 18-24 year old girls. This theory is easy to test, as attractiveness curves cross, and old women are far less attractive than old men - so the theory predicts that for 40+ demographics women would spam men with generic messages. My honeypots strongly disagree with this. And neither there is much spam by women with inherently low chances like those with children, no Photoshop skills, or bad spelling.

No, this is something gender-specific, and independent of attractiveness. Now the last wrong theory is that men simply care more and women are not that much into all the relationships and/or sex thing. And this would be completely wrong - just look at evidence of what kind of sex/relationship fiction men and women consume, and how much.

Men mostly consume visual pornography. Videos and photos of attractive women, some of it is outright "porn" of them having sex, but very large proportion of such material is simply attractive women posing, often mostly or even completely clothed. In other words - a suggestive picture (or even a myspace shot) of a non-nude woman fits very well within male porn consumption patterns.

And what kind of porn women consume? Yes, some of them watch "porn" porn, there's even a tiny category of porn specifically made for women - but this is a tiny niche. Most women read their porn. Romance novels, Twilight, slash fanfiction - these are all mainstream women's pornography.

And here we hit a major problem - because while content of women's dating profiles has what men need to feel a tiny initial spark of attraction - suggestive pictures - contents of men's profiles have nothing. It's almost as if it was exclusively men who designed dating sites, and these were the men with least clue on what women wanted (geek jokes go!).

I don't know how to turn this insight into dating site design advice - it is much easier to teach people how to make myspace shots than to teach them to write decent slash. And even if everyone was a good writer, it takes minutes to read through large blocks of text as opposed to seconds to look at a few pictures and decide, so it might not work within typical dating site flow.

So perhaps this theory is useless. Or perhaps whoever figures out how to exploit it makes the new Facebook.

And here's some misandrous, misogynist, and generally spiteful advice:

Men - learn to type. If you cannot fix the spam problem, you can at least outspam others. Have some quality copypasta ready. If you can program, Greasemonkey away better dating flow. Outright spambots might be too easily detected, but feel free to take the risk.
Women - learn to Photoshop, and Photoshop your pictures mercilessly. It really doesn't matter that you won't look anything like your picture. Also keep your profile brief and generic - if pictures are good profile serve mostly negative selection function. Feel free to lie outright.
Everyone - all browsers have built-in spell-checker, use it.

The real cost of oversized military

2010-05-04T17:05:00.000+02:00

In one of my previous posts I pointed to the elephant in the room - Greek economic problems are largely due to its oversized military - this confused people a lot.

Let's do some calculation, pulling all assumptions out of Google. These numbers are actually very noisy year to year, but let's just take some long term average.

Annual economic per capita growth 2.1% - this is world average since 1950
Annual population growth - 0% - most developed countries we speak about here have little in terms of population growth
Annual economic growth - 2.1% (by two above) - this number might seem low, but that's the facts, and there's not much we can do about it
Starting debt level - 50% GDP - fairly typical for a developed country
Interest on debt is 3% above inflation. This I base on historical rates of series I federal bonds before the recession, and adjust it a bit because most countries are less safe than States, so would need to pay somewhat more.
Baseline size of military is 1.7% GDP - EU average
Budget has 1% surplus before paying interest.
This results is interest payment of 3% (interest on debt) * 50% (debt to GDP) = 1.5% GDP.
This leads to deficit of 0.5% GDP.
Because economy keeps growing - 0.505 / 1.021 = 0.4946 - so debt level will very slowly fall as percentage of GDP even as total debt keeps rising.
Over 50 years, debt will fall to just 16.3%

This is all very reasonable. Now let's assume this country decided to have Greek style ridiculously oversized military - 4.3% of GDP. This is extra 2.6% spending. What happens in 50 years? Their debt grows to 175.5%.

Assuming anybody was still stupid enough to lend them money at the same rates, oversized military costs them 2.6% GDP (original spending) + 4.78% due to excess interest relative to baseline scenario. Imperialism costs far more than it seems to - and it has been a very long time since anyone made big money on empire building - usually the only benefactors are bankers and arm manufacturers in countries not engaged in hostilities.

A few inb4s:

This reasoning applies only to excess military spending.
Normal civilian government spending is typically useful, and if government slashed spending on healthcare, education, pensions, disability insurance etc., people would simply buy more or less as much of such services on private market.
Even worse - if government slashed spending on research, basic infrastructure and so on, where benefits are diffuse, it would be very likely that nobody would pick up the tab, and it would hurt the economy significantly.
Yes, there is some outright waste in civilian government spending, but similar waste exists in private sector as well.
Baseline scenario already includes EU average military spending - and even if you argue that optimal level of military spending is higher than 0% (something I won't be arguing one way or the other now) - I cannot imagine in what kind of bizarro world EU average is far below the point of rapidly diminishing returns.

I'm not done yet. Let's assume interest rates are not constant, but vary depending on country's prospects. So at 0% debt you only need to pay 1%, then it grows linearly to 3% at 50% (like in scenario), and so on. There is some evidence this growth might be ever faster than linear, but let's stick to that.

In 50 years of baseline scenario debt diminishes to 6.35% and nobody cares any more.
In imperialistic scenario, debt after 10 years is only 73.43% (as opposed to 70.89% with constant interest rates) - it doesn't make a big difference yet.
In 20 years debt reaches 110.62% as opposed to 93.7% - it start to get ugly.
In 30 years debt increases to 189.96%, as opposed to 118.61%, but the glorious Greek empire keeps borowing.
In 40 years debt reaches 531.73%, as opposed to 145.79%, and the interest rates are completely ridiculous over 22% per year.
When the imperial scenario ends in 50 years, debt stands at 738747192.9% GDP, interest rates are almost 5% a day, and everybody is looking for the new suckers into this Ponzi scheme.

Of course Greece had to either collapse or change its ways long before that happened.

Want democracy? Get a figurehead king

2010-05-03T11:55:00.000+02:00

Followup to: How much democracy in your democracy?.

I find the very concept of monarchy obscene - its premise is that some woman has a magical vagina, and whatever first comes out of said vagina is blessed with magic powers that somehow make it most well suited to rule a country. Alternatively, some man has a magical penis - but monarchies are older than DNA testing so while the theory often emphasized penises, the practice was mostly vaginal.

In either case, this is just so ridiculous, I cannot imagine how fucked up must someone brain be before they start treating it seriously, and yet they did! Some still do. But as Karl Marx said, being determines consciousness - and nobody ever came up with an idea so ridiculous, that some privileged people couldn't use it as an excuse for existence of their privileges. Cosmic Jewish Zombie? Magical Vaginas? Social Darwinism? Efficient Market Hypothesis? Anything goes, as long as it implies the privileged stay privileged of course.

Anyway, in spite of my repulsion with the concept of monarchy, I decided to run some data analysis on effects different systems of governments have on levels of freedom and democracy.

Data comes from Wikipedia - government systems, Democracy Index (by The Economist), and Freedom in the World Index (Freedom House, essentially CIA).

Monarchies are more free than republics

It doesn't take a sophisticated statistical analysis to see that monarchies are more free than republics. Here's Democracy Index, higher is better:

All countries: 5.6 +- 2.22 [n=163]
Monarchies: 6.37 +- 2.63 [n=28]
Republics: 5.44 +- 2.08 [n=135]

That's over 1/3 of standard deviation - about as much as the difference between Canada and USA - decent different, but maybe not worth writing a blog post about yet.

Just to verify, let's check Freedom in the World Civil Liberties and Political Right indexes (for both lower is better). Here's Civil Liberties:

All countries: 3.19 +- 1.81 [n=192]
Monarchies: 2.56 +- 1.7 [n=43]
Republics: 3.38 +- 1.8 [n=149]

And Political Rights:

All countries: 3.35 +- 2.13 [n=192]
Monarchies: 2.88 +- 2.15 [n=43]
Republics: 3.49 +- 2.1 [n=149]

So CIA and The Economist agree - that's again about 1/3 of standard deviation. Magical vaginas are not that bad.

It gets better

The results so far were not terribly interesting, so I tried something else - how important is having ceremonial head of state versus one with actual executive power? It's rather shocking (higher better):

Ceremonial: 7.47 +- 1.42 [n=54]
Executive: 4.67 +- 1.93 [n=109]

Civil Liberties (lower better):

Ceremonial: 2.01 +- 1.28 [n=70]
Executive: 3.87 +- 1.72 [n=122]

Political Rights (lower better):

Ceremonial: 1.96 +- 1.37 [n=70]
Executive: 4.16 +- 2.07 [n=122]

That's 1.25 σ for democracy, and a bit over 1 σ for the other two indices!

Combining two criteria gets expected results - figurehead kings > figurehead presidents > executive presidents > executive kings. Democracy Index (higher better):

Ceremonial Monarchies: 8.12 +- 1.41 [n=18]
Ceremonial Republics: 7.15 +- 1.32 [n=36]
Executive Republics: 4.82 +- 1.96 [n=99]
Executive Monarchies: 3.23 +- 0.67 [n=10]

Civil Liberties (lower better):

Ceremonial Monarchies: 1.76 +- 1.13 [n=29]
Ceremonial Republics: 2.2 +- 1.35 [n=41]
Executive Republics: 3.82 +- 1.75 [n=108]
Executive Monarchies: 4.21 +- 1.47 [n=14]

Political Rights (lower better):

Ceremonial Monarchies: 1.83 +- 1.42 [n=29]
Ceremonial Republics: 2.05 +- 1.32 [n=41]
Executive Republics: 4.04 +- 2.09 [n=108]
Executive Monarchies: 5.07 +- 1.71 [n=14]

So turning a king into a figurehead gets you 2.25 σ improvement (or 1.5 on CIA's indices), and even switching from American style executive president to German style figurehead president + prime minister / chancellor with real power scores you a free 1 σ or so.

Correlation and Causation

What does it mean? If correlation equaled causation then replacement of Prime Minister Gordon Brown with President Gordon Brown would turn UK into Russia or Pakistan, on average. Now we all know that correlation doesn't equal causation, but do you really want to risk that? So how about not fixing what's not broken, and fix what is broken - first past the post election system - first?

Cats would work as such kings and queens just fine - data clearly shows that the head of state should be a figurehead, so why not a cat? Is Cat King much sillier than Magical Vaginas, really? And even if it is, silliness is no excuse not to do science.

If someone thinks this post is a serious argument for monarchy, they might be interested in this kind of science too, just saying...

Boobquake experiment is bad science

2010-04-28T04:00:00.000+02:00

In spite of my best attempts I couldn't find any relevant cats, so as a next best thing I included some catgirls possibly participating in the Boobquake experiment

I'm as appalled by low standards of modern science as cleris Hojatoleslam Kazem Sedighi is about skimpy female outfits. And yesterday, we both had a chance to be outraged at the same thing - boobquake experiment makes travesty of science!

Background

The conflict comes from a long accepted paradigm of natural disasters, which explained them as God's punishment for human sins. Research supporting this theory goes as far back as the famous Lot's experiment [Gen 19], and countless other papers of similar antiquity. It has been very widely believed until even as recently as mid-18th century, when it was the most commonly believed explanation for the 1755 Lisbon earthquake - according to sources of the time supposedly due to slutiness of Portuguese women, and other human sins.

This paradigm has been largely replaced by current naturalistic paradigm which claims such events are mostly "random" and have no particular reason - replacement motivated more by wish to remove God from the picture than by any hard scientific data, and by crude analogy with cases like creationism vs evolution in which similar removal of God was actually supported by large volume of evidence.

So it should be no surprise that if even a well researched case of evolution still has many doubters, it should be even more true for less researched case of causes of earthquakes and other natural disasters, where the main argument is compatibility with Western secularism.

Given all this, we should be grateful that someone finally tried running an experiment to check the sin theory of natural disasters. Unfortunately the experiment was so far from standards of good science as to render it totally unconvincing.

What went wrong with Boobquake experiment

The boobquake experiment was an attempt to test the sin theory of natural disasters by having a large number of women wear sluttier outfits than usual, make God angry, and cause increased number of earthquakes.

Before even looking at its scientific value we should notice that such experiment grossly violates modern ethical standards by risking lives of countless people who didn't volunteer for it. In the future it would be prudent to at least limit such experiments to a singre geographic location, where all sinners would group before commencing the sinning - and which non-sinners would be able to evacuate in advance.

Now, for the experiment. No attempt was made to establish baseline sin level, or even baseline outfit skimpiness level - and unfortunately we have no way of knowing how large was the change due to boobquake relative to the daily fluctuation of sin. Even the most basic estimates suggest the effect to be minor - supposedly about 200,000 women tried to wear sluttier than normally on that day - such number is far below changes in a single country known for sinning like Sweden due to daily weather variations. Experiment also takes place in the aftermath of Eyjafjallajökull eruption (believed by some to be caused by the deadly sin of greed of Icelanders) - which caused major disruption of tourism, and it is universally accepted that people sin a lot more during holidays.

Moreover, this sample was not random. Experiment participants have been self-selected, and we have reasons to believe that people particularly unconcerned with God's wrath, and therefore with higher than baseline sinfulness level - probably constituted vast majority.

While data on most participant is missing, comparison between be the supposedly "most scandalous" outfit of the main researcher she wore during the experiment with her profile photo causes serious doubts, as they're about equally revealing.

In addition to lack of statistical soundness, and sample randomness, no attempt was made to blind the experiment. This difficulty is inherent, as God's omniscience makes proper double-blinding impossible, but participants knew well in advance that they're assigned to treatment group, so they may have consciously or subconsciously reduced their other kinds of sinning, a blatant violation of basic rules of scientific testing that would pass no peer review. What's worse, no attempt was even made to establish a control group!

Now it could be argued that blinding participants would be difficult - or that sinning requires conscious action, so both treatment and control groups would be equally in state of sin by even risking wearing revealing clothing - but this difficulty is present in many experiments, and at least serious attempt should be made to reduce placebo effect when it cannot be fully eliminated. Numerous examples of women wearing clothing far more revealing that they thought easily found on the Web suggest this kind of blinding is not fully impossible.

Not only experiment was of dubious quality, the hypothesis tested is far from the standard theory of sin causing natural disasters. Now the theory has many variations but even the basic Sodom and Gommorah version clearly disagrees with research assumptions:

Assumption was made of no delay between sin and punishment, while the theory says punishment only comes after sinning.
Assumption was made of near-linearity between sin and punishment, while the theory says God was willing to spare Sodom and Gomorrah if even as few as ten righteous people were found.

Given all these problems, the Boobquake experiment brings essentially no new knowledge to the discipline of causes of natural disasters, and we can only hope that future experiments are better planned, with better design, better experimental controls, and detailed publicly available photographic documentation of sinning.

With baseline like this, it is dubious Boobquake really changes average sinning level much

Related research

While Boobquake is clearly bad science, a much more relevant research was done by IPCC reports, which claim that we can expect high correlation between global warming and extreme weather effects.

This strongly confirms sin theory of natural disasters, as warmer weather causes many women to wear more revealing outfits, the results being among others "increased droughts, tropical cyclone activity, and tsunamis" (but curiously no mention of earthquakes).

Now correlation doesn't equal causation, and IPCC research doesn't directly test the sin theory, but it arguably provides stronger evidence than Boobquake experiment.

In the future, one would hope solid systematic research is done on variations of female outfit skimpiness - an area that has long been ignored by mainstream science.

Weekly review checklist

2010-04-27T11:03:00.000+02:00

Here's a quick list of steps to get my GTD system up to date. It differs somewhat from vanilla GTD, but so does everyone else's.

Preprocessing:

Go to mailbox, read all unread mails.
Go to mailbox, preprocess all blue-starred items. I use two kinds of stars: blue star means it's inbox item, yellow star is just to find an email quickly, and all activity related to it is tracked.
Preprocess every single item in paper inbox.
Go to both software repository (private and work), and commit or trash everything. If it cannot be completed right away it deserves an inbox item.
Preprocess every single item in inbox directory. On Linux I used ~/inbox, on OSX I normall just put stuff on desktop (and omg, I get my desktop to totally clean and empty regularly, I really do)
As result of this preprocessing everything is in the GTD system - currectly OmniFocus, but most is still crap.
A bit more preprocessing - quickly scan everything and mark completed items as done, do less-than-2-minute actions (the only reason such actions are even on the list is because they have been blocked before), mark completed waiting-fors as such - each such completion might trigger addition of extra stuff to inbox. The reason I do this is to save some sanity.
Now the big decision - look at my projects and other lists and decide if they are reasonably clean or a hopeless mess. If they're reasonably clean, leave them for now. If they're a total mess, move absolutely everything to inbox.

Processing - every item in inbox is processed. As everything is in one place and obvious trash, loudest items etc. has been mostly dealt with, there shouldn't be that many distractions - but they do happen.

What the fuck is it about?
Is it already done? If so, mark as completed.
Do I still even care? If not, delete.
Is it waiting for? Tag as @Waiting and move to appropriate project.
Should I push it to someone else? Write email right away and @Waiting
Is it a shopping item? @Shopping and move to appropriate project or generic "Area of Focus : Shopping" list. There are no specific subcontexts for shopping - I buy everything online including groceries - 90% of my shopping is Tesco online + Amazon online - and the main challenge is minimizing number of separate deliveries to reduce cost and annoyance. If I'm out for some reason anyway, I'll take a look at the list to see if there's anything I can buy on my way.
Is it calendar item? Move to calendar. I have very few of those so I'm using OmniFocus for calendar - it won't work well for most people.
Is it a random URL I might want to read someday? Move to "Someday/Maybe : Reading" list.
Is it a random idea I might want to follow through someday but not now? Move to appropriate sublist of Someday/Maybe.
Is it a well-defined action already? Is it simple enough to get it done in a few minutes? If so, do it (that might trigger another inbox item).
Is it a well-defined action that will take longer or I cannot do it right now because of wrong context? Move to appropriate project.
So it's something complex...
What am I trying to see accomplished anyway? If I'm very unclear about it, I might convert it to a new project + some @Mindmap / @Research actions.
If I'm clear about outcomes, do I want to move forward with it? If not, clarify description and move to one of the sublists of Someday/Maybe.
Am I clear about the next action? If so, turn this into a project, and put such action there.
If I'm not clear about next action, turn it into a project + some @Mindmap / @Research actions.

At this point inbox should be empty.

Go through project lists, make sure all projects have adequate actions. Some projects might already be completed, in which case mark them as complete.
Quickly go through contexts lists and make sure everything is sensibly contexted. Current list of contexts is: @Blocked @Calendar @Computer @Home @Linux @Mindmap @Outside @Research @Shopping @Waiting @Windows - but I change it if I need to.
Take a quick look at Someday/Maybe if there's anything I want to move forward soon. Usually there isn't, or at least not much.

After all this is done, I should be able to stay sane for a few more days.

What should South Korea do now?

2010-04-26T03:24:00.001+02:00

Nothing. Now the highly relevant Starcraft playing cat and kthxbye.

OK, you might be wondering why I think anyone would want to do nothing when another country sinks your ships and otherwise behaves provocatively for decades.

The first part of the answer is economy - entire economy of North Korea is worth PPP $40billion. That's about as much as Alaska or Luxembourg.

Other countries which might take part in conflict - some on South Korean side, some neutral, none on North Korean - are worth:

South Korea - 34x as much
Russia - 54x as much
Japan - 110x as much
China - 200x as much
USA - 350x as much

If everyone had similar levels of military spending as part of GDP, North Korea would be one big pile of rubble before I even had my second coffee. Now this isn't true, North Korea is the most militarized country in the world. It's hard to get good estimates, but according to some figures ratios of military budgets are:

South Korea - 4x
Japan - 8x
Russia - 10x
China - 13x
USA - 125x

It shows how insane North Korea is by even trying to catch up with its neighbours - some of which are low military spenders like China, and Japan; others are high military spenders like Russia and USA (and that "high" is still lower than countries like Greece), South Korea is surprisingly sane 2.7% considering environment it is in.

So there's absolutely no way North Korea can hope to actually "win" the war, if it actually started.

War is bad for economy

These ratios don't tell the whole story. Not only is North Korea ridiculously poor - it is about as poor as it was in 1953 and not improving at all.

South Korean economy is growing rapidly. Chinese economy is growing explosively. Japanese and American economies are growing - maybe not as quickly as those, but they reliably grow. Russian grows and shrinks as commodity prices change, that's a story for another post. And what about North Korea? It was doing fine until 1973 - at which point is was still better than the South - when it hit a wall, and stopped growing for tho decades, and then collapsed hard when the entire Communist world crashed in 1991, getting worse every single year ever since.

North Koreans are desperate. With each year, their chance of not winning but even putting up a good fight is diminishing.

This, paradoxically, means that everyone else would really really want to avoid a war. If you're getting richer and richer, while your enemy is getting more and more fucked up - why speed things up? Sure, militaristic dick waving looks good on TV and can get you some extra votes - but if you're anything resembling rational you'd prefer to postpone the conflict - assuming you want one in the first place.

How empires are built

This grow-economy-and-wait approach is how empires get created. Did Brits try to engage in European Continental Wars? They mostly let the Continentals bleed each others, while they kept themselves safe with the Royal Navy, picked only easy fights, and let the stability led to explosive economic growth which eventually led to their global domination.

That's how American Empire started as well - other than the infamous 1812-1815 war with Canada Americans mostly stayed isolationist, picking up Indians and growing their population and economy - after all other empires bled themselves to death twice USA became the world superpower by default.

A quick aside - it is commonly incorrectly believed that America has seen significant fighting in two world wars, or like ever. In the First World war it was rather laughable - Americans joined when the war was nearing the end, and by the time Americans reached Europe, it was over. Even countries like Bulgaria and Romania had seen far more fighting.

That of course without civilian deaths, Spanish flu deaths, countless wars following WW1, and such. WW1 was essentially Germany+Austria vs France+Russia+Britain.

WW2 was even simpler - being basically Germany vs Soviet Union, with a second unrelated front of Japan vs China - Americans and for that matter British had very little active role in either.

And the entire Western Front? For German soldiers fighting Soviets getting sent to the Western Front was like getting sent for holidays. The only reason Western Allies even managed to land in Europe was because Nazis were forced to move everyone they could to fight the Ruskis, and there was barely anyone left to defend anything.

America didn't get its empire by winning wars - it got it by not getting involved in wars too much. Lend-Lease , the Marshall Plan, and a few successful CIA operations here and there did far more for the empire than the actual military, which is only good at bombing whoever cannot defend themselves.

And this is precisely what all East Asian powers do. Why even bother spending buckets of money on armies? Japan and China have ridiculously small armies by international standards. Taiwan and South Korea are closer to world average - but considering what kind of danger they're facing close-to-world-average shows incredible levels of restraint. No, they just sit there quietly, making shoes, building iPhones, drawing tentacle porn, steadily growing their industries to catch up with the West - time is on their side.

And starting a war now? Not only your odds are worse than in foreseeable future, you risk messing up with your economic growth. There's nothing particularly Black-Swanish about it - wars take much more time and much more money than anyone predicts more often than not. Ask Americans about Afghanistan. Or Soviets about Afghanistan. Or Brits about Afghanistan. I'd like to see a list of wars which were actually won on budget - it would be a really short one I believe.

A small distraction - nukes

Every discussion of North Korea invariably gets sidetracked into an entirely irrelevant issue of nukes. Here's the only funny Subnormality strip ever:

This is highly accurate. Nuclear weapons never had any use. Manhattan Projects was a highly misguided idea of getting nukes to get long term strategic advantage over the Soviet Union - Japan never being a serious threat - and it lasted only 4 years before Soviets managed to catch up.

And as far as everyone knows, North Korea does not have nuclear weapons. Yes, they've probably managed to setup a nuclear explosion, but amount of damage a few tiny nukes without proper ICBMs can do is fairly low compared to the power of conventional artillery aimed at Seoul right now. This is the same mistake Nazis did with V2 - sure, they were cool, but they cost so ridiculously much compared to spending the same amount on conventional hardware Alies were very probably very happy about them.

Projects like North Korean nukes and V-2 are not military projects. They are half propaganda, and half massive cases of optimism bias in planning.

And even if North Koreans had large number of good nukes on good missiles, it's not at all certain they'd use them. Everyone during the Second World War - and you don't get more total war than that - had massive stocks of chemical weapons. And yet, they weren't used at all. At least against people who could retaliate in kind - Japanese had no qualms about mustard-gasing Chinese, just like Brits had no qualms about mustard-gasing Iraqis, and so on.

Demographic transition

And now we're getting to my main point - why South Korea is extremely unwilling to take part in any war. Unlike Americans, who can sit at home, eating supersized burgers, and drinking gallons of HFCS-laced soda while their missiles drop on Pyongyang, or even Japanese, Chinese, and Russians, who might get a few missiles here and there but will be mostly fine - the largest population center of South Korea is just 40km from the border. It doesn't require fancy guided ICBM to kill someone - a totally straightforward dumb howitzer can reach it just fine, and good luck evacuating a 25M people big metropolitan area in a hurry.

If war starts, there will be blood. Now you might be wondering - what happened to the world that it's so unwilling to have people die? Countries used to be totally blasé about a few thousand of their citizens here and a few thousand there dying. In some places like Afghanistan and most of Africa this is still the case. Yet - most of the world is highly unwilling to risk that. Even when wars take place - ridiculous amount of money is spent on protecting their soldiers' lives.

And the simple answer is demographic transition. In countries with high birth rates life is cheap. So what if someone dies? If your son dies, you make another - it's not terribly hard. Yes, you'd rather he didn't, but on the scale of most important things in life a small chance of child's death is not that high.

On the other hand once demographic transition happens, people have fewer children and have children later - it's all different. People spend ridiculous amount of money and effort on their children, are highly overprotective (limiting children's liberties in process, but that's another stories), and under no circumstances would they be willing to risk death or injury due to politics. And if your child dies? Not only you lose all investment you put into them, you are probably too old to have any more children, and your grandchildren aren't coming. There is probably some deep Darwinian reason why it works like that, I won't be getting into this - but the correlation between demographic transition and unwillingness to risk your lives in war is huge.

Notice this doesn't apply about lives of others - Americans don't give a shit if a few hundred thousand Iraqis here and there die; Israelis don't give a shit about bombing a Palestinian school here or a hospital there - seriously, who cares about "them" as long as "we" are fine?

Here's the best test case - WW1 and WW2. In 1913 total fertility was 6.3 in Russian, 3.52 in Germany 2.93 in UK, and 2.47 in France - they were at different stages of demographic transition, but still long before the point at which lives become scarce. So who cares if a million young men die meaninglessly in trenches?

In 1938 it was very different. Russia (I wonder how gapminder draws boundaries from these times, anyway...) was at 4.74 - still in the "so who cares if a ten million of our citizens die?". Germany was in "we should put serious effort into blitzkrieg - a few deaths due to quick war are acceptable, but painful" territory with 2.25. But France and UK were already well into "surrender if you can see tanks from Paris/London" territory with 2.13 and 1.84 - the only reason UK didn't surrender was its ability to wage war remotely, protected by the Royal Navy. And just look how ridiculous was their overreaction to small amount of bombing. So 43 thousand civilians died? These numbers are a rounding error compared to what was happening on the Eastern Front or in Asia.

Likewise Japan was very Kamikaze-happy with its 3.88 fertility rate - something Americans with their 2.22 would never try.

Once fertility drops, anyone's willingness to fight does too. And that's why we have so few wars in post-demographic-transition world. You still think I'm making this up? Look at fertility rates top 10:

Niger - somehow managed to get away with just a few coups, and some minor Tuareg rebellions.
Guinea-Bissau - had civil war 1998-1999
Afghanistan - had a war or another every year since 19th century.
Burundi - had civil war 1993 to 2005.
Liberia - one civil war 1989–1996, another 1999–2003.
Democratic Republic of the Congo - managed to pull one of the top 10 bloodiest wars of all times, just a few years ago; not counting minor ones.
East Timor - decades of fighting against Indonesia.
Mali - doing surprisingly well for the list, just some spillover conflicts from neighbouring countries and minor fights between government and Tuaregs.
Sierra Leone - civil war 1991-2002.
Uganda - one war, another civil war, plus participation in the Congo bloodbath on top of that.

And this is what the entire world looked like most almost all of human history. So who's on the other end? This is actually pretty good.

So other than some tiny territories - the lowest fertility and therefore least belligerent counties are exactly the most interested ones - South Korea and Japan.

By the way Czechs and Lithuanians are somewhat overstated here - they have fairly large migration to Western Europe, and many of their babies are born abroad. Their fertility is low, but not as drastically low as implied by the rank.

And no, North Korea isn't likely to be terribly trigger-happy either. Their fertility rate is 1.85 - soldiers will be difficult to replace. Not to mention they'll lose the war in a week.

Conclusions

South Korea should go to UN and get some sternly worded letter from Security Council and let it pass. Far more people could be saved by replacing a busy crossing with a roundabout than by stopping North Koreans from sinking your ships (or for that matter, stopping bin Laden from flying planes into your buildings).

Every argument shows that South Koreans win nothing by going into the war, and they will find loses unacceptable.

Now that I made Greeks and Koreans hate me, it's only 190 or so nationalities to go.

Notes on personal experience points system

2010-04-24T20:35:00.000+02:00

Here's a very quick post. I tried to run a personal experience points system, with software described in this article. I tried something like this before, it was described here on lesswrong.

So, the notes:

As I'm writing this, I have level 6 IRL avatar (251 XP)
In general, it didn't work terribly well. This isn't in itself a bad thing. As Edison said after failing and failing repeatedly "I have not failed. I've just found 10,000 ways that won't work", and then proceeded to steal Tesla's work.

Urgent vs important:

So one thing I learned doesn't work are big rewards for things I know I'm going to do at some point anyway. If I can get 25 points today - or play some video games today and I will get these 25 points in a few days anyway - this provides zero marginal benefit. It's just like people who always want to quit smoking next week.
With small recurring items like exercise - motivations due to these points sometimes managed to be just enough to push me in the right direction when I was undecided between doing that and doing something else.
I thought the system just doesn't work that well - but there was one exception. The "keep leechblock on whole day" item was a resounding success - at least at making me search new sites to waste time on but never mind that - somehow this urgency really works.
It seems faux urgency drives me more than actual importance. Unfortunately most important things in life are long term and not terribly urgent.
This is all because I am a one-marshmallower. I have no doubt that in the marshmallow experiment I would just eat that damn marshmallow right away - while children who waited would get another one, be more successful in life, and then get type 2 diabetes due to eating so many fucking marshmallows.
By the way, standard economics with its silly exponential discounting of time cannot explain this urgency - but it all makes a lot more sense if you use hyperbolic discounting. And life is definitely hyperbolicly discounted - if you doubt it ask the Egyptian pharaohs who prudently granted land to their temples so that it would maintain priests who would pray to the gods for them until end of time how is compound interest working for them. Exponential function is merely a mathematical artifact, nothing more.

User interface:

Scripts to record XP-able actions are not terribly annoying. This makes the most awesome keyboard in the world even more awesome. I'm wondering if it would be better to get it integrated with OmniFocus inbox, but annoyance is low enough for it to not be worth the effort.
XP bar is nowhere near flashy enough, and slowly filling up bar doesn't provide much motivation at all.

Economics and video games:

Video games solve this by giving level-dependent bonuses - unfortunately life is not like that, and I cannot think of any meaningful ways of granting myself a level bonus (also see the point about one-marshmallowing).
One way to think about would be compound xp interest - increase my XP level by some % every day to restore exponential discounting behaviour.
Another idea would be extra XP points for streaks of positive actions - like "3rd day with leechblock on" / "exercised 3 times in the week already" etc.
I would really like to find something less blunt than leechblock - the problem is that I actually need to access those websites every now and then for legitimate reasons - and I just want to make sure I don't fall into tvtropes trap (I accidentally clicked on this link and tvtroped myself dammit).

So I think I'll try another iteration based with far more urgency in it. Let's see how it works out.

How to configure OSX for software development

2010-04-23T07:27:00.000+02:00

In my previous post about OSX and Linux I mentioned that I maintain a very long document explaining how to turn a fresh OSX into something usable for software development and general use. Well, here it is - but be warned I only mildly cleaned it up.

This is all based on OSX 10.5 Leopard. For some reason OSX 10.6 Snow Leopard crashes like Windows 98 - at least once a day - and I cannot work like that, so I downgraded back after the upgrade. This seems to affect quite a few people, and 10.6 enhancements are nowhere near good enough to deal with such crashes.

Install all necessary applications

First of course - upgrade the system, as installation DVD is always rather outdated. Then install applications - this is more difficult than apt-get line, so here's list of links, sorted by category. All is open source or at least free download unless noted.

Browsers:

Firefox - Safari is IE for Mac. Here's my post on best Firefox 3.6 extensions - it's the same on any system. Normally that's the only browser you need, but for web development testing I also install:
Opera
Google Chrome

Drivers:

MacFUSE - drivers required by NTFS drivers
NTFS-3G drivers - NTFS is essentially the only reasonably portable filesystem for external drives as FAT doesn't support files larger than 4GB.
Arduino drivers - ok, you most likely won't need these.
Microsoft Keyboard drivers (I like binding those extra keys to something useful - Mac doesn't even see them by default). You won't need it unless you have Microsoft Keyboard or care about such extras.
Perian - a pack of media codecs.

Software development:

Xcode - this is a bundle of gcc and other development essentials - but instead of giving it a straight download Apple hides it under registrationwall on some horribly failful page.
TextMate - the only reason I even use Mac. Only trial version is free.
Xulrunner - Firefox-based environment in which you can make your own XUL-based applications

Basic system functionality:

Witch - it fixes alt-tab to work. Alt-tab is broken by default in OSX, has always been, and they don't plan to fix it ever - so this is absolutely essential. It's supposedly a trial version
Unarchiver - Mac supports most common archive formats, but you will need this sooner or later.
mysql - I don't remember exactly what was wrong with MacPorts' version (it was long time ago), but due to some brokenness I use this DMG version.
MacPorts - however much it sucks, it's absolutely necessary. Fink might seem like a nicer (apt-get based) alternative, but it just lacks most packages, and mixing them is a recipe for fail.

Internet communication:

Adium - multi-protocol IM program
Skype - another IM program (Adium doesn't support Skype) - these two should cover all your IM needs
Transmission - all bittorrent clients for OSX suck, but this seems to suck relatively less.

Desktop applications:

OpenOffice.org - there's native version these days, or at least decently native-ish
Stepmania - if you're into DDR
VLC - music player (OSX version of mplayer isn't as good as Linux's, so it's nice to have a backup)
Xee - image browser (OSX has none)

Others:

Growl - notifier system
Isolator - a really nice gadget which lets you blur all but active window, if you need to concentrate.

System settings

After basic applications are installed it's useful to create some symlinks that make OSX more Linux-compatible (sudo ln -s /Volumes /mnt; sudo ln -s /Volumes /media; sudo mv /home /home-old; sudo ln -s /Users /home) in case your scripts or you want to switch seamlessly between the two.

Then clean up the dock - it's filled with every stupid Apple program ever made (Garage Band ...), except for the most useful one (Terminal). So remove everything you won't be using, add all you will be (start application; right click; "Keep in Dock"), and mark applications you want to start when you login as such - typically Adium/Skype/Terminal/Firefox (start application; right click; "Open at login")

Now here's a long list of configuration tweaks:

Settings > Mouse > faster - I wish installer asked me if I have Parkinson's disease or not, or default to assumption that I don't. Protip for OS designers: Most people have reasonable control over they hand movement.
Settings > Keyboard > faster - likewise
Settings > Keyboard > Use all F1, F2, etc. keys as standard function keys - you use them about 50x more often than laptop controls, and the only common control - volume up/down/zero - has dedicated keys on my external keyboard anyway
Settings > Keyboard & Mouse > Ignore trackpad when mouse is present - it was extremely useful, except the most recent update of OSX 10.5 broke it - it's no longer there, even messing with registry file won't give it back
Settings > Universal Access > enable - needed by witch driver
Settings > Energy Saver > Power Adapter > Performance + no brightness reduction - it's just annoying
Settings > Displays > Automatically adjust brightness as ambient light changes > off - equally annoying
Settings > Displays > Make external display the main one
Settings > Displays > Show displays in menu bar (so you can quickly fix resolution if Mac autodetects it wrong as it does every 10 or so times)
Settings > Time Machine > Show Time Machine status in menu bar > off - unless you use Time Machine
Settings > System > Play user interface sound effects > off - now to stay sane you absolute need to turn off system bell - this setting needs to be repeated in multiple places
Settings > Sound > Alert volume 0%
Terminal > Advanced > Audible Bell off
Terminal > Shell > Close if exited cleanly - no idea why it's not a default, get rid of useless dead windows
Terminal > Window > Dimensions - disable - less clutter
Terminal > Text > Antialias text - without antialiasing Terminal.app looks like shit. Not antialiasing in terminal is just like making "look like shit" option and turning it on by default. WTF Apple?
Terminal > Monaco 14pt font, line and character spacing 0.90/0.90 - somewhat nicer than defaults.

Because I want access to development console in Safari, I need to mess in registry: defaults write com.apple.Safari IncludeDebugMenu 1.

It's also a good idea to get rid of crash reporter - especially if you develop XUL applications wink wink - defaults write com.apple.CrashReporter DialogType none.

Configuring applications and shell

Now let's go to shell and copy my .bash_profile and such.

It's a good time to find that TextMate license and install it.

Then get the most recent bundles:

mkdir -p /Library/Application\ Support/TextMate 
cd /Library/Application\ Support/TextMate/svn co http://svn.textmate.org/trunk .

I have some TextMate hacks which I install at this point - I'll write about them in another post.

Now start Adium / Skype and enter your logins and passwords.

Start Transmission, make sure incomplete downloads are in ~/Downloads/Incomplete/ and are not mixed up with completes. You may also change your share ratio settings at this point if you're an asshole ;-)

StepMania needs a link to songs located on external drive:

rm -rfv /Applications/StepMania/Songs/
ln -s ~/e/Dancing/Songs /Applications/StepMania/

And now install some packages from MacPorts/rubygems/CPAN. This set is related to what I'm working on right now and will probably differ for you - but everyone needs git, wget, unrar, pstree etc.:

sudo port -v selfupdate
sudo port install mplayer unrar ruby19 mc git-core pstree rb-rubygems wget mp3infosudo port install octave ImageMagick id3v2 p5-libwww-perl p5-dbi p5-heap p5-yaml
sudo gem install rak hpricot objectiveflickr url_escape mongrel s3sync mechanize RMagick
sudo cpan Sys::Mmap DBD::mysql Socket::GetAddrInfo

Have fun

At this point it's almost usable. There are some extra steps related to building stuff in my repository, configuring mysql for my applications etc. - all totally me-specific and of little use to anyone else.

Not that the rest of the advice is in any way universal - but it should be a good starting point for you.

OSX vs Linux

2010-04-21T19:19:00.000+02:00

I hate OSX and yet I use OSX because alternatives (by which I mean Linux, obviously) seem even worse. Here's a very short list of pros and cons as I see them.

Most important things:

OSX pro: TextMate. I really cannot use any other editor without feeling physically ill, they just do everything wrong. There are some attempts at getting semi-decent editors for Linux like redcar, but I have no idea how good they are.
OSX con: MacPorts - source based distribution systems are just failful, and MacPorts is far more failful than most. It is completely routine that upgrade will leave you with broken packages - neither upgradable nor downgradable.
Linux pro: apt-get. I wouldn't even seriously consider any Linux without apt-get for anything.

Extra points:

Linux pro: same software on desktop and servers. It saves me a lot of complications, 32bit vs 64bit is annoying enough without OSX vs Linux.
Linux pro: better development tools. strace, ltrace, and LD_PRELOAD for example. It is system made for developers by developers and it shows.
Linux pro: virtual desktops. Spaces on OSX doesn't work and it will never work as long as OSX stays Application-centric instead of document-centric like Windows and Linux GUIs.
Linux pro: better internationalization. Especially if I want to type in multiple languages. There's proper Polish Dvorak keyboard for example. (I remember I once haxored such for Windows 95 by editing binary drivers... these were good times).
Linux pro: Amarok. It's vastly more awesome than any OSX music playing program, especially shit like iTunes. Even old Winamp is far better than iTunes.
Linux pro: shell. Ubuntu bash has smart tab-completion, command not found suggests what to install, and everything else in shell got as much attention to detail as GUI got for OSX.
Linux con: Support for laptop hardware. Last time I tried it, even something as simple as getting external monitor in and out was a massive pain.
Linux con: GUI usability: Now that Windows 7 finally catched up, KDE is in general the second least polished GUI, the absolute worst being GNOME of course. It's not just looks, it's all the minor things that make it work seamlessly.
Linux con: GUI programming. C++ based Qt, and fake-OO-C based Gtk are both massive pain in the ass compared to even less than amazing Cocoa. There are Ruby etc. bindings for all, but it's just like putting lipstick on a pig. Everyone develops GUI for web these days - that even dealing with MSIE6 and all ridiculous web limitations is still far less pain than desktop GUI programming shows how bad it is.
OSX pro: It's prettier. Font rendering is a particularly good example
OSX pro: Omnifocus. Every other GTD software I tried was so bad I simply stayed with paper, but Omnifocus suckiness is fairly moderate.
OSX pro: Development for mobiles. Right now the only categories of mobiles are iPhones and turdbricks - one needs OSX, the other nobody wants to develop for. It might change in 2-5 years, but right now I need OSX if I wanted to develop for OSX.
OSX con: Support for laptop hardware. It's ridiculously bad, even though it's Apple's own hardware. The only way to turn off laptop monitor is by sacrificing a chicken (put computer to sleep; wake it up without fully opening the laptop screen; then maybe it will work, maybe not) - and routine software upgrade broke the "turn off trackpad when mouse is in" option.

I see no chance of OSX ever catching up with Linux before I get old.

Linux would need at least a decent editor - and if redcar won't work, I'm out of hope - Emacs and vim are stuck in 80s and haven't improved the slightest since I started using Linux. All KDE/GNOME-based editors are made without any clue.

Laptop hardware support gets better each year, while Mac seems be to getting worse somehow, so that problem doesn't bother me that much. And I can live with ugly fonts and less GUI polish.

I might miss Omnifocus a bit, but that's definitely something I can live without, especially since Omnifocus is OK but not that awesome, and fairly moderate amount of GUI programming should make it possible to make something more or less as good, if only GUI programming wasn't such pain.

So I'm quite tempted to switch to Linux. Am I missing some big pro or con of either system?

How much democracy in your democracy?

2010-04-19T22:31:00.000+02:00

As elections in both UK and Poland get closer, I want to write a bit about some severe flaws in modern Western-style representative democracy. Unfortunately it's very difficult to hold a meaningful discussion about such subjects, as far too many things get collectively labelled "democracy".

This is not at all unreasonable, as they tend to cluster together. So rule of law, constitutional limits on government powers, freedom of speech and press, ability to run business independent of government connections, high standards of living, low corruption, regular elections, existence of multiple political parties, and even bicameral Parliaments tend to be present all at once in the same set of countries, or to be absent all at once in others.

Yet, there's nothing causal about this, and this is largely a historical accident. Many people are under impression that the kind of governments we have are a result of long careful thinking, or many experimentations with different government design - and nothing could be further from truth - it's all one big historical accident.

History lesson

Between death of Julius Caesar and establishment of United States, there was only one large country which could be even vaguely described as "democratic" - the Polish–Lithuanian Commonwealth. And like most things in history, it all happened by accident - after the main line of Piast kings went extinct in Poland in 1370, Polish lords figured out that instead of electing a new dynasty, they will be approving every new king one by one.

Normally it would bother the king a lot, as most kings had been obsessed with making sure their children inherit their realm, but Jagiellons weren't that terribly bothered as they had still been guaranteed succession in Lithuania, and Polish lords could be relied on to elect someone from the Jagiellon family if they wanted to keep the Polish-Lithuanian union going. The only thing they had to agree to were fairly modest limitations on their powers - limitations that would get larger and larger with each next generation, and power was shifted further downwards from big lords to wider nobility.

By the time of death of the last Jagiellon in 1572, the system was something like this:

There was a bicameral Parliament - upper house Senate (Senat) consisting of major lords and bishops
Lower house House of Representatives (Izba Poselska) consisting of representatives of various districts elected freely by all noblemen living in a given district.
King was elected for life by all noblemen.
All legislation had to be approved by both houses of parliament and the king.
No war could be waged without approval of the Parliament.
All taxes and tariffs required approval of the Parliament.
Before the coronation, the king had to take an oath to uphold basic constitutional right.
Polish "nobility" (szlachta) was a very wide social class - comprising of about 10% of population, and in some parts of the Commonwealth as much as 30% (of that only males could vote, not surprisingly) - this is far more than even in most of 19th century republics, the main different being that in Poland this status was hereditary while in most 19th century republics it was based on wealth.
There was no slavery as such, but there was very harsh serfdom system in some regions.
While Catholic Church still had semi-official status, unprecedented levels of freedom of religion was guaranteed.

This is essentially identical to American system of government as founding fathers made it, and very very close to American system of government as it exists today. Modern American system of government, and also those of other democratic countries - is based on such accidents from a half millennium ago, similar accidents from British history, more accidents from 19th century conflicts between demising landed aristocracy and growing bourgeoisie, and on top of that those due to 20th century's rise of social democratic welfare state.

Is this the only way to organize a democratic state? Not at all! Even a very quick look at Athenian democracy, Roman Republic, and Medieval Republic of Venice - three particularly notable historical example - show how ridiculously vast is the range of possible democratic models. According to some ranks there are 80 or so countries which are reasonably "democratic", but nearly every single one of them a tiny variant of essentially a single political system, with neither theory nor experimentation to show that it's any better than the alternatives, and even outliers like Switzerland are not that terribly far off.

How democratic are Western democracies?

It's not terribly difficult to make up some ad hoc stories explaining why this particular historical accidents just "happen" to be the best system possible - every political system, every feature of social organization, essentially everything anyone has ever done had plenty of apologists of this kind.

But can we actually measure it? Yes we can. Let's start with one very simple metric - what percentage of votes did the ruling coalition get. Surely we should expect at least 50%, right? And we would prefer if policies set had support of reasonably solid majority, let's say 55%-60%, instead of the country being governed by what is essentially a rounding error?

Not so! Even without considering tactical voting, electoral systems tend to be ridiculously flawed, and in many countries it is actually completely common for well placed minority to be able to hold control over the government.

Going down the list according to Democracy Index score, ruling coalitions in the top democracies had such shares of popular vote:

So skipping the Swiss, of 20 supposedly "most democratic" countries in the world - governments of only 7 got majority of votes - and this doesn't even account for tactical voting - as I'm sure plenty of Americans would rather vote Green or Teabag, but the system essentially forced them to choose between the two leading parties or staying at home, so all these numbers seriously overestimate actual support for governments.

And even in spite of this overestimation, three governments are so far from majority of votes - UK, Denmark, and Canada - that their inclusion among "democracies" is highly dubious. OK, current situation in Denmark seems to be unusual, but no government in United Kingdom received majority of votes since 1931. And it isn't terribly likely to change - unless a miracle happens in the next elections and in the hung Parliament Liberal Democrats push a Parliamentary reform through.

Three more governments - Spain, New Zealand, and Australia - are still in highly dubious low 40s, and like with UK and Canada this seems to be the norm. So in one third of top 20 "democracies" governments rule thanks to quirks of electoral systems, not thanks to actual popular support - before tactical voting is even accounted for!

I haven't studied all 20 systems in detail, so this might be a serious oversimplification, but it seems to me that we could divide them into 3 groups:

Most of continental Europe has proportional representation with mild distortions due to electoral system and low levels of tactical voting - typically government support in high 40s / low 50s
UK and its former dominions tend to have first-past-the-post or otherwise highly distorting systems, but moderate level of tactical voting - typical government support in high 30s / low 40s - but once you account for tactical voting often even less
USA - first-past-the-post with very high distortions (including routine active manipulation of voting system via gerrymandering etc.) and very high level of tactical voting - typical government support in high 40s / low 50s - but this is mostly illusion thanks to very successful exclusion of third parties, and real support is much lower (not that parliamentary majority in USA matters that much, but presidential elections are about equally distorted)

In other words top democracies are either borderline democratic "rule of the rounding error" of continental Europe, or simply not democratic at all in the British and American world. And in case you think I'm exaggerating about United States - polls show only 23% or so of voters approve of what Congress is doing. I wouldn't be surprised if even Chinese Communist Party and Hugo Chávez had far more support than that.

So if voters have so little influence, what good is Western-style representative democracy? Yes, it correlates with many good things like civil liberties, and general welfare, but which way does the causation go if there is any at all? And shouldn't countries with less voter control like UK and US be far worse off than those with more voter control like Iceland or the Netherlands? Or maybe they actually are... I'll leave that for another post.

Dating Potential PageRank

2010-04-15T20:31:00.000+02:00

Some people, when confronted with a problem, think “I know, I’ll use regular expressions”.
Some people, when confronted with a graph problem, think "I know, I'll use PageRank".

Being a member of both groups, I was wondering about all the data analyses okcupid was running - and how can we actually measure dating potential correctly? Using chance of getting a reply to your messages like okcupid's analyses is strongly biased towards Internet trolls who are really good at getting replies - but that's not concern of this post. Let's say we actually have a good measure of how attracted one person is to another (and for simplicity assume it's between 0 and 1, values in between being interpreted as either levels or probabilities).

The problem is that even such simple kind of attractiveness requires a graph analysis - counting how many people are attracted to someone is not terribly useful as it ignores some distinctions that proper graph analysis would catch:

how attractive are people being attracted to you? (by recursive graph analysis)
to how many other people are they attracted in addition to you (or do they simply have low standards?)
is it mutual or are you attracted to wrong kind of people?

What I'm going to do is to turn every person into a website, attractiveness into linking, and compute PageRank on the entire dating market. Scenarios presented here are hypothetical, as right now I'm more interested in figuring out proper analysis tools than in running any real analysis before I know if I'm measuring a sensible thing or not.

By the way, these graph measures works surprisingly well in presence of an arbitrary mix of hetero-, homo-, and bisexuals in the network, and it's fine if some people are homophobes or ... let's call them "biphiles" - guys who are really into lesbians (think threesomes) and girls who are really into gays - and in either case typically preferring a bisexual opposite gender partner to a heterosexual one - we can just note that in their attractiveness links.

AttractiveRank

The most naive thing to measure would be PageRank of "link to someone if you're attracted to them".

Scenario 1. Let's populate the world with 10 ensembles of girls and 10 ensembles of guys separated by which decile of physical attractiveness they fall into. People are only attracted to those at least as attractive as they are.

The results are quite drastic. Because the 10s are desired by everyone, yet extremely picky, they get really high scores - AttractiveRank by decile is 0.110 0.122 0.138 0.158 0.186 0.227 0.292 0.418 0.759 7.591 - and that's only because the model includes a 10% leak - traditionally meant to represent "user getting bored and typing a new URL" - here maybe "someone getting really really drunk at some party".

Let's modify it a bit and create Scenario 2 - with people attracted to those of just one decile less too. So 7s are attracted to everyone from 6 to 10 and so on.

This is great news for 9s, and pretty good news for everyone except 10s - now the ranks are 0.124 0.140 0.162 0.194 0.253 0.394 0.780 1.715 3.119 3.119. All that thanks to just slight lowering of everyone's standards.

ReciprocityRank

There's a serious problem with this anyway. In scenario 1, 10s could have any 3s or 8s if they wanted - but they very much don't. So everybody is actually is a very similar situation - 3s have as many other 3s to date, as 10s have 10s.

Let's introduce ReciprocityRank - which is just like regular AttractiveRank except non-mutual links are dropped. So it's you're a 7 and you wouldn't actually mind that 9 over there if you thought you had any chances, but as a good graph analyst you know you don't - you forget about that 9 in no time.

For scenario 1 ReciprocityRank correctly identifies that everyone has the same rank 1. Scenario 2 is more interesting - with ranks of 0.781 1.098 1.058 1.036 1.027 1.027 1.036 1.058 1.098 0.781. First, it is strangely symmetrical - it sucks as much to be a 10s as it does to be a 1s - in one case because your standards are too high, in other because everyone else's standards aren't low enough - but the effect is the same. What might be more surprising is how 2s and 9s are in the best position - not only can they date other 2s and 3s / 8s and 9s respectively - they have access to large number of fairly desperate 1s or 10s. Yes, 10s' desperation comes from unreasonable standards, but it doesn't matter (mathematically that is).

DateRank

That's probably not what we're looking for. You'd think that these 10s are in better situation - they might not have much choice but it's all quality choice. And if you're a really desperate 10 you can always change your mind about one of those 8s that stalks you. A 1 doesn't have such luxury. So let's count unreturned links, but only for 1/5 as much as returned ones, sort of like ref="nofollow" about which search engines changed their mind so many times.

Scenario 1 results in DateRank by decile of 0.147 0.167 0.194 0.229 0.281 0.359 0.493 0.759 1.474 5.897 - quite reasonable. Scenario 2 results in 0.210 0.282 0.336 0.424 0.569 0.808 1.194 1.755 2.408 2.013 - 9s are strangely still better than 10s - but notice that everyone is equally attracted to both, it's only 9s' lower standards which increase their dating potential.

Now that our links have weight, let's try scenario 3 - people are attracted to those less attractive than them only 50% as much as to those equally or more attractive than they are. Surprisingly it only helped 9s and now DateRank is 0.188 0.235 0.278 0.349 0.469 0.686 1.080 1.730 2.558 2.426.

Height and education

Now that we have a sensibly defined PageRank - let's try something closer to reality. Let's say guys and girls come in three tertiles by height - short, medium, and tall; and in three independent tertiles by intelligence and education - let's say dumb, average, and smart. Of course they're not directly comparable across gender lines - what counts as tall for a woman might count as medium for a man.

So the rules of scenario 4 are:

Women don't like men shorter than them
Women don't like men dumber than them
Men don't like women taller than them
Men don't like women either smarter on dumber than them

It's all loosely based on this paper, I'm not making this up. Could you guess who has it easier and who has it harder in such situation?

Here's table for guys (rows by height, columns by intelligence):
   0.240   0.401 0.985
   0.380   0.703 1.873
   0.550 1.047 2.821

And here's one for girls:
   0.652   1.091   2.676
   0.436 0.730   1.789
   0.265   0.414   0.946

So how good was the analysis? Let's look at girls' table first. It's best for the girl to be short and smart. We could have predicted the shortness - guys not liking taller girls was explicitly in the rules - but why would girls want to be smart if equally many guys will be attracted to her regardless? This is a good example of why graph analysis matters. If a girl is dumb, plenty of men will be attracted to her, but these will all be dumb men shunned by most. If a girl is smart, she gets exactly as many men to choose from, but these are some quality men.

And unsurprisingly it pays for a man to be either taller or smarter - but intelligence is a far greater factor than height. How did that happen? And why does the same hold for girls? (except with reverse order of height) In this model men are more selective on intelligence than on height - and only same-intelligence combinations are possible (3 combinations out of 9 are ok), while plenty of mismatched height combinations are possible (6 combinations out of 9 are ok). You decide how meaningful this is.

Age

A couple months ago OkTrends wrote an interesting post about men not being terribly attracted to older women. Now I'm not going to use their data because I'm too lazy to convert pictures into numbers, and it's rather noisy, so the rules of scenario 5 are:

There are 31 ensembles of men and women, for each age between 18 and 48.
Women are attracted to men no older than their age + 6, and no younger than their age - 4; and half as attracted to men between their age - 8 and their age + 10.
Men are attracted to women at most 4 years older than them, and this is a hard limit, no exceptions.
Men are most attracted to women older than their minimum target - which grows linearly from 18 at 23 to 32 at age 48; however they're still open to younger girls (0.5x).

And these are the result - unsurprisingly men peak later than women (around 28, not 22), and while they're not as hot at their peak as women, they don't crash as hard as women either.

There are some limitations of this analysis - first, because okcupid's data used hard cutoff of 18-year, it discounts the possibility that a given 18 year old might wish to date someone who's 16 or 17 - their actual potential is higher than indicated - and likewise for older people. By the way it is blatantly obvious in okcupid's graphs that there are plenty of early-30s women who lie and pretend they're exactly 29. And if you've ever been there you'll know that there are quite a few under-18s who pretend to be 18. So take their data with a pinch of salt.

Another problem is assumption that all ensembles are equally sized - in reality while there are about as many 48 old women alive as there are 22 year old, there are far fewer of them on the dating market. Let's run scenario 6 - with ensembles of under-30s are 5x as numerous as ensembles of over-40s, with sizes decreasing linearly during the 30s.

The peaks stayed where they were - but the fall with age is far more drastic, especially for women.

Offensive misogynist advice

Guardian-reading liberals are asked to skip this section.

To the extent that these models are valid - the advice they give to men seeking long term relationships would be to go after teenage girls before their peak; while men looking for a quick hookup would do better going after women in their late 20s or early 30s - when they are still hot but already getting desperate. And in either case go after taller girls to avoid competition.

Model's advice to women is - get pregnant while you can. Unless you're still a teen, your dating value is only going down, and quality of men you will be able to attract for a serious relationship is getting worse each year. And remember - your biological clock is ticking, and setting up a family takes years (accidents excepted). You might think that you're still hot now, but what if after 2 years with someone you decide he's not the right guy, then you spend 2 more years with someone else, then you decide to perhaps maybe try getting a baby - nowhere near as quick and easy as it seems in your 30s - all while your younger friends look more and more tempting to your man; while your eggs are less and less cooperative. The legal system is what it is - in most countries getting married and having a few babies sets up woman for life, so even if it doesn't seem terribly appealing now, you need to decide thinking what will be appealing in a decade.

That, or freeze your eggs and start saving for regular botox. Works either way.

Don't blame me - it's the reality which is misogynist.

Code and mathematics

Code can be downloaded here.

One problem with this approach is existence of two completely arbitrary numbers - booze factor (0.1) and stalker factor (0.2) - they are not derived from anything meaningful and changing them can completely change the results. In fact the booze factor gives a hard floor to how low can anyone's attractiveness fall, and three values of stalker factor (0.0, 1.0, 0.2) result in three completely different ranks discussed here.

Another problem is assumption of constant ratio of mutual to unreturned attraction. We might as well assign constant percentage of links to mutual attraction and the rest to all links bundled together. It matters as people with low standards would focus on attraction returned instead of wasting their link power / time on stalking. It would be a fairly straightforward modification, and I don't know which version would work better.

And remember - all models are wrong, some are useful.

First look at kUbuntu 10.4 Lucid Lynx or why I hate KDE4

2010-04-10T02:31:00.000+02:00

One thing that still amazes me is how in 2010 operating systems still require ridiculous amount of work from inserting installation DVD to having a semi-decent system - even though no technical reasons prevent that from being a 5 minute zero-click operation.

First, installer decided to crash a few times. It is only beta 2 I've been testing, not final release - but seriously, that just shouldn't happen. Random installer crashes a couple of weeks before Long Term Support release?

Next there's a few classic system configuration questions - region, timezone, keyboard - and shouldn't they just set the defaults based on DHCP + geoip? Anyway, it's a small issue compared to those to come.

And here we come to my first big complaint - partitioning. First, why the fuck do they still use swap partitions instead of swap files like every other operating system? Don't make it more difficult for users for some marginal gain in performance. If they cared about performance that much they wouldn't be using KDE, you know. And second - how about some sane defaults? There was an ext3 partition from some old Linux and ntfs partition on that computer, but I had to manually tell the installer what I want to do with them - as if it was so difficult to suggest / and /media/c or somesuch so as defaults.

Did I mention installer was ridiculously slow? Did they use framebuffer graphics driver or what? Anyway, a few clicks, and the system is installed - this is of course only the beginning.

Go to the OSX-ripoff system settings panel (Applications > Settings > System Settings), and let's start fixing things.

It's been an ancient tradition in all operating systems I know that mouse and keyboard defaults are designed for old people with Parkinson's - so we need to go to keyboard, reduce repeat delay and increase repeat rate to saner values. Of course unlike every system since probably Windows 3.11 KDE didn't even bother including test box in its keyboard speed configuration dialog. Seriously guys? If you cannot get even something as obvious as that right, what chance do you have for even chasing the 2x overpriced Apple, let alone Microsoft?

It's even worse when it comes to configuring mouse. Instead of one mouse speed slider there are 6 confusing numeric entry boxes. Tip: increase the first from 2x to 5x to restore sanity.

While in keyboard and mouse settings I was wondering if I can make one of special keys launch a browser - but application launching didn't seem to be available anywhere. And I stumbled upon something really ridiculous - monitor brightness controls are under "KDE dæmon". With a ligature. You're trying to outhipster Macs or what? Seriously, fix that spelling - or even better label these "monitor settings" or something like that.

Now it was time to install some software to make the system useful. Konsole was hidden deep is "Applications > System" and I needed to scroll down to get to it. I know it's a trivial inconvenience, but even Windows fucking 95 had that right. Do not use tiny constant height widgets for Start menu - and to add insult to injury make them use huge icons. It can only display 8 programs without scrolling, and they're not even arranged in any sensible order (it's supposedly alphabetic, except "Konsole" is under T for "Terminal"). Are you trying to repeat the spectacular success of "let's throw away everything and reinvent GUI" OLPC?

So I was looking for apt-something, and because it was not obvious I just typed "apt" in the program search box - it came with literally 15 results, and KPackageKit was not even in the first half.

While I looked around the menu I accidentally clicked Akregator - there's zero pixels of inactive margin between clickable areas to run programs - Fitts' Law called me and told to says hello - and instantly proceeded to quit it. Instead of quitting it showed me the "You're not gonna catch me, I'm going to stay in the system tray" message - I tried looking for it but the system tray in fresh KDE is as filled with unnecessary garbage as a Windows 98 installation which has been running since literal year 1998. Why do I need half of that? Anyway, I located some icon which looked like Akregator, asked it to quit and it proceeded to enrage me with "Are you sure you want to quit Akregator" dialog? What kind of question is that? I've told you to quit twice in the last 30 seconds you dumb program! And there are no unsaved documents or anything there - you're a fucking RSS reader! If there were unsaved documents you should tell me about them - and there is never ever under any circumstances any reason why a program should ask if it should quit or not, other than unsaved documents. How is it that every minute of using KDE I stumble upon such a basic usability failure?

So, KPackageKit started - let's install Firefox because you won't catch me using Konqueror even for web testing and I'm the kind of person who still tests in IE6. Actually I wanted to check if Firefox is maybe already installed - the useless Konqueror is prominently displayed in Start menu, but maybe Firefox is somewhere there. So I typed "firefox" in start menu search box and clicked on something called "Mozilla Firefox Web Browser Installer" - the only result this time. It steals the whole screen for some reason (all right, some video came installers use fullscreen mode, but I've never seen any regular installer do that before...) and when I click "Install"... it tells me that "Requested packages are already installed".

So is Firefox installed or not? KPackageKit search for firefox only lists "kubuntu-firefox-installer" package - exactly the same one which fails so hard. I even tried to search for "iceweasel" - in case kUbuntu shares Debian's obsession with legalistic minutiae of its trademark amendment to DFSG but no. I typed "firefox" in Konsole to get "Program 'firefox' is not installed. You can install it by typing sudo apt-get install firefox" - apt is the most awesome thing that ever came out of Linux - but when I tried that I've been told firefox package is not available.

Wait, did they even get apt repository listings? I assumed they must have done it automatically, as KPackageKit showed no button or anything suggesting that manual action was required - but "sudo apt-get update; sudo apt-get install firefox" actually did the trick. I wrote a whole post about configuring Firefox 3.6 a couple of months ago, so I won't be repeating myself now.

So, time for some serious package installation. Now I've noticed something I'm rather ambiguous about. Dragging a window to the left or right edge of the screen semi-maximizes it to full height / half width - which is an idea so brilliant that if I ever meet the person who came up with it they're getting some free booze. Unfortunately X is also the only graphics system with usable virtual desktops (if any of you ever mentions Mac Spaces, God is going to drown some puppies) - and usable virtual desktops work even better if screen edges move between desktops - especially since by default there are no keyboard shortcuts to move windows either - Mouse and Keyboard > Global Keyboard Shortcuts > Window to Desktop 1 > Custom > Ctrl+Shift+F1 etc. fixes that fortunately. I like having both ways of moving stuff from one virtual desktop to another, but at a very least one way should be provided, no? (right clicking on title bar doesn't count)

So package installation. First, 159 packages got updated since beta 2 release - maybe not that surprising so shortly before the big release. And upgrading programs with apt is such a joy after suffering of MacPorts.

Back to Firefox for a moment. Why is there no Bing in default search engines list, when they have ever obscure ones nobody uses like ask.com and creative commons? Is it all Firefox or just kUbuntu's? It's not a big deal by any means, but if you want Microsoft to offer users choice of browser with its operating system, why not offer users choice of search engine with your browser? And speaking of browser choice - why nobody asks me which browser I want as default? Because the answer is very much not Konqueror. Konqueror is the MSIE of ... well, of not-Microsoft. I have about as much interest in using Konqueror as I have in using Safari or IE. Basically, default browsers for all platforms somehow suck massively, and all sane people use Firefox... or maybe Opera. So I went to youtube to see if Flash works. I've been greeted with the usual "Missing plugins" message, so I clicked some buttons - and the relevant plugins have not been found obviously - why would it work if it can rather not? Fortunately apt-get install flashplugin-installer worked - by this time I abandoned pretty much all hope of installing anything from GUIs and went straight to good old console, as if it was 1999 and I was still using Debian 2.1 slink... By the way did you know youtube had ads? Wow, sometimes I don't realize how different the Internet for smart people is from the Internet for everybody else.

This installation guide/diary gets ridiculously long, and we're still quite far from a usable system. From memory I just typed apt-get install build-essential irb1.8 irb1.9 octave3.2 unrar ttf-mscorefonts-installer mplayer mc git subversion mongodb - I'm sure I forgot half the programs needed, but I'll figure them out someday. By the way, how hard would it be to just have apt-get install everything? Yes, there will be a couple of conflicts, complicated packages, and such, but 90% of packages should be installable everywhere without any hassle. I couldn't find Sun Java, and I know past Open Source javas like Kaffe and gcj were essentially useless - but I'll give that OpenJDK a try. Once. Like I did with Tesco Value.

At this point the system is reasonably close to usable except for a few things which I'll keep open:

Why don't PageUp / PageDown / Home / End work in Konsole (or xterm)? They work in Firefox just fine and my keyboard is not unusual - just awesome.
How do I make Konsole and Firefox start when I log in? It's right there in dock menu on Mac, so it shouldn't be too hard in KDE either.

How to kill all your children

2010-04-05T15:15:00.000+02:00

I'm sure you all had this problem a few times - you tell your children to do something, and they just won't obey. At first you wait, but your patience grows thin until you come to face the inevitable - you have no choice but to kill all you children. But first you need to find them.

Unfortunately there's no portable API for traversing process tree to find all your descendant processes. ps, pstree and similar programs use operating system-specific hacks, like reading procfs (Linux) or reading tea leave patterns (OSX). Even command line options for ps seem to follow three different mutually incompatible systems. And careful tracking of return values of fork won't do you any good, as forked processes might have kept spawning.

So we'll need more than one line of Ruby. Maybe even as many as five!

First, let's get raw data out of the operating system. If it was Linux-only exercise, procps would work fine, but some experimentation shows that both Linux and OSX versions of ps support -eo pid,ppid options for getting process parentage graph.

$ ps -eo pid,ppid
  PID  PPID
    1     0
   10     1
   11     1
   12     1
   14     1
    ... 
 3367  3366
 3382  3367
 3441  1240
 3442  3441
 3460  3442

Before doing anything else, let's just turn it into a hash.

Hash[*`ps -eo pid,ppid`.scan(/\d+/).map{|x|x.to_i}]

Now we know what's parent of every process and the only problem is reverting this graph - something that's strangely missing from the usual Hash/Array toolkit that Ruby and Perl are based upon.

A brief pause time. You might be wondering what happens when process's parent dies. Normally it is the reparented to its grandparent, or init process, whichever is more sensible based on process groups and other things we don't need to concern ourselves with. So process graph is always proper, unless something weird happened during execution of ps (atomicity in my Unix? it's less likely than you think).

Now it's time to do a lazy half-unification. Create a list for every process containing just its id, and then go through all processes and add reference to its list to its parents' list. When we're done every process's list will contain at some depth ids of all its descendants. Now it's just flatten and remove yourself from the list to avoid a suicide.

def Process.descendant_processes(base=Process.pid)
  descendants = Hash.new{|ht,k| ht[k]=[k]}
  Hash[*`ps -eo pid,ppid`.scan(/\d+/).map{|x|x.to_i}].each{|pid,ppid|
    descendants[ppid] << descendants[pid]
  }
  descendants[base].flatten - [base]
end

Once you've found all your children and further descendants, it's just a matter of a quick kill -9 to finish them all off.

As a bonus, here's the version of the same function in Perl.

sub flatten {
  map{ (ref($_) eq "ARRAY") ? map{flatten($_)}@$_ : $_ } @_;
}
sub descendant_processes {
  my ($base) = (@_, $$);
  my %parentage = (`ps -eo pid,ppid` =~ /\d+/g);
  my %reverse = map { ($_, [$_]) } %parentage;
  while (($pid,$ppid) = each %parentage){
    push @{$reverse{$ppid}}, $reverse{$pid};
  }
  shift @{$reverse{$base}};
  flatten($reverse{$base});
}

Notice the advantage Ruby has over Perl - and how workaround while annoying are not that difficult:

There are no default arguments, but we can easily hack them with an equivalent of [*arguments, default] for one-argument functions.
I wanted to say in Perl we don't need to_i - but then it doesn't really change anything in Ruby either, operating on pids like 1234 instead of "1234" just feels saner.
We can use Ruby autovivification which takes blocks, but Perl autovivification is useless as we want hash values to default to [key] not []
There's no flatten so we need to implement it (and it's such an extremely common function - the reason it's not in Perl is because Perl started without any support for nested arrays whatsoever)
There's no way to subtract lists from each other like Ruby's [1,2,3] - [2]. Fortunately we know own pid is always first, so we can just shift the list.

As always, C++ version left as an exercise for the readers.

Personal experience points and OSX menulets

2010-03-29T03:42:00.000+02:00

Doesn't it seem odd how people are willing to spend so much time and effort on doing everything that's best for their in-game characters, and yet they never do anything for their real lives?

And it's not just highly complex MMORPGs like World of Warcraft and Eve Online - look how many people spend how much effort on FarmVille - which is about logging it at scheduled times and clicking harvest/plow/plant seeds on a large number of virtual squares! Many of them are the same people who avoid putting any effort into their real life characters as much as possible.

There's been some interesting discussion about this real world - gaming world divide, started by this TED talk by Jane McGonigal - like vast majority of TED talks it's totally wrong, and totally worth watching for entertainment value and some intellectual stimulation:

By the way - to people who are new to this blog - links here are like on Wikipedia - they usually lead somewhere interesting.

Anyway. Assuming you want to achieve some real life outcomes but your laziness and disorganization stops you - like most people - why couldn't you simply approach life just as if it was a game? Well for one thing games give you a lot of immediate feedback that you're doing well, but real life doesn't.

So why not fix the problem and create such immediate feedback? This "feedback" is usually little more than just some numbers, progress bars, and make some badges or other icons. Let's do it!

Experience points log

Well, first thing you need is log actions which give your real life character experience points.

A log can be a simple semi-structured text file like this.

= RULES =
# Every time
+10 write a blog post
+2 read a book
+1 go to gym
+1 play with cat

# Just once
+100 setup a backup system
+500 organize free elections in Belarus

= 2010-03-27 Sat =
+10 blog post
+1 played with the cat

= 2010-03-28 Sun =
+1 played with the cat
+2 read a book

Example is completely made-up but you can think of some points for yourself. And of course you can go back and change these rules and numbers any time you want.

Recording actions

Unfortunately there are no iPhone apps for that yet, so we'll have to record our actions manually - but let's make it as easy as possible. The log is located at /home/taw/all/xp/xp.txt and I have the bound one of the keys on the most awesome keyboard in the world to open this log in case I want to edit it.

And because I almost always have terminal open, I wrote a script which appends current date header if it changed, and if you passed it any arguments it appends them to the log, otherwise it opens it. Actually now that I think about it, it wouldn't be a bad idea to get the keyboard shortcut to do this date change too... The script also edits itself if I pass it --edit

#!/usr/bin/env ruby1.9
exec "mate", __FILE__ if ARGV[0] == '--edit'

require "time"
require "date"

fn = "/home/taw/all/xp/xp.txt"
last_day = Date.parse(File.read(fn).scan(/^= (\d{4}-\d{2}-\d{2} \S{3}) =$/)[-1][0]) rescue nil

out = []
unless last_day == Date.today
  out << Date.today.strftime("\n= %Y-%m-%d %a =\n")
end
out << ARGV.join(" ") unless ARGV.empty?
unless out.empty?
  open(fn, "a"){|fh| fh.puts out.join}
end

system "mate", fn if ARGV.empty?

It's all highly portable so far as long as you adjust paths etc. - nothing OSX-specific about it.

Drawing progress bar

It's fairly simple to think of a regexp to parse experience points out of the log - the only nontrivial bit is skipping rules section. But we'd much rather have pretty icon than numbers.

Well, first let's parse XP and convert it to levels, in a typical quadratic system (code assumes Symbol#to_proc; upgrade your Ruby to 1.8.7 or paste it from core_ext if it dies).

module XP
  class << self
    def read_xp
      log = File.read("/home/taw/all/xp/xp.txt").sub(/\A.*?= \d{4}-\d{2}-\d{2} \S{3} =\n/m, "")
      log.scan(/^\+\d+/).map(&:to_i).inject(&:+)
    end
    def threshold(level)
      10 * (level ** 2)
    end
    def stats(xp=(ARGV[0] || read_xp).to_i)
      level = 0
      level += 1 while xp >= threshold(level)
      [xp, level, xp-threshold(level-1), threshold(level)-threshold(level-1)]
    end
  end
end

XP.stats returns an array of [experience ponits, level, XP since acquiring current level, XP needed from current level to next] so code drawing pictures doesn't have to think about it.

Now the drawing part. We want 32x16 icon - height is constrained by OSX menu bar height, this ratio just looks good. On the bottom we have progress bar, on the top 1-4 colored circles showing your level - levels 1-4 are green, 5-8 are cyan etc. Here are some examples:

And the code:

require "rubygems"
require 'RMagick'

canvas = Magick::Image.new(32, 16){
  self.background_color = 'transparent'
}
Magick::Draw.new.instance_eval {
  stroke('black')
  fill('none')
  rectangle(2, 9,  30, 14)

  xp, level, xp_got, xp_needed = XP.stats

  stroke('none')
  fill('#0F0')
  rectangle(3, 10, 3 + (29-3)*xp_got/xp_needed, 13)

  level_colors = %w[#0FF #FF0 #F0F #00F]
  while level > 4 and !level_colors.empty?
    fill(level_colors.shift)
    level -= 4
  end
  level.times{|i|
    x = 4+8*i
    circle(x, 4, x, 1)
  }
  draw(canvas)
}
canvas.write(ARGV[0] ? "xp-#{ARGV[0]}.png" : "/home/taw/all/xp/xp.png")

In both parts of the code weird ARGV[0] code is just there for making sample images instead of reading the actual log. Not exactly a masterpiece of modularity, but that's real world Unix scripting for you ;-)

Putting it in OSX menu bar

All this work is fairly useless we put the icons generated somewhere no display - like on OSX menu bar. So far everything has been reasonable system-independent - this part is highly OS-specific - unfortunately all cross-platform GUI toolkits take lowest common denominator approach, and look like shit on all systems equally. But it's really simple code. It's really easy to make the menu icon interactive but then I don't have any particular need for it to be interactive yet.

It must use OSX-shipped ruby interpretter (/usr/bin/ruby), not one from MacPorts unless you want to deal with some serious library hell.

#!/usr/bin/ruby

require "osx/cocoa"
include OSX

class Timer
  def init
    tick
  end
  def tick
    system "./draw_xp.rb"
    $statusitem.setImage(NSImage.alloc().initWithContentsOfFile("/home/taw/all/xp/xp.png"))
  end
end

app = NSApplication.sharedApplication 
$statusitem = NSStatusBar.systemStatusBar().statusItemWithLength(NSVariableStatusItemLength)
NSTimer.scheduledTimerWithTimeInterval_target_selector_userInfo_repeats(1.0, Timer.new, 'tick:', nil, true)
app.run

Alternatives

One thing I found while writing this blog is Chore Wars, which seems to be like a website for getting XP for performing real life actions - there might even be a Firefox port for constant visibility, but making it public is kinda creepy. You don't want Belarusian secret police finding out about your "+50 sent information about Lukashenko's secret Swiss bank account to WikiLeaks".

Small tips for making Unix programming nicer

2010-03-27T01:39:00.000+01:00

Every Unix programmer - and I tell it authoritatively based on my statistically significant sample of 1 - creates millions of tiny scripts that he never bothers publishing. Because they're small, so overhead of cleaning up, documenting, and publishing such script would be immense. And not obviously googleable/bingable/baiduable or whatever people call it these days - it might surprise you but most things in the world are not described by a few well-defined key phrases.

I decided to get a few of such scripts, and throw a bunch of tips on top of them - hopefully you'll find a useful trick or two here.

How to avoid Unix destroying your files

First, we need to fix some of the famous Unix brain damage - cp, mv, and bash overwriting your files.
Now a case could be made for rm removing files without asking for confirmation - that's what rm stands for.
But how in the world is this:

mv most_awesome_song_ever.mp3 ~/Music/

supposed to quietly overwrite existing ~/Music/most_awesome_song_ever.mp3?
In 99% of cases this is behaviour you do not want - and if you actually do,
you know to type this instead:

mv -f most_awesome_song_ever.mp3 ~/Music/

So open your ~/.bashrc and put these commands there (at least ones for mv and cp):

alias rm='rm -i'
alias mv='mv -i'
alias cp='cp -i'

There's one more way Unix can destroy your files - command >file redirection will overwrite file without asking, so it's pretty easy for accidents this way. You really don't want

cat >~/notes/girlfriend

to lose information about all your exes, do you? (example purely speculative) Get yourself into habit of always saying command >>file - appending to file. 99% of time the file in question won't exist - so the result will be the same - and in cases where file exists, you remove it first. If you make a mistake, you just ^C and you're back when you started. I haven't used overwriting redirection is years - it's completely purged from my Unix dictionary. Do the same thing.

By the way some Unix distributions do these by default. And some shells have options for asking for confirmation if > would override files. Do these anyway, just to be sure.

Make shell history useful

By default shell history stores entire 500 entries - a sensible decision back when you had 4MB of RAM, most of which taken by Emacs. It's just ridiculous these days so as first line of .bashrc put export HISTSIZE=1000000. It must be first, because if bash ever decides to exit without HISTSIZE set to a sane value, it might decide to trim your history file and lose all your history. Of course if you want to tempt the fate...

Get rid of .pyc file everywhere

Python's habit of creating .pyc files all over the place gets on my nerves - and it's dubious that they really improve performance that much. If they want to fix performance, how about dealing with GIL first instead of resorting to such hacks? Anyway export PYTHONDONTWRITEBYTECODE=1 totally fixes the problem without any side effects.

Saner output from Unix commands

Compared to fatal loss of data, these are just minor annoyances, but they're really easy to fix, so let's do it.
For stupid reasons du and df commands give sizes in units of 512 bytes or something like that.
Probably some ancient BSD file system allocated files in multiplies of that. That they care more about file system
implementation details than about human usability tells you something about the Unix mindset (if these were at least kBs
that would make sense but no...) - and in any case these assumptions are no longer true on modern operating systems,
which don't rely so mindlessly on blocks. So add these two lines to get human-readable output:

alias df='df -h'
alias du='du -h'

By the way alias command only applies to what you type in the terminal, not what scripts do. So if you run mv or df from scripts, it will not have this default -i/-h. Be cautious.

Final touches for .bashrc

If some commands require root access, and you're tired of typing sudo this and sudo that, just add a few of:

alias port='sudo port'
alias gem='sudo gem'
alias apt-get='sudo apt-get'
alias reboot='sudo reboot'

to your .bashrc. They only save you a little typing, and don't chance anything about security (you still need your admin password etc.), but why type more if you can type less.

Colorful shell

Depending on your distribution you might already have colors in your shell or not.
export CLICOLOR=1 in .bashrc will convince many commands that you want nicely colored output.
To tell that to git, you need to add to your .gitconfig:

[color]
 diff = auto
 status = auto
 branch = auto

And now your Unix comes with more rainbows.

GNU grep must die

Never use GNU grep. Use pcregrep for everything - or rak/ack if you want coloring, automatic .git directory skipping etc. Unfortunately pcregrep is ridiculously slow to type so do yourself a favour and add

alias gr='pcregrep'

to make your life even easier.

wget HTTPS nonsense

I don't know if it's just MacPorts' version of wget, or is it universal, but it seems to miss all HTTPS root CAs (take that VeriSign!). Another alias solves the problem. By "solves" I mean it opens a massive security vulnerability, but we already know that CAs will create fake certs for NSA, Mossad, RIAA, and Hackney Borough Council if asked, so the vulnerability is much less than it seems at first.

alias wget='wget --no-check-certificate'

Use ruby or perl for nontrivial actions

In early 1980s, before Perl got invented to solve exactly this problem I talk about,
people would write insanely complicated shell scripts to automate their Unix actions.

Unfortunately you cannot really serve two aims at once - being highly accepting for casual real time input,
and being highly robust for programmable interfaces - so shell sucks at both. Fortunately the problem
is solved since December 18, 1987 when Perl got invented just for this reason.

And yet - many people act as if 12/18 never happened. Wake up sheeple!

Shell is ridiculously stupid. It has arcane and fragile escaping rules.
It cannot even reliably expand a file list:

$ ls ~/porn/*.jpg
-bash: /bin/ls: Argument list too long

How useless is that? And don't even get me started with xargs, find, awk and the rest horrible mess.

How do you find top largest MP3 files - in any subdirectory?

ruby -e 'Dir["**/*.mp3"].sort_by{|fn| -File.size(fn)}[0,10].each{|fn| system "ls", "-l", fn}'

And yes, I'm calling ls 10 times here, as otherwise it would see fit to rearrange them alphabetically just for lulz.
This Ruby code is really easy and really obvious.

What would be shell solution? Something like this:

find . -name '*.mp3' -print0 | xargs -0  ls -l | sort -k25 -rn | head -n 10

You need find and xargs to avoid "argument list too long" error, then you need -print0 and -0 because file names can contain single quotes or - longcat forbid - spaces! Then you need to manually count at which position of ls -l's output is file name (conveniently ls -l uses something close enough to fixed column width to make sort work - otherwise you'd have to do some heavy awking around it), then you finally head. Personally I prefer waterboarding to having my brain suffer any of this.

Learn GUI integration basics

The chasm between nice programmable world of Unix terminals and hostile world of closed GUI programs can be to some extend lessened.

If you use KDE, dcop command gives you decent level of control over GUI programs and you can explore the interface from command line.

On OSX there's a convenient open command for opening URLs and files with the most sensible program. And osascript command, which is about as powerful as KDE's dcop except far more painful to use by trying to be "friendly" too hard, resulting in unsurprising failure. Google will help.

Editing your scripts made easy

You probably have five billion scripts in your path (you have ~/bin or ~/local/bin or ~/gitrepo/bin or such in your $PATH right?) - and you're tweaking them all the time.
Typing mate `which some_script.rb` takes forever and is not easily tabbable (Ubuntu has really good
bash autocompletion package which might alleviate this problem a lot - but most distros don't).

Wouldn't it be easier to just say some_script.rb --edit? It would also be far easier to type - somTAB --edit. It's really easy. Just put this below the shebang line of all your scripts:

# Ruby
exec 'mate', __FILE__ if ARGV[0] == '--edit'

# Perl
exec "mate", __FILE__ if $ARGV[0] eq '--edit';

# Python
import os, sys
if sys.argv[1:2] == ['--edit']: os.execlp("mate", "mate", __file__)

If someone passed --edit as first argument it will start the editor instead of running the script - otherwise it will not affect it in any way.

Python code is fairly painful because Python decided to keep low level C interface to exec* instead of providing sane Perl-style interface. And you know something is wrong if Perl is described as "sane" compared with you.

Feel free to figure out how to get this effect with C++.

Find kittens for your blog

I only want CC kittens, so nobody sues my blog. Except for defamation, that I don't mind. Here's the script

#!/usr/bin/env ruby
uri = "http://www.flickr.com/search/?q=#{ARGV.join '+'}&l=cc&ss=2&ct=0&mt=all&adv=1&s=int"
# On OSX
system "open", uri
# On Linux
#system "firefox", "-new-tab", uri

By the way could someone get Linux distros to copy open command? It's really simple and really useful.

That's it for today. Enjoy your Unix.

taw's blog

Friday Night Magic - Red Deck Wins

Draft

Deck

Games

Cards

Future

My Christmas wishlist - redcar and desktop Linux

Macs suck

Desktop Linux sucks

Redcar

I'm not done yet

Blogging about Robin Hanson is not about blogging about Robin Hanson

Science is not about science

Medicine is definitely about health

Neolithic Counter-Revolution in Diet

Diet by era

Anonymous Heroes of Science

We need syntax for talking about Ruby types

Basic types

nil and other ignored issues

Booleans

Duck typing

Technical explanation of to_str and friends

Collections

Literals and tuples

Functions

Type variable and everything else

The End

If only Ruby had macros

Ruby is now more popular than C++

Another example of Ruby being awesome - %W

Arrays are not integer-indexed Hashes

What would work

What doesn't work

Random sampling or processing data streams in Ruby

When size is known

When size is unknown

This isn't what stream processing looks like!

Synchronized compressed logging the Unix way

Palestinian problem fixed

Alternatives

Best sources of DHA omega-3 essential fatty acid

Best Sources of DHA

What is Internet good for?

Economics 101

My logs

What is Internet good for?

Empire Total War mods - no walls, libertarians everywhere

Libertarians everywhere

No walls

How to install

Greasemonkey and jQuery easier than ever

Show spoilers on tvtropes

Always sort by seeders on The Pirate Bay

Download Stewart and Colbert from bittorrent

Everybody Draws Muhammad Day and lessons in cultural relativism

Lesson #1 - Principle of dissimilarity

Lesson #2 - Why is Muhammad widely considered a pedophile?

Lesson #3 - Freedom of speech

Very simple parallelization with Ruby

Exception handling interlude

Execute N tasks in parallel

What about other comics?

Steam is ripping you off

Greeks can pay, Greeks will pay

Basics of money lending

The final assurance

Audible review

DRM

Speech synthesis

Why Audible sucks

Progress bar for Unix pipes

Brief history of human diet

Case study - USA

Case study - Germany

Case study - Japan

The unified theory of online dating and porn

The real cost of oversized military

Want democracy? Get a figurehead king

`nil` and other ignored issues

Technical explanation of `to_str` and friends