Empire Total War mods - no walls, libertarians everywhere

Rome Total War was highly moddable. Medieval 2 Total War somewhat less as all data files were in obfuscated packages - but once you unpacked them it was as good as Rome. My M2TW mod is generated by a bunch of simple regexps, with which I can experiment as much as I like, turning features on and off and changing their magnitude in seconds.

I knew Empire Total War won't be as easy, but nothing prepared me for the pain I suffered. But I'll complain as I go.

Libertarians everywhere

First to see how modding works I wrote a very small mod - turning off taxes adds +10 happiness. Not quite literally, as it seems impossible to have anything triggered by zero taxes, so I gave all non-zero taxes extra -10 happiness, and every government type gets extra +10 happiness - net result being what I wanted, except it looks a bit silly in game.

Now why did I do so? The answer is my favourite "less micromanagement". I don't particularly having to babysit conquered provinces, chasing rebels around, and counting how many units I can move and how many I need to leave. This is simply not fun. So I decided that every province has a sizable population of armed libertarians who will gladly shoot every protester for me as long as I set taxes to zero. When taxes are not zero they blog climate change denial or something, I don't care.

This is all a stop-gap measure for the first few turns after conquest - if you ever want to get any taxes out of the province, as you usually do, you'll need to deal with taxpayers' happiness eventually. By the way zero taxes essentially means provinces loses money every turn, as it increases administration cost of every other province in your empire.

Happiness +10 is enough most of the time, but when you conquer someone's capital and have to face -30, on top of all industrialization, religion etc., you might need to deal with rebellions anyway. And as such regions usually bring a lot of money, you probably want to set higher taxes anyway.

If you want to tweak this bonus, use DBEditor for it.

No walls

And now the big mod - removing all walls. I'm not ideologically opposed to settlement fortifications - in fact my M2TW mod makes settlements more difficult to take. However:

• ETW sieges are broken due to stupid AI and stupid pathfinding
• Because they're broken, all ETW sieges follow just two boring scripts:
  
  
  • either: approach diagonally with infantry from 3 directions, bayonet charge everything;
  • or: approach diagonally with howitzers with carcass shot and a few line infantry units, once you start bombarding them AI will charge you one by one and you win without loses
  everything else fails as your units are too stupid to pathfind in more interesting strategies
• After 1710 or so nearly every settlement outside Americas has city walls
• And so after the first few years, 90% of battles is unbelievably boring

Field battles on the other hand are much more interesting. An unfortunate side effect of this mod is that while in vanilla town watch can use settlement fortifications to defend it from small enemy armies reasonably well, now it's completely useless. I can live with that.

Unfortunately, this has second order side effect of forcing you to be more aggressive. If before you'd be quite willing to have your armies further away from the front as town watch could serve as good enough first line of defense, now you need to have your armies closer to the borders, preferably on their side of it.

So how did I write this mod? It was truly painful. First, ETW has building features, and the fact that settlement fortifications building creates walls in the battle is supposedly encoded by such features. I ran into the first problem, as DBEditor is incapable of removing features - only adding or modifying them. To do such thing you need to go to PackFileManager, clone a table, make sure it's named exactly like the original one (so it will be overridden, not merged), and remove rows from that in DBEditor.

Except for some reason PackFileManager incorrectly mixes up slashes and backslashes, so I needed to fix the mod file from a hex editor.

Except it turned out in the end ETW ignores this feature, and simply hard-codes walls. So much for my effort.

It was plan B time. First I needed to remove all walls. And while db files are pain to edit it is nothing compared to the pain of editing startpos.esf which contains campaign information:

• Start campaign, look at every single settlement on a map and write down which settlement has walls (as there's no search function in EsfEditor)
• Manually find every such settlement in esf tree, and do all necessary changes (change True to False in one node, delete another node).

This is of course wholly incompatible with any other esf mod, like those which enable you to play minor factions.

After that I needed to make it impossible to build walls. I haven't figured out how to do that (I suspect if I remove walls completely the game will simply crash, and no other building in unbuildable) - so I made it require late technology of mass production, and take 999 turns to build. Sort of good enough.

And all this was possible only after a lot of effort of modding community members who created tools like DBEditor, EsfEditor, PackFileManager, documented what they could etc. This is borderline unmoddable, and I fully expect the next generation of Total War games after Napoleon (which is essentially extra scenario for ETW) to be completely impossible to mod. But hey, maybe they'll sell more DLC this way.

How to install

In case you want to install these mods, you need to:

• Download starpos.esf and replace one in C:\Program Files (x86)\Steam\steamapps\common\empire total war\data\campaigns\main (or similar), first of course making a backup copy - this will remove starting walls. (small warning - it's a big file and the server I'm hosting it on is pretty slow; other files are tiny)
• Download mod_no_walls.pack and put it in data directory together with all other packs - this will make walls unbuildable.
• Optionally download mod_tax_break.pack and put it into data directory - this will give you +10 happiness bonus for no taxes.
• Download ModManager and unpack it.
• Start ModManager, select mods you want to use, and click Launch.

Greasemonkey and jQuery easier than ever

Last year I wrote a short Greasemonkey tutorial, in which I explained how to use it with jQuery for some really simple scripts. Since then it became even easier, so here a few more useful scripts.

Since version 0.8 Greasemonkey has @require feature, in which your scripts can be made to depend on some external Javascript files - like jQuery. Unfortunately there are a few gotchas:

• jQuery 1.4 doesn't work with Greasemonkey, you need to use jQuery 1.3 (or keep pestering them until it's fixed)
• @require only works on installation time, you cannot use it with "New User Script..." feature, nor can you change @requires from the editor. If you simply use published scripts and don't write anything - you don't need to worry. If you write your own scripts you need to prepare something.user.js file somewhere, then open it from Firefox (copy&paste it's file path to Firefox URL bar, then click Install).

Show spoilers on tvtropes

Let's start with something really trivial. I don't care about spoilers. Maybe 1% of "spoilers" significantly diminish viewing pleasure, vast majority of them do not. I knew that Vader is Luke's father, that Snape killed Dumbledore, and that Titanic sunk before watching/reading, and it made it no less enjoyable.


On the other hand I'm quite annoyed that to read tvtropes I needed to constantly select "spoiler" text for everything to see it - but no more. This trivial script solves it entirely:

// ==UserScript==
// @name           Show all spoilers on tvtropes
// @namespace      http://t-a-w.blogspot.com/
// @include        http://tvtropes.org/*
// @require        http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js
// ==/UserScript==

$(".spoiler").removeClass('spoiler');

It's really simple - @name is unique name for the script, which should really be a very short description. We're not going to rely on @namespace at all, so put anything you feel like there.


@include is a pattern of URLs for which the script should be executed. @require is the jQuery library we're including. And thanks to jQuery, removing spoiler tags is really easy.

You can download this script here.

Always sort by seeders on The Pirate Bay

And extremely annoying thing about The Pirate Bay search engine is how it sorts results by its idea of "relevance" - usually giving you some dead ISO of three year old Ubuntu version when you obviously want the most recent one. And nearly always sorting by seeders is the right thing to do.

There is actually a GreaseMonkey script that claims to do exactly that, but it doesn't always work correctly, and the author decided to obfuscate the source for lulz. I'll have none of that, so I just wrote my own. Again, thanks to jQuery it's truly trivial.

// ==UserScript==
// @name           Always order by seed count
// @namespace      http://t-a-w.blogspot.com/
// @include        http://thepiratebay.org/*
// @require        http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js
// ==/UserScript==

$("input[name='orderby']").val(7);

You can download this script here.

Download Stewart and Colbert from bittorrent

Now a more complicated and perhaps more useful script. One of the recent developments on the Internet I hate the most are geographic restrictions - Americans can view whatever they like, for everyone else it's "Sorry, Videos are not currently available in your country" or other such nonsense.

Personally I don't care about licensing restrictions which led to this - I'm not going to accept development like that if I can help it. Fortunately in this case I can - all Stewart and Colbert is available on bittorrent.

The script does the following:

• Find all dates on the page. They're not kind enough to use consistent class for those, I found at least 3, and perhaps I still missed a few.
• Convert every date to from "Month DD, YYYY" to "YYYY MM DD" format, as this is what most torrents use. Half of the script is just doing that because Javascript doesn't have anything like Ruby's Date.parse.
• Append link to bittorrent site next to the date. I'm using yourbittorrent here, as ThePirateBay is down due to overload far too often.

So every time someone on Reddit posts a "omg Colbert totally pwned Glenn Beck" link, I can actually see the pwnage now thanks to this script.

// ==UserScript==
// @name           View Stewart and Colbert on bittorrent
// @namespace      http://t-a-w.blogspot.com/
// @description    No more "Sorry, Videos are not currently available in your country"
// @include        http://www.thedailyshow.com/*
// @include        http://www.colbertnation.com/*
// @require        http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js
// ==/UserScript==

var month_names = {
  'January': '01',
  'February': '02',
  'March': '03',
  'April': '04',
  'May': '05',
  'June': '06',
  'July': '07',
  'August': '08',
  'September': '09',
  'October': '10',
  'November': '11',
  'December': '12'
};

$(".date, .airDate, .clipDate").each(function() {
  var txt = $(this).text();
  var m = txt.match(/(\S+)\s*(\d+),\s*(\d{4})/);
  var date = m[3] + "+" + month_names[m[1]] + "+" + m[2];
  var url;
  if(window.location.hostname == "www.thedailyshow.com") {
    url = 'http://www.yourbittorrent.com/?q=Daily+Show+' + date;
  } else {
    url = 'http://www.yourbittorrent.com/?q=Colbert+Report+' + date;
  }
  $(this).append("<div><a href='"+ url + "'>Download on bittorrent</a></div>");
});

You can download the script here.

See how easy it all was? Now start writing your own scripts and share them.

Everybody Draws Muhammad Day and lessons in cultural relativism

May 20th is Everybody Draw Mohammed Day - a day when you too can join the defense of Free Speech by drawing a cartoon. Or if you really suck at drawing you can make a shop of Muhammad like me.

Prophet Muhammad, artist's conception
If you think I'm wrong, draw a better one

Lesson #1 - Principle of dissimilarity

Recently I have been convinced by some TTC audiobooks that Jesus might have probably existed as a historical person. You know what's the best argument for it? The criterion of dissimilarity.

It's not any pomo deconstructionism - just plain old historical analysis. The idea is that authors have an agenda, but life doesn't always agree with it. So every time the text admits to something that is clearly against the authors' agenda, it suggests it probably had some basis in reality - because they wouldn't make this up on purpose. A few examples from Jesus' life:

• Crucifixion was embarrassing kind of death reserved for slaves, rebels, and other lowest status scum. If someone was making up the story they would have Jesus die in battle, or die some other "respectable kind of death". So actual Jesus was most likely actually crucified.
• All the messy explanations how "everybody thought Jesus was born in middle-of-nowhere Nazareth but actually he was born in Bethlehem like King David" suggest that Jesus was probably born in Nazareth. If they were making this up, they'd skip the Nazareth story altogether.
• Jesus was baptized by John the Baptist - being baptized by someone was signaling submission and inferiority to that person. This story is clearly embarrassing to Bible writers, they put words like "I should be baptized by you" in John the Baptist's mouth, and the last Gospel omits it altogether.

The sheer volume of such cases suggests that Bible is an "enhanced" story of some actual life, as opposed to being completely made up. There's no question that all the supernatural bits have been inserted, and it's widely believed that the real life stories were quite significantly massaged - but it does such a bad job at covering a lot of embarrassing parts like these that the hypothesis "Bible is vaguely based on a life of a real person, and these embarrassing stories were widely known so couldn't be ignored" is much more compatible with the evidence than "Bible is completely made up" - and as their Bayesian priors are not terribly different in the first place, historicity of Jesus can be reasonably believed in. The "Bible is real" story on the other hand is both against the priors and against the evidence, so let's not even go there.

Lesson #2 - Why is Muhammad widely considered a pedophile?

The criterion of dissimilarity must be truly infuriating to the true believers - it essentially says that every time they like some part it's probably false, and every time they dislike some part it's probably true. By the way this applies to all historical texts, not just religious ones - parts of Commentarii de Bello Gallico that are overly sympathetic to Julius Caesar should be looked at with more suspicion than parts which talk about his failures, and so on.

So, what does it all have to do with Muhammad being a pedophile? Muslim texts clearly say that he was married to Aisha when she was 6, and fucked her when she was 9 year old.

Now how likely is it that this particular bit was made up? If you had a prophet who didn't fuck children, would you casually make up a story that he actually did? That'd go against all rules of proper writing. And it's not one isolated verse somewhere - as Wikipedia says "references to Aisha's age by early historians are frequent", and nobody questioned that back then.

So regardless of our believes, we can be fairly certain that

Muhammed fucked a 9 year old girl

And now some time for cultural relativism - this was considered fairly unremarkable back then. Our modern culture is obsessed about young people's sexuality and as a civilization we essentially lost the ability to propagate the species, but historically it was entirely normal for much younger people to marry, fuck, and have babies - biologically late teens are the optimal time to have children. In completely typical Ancient Rome - girls typically married in their mid teens, very soon after puberty. The idea was simple:

• Regardless of societal constraints, many people will get sexually active once they hit puberty
• Of people who get sexually active, many will get pregnant
• Nearly everywhere except for modern times, it's really difficult to either provide for your kids without a husband, or get a husband if you already have kids
• To avoid risking that, it's better to marry your daughters sooner

This was the baseline normal case for girls.

Western pedophile obsession (where all sexuality of people under 20 or so is a big taboo - this has little to do with what psychology calls "pedophilia") is a highly atypical case. 7th century Arabic children marriage, and fucking of prepubescent 9 year old girls was also rather atypical.

Not that it matters what's typical and what's not - abject poverty, illiteracy, and lack of broadband Internet are historical norm, and yet I much rather prefer atypical modern situation.

Anyway, what we see here is a conflict of values - Muhammed fucking a 9 year old was acceptable within his culture, and is not considered acceptable within ours - not even for most modern Muslim countries. Aisha most likely didn't mind getting fucked, and nobody was weirded out by this, in spite of modern fiction of "age of consent". She was most likely not scarred for life or anything like that - these laws exist primarily to make adult bigots feel good, not to "protect" children, and evidence is clear that plenty of teenagers (and an occasional pre-teen) have sex and enjoy it.

Lesson #3 - Freedom of speech

But that's not all. On top of one value dissonance about sex with pre-teens we have a second value dissonance about free speech and religious respect.

In Western culture since the Enlightenment, we have been very strongly attached to the idea that there is nothing that cannot be criticized. Yes, laws of different countries prohibit different kinds of speech (including United States, Supreme Court's favourite activity is making exceptions to the First Amendment and Elena Kagan doesn't seem any different here) - but every time such law is applied the media get freaked out and people feel highly uneasy about it. Even people who want to limit freedom of speech considerably universally consider it to be the default case - exceptions to be few and made only when "necessary".

These believes are not shared by many, apparently including modern Muslim societies. It seems that they approach Muhammed cartoons from another direction - that people's religious believes should be respected, and their holy figures shouldn't be mocked - freedom of speech being far lower in their order of priorities than that.

They don't even see the cartoons as a freedom of speech issue, just like 7th century Arabs didn't see fucking a 9 year old girl as a child abuse issue, and we don't see the cartoons as a blasphemy issue. Different cultures have different perspectives.

This basic cultural relativism does not mean that all cultures are equally wrong, or equally right. It would be intellectually dishonest to think that your culture is uniquely correct about things, and its beliefs are some sort of human universals - history shows that there are hardly any true human universals, and we might even get rid of death and taxes one day. But you are still free to follow your own culture's value system.

If you think that freedom of speech is valuable, and fucking prepubescent girls is creepy, mocking that and not giving a shit about angry Pakistanis is all fine.

In other words - enjoy the Everybody Draws Muhammad Day.

Very simple parallelization with Ruby

I hate waiting. And all too often when I tell computer to do stuff, the only reply I get is a progress bar with ETA far in the future.


Fortunately plenty of such things are highly parallelizable. For this post let's focus on a simple task of downloading every single Questionable Content strip ever released because QS has really simple URLs. It's real simple to do it serially but waiting time of about 23 minutes is killing me:

(1..1665).each{|i|
  system "wget", "-q", "http://www.questionablecontent.net/comics/#{i}.png"
}

Notice how I didn't even bother with shell, and went straight for a real programming language. Anyway, Ruby has very easy threading system - just use Thread.new{ ... } to spawn, Thread#join to join, avoid global mutable state, and you're done.

We can get it running in no time:

module Enumerable
  def in_parallel
    map{|x| Thread.new{ yield(x) } }.each{|t| t.join}
  end
end
 
(1..10).in_parallel{|i|
  system "wget", "-q", "http://www.questionablecontent.net/comics/#{i}.png"
}

The only thing that changed was replacement a one-liner middle-ware method, and replacement of Enumerable#each with Enumerable#in_parallel - so far really good.

On the other hand, we don't want to spawn 1665 threads all at once - neither our network connection nor QS's servers would appreciate that much. Let's write some code which would create 10 threads and keep them all busy instead.

Exception handling interlude

A very common problem with Ruby multi-threaded programming is that we don't want the program (or even thread) to die just because of some error - usually we want to capture, print, and otherwise ignore exceptions. Here Ruby fails hard - Exception has no method to generate user-friendly message of the kind it prints when exceptions fall out of the main loop. There are some gems that handle that, but here I'm rather unconcerned with this - and instead I'll just use a really simple catch-print-ignore method - which does not even bother printing backtrace.

def Exception.ignoring_exceptions
  begin
    yield
  rescue Exception => e
    STDERR.puts e.message
  end
end

Execute N tasks in parallel

How do we communicate with N threads? A simple and wrong way would be to create N queues, and feed them in a round-robin manner. Unfortunately unless tasks always take nearly the same amount of time (this is almost never even close to true) - this results in a lot of threads waiting aimlessly.

We need exactly one todo queue. Now it would be better if this queue had a size limit - but SizedQueue was giving me troubles, so I did it with the plainest possible Queue.

The main thread is really simple:

require 'thread'

module Enumerable
  def in_parallel_n(n)
    todo = Queue.new # Create queue
    ts = (1..n).map{ # Start threads
      Thread.new{
        # Do stuff in threads
      }
    }
    each{|x| todo << x} # Push things into queue 
    ts.each{|t| t.join} # Wait until threads finish
  end
end

Now the main question is how do we tell threads there are more tasks. Unfortunately queues have only two states:

• Queue not empty - dequeue one element, do work, go back for more
• Queue empty - sleep and wait for the queue

While we need three:

• Queue not empty - dequeue one element, do work, go back for more
• Queue empty, producer not finished - sleep and wait for the queue
• Queue empty, producer finished - finish thread

Now I could go for a more complicated signaling solution - but that would either introduce global mutable state (hell no!), or require too much complexity - so I simply went for the simplest possible "flood queue with nils once you're done" way.

And because I don't want the program to break when object we're iterating over contains legitimate nils I'm passing whatever we get in a one-element array (Ruby probably has equivalent of Ocaml's 'a option / Haskells's Maybe monad etc. - but for this the simplest solution will work just fine).

require 'thread'

module Enumerable
  def in_parallel_n(n)
    todo = Queue.new
    ts = (1..n).map{
      Thread.new{
        while x = todo.deq
          Exception.ignoring_exceptions{ yield(x[0]) } 
        end
      }
    }
    each{|x| todo << [x]}
    n.times{ todo << nil }
    ts.each{|t| t.join}
  end
end

And now we can happily get the comics. It took 4 minutes 11 seconds instead of 23 minutes - 8x speedup!

(1..1665).in_parallel_n(10){|i|
  system "wget", "-q", "http://www.questionablecontent.net/comics/#{i}.png"
}

What about other comics?

Now unfortunately far too many comics don't have pretty URLs like that. You can always (well, nearly always) resolve to wget --mirror or find a greasemonkey script which makes the comics' website bearable. Fortunately quite often we can handle even somewhat more complicated URL schemes.

First, for comicses like sinfest, which use dates in URLs, it's only marginally more difficult:

require 'date'
(Date.parse('2000-01-17')..Date.parse('2010-05-17')).in_parallel_n(10){|d|
  system "wget", "-q", d.strftime("http://www.sinfest.net/comikaze/comics/%Y-%m-%d.gif")
}

3536 files, 7 minutes 52 second.

Because Ruby dates are first class objects and Range class works very sanely with them - it took nearly no effort to code that, other than trying to remember wtf were all those strftime %-codes (or you could use Date#year etc. and plain string interpolation - it would probably be better for your sanity actually).

And wget will quietly ignore proper 404s, so you don't need to wonder if they're updating on weekends too, or only on weekdays, or what.


For other comicses like xkcd you might need some hpricoting (and put alt text in metadata, then teach image viewer to read them? Or use imagemagick to put them into the image? choices, oh all the choices...). Or you could use This awesome Webcomics Reader Greasemonkey script. Unfortunately it only knows about a handful of the most popular ones.

Now this technique is applicable for a lot of uses other than downloading funnies - like handling S3 backups, encoding large number of MP3s and so on. Everything where it makes sense to run multiple threads in parallel, but not too insanely many will be really easy to do now thanks to Enumerable#in_parallel_n or similar code.

It works with both 1.8, and 1.9 equally well, in spite of their completely different threading systems.

Enjoy.

Steam is ripping you off

Online distribution of content should be cheap and easy. Compared to the complexities of making and moving around DVDs and such, moving bits is pretty much free. So it's a no-brained that games on Steam should be cheaper than the same games on DVDs, right? Of course that's not how it works.

These are top 10 Steam's best-selling games - this list is rather generous for Steam, as people would prefer to buy games on Steam which were better deal on Steam than on DVD, and would prefer to buy games on DVD which were better deal on DVD - so it's reasonable to expect comparing other games Steam would come out even worse.

As for DVD alternative, I wasn't looking too hard - just Amazon. I'm sure there are ways to get the games even more cheaply, but let's stick with the default online shop. Here are the prices, all Amazon's are with free shipping:

• Call of Duty: Modern Warfare 2 - Amazon £29.65; Steam £39.99 - Steam costs 35% more
• Battlefield: Bad Company 2 - Amazon £24.99; Steam £29.99 - Steam costs 20% more
• Left 4 Dead 2 - Amazon £19.98; Steam £19.99 - essentially equal
• Counter-Strike: Cource - Amazon £14.99; Steam £13.99 - Steam costs 7% less
• Grand Theft Auto: Episodes from Liberty City - Amazon £14.99; Steam £19.99 - Steam costs 33% more
• Civilization V (pre-order) - Amazon £30.99; Steam £29.99 - Steam costs 3% less
• Mount & Blade: Warband - Amazon £16.97; Steam £24.99 - Steam costs 47% more
• Borderlands - Amazon £11.78; Steam £19.99 - Steam costs 70% more
• Just Cause 2 - Amazon £23.92; Steam £29.99 - Steam costs 25% more

In other words - median cost on Steam is about 25% higher than on Amazon. That's even though physical DVD costs quite some money to make, packaging, ship etc., while for Steam it's just pure profit.

And please don't even mention "Recommended Retail Price" - if you fall for this kind of blatant anchoring effect, there's no hope for you.

So to summarize: Steam games cost a lot more, and to make matters worse have no resell value, they're simply ripping you off.

PS. Amazon prices are for NEW games. These are not used titles. Anyone claiming otherwise has no idea what they're talking about.

PS2. Prices I gave are for UK (notice the £s). acct_rdt made a similar list for US prices, which shows exactly the same thing - US Amazon is cheaper than US Steam too, even though the difference is not as drastic (median 7%)

• Call of Duty: Modern Warfare 2 - $37.80 new on Amazon, $59.99 on Steam
• Bad Company 2 - $43.83 Amazon, $49.99 Steam
• Left 4 Dead 2 - $29.99 (both)
• Counter-Strike: Source - $19.99 (both)
• Episodes from Liberty City - $27.99 Amazon, $29.99 Steam
• Civilization V - $49.99 (both)
• Mount & Blade: Warband - $24.53 Amazon, $29.99 Steam
• Borderlands - $27.99 Amazon, $29.99 Steam
• Just Cause 2 - $39.99 Amazon, $49.99 Steam

Greeks can pay, Greeks will pay

Most normal blogs' favourite countries to rant about is US, or Israel, or China, or something like that. Not on this blog - my number one target, at least for the last month - is Greece. And I'm really shocked that some of the otherwise sane people I know buy into the whole "can't pay, won't pay" nonsense, so I wrote this short post explaining how it all really works.

Basics of money lending

Let's get back to basics. Why would anybody lend anybody else their money? The most common reason is they want to get even more money back. Now this is in no way the only reason - people routinely lend money to their family members and friends, IMF lends money to countries facing collapse, USA and Soviet Union lent money to support whichever dictator seemed more aligned with their interest, Europe lends money to shit-poor countries out of pity, AIG lent itself taxpayers' money by bribes and threats, and CIA lent money to Osama bin Laden because they're idiots - but while these are all very important cases, most money in the world is lent for profit motive.

This profit-motivated lending is not only largest in volume - it's also most reliable. Political will can evaporate overnight - ask South Vietnam or Cuba if you don't believe me - but speculators can be reasonably counted on to keep lending as long as it makes business sense. Even in the middle of the latest recession and the Great Depression solvent borrowers had no trouble getting loans - the problem was sudden decrease in solvency, not disappearance of the profit motive.


Political and charitable lending is highly complicated and varied form case to case, but for-profit lending is quite straightforward from game theoretic point of view:

• Lender decides to give borrower money or not
• Then borrower decides to repay or not
• Because borrower would much rather get money, and not repay it - no lender would be willing to lend anything without some pretty hard assurance of seeing their money back
• On the other hand, as repayment might be impossible due to objective circumstances, borrowers would be unwilling to provide too much assurance
• Level of assurance is based on balance of these two factors.

And what forms of assurance are available? For private individuals there's debtors' prison - a barbaric institution which is still used in child support and tax cases - not coincidentally both being the kind of "debt" which is incurred unwillingly, where the "borrower" has no leverage whatsoever.

Another assurance is forceful confiscation of property, and various kinds of physical abuse - these were routinely used as international debt collection in 19th century, see Egypt, and Haiti for examples - but these rarely happen these days.

Yes, lenders could try to sue Greece or another unwilling country in foreign or domestic courts, but it would be mostly a PR stunt, and they wouldn't even get enough to cover their legal costs. If a country doesn't want to pay, lenders cannot do shit.

Hamster showing what debtors' prison looks like

The final assurance

So what keeps borrowers repaying and lenders borrowing? It's lenders not being stupid. They know very well that if someone didn't honour their debts in the past, they're unlikely to do so in the future - so if you tell the banks to go fuck themselves, you saved yourself a huge pile of money, but you have no chance of getting any money from them in the future. 

When you think about it - it's a pretty weak assurance - it only works when the country is moderately-screwed-up:

• If country's economic situation is truly screwed up - they won't be able to repay - so banks lose
• If country's economic situation is bad but not horrible (like most countries now) - they cannot afford losing their credit lines - so they keep repaying - and banks win
• If country's economic situation is some awesome they don't care about future loans - they can tell banks to go fuck themselves - and banks lose too!

Fortunately for banks, the last case is very difficult to achieve - not only you need zero deficit (before interest) now - something already very hard - you need to be certain of being able to keep this zero deficit essentially indefinitely. Recessions, commodity price fluctuations, wars, terrorist attacks, aging population, floods, tsunamis, volcanoes, epidemics, electing Conservatives, and plenty of other natural disasters can screw your budget essentially overnight - all sending you crawling back to the banks begging for money. And bankers will remember what happened to their old loans.

I cannot think of any country in the world which is financially healthy enough to pull that off. What makes it even less likely is that countries with most balanced budgets tend to have lowest debt levels and borrowing costs as a rule (debts are results of past performance, and past performance is the best predictor of future performance there is) - so for them repaying debts is nearly painless, and provides good insurance in case something screws their economies in the future.

Essentially the only case in which a country can tell the bankers to fuck off, and has debts high enough to make it worthwhile - is one which used to be horribly mismanaged for many decades, and then turned into one of the healthiest economies in the world essentially overnight.

Does it sound like Greece? If you believe so, I have some sovereign debt credit default swaps that you might be interested in.

Greeks simply cannot afford not paying. They can negotiate better terms of repayment, and get some help from EU and IMF, but essentially they will have to get their country in order - cut their bloated civil service and bloated military, cut - and repay their debts, because they simply cannot afford losing their credit lines. And so they will pay.

Audible review

Before the posting frequency either reaches singularity or crashes like Nick Clegg's hopes, here's a brief review of Audible.

I have listened to ridiculous amounts of audiobooks (a term I'm going to use for essentially any spoken word - whether an original material or a spoken version of a paper book) for many years now - a very long time before they became anything even vaguely approaching mainstream. And obviously I got most of them from Linux ISO download sites, not counting a very large number which were officially free as podcasts and such.

Then, my cat threw my iPod into a bathtub full of water, I bought a new and much better MP3 player, and with it came free subscription to Audible, which I actually extended for a few more paid months more out of curiosity how this world of paid audiobooks operates than anything else.

DRM

First - DRM is as bad as you'd expect. Devices cannot be activate on anything except Windows (the stories about iTunes being able to activate MP3 players are a lie). Then after I formatted the MP3 player, it lost activation and it was impossible to get it back. There was no error message, nothing. Activated successfully, still doesn't work. I had to email Audible for them to fix it... it was really one big pile of ridiculousness, all horror stories about DRM basically came true.

Does it at least DRM the books? Not at all. First, they left CD burning as an option, so anyone who is desperate enough can get virtual CDs and then rip them. And you can download Audible-ripping software which will happily hack into Audible drivers on any activated PC, and convert all your DRMed files into MP3s, FLACs, or whatever. Essentially they fuck over paid customers, and the pirates will get their warez anyway.

This is all good enough reason for not using Audible even if they didn't suck in other ways, but suck they do...

Speech synthesis

But before I get to the main dish of Audible suckiness, a brief interlude for state of speech synthesis. 60 years ago, when computing started, people believed that in no time computers will be as smart as humans. You know - 10 years, 20 tops. Which as it turned out in computing world means essentially "never".

I'm just amazed at how filled with wrong are predictions of people who expect rapid arrival of AI - like most users of the ironically named LessWrong community blog. How the fuck are we going to see AI go "foom" if during the last 60 years there has been essentially zero progress at:

• Speech recognition
• Handwriting (and most importantly hand-written math/diagrams/mindmaps) recognition
• Speech synthesis

Yes, they expect a big foom - but better be worried in case this foomy future Kindle 2020 or whatnot will turn out to be unfriendly - and let's say will read the latest Twilight sequel in a very ironic voice...

The truth is - automated speech synthesis of anywhere near human quality is not coming anytime soon. It just isn't. Amazon knows it as well, while building gimmicky crappy speech synthesis into Kindle, they also purchased Audible for $300M whose business model totally relies on speech synthesis being crappy for decades to come! How is this $300M for a prediction market? Are you shorting Amazon stocks already? I didn't think so.

And while I'm at it, remember how one of my favourite crackpots - Raymond Kurzweil boldly predicted that by 2009 "[people] interact with their computers primarily by voice and by pointing with a device that looks like a pencil. Keyboards still exist but most textual language is created by speaking". Obviously this crackpottery never came true, I'll keep my Microsoft Natural Ergonomic Keyboard 4000 thank you very much. I doubt even Apple would be stupid enough to go into the whole voice thing - but then with iPad they sort of have proven that they can sell absolutely anything, and hipsters will buy it...

Why Audible sucks

And now the main part - why Audible sucks. To explain this think for a moment what an Internet shop is. Yes, it needs to have some stuff to sell, and some way to deliver it, and some way to accept payment, but primarily:

An Internet shop is primarily a search engine

That's right! Everything else is just an insignificant attachment to the main functionality of letting people find the stuff they want to buy. Go to Amazon - the archetypal Internet shop. What's there - a huge search engine. With user reviews, recommendations based on your previous purchases, and purchases of other users, and a huge wealth of information which helps you find things you might want to buy. Why nobody has copied Amazon yet? Well, there are some economies of scales involved, but dis-economies of scale exist as well. This is not the answer.

The primary reason is that nobody else has this massive database of data which helps people find what they want. Even if you had all the products Amazon has, at identical prices and terms of delivery, without user reviews and contextual recommendations your shop would be hopeless. And without even as much as Amazon search system? Forget about it.


By the way, I'm probably biased as most of my commercial career was in one way or another related to building custom search engines, just saying for the sake of full disclosure...

Anyway, Audible fails so hard at search it's not even funny. You go to category - and it's sorted by release date. How is this sane? You can sort by customer review - but most of their books have at most one review and there's no Bayesian filtering applied, so everything on the top will be audiobooks with a single 5-star review. Yes, it's all great that ONE person liked it, but it has extremely low predictive power on how good it actually is. About the only useful way to sort anything is by top selling.

So what do we see - title, author, price, star rating, a picture (wtf? these are audiobooks, these pictures are essentially just some random stock photos of no relationship with the product) - and that's it. And max displayed at once is 30 - but don't worry, it will change back to the default useless value of 10 results as soon as you change anything.

This is not Google! You don't want to go from search term out of their site in less than a second - you want to figure out what's actually any good! The logic of having smaller number of results so they're faster doesn't apply at all!

So how do you figure out if the book is any good at all? Well, there are very rarely any customer reviews (just a few star ratings is typical) and the ones there are are typically as useless as "Took me a little while to get into this book, but when I did I absolutely loved it! Have just downloaded the second book and can't wait to begin!". How does this provide any information whatsoever?

There is usually a short publisher-provided blurb telling what the book is about - and you can probably guess what I think about such publishers' honesty.

The only thing that's of any use at all - and without which the entire Audible website would be utterly useless - is the audiobook sample. Unfortunately it doesn't work terribly well. One thing you can figure out is that quality of recording tends to be top notch - unlike some other sources which compress audiobooks as if people were still downloading them on dial-up modems, audible handles quality properly. And you can also find out if you like the narrator or not. Which might be very highly relevant for their "erotic" section at least... (seriously, go there for some major lulz and/or facepalming..., but then maybe I'm not the target audience...).

What you'll rarely find out is if the book is actually any good, which is what we want to establish. Random pages from the middle of the book rarely tell you that. Even worse, I've seen quite a few books for which the random sample was actually legal blurb + acknowledgments from the first few pages... that definitely is some fascinating book, right !?


Unfortunately that's all your search options. Audible won't search inside books to figure out what they're about (speech recognition obviously being crap, but on the other hand - they have access to written original most of the time, so that's a weak excuse), and there's no other information on their website.

If you want to find out a good audiobook, you need to check every audiobook's written equivalent's page on Amazon - for reviews there. Somebody should write a Greasemonkey script for that... (this problem also affects UK Amazon to some extent - often US version has 10x as many reviews, but there's no link and the only way to get to those reviews is Googling... it gets me raging every time...)

How difficult would it be to link it up with the rest of Amazon? Seriously now.


There's also another issue - the choice on Audible is simply less than on "Linux ISOs download sites" - so when you want an audiobook it's very likely you won't find it there, even if it exists. The opposite - Audible having something which pirates don't also happens, but usually for more obscure titles.

This is actually an opportunity for them - sure, many people will torrent Dan Brown, but there's long tail to explore - except it is exactly this long tail which is it direst need of good search functionality. And so Audible fails.

So maybe it's a good idea to short them after all...

Progress bar for Unix pipes

This is getting quite ridiculous, as it's the third post I'm writing in just one day - that's more than I typically wrote monthly over the last two years. This is what you get for finally getting organized - a big pile of 90%-finished stuff which while useless on their own can be very quickly be turned into something of high utility. OmniFocus and GTD are truly awesome. (unfortunately this means there are now two apps for Mac I care about - TextMate and OmniFocus - so switching away from Mac will be even hander)

The script I want to show you today solves one of the most severe problems with Unix pipes - lack of progress indicators. Normally you'd start a pipe, and until it actually finished what it was doing, you'd have no way of finding out if it's 1% done or 99% done, and how fast it is progressing.

There are some nasty hacks. Many times I used strace or looked inside /proc to figure out how much progress has been done - but these are painful waste of effort for something that should be builtin. Feedback is the basic principle of good UI design.

Anyway, here's the final solution to progress bar question:

#!/usr/bin/env ruby

STDERR.sync = true

$bytes = true
$max = nil
$count = 0

until ARGV.empty?
  case (arg = ARGV.shift)
  when '-l'
    $bytes = false
  when '-b'
    $bytes = true
  when /\A(\d+)([kmg]?)\Z/
    units = {'k'=>2**10, 'm'=>2**20, 'g'=>2**10, ''=>1}
    $max = $1.to_i * units[$2]
  else
    raise "Unrecognized argument: `#{arg}'"
  end
end

$max = STDIN.stat.size if $bytes and STDIN.stat.file? and $max.nil?

Thread.new{
  last_count = nil
  while true
    if $count != last_count
      if $max
        STDERR.print "\r#{$count}/#{$max} [#{$count*100/$max}%]"
      else
        STDERR.print "\r#{$count}"
      end
      last_count = $count
    end
    sleep 1
  end  
}

begin
  while data = ($bytes ? STDIN.read(2**12) : STDIN.gets)
    STDOUT.print(data)
    $count += $bytes ? data.length : 1
  end
  STDERR.print "\n"
rescue Errno::EPIPE
end

Explanation time:

• You can use this script at any point of the pipeline. foo | bar | progress | blah.
• The script will take advantage of the fact that its STDERR is still linked with your terminal, and output progress information there. It will then clear and overwrite the same line every second with new information.
• There is no support for multiple progressbars in one pipeline - his is not terribly difficult to do (one would be master, others would send info to it via socket based on tty's inode), but I never found a good use case for it, so I never bothered implementing it.
• progress script works in two modes - by default it counts bytes (-b), but it can count lines (-l) as well.
• You can specify what counts as 100% if you want percentage information - with progress -l 1234, progress -b 700m etc. If you specify wrong size of course you get garbage.
• If you operate in the default byte mode and input is a file - the script will figure out file size automatically. This doesn't happen in line mode, as it would require a potentially expensive wc -l - it's easy to do it manually if you want.

Here's a "screenshot":

$ ./progress < kubuntu-10.04-beta2-desktop-amd64.iso | openssl md5
287244288/708704256 [40%]

Enjoy.

Brief history of human diet

Previously in series: The myth of 5 a day - people never ate that much fruit and veg, Dietary consequences.

I've been listening to an audiobook about history of potato cultivation, which I purchased legitimately, and then proceeded to strip DRM of it in spite of Audible insisting in its FAQ that it's impossible. Are you getting WTF overload already?

Anyway... for people who are puzzled by the recent obesity/diabetes/metabolic syndrome clusterfuck, here's a quick reminder of history of human diet:

• Humans did a lot of hunting and gathering with diet composition being highly varied depending on local circumstances. Normally hunting would bring most high quality protein, while gathering would bring most calories, and diversity of food sources would ensure good dietary balance and protect against bad seasons.
• Then humans switched to agriculture. This happened at least 7 times, independently, almost at once, after millions of years during which it could have happened and didn't. Amount of evidence we have is just ridiculously high compared to all other human evolutionary transitions (like language, behavioral modernity, industrial revolution, and scientific revolution - all happening just once) that it's a shame that we don't have any convincing theories explaining why it happened when and where it happened.
• Invariably, this switch resulted in severe health problems. Calories were more plentiful, but reliance on a small number of food sources virtually ensured micronutrient deficiencies, and regular famines due to bad weather. Farmers were shorter, died earlier, and generally suffered. This is well documented everywhere transition to agriculture happened.
• Gradually, people figured out how to diversify their food sources, and in time farmers got as healthy as hunter-gatherers before them. There were more kinds of food, better agricultural practices, better preservation methods, better cooking methods, varieties of food grown got changed, and to a small extent even human biochemistry changed to adapt to agriculture.
• There was still one huge problem - people were spending insane amount of time on food preparation. According to some statistics, even as recently as a century ago, average wife spent 45 hours a week on activities related to food preparation. Stay-at-home mums were not a cultural quirk - they were a necessity (unless you could afford servants).
• A new revolution in food production happened in the last century. There was quantitative increase in productivity, but most interesting part of it was switch from highly home-labour-intensive food which required rigid gender role separation to semi-prepared food which freed women into general labour force.

Yes, it was not feminist nonsense which brought equality, or birth control, or anything like that - it was primarily food processing industry. You might protest that you don't eat that much processed food - but you're fooling yourself. How many of you have milked your own cows, churned your own butter, baked your own bread (milling flour has been industrialized ages ago - but at first it was also done at home), made your own pasta etc.? We have gone so far up the food processing ladder than we forgot how much processing goes into all those seemingly simple foods that form bulk of our diets. Actually, I do bake my own bread sometimes, but with a bread machine, so it hardly counts, and mostly because British bread is atrocious spongy shit.


Unfortunately food is a complex matter, and this transition went not without causing some serious problems, just like the original switch to agriculture. If you think about it - these problems aren't terribly severe. Yes - untreated obesity is live-threatening, and we haven't found good cure for it, but we have been really good at treating the symptoms. There are drugs for diabetes and hypertension, cars have replaced the need to even as much as walk longer distances, and loss of physical attractiveness can be largely covered by learning the art of myspace shots. People still live longer each year, and don't seem that terribly distressed.


Moreover, it seems that obesity rates have plateaued in States in the last decade - at a ridiculously high rate, but it's not unlikely that the worse is already over, and they'll only go down from there.

If the rates start indeed falling, then perhaps the transition to the new kind of food will have went surprisingly smoothly, compared to all such past transitions.

Case study - USA

Let's take a look at how eating patterns changed in one of the worst affected countries - USA. It is so commonly repeated that people eat more sugar, fat and animal products that it gets on my nerves. Here's the definite table of all major food groups (at least 1% share at either date), based on FAO data, as percentage of caloric intake, ordered by relative change. I'd love to have data for 1900, as 1961 was midway the shift from unprocessed to processed food, but we have to do with the data we have, instead of despairing for the data we wished to have.

Product	1961	2005
Rice	0.93%	2.46%
Poultry Meat	2.23%	5.48%
Cheese	1.67%	3.94%
Vegetable Oils	9.58%	18.58%
Groundnuts	1.1%	1.55%
Maize	2.05%	2.52%
Vegetal Products	64.93%	72.77%
Fruits - Excluding Wine	2.77%	3.03%
Beer	2.36%	2.57%
Meat	11.63%	12.0%
Alcoholic Beverages	3.77%	3.84%
Cereals - Excluding Beer	21.76%	21.44%
Sugar & Sweeteners	17.89%	16.92%
Pulses	1.25%	1.14%
Potatoes	2.69%	2.43%
Wheat	17.5%	15.5%
Vegetables	2.22%	1.93%
Starchy Roots	3.14%	2.57%
Animal Products	35.07%	27.23%
Other Alcoholic Beverages	1.19%	0.92%
Pigmeat	4.41%	3.36%
Milk - Excluding Butter	13.29%	9.9%
Beans	1.13%	0.79%
Bovine Meat	4.36%	3.02%
Eggs	2.33%	1.45%
Butter, Ghee	2.26%	1.05%
Animal Fats	6.92%	2.91%

That is:

• Americans eat more vegetal, and less animal products - and within this category there's shift towards leaner meats
• Americans eat about as much sugar & HFCS
• Americans eat absolutely insane amounts of vegetable oil

And these are the biggest losers. I will slap around with a large healthy trout everyone who dares to suggest that HFCS is responsible for obesity. HFCS is damn sugar - statistics say it's vegetable oil you eat too much of.

"HFCS causes obesity" is the new "vaccines cause autism"

Case study - Germany

For a second case study, I took a less drastically affected country - Germany. Germany is taking the transition somewhat better than States, and your guess is as good as mine if this is a genuine success, or the worst is simply still ahead of them.

I wished to simply have EU15 average or something like that, but FAO lacks good aggregate data, so here's the next best thing.

Product	1961	2005
Maize	0.4%	2.44%
Rice	0.53%	1.34%
Poultry Meat	0.66%	1.53%
Wine	0.61%	1.32%
Nuts	0.55%	1.1%
Cheese	1.93%	3.76%
Other Alcoholic Beverages	0.81%	1.39%
Vegetables	1.08%	1.8%
Fish, Seafood	0.77%	1.17%
Vegetable Oils	8.81%	11.4%
Wheat	14.5%	18.41%
Sugar & Sweeteners	11.71%	14.22%
Alcoholic Beverages	5.71%	6.63%
Pigmeat	6.25%	6.96%
Milk - Excluding Butter	8.09%	8.89%
Meat	9.15%	9.79%
Vegetal Products	67.45%	69.96%
Cereals - Excluding Beer	24.94%	25.47%
Animal Products	32.55%	30.04%
Beer	4.3%	3.91%
Fruits - Excluding Wine	3.75%	3.28%
Stimulants	1.07%	0.91%
Eggs	1.7%	1.31%
Animal Fats	12.43%	8.76%
Butter, Ghee	6.56%	3.6%
Bovine Meat	1.99%	1.02%
Potatoes	8.81%	4.06%
Rye	8.2%	2.39%

Interesting points:

• Changes are much less drastic than in States.
• There's big shift away from local rye and potatoes to more international wheat, maize, and rice.
• Just like in States, there's shift from butter to vegetable oil, and from fatter to leaner meat - but it's less significant.
• Other than that, German food preferences are remarkably stable over such a long period of time, and such drastic changes in actual agricultural practice.

Case study - Japan

And finally the country which takes transition remarkably well - they are essentially free of Western metabolic syndrome troubles. You'd think they sticked to their traditional food? No such luck.

Product	1961	2005
Maize	0.16%	2.74%
Poultry Meat	0.18%	2.03%
Pigmeat	0.36%	3.37%
Meat	1.07%	6.38%
Vegetable Oils	3.26%	12.21%
Milk - Excluding Butter	1.42%	3.78%
Animal Products	9.64%	20.58%
Eggs	1.45%	2.75%
Beer	0.74%	1.3%
Other Alcoholic Beverages	1.51%	2.54%
Demersal Fish	1.19%	1.83%
Fruits - Excluding Wine	1.36%	2.0%
Sugar & Sweeteners	7.01%	10.01%
Wheat	9.73%	13.32%
Animal Fats	1.05%	1.38%
Fish, Seafood	4.52%	5.93%
Alcoholic Beverages	3.56%	4.65%
Vegetables	2.4%	2.82%
Pelagic Fish	2.5%	2.21%
Vegetal Products	90.36%	79.42%
Soyabeans	4.29%	3.52%
Potatoes	2.09%	1.5%
Cereals - Excluding Beer	60.34%	38.68%
Fermented Beverages	1.3%	0.68%
Rice	46.6%	22.09%
Beans	1.24%	0.54%
Starchy Roots	5.87%	2.45%
Pulses	1.44%	0.6%
Sweet Potatoes	3.31%	0.64%
Barley	3.6%	0.17%

Or:

• Changes are even more drastic than in other case studies.
• There has been huge reduction in consumption of rice, barley, sweet potatoes.
• There has been huge increase in consumption of animal products, and vegetable oil, and big increase in consumption of sugar.
• Japanese drink far more milk than they used to - and fuck lactose intolerance.

So to sum it up... I makes little sense. There's no going back. Stay-at-home mums are as much a thing of the past as bands of hunters chasing mammoths across tundra. You have to switch. The question remains - what to switch to.


The optimistic answer to this is simply follow the crowd, and hope they'll eventually figure out what works. Even if it ends up as badly as the idea of replacing animal fats with partially hydrogenated vegetable oils and simple sugars (yes, these were the original "healthy foods") - someone will come with a pretty good drugs for diabetes by the time you need them. Maybe we'll even stem-cell diabetes away entirely?

The pessimistic answer is to just copy the Japanese.

Enjoy your whale sushi.

The unified theory of online dating and porn

Most information in this post is based on absolutely no citable sources - it's either private original research or outright ass-pull, and I won't even tell you which is which to protect anonymity of certain sources. Wikipedia protesting is futile. Now the subject of the post is:

There is huge gender asymmetry in online dating behaviour

In case you haven't tried online dating, and/or haven't set up large number of research honey pots it looks like this:

• Everyone writes profiles with basic data, a few pictures, and a few fairly generic paragraphs about themselves.
• Men message every woman with whom they see even a remote possibility of attraction developing - which is to say every woman with average-or-better picture (this implies that men who type faster get laid more - as if there weren't enough reasons to learn to type fast already). Unless there's something immediately disqualifying in profile like text-speak, in which case even pretty pictures won't help.
• Women get swamped by these messages. They are invariably rather generic like they profiles, and few get a reply. Fortunately the criteria seems to include proper spelling, for another easy win.
• Out of this maybe some conversation develops, and maybe some actual dating happens.

So what causes this asymmetry in messaging behaviour? First, we can reject a few ideas outright:

• While this looks like a self-reinforcing equilibrium (men who don't spam don't get any replies, women don't have any reason to spam if they're getting spammed anyway) - this is not one of two possible self-reinforcing equilibria - the same result is reported on almost all dating sites.
• Alexa insists that ratio of men to women on most dating sites in quite close to 50:50, and so do all dating sites. So it's not one gender being more numerous.
• Another obvious theory would different age-dependent attractiveness curves for men and women - nearly every man, regardless of age, is attracted to 18-24 year old girls. This theory is easy to test, as attractiveness curves cross, and old women are far less attractive than old men - so the theory predicts that for 40+ demographics women would spam men with generic messages. My honeypots strongly disagree with this. And neither there is much spam by women with inherently low chances like those with children, no Photoshop skills, or bad spelling.

No, this is something gender-specific, and independent of attractiveness. Now the last wrong theory is that men simply care more and women are not that much into all the relationships and/or sex thing. And this would be completely wrong - just look at evidence of what kind of sex/relationship fiction men and women consume, and how much.

Men mostly consume visual pornography. Videos and photos of attractive women, some of it is outright "porn" of them having sex, but very large proportion of such material is simply attractive women posing, often mostly or even completely clothed. In other words - a suggestive picture (or even a myspace shot) of a non-nude woman fits very well within male porn consumption patterns.

And what kind of porn women consume? Yes, some of them watch "porn" porn, there's even a tiny category of porn specifically made for women - but this is a tiny niche. Most women read their porn. Romance novels, Twilight, slash fanfiction - these are all mainstream women's pornography.

And here we hit a major problem - because while content of women's dating profiles has what men need to feel a tiny initial spark of attraction - suggestive pictures - contents of men's profiles have nothing. It's almost as if it was exclusively men who designed dating sites, and these were the men with least clue on what women wanted (geek jokes go!).

I don't know how to turn this insight into dating site design advice - it is much easier to teach people how to make myspace shots than to teach them to write decent slash. And even if everyone was a good writer, it takes minutes to read through large blocks of text as opposed to seconds to look at a few pictures and decide, so it might not work within typical dating site flow.

So perhaps this theory is useless. Or perhaps whoever figures out how to exploit it makes the new Facebook.

And here's some misandrous, misogynist, and generally spiteful advice:

• Men - learn to type. If you cannot fix the spam problem, you can at least outspam others. Have some quality copypasta ready. If you can program, Greasemonkey away better dating flow. Outright spambots might be too easily detected, but feel free to take the risk.
• Women - learn to Photoshop, and Photoshop your pictures mercilessly. It really doesn't matter that you won't look anything like your picture. Also keep your profile brief and generic - if pictures are good profile serve mostly negative selection function. Feel free to lie outright.
• Everyone - all browsers have built-in spell-checker, use it.

The real cost of oversized military

In one of my previous posts I pointed to the elephant in the room - Greek economic problems are largely due to its oversized military - this confused people a lot.


Let's do some calculation, pulling all assumptions out of Google. These numbers are actually very noisy year to year, but let's just take some long term average.

• Annual economic per capita growth 2.1% - this is world average since 1950
• Annual population growth - 0% - most developed countries we speak about here have little in terms of population growth
• Annual economic growth - 2.1% (by two above) - this number might seem low, but that's the facts, and there's not much we can do about it
• Starting debt level - 50% GDP - fairly typical for a developed country
• Interest on debt is 3% above inflation. This I base on historical rates of series I federal bonds before the recession, and adjust it a bit because most countries are less safe than States, so would need to pay somewhat more.
• Baseline size of military is 1.7% GDP - EU average
• Budget has 1% surplus before paying interest.
• This results is interest payment of 3% (interest on debt) * 50% (debt to GDP) = 1.5% GDP.
• This leads to deficit of 0.5% GDP.
• Because economy keeps growing - 0.505 / 1.021 = 0.4946 - so debt level will very slowly fall as percentage of GDP even as total debt keeps rising.
• Over 50 years, debt will fall to just 16.3%

This is all very reasonable. Now let's assume this country decided to have Greek style ridiculously oversized military - 4.3% of GDP. This is extra 2.6% spending. What happens in 50 years? Their debt grows to  175.5%.

Assuming anybody was still stupid enough to lend them money at the same rates, oversized military costs them 2.6% GDP (original spending) + 4.78% due to excess interest relative to baseline scenario. Imperialism costs far more than it seems to - and it has been a very long time since anyone made big money on empire building - usually the only benefactors are bankers and arm manufacturers in countries not engaged in hostilities.

A few inb4s:

• This reasoning applies only to excess military spending.
• Normal civilian government spending is typically useful, and if government slashed spending on healthcare, education, pensions, disability insurance etc., people would simply buy more or less as much of such services on private market.
• Even worse - if government slashed spending on research, basic infrastructure and so on, where benefits are diffuse, it would be very likely that nobody would pick up the tab, and it would hurt the economy significantly.
• Yes, there is some outright waste in civilian government spending, but similar waste exists in private sector as well.
• Baseline scenario already includes EU average military spending - and even if you argue that optimal level of military spending is higher than 0% (something I won't be arguing one way or the other now) - I cannot imagine in what kind of bizarro world EU average is far below the point of rapidly diminishing returns.

I'm not done yet. Let's assume interest rates are not constant, but vary depending on country's prospects. So at 0% debt you only need to pay 1%, then it grows linearly to 3% at 50% (like in scenario), and so on. There is some evidence this growth might be ever faster than linear, but let's stick to that.

• In 50 years of baseline scenario debt diminishes to 6.35% and nobody cares any more.
• In imperialistic scenario, debt after 10 years is only 73.43% (as opposed to 70.89% with constant interest rates) - it doesn't make a big difference yet.
• In 20 years debt reaches 110.62% as opposed to 93.7% - it start to get ugly.
• In 30 years debt increases to 189.96%, as opposed to 118.61%, but the glorious Greek empire keeps borowing.
• In 40 years debt reaches 531.73%, as opposed to 145.79%, and the interest rates are completely ridiculous over 22% per year.
• When the imperial scenario ends in 50 years, debt stands at 738747192.9% GDP, interest rates are almost 5% a day, and everybody is looking for the new suckers into this Ponzi scheme.

Of course Greece had to either collapse or change its ways long before that happened.

Want democracy? Get a figurehead king

Followup to: How much democracy in your democracy?.

I find the very concept of monarchy obscene - its premise is that some woman has a magical vagina, and whatever first comes out of said vagina is blessed with magic powers that somehow make it most well suited to rule a country. Alternatively, some man has a magical penis - but monarchies are older than DNA testing so while the theory often emphasized penises, the practice was mostly vaginal.

In either case, this is just so ridiculous, I cannot imagine how fucked up must someone brain be before they start treating it seriously, and yet they did! Some still do. But as Karl Marx said, being determines consciousness - and nobody ever came up with an idea so ridiculous, that some privileged people couldn't use it as an excuse for existence of their privileges. Cosmic Jewish Zombie? Magical Vaginas? Social Darwinism? Efficient Market Hypothesis? Anything goes, as long as it implies the privileged stay privileged of course.


Anyway, in spite of my repulsion with the concept of monarchy, I decided to run some data analysis on effects different systems of governments have on levels of freedom and democracy.

Data comes from Wikipedia - government systems, Democracy Index (by The Economist), and Freedom in the World Index (Freedom House, essentially CIA).

Monarchies are more free than republics

It doesn't take a sophisticated statistical analysis to see that monarchies are more free than republics. Here's Democracy Index, higher is better:

• All countries: 5.6 +- 2.22 [n=163]
• Monarchies: 6.37 +- 2.63 [n=28]
• Republics: 5.44 +- 2.08 [n=135]

That's over 1/3 of standard deviation - about as much as the difference between Canada and USA - decent different, but maybe not worth writing a blog post about yet.

Just to verify, let's check Freedom in the World Civil Liberties and Political Right indexes (for both lower is better). Here's Civil Liberties:

• All countries: 3.19 +- 1.81 [n=192]
• Monarchies: 2.56 +- 1.7 [n=43]
• Republics: 3.38 +- 1.8 [n=149]

And Political Rights:

• All countries: 3.35 +- 2.13 [n=192]
• Monarchies: 2.88 +- 2.15 [n=43]
• Republics: 3.49 +- 2.1 [n=149]

So CIA and The Economist agree - that's again about 1/3 of standard deviation. Magical vaginas are not that bad.

It gets better

The results so far were not terribly interesting, so I tried something else - how important is having ceremonial head of state versus one with actual executive power? It's rather shocking (higher better):

• Ceremonial: 7.47 +- 1.42 [n=54]
• Executive: 4.67 +- 1.93 [n=109]

Civil Liberties (lower better):

• Ceremonial: 2.01 +- 1.28 [n=70]
• Executive: 3.87 +- 1.72 [n=122]

Political Rights (lower better):

• Ceremonial: 1.96 +- 1.37 [n=70]
• Executive: 4.16 +- 2.07 [n=122]

That's 1.25 σ for democracy, and a bit over 1 σ for the other two indices!
 
Combining two criteria gets expected results - figurehead kings > figurehead presidents > executive presidents > executive kings. Democracy Index (higher better):

• Ceremonial Monarchies: 8.12 +- 1.41 [n=18]
• Ceremonial Republics: 7.15 +- 1.32 [n=36]
• Executive Republics: 4.82 +- 1.96 [n=99]
• Executive Monarchies: 3.23 +- 0.67 [n=10]

Civil Liberties (lower better):

• Ceremonial Monarchies: 1.76 +- 1.13 [n=29]
• Ceremonial Republics: 2.2 +- 1.35 [n=41]
• Executive Republics: 3.82 +- 1.75 [n=108]
• Executive Monarchies: 4.21 +- 1.47 [n=14]

Political Rights (lower better):

• Ceremonial Monarchies: 1.83 +- 1.42 [n=29]
• Ceremonial Republics: 2.05 +- 1.32 [n=41]
• Executive Republics: 4.04 +- 2.09 [n=108]
• Executive Monarchies: 5.07 +- 1.71 [n=14]

So turning a king into a figurehead gets you 2.25 σ improvement (or 1.5 on CIA's indices), and even switching from American style executive president to German style figurehead president + prime minister / chancellor with real power scores you a free 1 σ or so.

Correlation and Causation

What does it mean? If correlation equaled causation then replacement of Prime Minister Gordon Brown with President Gordon Brown would turn UK into Russia or Pakistan, on average. Now we all know that correlation doesn't equal causation, but do you really want to risk that? So how about not fixing what's not broken, and fix what is broken - first past the post election system - first?

Cats would work as such kings and queens just fine - data clearly shows that the head of state should be a figurehead, so why not a cat? Is Cat King much sillier than Magical Vaginas, really? And even if it is, silliness is no excuse not to do science.

If someone thinks this post is a serious argument for monarchy, they might be interested in this kind of science too, just saying...