Make your own Medieval 2 mod, with blackjack, and random campaigns!

I'm as biased about it as it ever gets, but I'm pretty sure this is one of the most awesome things that happened to Total War modding ever.

Here's what in the release:

• Concentrated Vanilla minimod - essentially vanilla with a ton of minor fixes and enchantments for more dynamic gameplay.
• A program you can use to easily make your own minimod, customizing your gaming experience in many ways.
• Random campaign system, allowing every single campaign you play to be completely different.

Pics or didn't happen

For some nice campaign maps, go here, here, and especially here.

Here are some screenshots of just some of the things you can customize:

Installation

This is first public release of the mod builder, so it wasn't extensively tested. You might have problems with any of these steps - in such case just contact me.

• Install JRuby. Download link.
• Install 7zip. Download link. (in case you don't have 7zip or another unpacker for .7z archives)
• Download mod builder and unpack it. Download link.
• Start mod builder by double-clicking start.bat. You should see GUI like one in screenshots above.
• Customize in any way you want. If you want random scenarios, be sure to tick "Enable random scenarios" checkbox.
• There are two presets you can choose with File > Open. "Default" preset is Concentrated Vanilla minimod. "Minimal" preset is almost vanilla Medieval 2 Grand Campaign. You can then customize them any way you want from either of these starting points, including enabling random scenarios for both.
• Build your mod. It's as simple as using File > Build command. If there are any errors, you'll see them in a separate window. (if so, screenshot and send to me if you need help)
• Install your mod. If your game installation was detected correctly, File > Install command will install the mod.
• Run your mod [Retail version]. If you use retail (or cracked) version, use File > Run command, or double click concentrated_vanilla.bat file in your game directory.
• Run your mod [Steam version]. If you use Steam version with builtin launcher, start Kingdoms (not Medieval 2) from your Steam Library, then select Concentrated Vanilla from launcher's menu. This is least tested way. If in doubt, try to get retail version.

Troubleshooting

In case of trouble, screenshot whatever errors you see and send it straight to me.

If installer doesn't work, try running it as administrator.

If builder can't find your (retail) game installation, copy and paste all files from output directory to your game directory, then run concentrated_vanilla.bat from your game directory.

Support for Steam launcher is extremely experimental, and I just finished it moments ago, so if it crashes, just provide me the best information you can and I'll try to figure it out. For some reason starting launcher as "Medieval 2" rather than "Kingdoms" invariably results in a crash on my computer. (even though the mod actually uses Medieval 2 not Kingdoms engine).

Saving your work

You can save your settings with "File > Save", but format of that file is not guaranteed to be compatible across upgrades. For now it probably won't be.

Random seeds should result in the same map as long as you use the same version of the mod builder, and identical scenario settings. Different machine, or different settings for anything else are not meant to affect it, but no guarantees. In the future in would be nice to make random seeds shareable like in Minecraft.

Using Install command will overwrite previously installed version of Concentrated Vanilla. It won't touch your saved games.

Using Uninstall command will uninstall Concentrated Vanilla, but will not touch your saved games.

Limitations

This is early release so it has many limitations. Just some of them:

• Only Concentrated Vanilla map is supported (which has 7 extra regions over vanilla map). In the future vanilla map, and possibly other maps will also be selectable. Each map requires a bit of support code for random scenario builder. If you have any map you want supported, send me all the files, and I'll try to include it in a future release.
• Mod builder is somewhat fragile and hasn't been extensively stress tested. Unusual combinations of options might not work. Error reporting is currently minimal. If you find any errors screenshot errors, email me with as much information as you can.
• Scenario builder doesn't have any interface for ton of features it can do programatically. Features like leaving pope in Rome while shuffling everybody else around, adjusting city sizes randomly, or adjusting regions' religions to match ruling factions - and many others - would be fairly straightforward except for lack of interface. If there's any particular feature you want to see, just contact me.
• Only features I needed for Concentrated Vanilla are available in the mod builder GUI. I'm sure ton of other automated changes could be added. If you have any needs, contact me and I'll see what I can do.
• Medieval 2 engine can only run one mod at a time, but mod builder can add minimods to mod you're building. Currently only 3 such minimods are included. If you want your minimod included, contact me right away. (I especially need better CAI mod).
• Installer, especially for Steam launcher, is only minimally tested (and only on English 64-bit Windows 7).
• In principle this kind of customization system can work with Rome, or any existing mod. It would require some porting effort, but if there's significant interest, I don't see why not.
• A lot of people requested hotseat campaign mod. This would require switching the mod from medieval 2 engine to Kingdoms' retrofit mod as a base. It will probably happen someday, since Kingdoms gives us access to a bunch of extra features.

Good modding, and have fun.

Credits and links

Concentrated Vanilla 0.60 includes some files from Medieval 2 Total War game by Creative Assembly, and the following minimods:

• Sinuhet's Battle Mechanics
• Lusted's Better BAI/CAI
• Agart's Cities/Castles Strat

Download link.

If you're interested in how this mod evolved, posts about previous versions (with links inside):

• 0.53
• 0.52
• 0.51
• 0.50
• 0.08
• 0.07
• 0.06
• 0.04
• 0.01

Easy Windows registry editing with JRuby

Like all people who came out of Unix tradition I approached Windows registry as something not to be touched even with a long stick, but it turned out not to be that bad.

The first thing you need to know about Windows registry is that it has multiple roots. All our viewing and editing will apply to particular key only (usually HKEY_LOCAL_MACHINE).

It looks like a pretty stupid decision, but then it comes from people who use C: D: etc. instead of single directory tree.

  require "win32/registry"
  def hklm
    Win32::Registry::HKEY_LOCAL_MACHINE
  end

Reading information from registry, like installation paths of various programs, is very easy:

  hklm.open('SOFTWARE\Wow6432Node\SEGA\Medieval II Total War')["AppPath"] rescue nil
  hklm.open('SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\Steam App 4700')["InstallLocation"] rescue nil

If you want nil instead of exception just rescue nil the entire thing.

Writing to registry is very easy as well, here's actual example:

    cv = hklm.create('SOFTWARE\Wow6432Node\SEGA\Medieval II Total War\Mods\Unofficial\Concentrated Vanilla')
    cv["Author"]="Tomasz Wegrzanowski"
    cv["ConfigFile"]="concentrated_vanilla.cfg"
    cv["DisplayName"]="Concentrated Vanilla"
    cv["FullName"]="Concentrated Vanilla"
    cv["Language"]="english"
    cv["Path"]="mods/concentrated_vanilla"
    cv["Version"]="0.60"
    cv["GameExe"]="medieval2.exe"

And deleting things to uninstall:

    hklm.delete_key('SOFTWARE\Wow6432Node\SEGA\Medieval II Total War\Mods\Unofficial\Concentrated Vanilla', true)

And that's about it. If you want to explore the registry either start regedit program, or start jirb and play with it from JRuby REPL.

JRuby Swing GUIs with cheri gem

I am not in any way a fan of desktop GUI toolkits - HTML5 and jQuery totally spoiled me, so I resisted for a very long time making GUIs for my Total War tools - and happily enough, other people would sometimes make them for me.

But this time I decided to make a desktop GUI, in JRuby, and that means one of the awful non-HTML toolkits.

So my first idea was of course making a big window with a menu calling some functions, and big embedded HTML form with all stuff in HTML. I was even getting somewhere since Java Swing has HTML widget, but then it turned out it's HTML 3.2 only, no Javascript whatsoever, and serious pain to get data into and out of it.

I also tried SWT, and hoped danlucraft's cookbook would help me get somewhere with it, but I couldn't figure out most of the things I wanted to try, so I kept looking.

Finally I found this lovely cheri gem, which didn't seem to such too hard. I've heard mostly horrible things about Swing API, but it was only as bad as the rumor says, at least for my simple use case.


I'll put all that code for public view eventually, but it's pretty massive, so here are just some tips for working with Swing and cheri.

Basic window creation

Start a class and include Cheri::Swing module. What you probably want to match HTML-ish behaviour is actually not a single layout manager but GridBagLayout (for actual layout) within ScrollPane (so you get scrollbars when content).

That's the code:

class ConcentratedVanillaBuilder
  include Cheri::Swing

  def initialize
    @controls = {}
    @frame = swing.frame('Concentrated Vanilla builder'){ |frm|
      size 800, 800
      default_close_operation :EXIT_ON_CLOSE
      build_menu!
      scroll_pane {
        panel {
          grid_bag_layout
          grid_table {
            background :WHITE
            build_form!
          }
        }
      }
    }
    load_settings! load_settings_file("settings/default.txt")
    @frame.visible = true
  end
end

This initialization is pretty generic (other than trivial matters of default window size and title), other than four italicized lines.

Separate form buildings from settings

That's advice for GUIs that simply configure some settings and then run some script. You want to keep your settings in a nice Hash, and don't mix GUI code with settings defaults.

So what you want are helper methods like these:

  def checkbox(name, description)
    grid_row{
      @controls["checkbox-#{name}"] = swing.check_box description, :a => :w, :gridwidth => 3
    }
  end

And then use methods on @controls[something] to both get and set various fields. That's far easier than ton of on_change callbacks or whatever is their Swing equivalent.

Use text_area not label for labels

Label widgets are pretty dumb, and non-editable text areas can handle things like multiline text and formatting a lot better.

Just add some helper methods and pretend you're coding HTML:

  def div_helpmsg(msg)
    grid_row{
      text_area(:a => :w, :gridwidth => 3){
        editable false
        text msg.gsub(/^\s+/, "")
      }
    }
  end

  def h1(msg)
    font = java.awt.Font.new('Dialog', java.awt.Font::BOLD, 24)
    grid_row{
      text_area(:a => :w, :gridwidth => 3){
        set_font font
        editable false
        text msg
      }
    }
  end

If things get too complicated, you can always go for full HTML widgets.

Result

Half-finished result looks something like this. Not amazing, but it will do the trick.

It will get released sometime soon, and then you'll be able to play random scenarios everybody's waiting for.

By the way if any Java / JRuby experts has better ideas, go ahead. Googling was unusually unhelpful to me here, and IRC and StackOverflow were as useless as they always are.

Yu-Gi-Oh! - first impressions

I'm back from Hyper Japan and one of the things I've done there was a demonstration game of Yu-Gi-Oh! I had no idea what the game is about, other than the thing Magic: the Gathering players make fun of.

They gave away free demonstration deck and one booster.  I am going to open them right now as I'm writing this post, and describe my impressions.

Just a reminder - I have no idea what I'm talking about. This is a first impression post. I'm posting this without even checking Wikipedia, so hilarious misunderstandings are very likely to ensue.

Demonstration Deck

It's 20 cards. Surely you must play "real" Yu-Gi-Oh! with more than that right?

There are 12 monster cards, 3 spells, and 5 traps.

They have types in upper right corder and different colored borders. For spells and traps there seems to be 1-to-1 correspondence between frame colors and card types.

For monsters the symbol says dark/earth/light, but frame color seems to be unrelated to it. It seems that one color is for vanilla creatures, other for non-vanilla creatures. I bet there are more types, just like with Pokemon.

There don't seem to be any rarity indicators, no idea yet if it's also true for cards from the booster.

Art is pretty crappy, and there are no artist attributions. It's probably drawn by Chinese prisoners or something like that.

One thing I really like about it is very high numbers it uses. First card has attack 2500, defense 1200. I thought they're all multiplies of 100, but there's another with Attack 450, defense 600.

Magic really suffers a lot from not being able to use fractional numbers for power, toughness, life, mana and so on. The chasm between 2 and 3 is just too vast, and forcing Thragtusk to have toughness 3 while every single first striker in Standard has power 2 at best is part of the problem with current meta. If Thragtusk had toughness 2.5, and at least some of these first strikers had power 2.5 - it would have been a very different story. Or if Snapcaster had only power 1.5. Now fractions look silly, but if Magic started alpha with all numbers x10 (or otherwise wider scale), it would have saved itself a ton of problem.

Here's contents of demonstration deck:

• Summoned Skull
• Dark Blade
• Sabersaurus
• Gene-Warped Warwolf (yes, that's exactly what it says)
• Man-Eater Bug
• Hayabusa Knight
• The Fiend Megacyber
• Marauding Captain
• Exiled Force
• Des Koala (featuring literally a koala, meanwhile Magic creative is too much serious business to get even one squirrel...)
• Majestic Mech-Ohka
• The Calculator (I so fucking hate cards with "the" in name, like The Mimeoplasm, nothing about card names get me more annoyed than "the Something" cards)
• Fissure
• Upstart Goblin
• The Warrior Returning Alive
• Mask of Weakness
• Secret Barrel
• Threatening Roar
• Rising Energy
• No Entry!

I really hope it all sounds less ridiculous in Japanese.

Oh, and three more things come to mind:

• Does Yu-Gi-Oh! have anything like magiccards.info (or even its crappier official equivalent Gatherer - which I'm not going to link to because it sucks too much)?
• Does it have anything remotely like Comprehensive Rules? I've noticed a few people with real decks play, and it seemed pretty thoughtless as well, but then this wasn't exactly a high level tournament.
• Templating on Yu-Gi-Oh! cards sucks so hard. I'm a huge fan of legalistic precision of modern Magic templating. What I see here reminds me of alpha's "let's vaguely describe what it does" templating style. But then if Yu-Gi-Oh! is so much simpler than Magic, maybe that's enough.

Booster

Now I'm going to open "Return of the Duelist" 1st edition booster.

It says that it includes 9 cards, and full set is 48 commons, 20 rares, 14 super rares, 10 ultra rares, and 8 secret rares. I already hate the fuckers. Mythic rarity was such a massive mistake for Magic - and the main reason they gave was "every other card game has more than 3 rarities, so we'll introduce mythics to force players to buy more boosters".

Anyway, opening now.

First, art quality is a lot better. Madolche Chouxvalier and Madolche Lesson are adorably chibi, instead of crappy shonen dark bleh.

For some reason Madolche Couxvalier has foil label. The rest of the card is nonfoil. Does it have something to do with rarity?  My prediction that there will be more types than Dark/Light/Earth so far isn't working.

Judging from text of cards, they are parasitic as hell. Just these 9 cards refer to "Madolche", "Xyz", "Spellbook", "Machine-type", "Chronomaly" cards - and how many are just from this particular set? Magic managed to avoid this parasitic design, mostly.

For reference, the cards:

• Madolche Chouxvalier
• Amores of Prophecy
• Damage Mage
• Chronomaly Mayan Machine
• Madolche Lesson
• Lucky Punch
• Chronomaly Pyramid Eye Tablet
• Generation Force
• Spellbook of Power

tl;dr I'll stay with Magic.

PS. Kakigori is delicious.

How to make a good random maps for Medieval 2 Total War

Some news about Medieval 2 random scenario generator I wrote about yesterday.

Today I fixed most of the known bugs in scenario builder (except AI passivity, which would require playing an actual campaign to figure out), and I was looking at JRuby options for a simple GUI so everybody can enjoy scenario builder, and customize their Concentrated Vanilla experience any way they want - including simply turning all options off and playing vanilla with bugfixes and random scenarios if that's your thing.

Sadly making GUIs looks like a lot more work than expected. I got so used to the happy universe of HTML5 and jQuery on frontend with Ruby on Rails on backend that my first contact with Java GUI libraries was like hitting a wall. I'll figure something out eventually - in the worst case I'll just make the script start web browser and present you the mod customization form in browser window.

Anyway, scenario builder.

How to make good random maps

Assigning settlements completely at random means a few fun campaigns, but it confuses AI too much, and doesn't really lead to much strategic gameplay since whoever you just attacked probably can't respond since all their other settlements are in completely different part of the map.

So my next idea was % cluster / % random allocation, but there are many things I don't like about it. Here are some ways how cluster algorithm fails them, while random allocation worked reasonably:

• 5 settlements have no neighbours and 6 have only one. Cluster allocator assigns most of them to rebels.
• There are some false neighbours like Dongola / Benghazi, Tripoli / Timbuktu, Arguin / Marrakesh - they are next to each other on a map but you'd need to move through unpassable desert.
• Since factions were allocated in order, factions late on the list were most likely to get scattered settlements.
• Very often huge areas are all rebel, while there are no rebels in other places. On 67% cluster map there are no rebels in Spain, France, and Germany, but there's a massive 11-region rebel cluster in Anatolia and Balkans, and second 7-region cluster on South/East Baltic Sea coast. This is not due to algorithm since rebels just got whatever was left. Random chance will lead to some clusters, but rebel clustering is far too strong.

So I tried many things like:

• Allocating settlements to different factions in random order, one at a time.
• Allocating settlements to rebels first rather than last (without clustering), or one at a time as if they were normal faction (except without clustering) 
• Making rebels not take settlements with 0/1 regions if possible, so factions would get them instead.
• Using XY distance between cities instead of map neighbourhood, or some combination of both.

Eventually after looking at countless maps for far too long, the following algorithm seems to work best:

• Fully at random, allocate one settlement to each faction, and all rebel settlements.,
• Then allocate other settlements in random order. Some % will be neighbours of existing settlements, the rest will be random.
• If there are no neighbours available since all have been allocated to other factions, try neighbours-of-neighbours instead.
• If neighbours-of-neighbours still doesn't work, just do random allocation. (because of this % in this post are much higher than in previous post - you actually get significant random allocation because no other is possible)
• Neighbours of a settlement are 5 geographically closest settlements. Then we add arrows back, but only if they are geographically connected. This seems to work best. 

And now the maps. Muslim/Orthodox factions get 3+d6 regions, Catholics get 1+d6 regions, rebels get whatever's left.

Each with 10 different random seeds - for ton of maps.

100% clustering (only random when forced)

80% clustering

70% clustering

Next steps

So much map porn for today. The next step is making some kind of interface so people can run this without going into command line and/or editing code.

Testers very much welcome. If you want to play some random map soon, and don't mind possible technical difficulties, contact me ahead.

Also, if anybody knows how to run a mod on Steam version of Medieval 2 without going through massive pain (the way it works on retail and pirate versions), definitely tell me. I've heard rumors there is a way, but I've never seen anything reasonable.

By the way since maps will be mostly clustered, I'll probably move distance to capital penalty somewhat higher than it was on fully random map.

I've also been asked about hotseat mode. As far as I understand, this would require switching from Medieval 2 engine to Kingdoms engine with retrofit mod as base. I'll probably do that some day, but for the time being the mod will most likely stay a single player only thing.

Random campaign scenarios for Medieval 2 Total War

Total War games have somewhat limited replay value, since campaign map is the same every single time, and there are only so many mods you can play.

In my Concentrated Vanilla mod I tried to make things somewhat more interesting by introducing large number of tweakable settings - so for example in one campaign cavalry would be really nerfed and I'd need to play infantry game, some other time mining income was far greater than any other income so a small number of key settlements held key to victory, or I removed distinction between cities and castles and so on. Every campaign slightly different.

That still doesn't get us very far, since campaign map looked the same each time.

Here's the campaign map (it's like vanilla map except a bunch of extra regions added to fill some gaps):

The first thing I've done was add scenario builder to Concentrated Vanilla script, and used it first to make life much more difficult for Byzantium (which also started at war with all its neighbours). It was some serious fun, especially early game.

I could of course keep making new scenarios manually (it's not that hard), but then I got an even better idea - why not assign settlements to factions at random?

I started a campaign and I'm writing AAR as I go (sadly I didn't have much time this week).

I'm playing as Portugal (white with blue borders), which starts in Arhus, Alexandria, and Cagliari. You can get details in AAR.

This is fun and somewhat bizarre experience, but then it's a bit too easy - AI is very confused with its provinces scattered all over the map, and it's too easy to attack AI, take its one province, and then get it to agree to ceasefire since you're not longer neighbours. Rinse and repeat.

So I tweaked the scenario generator. Instead of allocating settlements randomly X% of the time it allocates a settlement next to one already allocated, otherwise (or if none are available), it picks a random one.

75% cluster, 25% random:

67% cluster, 33% random:

50% cluster, 50% random:

These maps are very interesting, and probably won't lead to AI problems like with my AAR.

Technical details

As for some extra details - settlements are neighbours if they have either land link or land bridge, so Denmark is neighbour of Sweden (land bridge), but Cyprus isn't neighbour of anything.

In my AAR playthrough I put minimum number of settlements at number of family members, since I had no other way to deal with them, now they'd just be places somewhere near existing cities.

If faction gets more cities than it had originally, extra cities get filled with some basic units (IIRC I set it to 2 Spear Militia or something similar + 2 basic missile units for all factions).

Muslim and Orthodox factions get more settlements on average, since otherwise it would be too easy for Catholics.

The script can be tweaked to always allocate some settlements to some factions (like Rome to Pope), allocate different units based on settlement size and type, and do other interesting things.

All navies are disbanded to simplify matters.

I could generate correct maps for campaign start menu, but it's better for suspense not to reveal factions' locations there just yet.

There are some bugs to fix (Durazzo gets skipped always, churches should get demolished if faction of different religion gets settlement).

Can it be used it in other mods?

The obvious question is - can you use it in vanilla, or other mods?

In principle it should be possible, as long as the mod is not too heavily scripted (like Third Age).

There would need to be a lot of tweaking for each mod. Things like lists of factions, protected regions and factions (here - Americas, and aztecs/timurids/mongols), what kind of units should factions get, etc. These aren't particularly complicated issues, but each mod needs to solve them differently.

The scenario script is not separate from the rest of Concentrated Vanilla, but if you need some assistance with porting it to another mod, we should be able to sort things out somehow.

I don't see any major reasons why it couldn't be adapted to run on anything from Rome to Medieval 2 to Kingdoms, with some work.

Future possibilities

In theory it would be totally possible to go further with it, and make fully random maps for Medieval 2, with one of random map generating algorithms other strategic games use.

I don't have any immediate plans to go there, but one day, who knows?

When can I play?

I'll release it sometime soon, there are still some bugs to be fixed, and I need to figure out some kind of user interface so non-programmers can run the script.

If you're very impatient just email me and I'll send you what I've got.

Think Like a Programmer by Anton Spraul - book review

The book "Think Like a Programmer" tries to teach people relatively new to programming how to approach problems and program solutions.

The idea is good, but the book has one massive flaw - it uses C++. I'm not going to kick the dead horse and repeat all reasons why C++ is horrible in general - but it's especially horrible for teaching problem solving to beginners.

Irrelevant low level details caused by limitations of C++ keep getting mixed with solutions of discussed problems throughout the book. Many problems like constant sized arrays, pointer lifetime management and so on, lack of usable containers, I/O, and string functions in standard library and so on simple don't exist in civilized language.

And sadly such basic ideas as:

• experiment with your problem in REPL
• start with writing unit tests for your problem
• check if standard library can solve part of your problems
• check if any easily installable third party library (via CPAN, rubygems, or whatnot) can solve part of your problem

are completely alien to C++ thinking.

Other than this one huge flaw the book is really nice. All code is very nicely formatted with inline bullet point references and shading for commentary, all problems are explained with liberal use of relevant diagrams, methodology of breaking problems into steps and solving them separately is executed quite well, and even various pain points of C++ tend to be handled as reasonably as they can.

Other than choice of C++, the only other problem is that while problems tend to be broken into small steps to be solved one by one, solutions to these small steps are then typically put together into one big lump of code instead of being left alone as separate functions or methods, and this code tends to be much more complex than it should be. I blame C++ for that as well.

I urge the author to write a Python or Ruby version - or at least Java version for that matter. Using C++ in a book for beginners mostly just shows the basic problem solving mistake - not choosing the right tool for the job, and it really distracts from what the book is supposedly trying to do.

Summary

If you're a beginner to programming and your school or university forces you to learn C++ in 2012 this book might be really useful to you. Also find a lawyer and sue your school for emotional trauma caused by C++.

If nobody makes you learn C++, I'd skip it. It's very C++-centric, and many of its lessons are not directly applicable to more sensible languages.

If you know a bit about programming from point of view of a high level languages, and are trying to learn C++ now, it might actually be reasonable supplement to C++ textbook.

If you're coming from reasonable C to C++ world, you probably won't find much of it useful. You'll presumably know most of it already, and advanced C++ issues like boost libraries won't be even mentioned here.

And in interest of full disclosure, I got this book as a free review copy.

Elections happened and Intrade lost

So elections happened, and Intrade lost big.

There's no need defending them that they still had Obama as leading candidate. Here are some of their minor markets one day ago:

• Obama to win Florida - 32%
• Obama to win 320 or more electoral college votes - 26%
• and so on

Intrade massively overestimate Romney's chances and possible margin of victory and underestimated Obama's.

Meanwhile it seems that Nate Silver got 49.5 out of 50 correct, since he gave Florida exactly 50.0% chances.

Anyway, I'm sure Nate Silver will be getting more congratulations than Obama over the next few days, that's not what I want to write about.

Markets vs Models

My post from two days ago caused a bit of confusion, and perhaps I wasn't entirely clear (but look at the kittens!).

I'm not claiming that yesterday's elections were a definite resolution of markets vs models question. Such resolution is in principle impossible.

Even if Nate Silver gave Obama 99% chances, and Intrade gave him 1%, even that wouldn't really be definite.

First - we have a independent sample size of one. There were multiple predictions made by both Intrade and Nate Silver, but they are not independent at all. By any measure probability of winning Ohio and probability of winning elections were very highly correlated, so we have just one observation.

Second - even if both Intrade and Nate Silver keep predicting subsequent elections, these are not independent events at all! Pollsters, polling aggregators, and market participants will keep changing their methodology based on how their previous predictions worked, so every new prediction is dependent on all previous predictions.

And third and worse of all - we don't even have independent sample size of one. Market participants had full access to Nate Silver's predictions, and could have priced them in, so that's not independent at all.

It's somewhat better in the other direction since Nate Silver claims to have made his model long time ago, and not to have made any tweaks, but then Intrade numbers were widely quoted, and they might have affected both voters and polling organizations, affecting inputs to Nate Silver's model. Compared with other problems this one is probably the least important, but in principle nothing at all is independent!

What was Intrade thinking?

Intrade numbers were so strangely optimistic about Romney it's difficult to take them at face value. Any of these alternatives makes more sense than Intrade traders honestly and rationally expecting Romney to have 30% chance:

• Deliberate market manipulation on Intrade
• Completely irrational bubble
• Traders believing in some kind of vast left-wing conspiracy by pollsters, who all lie about voters' preferences 
• Traders believing in some kind of vast right-wing conspiracy by voting machine manufacturers or some other kind of massive pro-Romney voting fraud

I'm not joking about these two conspiracy theories - they were both pretty widely discussed all over Internet and apparently some "fair and balanced" cable news stations.

Nate Silver's model actually included some small voting fraud effect of Republican voter suppression efforts, but Intrade traders would have to believe in something much greater than that to reach these numbers.

I think the most likely answer to what happened was some combination of market manipulation and bubble. We know there were attempts to manipulate Intrade by Republicans in both 2012 and 2008 elections.

In theory such efforts should always fail - if there are enough people who care only about making money - and this tends to be a safe assumption on most markets - they'd take advantage of such manipulation attempts and market would go back to rational equilibrium.

However if there's a sufficiently large number of people who take high Intrade prices as solid evidence that actual chances must be high - they can create positive feedback which turns any movement (either random or deliberate) in prices into long term price change.

This sounds speculative, but I have no other explanation for why Intrade's odds of Supreme Court overturning Obamacare increased from 2:1 to 4:1 while nothing whatsoever was happening! Someone pushed prices for whatever reason, people saw that prices are increasing and assumed that this someone has either insider knowledge or some kind of insight, and jumped the bandwagon.

No kind of rational market process can generate both 2:1 odds and 4:1 odds with zero new information.

Now stupid kind of market manipulation - throwing ton of money on the market very quickly - will probably end up being defeated since it's too obvious to everyone. But if you push the market slowly then you might very well generate a lot more positive than negative feedback.

The case for manipulating prediction markets

That's for some speculation on why market manipulation might work in practice better than in theory, but in leaves a big question - why?

After all even if you successfully manipulated the market, you're very likely to still end up losing money on election day, and most voters don't pay much attention to Intrade.

But there's one route how Intrade odds can plausibly affect the results:

• Candidate manipulates his odds on Intrade
• Candidate appears more likely to win to donors, volunteers, journalists and so on
• Candidate gets a lot more donations and coverage
• More campaign money and more coverage affect voters
• Polling shows this
• And there's positive feedback look

I don't think this would matter too much in presidential elections, but in primaries perceptions of candidates' seriousness is a huge deal. Intrade's preferred candidate Mitt Romney won against everybody else even though he never lead in any polls, and it was clear that he was very far from what the Republican base wanted.

So was Romney campaign or his supporters manipulating Intrade during primaries (and later during elections)? I doubt we'll ever see any hard evidence, but this scenario is broadly consistent with data.

I'll just leave it here as an interesting scenario to think about, not any serious conspiracy theory.

What does it mean for the future?

Actually not that much.

After this loss, Intrade traders in future elections will probably pay a lot more attention to models like Nate Silver's and election fundamentals, and will be much less likely to jump on any bandwagon of either deliberate market manipulation or random bubble.

Unless you're willing to outspend every single market participant put together, successful manipulation is only possible if you can get bubble going, and traders will probably now be much less willing to put too much weight on Intrade prices.

For prediction markets enthusiasts, this isn't even much of a problem in theory - Intrade is very far from a perfect market, with all kinds of transaction costs, and other limitations, so even if Intrade was definitely proven to be biased and bubble-prone (and we have no such proof, just some suggestive data points) - they can keep insisting better prediction market would still be "rational".

Like belief in Efficient Market Hypothesis, belief in rationality of prediction markets is simply immune to data.

Anyway, that's probably the final post about politics, I'll be back to your regular kittens and technology posting soon. (especially once Raspberry Pi I ordered arrives)

Tomorrow is the most important election in history of the future

Tomorrow millions of Americans will choose between rule guided by wisdom of the crowd embodied in free markets and rule guided by wisdom of elite experts embodied in technocrats' statistical models.

And by that I mean obviously Intrade vs Nate Silver.

What government is all about

Make no mistake - this isn't some obscure issue a few wonky bloggers care about - this is one of the biggest issues the future will face - the problem of predicting itself. Governing is fundamentally about just two things:

• Determining how you value various possible outcomes - how much do you care about low unemployment vs economic growth vs showing off your military power vs not offending God with too much gayness vs other such important outcomes of government policy.
• Predicting how various choices made by the government will affect these outcomes.

Whenever there's full agreement on both, there's really no controversy as for what government should do. You apply Bayes' Theorem and it will tell you precisely which of many possible policies to adopt.

Whenever there is disagreement, it must mean either different values, or different predictions of the future, or both.

Values

Contrary to what you'd expect agreement on values is the easy part. Most things affected by government policy are economic in nature, and people mostly agree on the same things being good - like higher incomes, lower unemployment, cheaper energy, better education, healthier environment, less poverty, and so on - they might disagree on relative weights to be given to these issues, but there's a natural utility metric - one happy voter is one point of utility - so you can just estimate how median voter feels about various issues to get a pretty good starting point for the compromise.

This won't resolve some conflicts, like the one between "people against murdering unborn babies" vs "people against forcing women to carry their rapists' babies", or whatever were the politically correct terms for them these days. Well, too bad, but there's literally no solution that will make everybody happy anyway. Such problems are fortunately fairly rare, and it's not like the government is doing a great job handling them anyway.

Predictions

A much more common and more important source of disagreements about the future is with predicting outcomes of various policies.

If one seriously believes that tax cuts for the rich lead to economic renaissance, and that welfare just makes people lazy, then Romney's policy makes perfect sense.

And if one believes that Obamacare will make healthcare available to everyone at lower cost, and that massive budget deficits saved the world from the Second Great Depression, then Obama's policy makes perfect sense.

There's little difference in values here - both Romney and Obama and most of their supporters want high economic growth, high employment, affordable healthcare, and fewer financial crises - but they have very different ideas on what's the best way to reach such goals.

How to predict the future?

So the problem of governing reduces to problem of predicting the future. And only two serious approaches exist - prediction markets and experts' statistical models.

Prediction markets like Intrade, or for that matter - stock and bond markets - sound a lot like gambling to many, but that couldn't be further from truth. Every transaction on prediction markets is a disagreement about the future - with winner taking loser's money. If you truly believed that Ron Paul was going to win Republican primaries, or that Nokia's N series was going to destroy iPhones, or that deficit spending would result in hyperinflation, you can put your money where your mouth is (on  respectively Intrade, stock market, and bond market) and if you were right then you could be an overnight millionaire. Or if you were wrong, you'd lose your money.

This is a pretty good motivation, and prediction markets tend to be pretty good most of the time. It's really most unfortunate that we don't have any widely traded prediction markets for economic growth, unemployment, and other such macroeconomic indicators, but markets we have tend to perform quite well.

The only second approach is statistical modeling based on reference class forecasting. You accumulate a lot of data about the past, and based on assumption that the future is likely to be broadly similar to the past you can make some really good predictions. This is far from automatic exercise and requires a lot of judgment as to which variables are relevant and how they might be connected, and in the end you often get pretty wide margin of error - but if reference class forecasting tells you we don't know better that's probably because we don't.

I'm completely ignoring all so called "experts" who don't base their predictions on any statistical models and just talk out of their ass, and all claims of "markets" that are not backed by actual money on the claim. These have track record more dismal than asking my cat's opinion.

So how do we decide between prediction markets and experts' statistical models?

Intrade vs Nate Silver

Unfortunately we have very few serious statistical models, and very few serious prediction markets, and even less overlap. Pretty much the only area where we can compare them against each other is predicting outcomes of American elections. And unfortunately for us, in the past they've been mostly in very good agreement with each other.

However, something really interesting happened. As of today, prediction markets and statistical models disagree more than at any time in history. Here's quick summary of Obama's vs Romney's chances of victory (normalized to 100%):

• Intrade (popular vote by at least 0.5%) - 58.1% to 39.1% (1.48 to 1)
• Intrade (electoral college) - 67.3% to 32.7% (2.06 to 1)
• Intrade (Ohio) - 68.7% to 31.3% (2.20 to 1) 
• Iowa Electronic Markets (popular vote) - 74.0% to 26.0% (2.85 to 1)
• betfair (electoral college) - 79.3% to 20.7% (3.83 to 1)
• 538 (popular vote) -  80.2% to 19.8% (4.05 to 1)
• 538 (electoral college) - 86.3% to 13.7% (6.30 to 1)
• 538 (Ohio) - 87% to 13% (6.70 to 1)

Don't be misled by the fact that they both predict Obama's lead - they differ so drastically in predicting how big is the lead it's as if they were seeing different elections. 538 offers odds more than three times Intrade's. Difference is very similar for popular vote and for Ohio winner.

If Romney wins after all, this will be a stunning victory of prediction markets over experts' models, and Nate Silver will most likely go back to baseball statistics.

If Obama wins, especially if he wins by a big margin, it will be a major defeat of prediction markets, especially Intrade.

Now you may not think a single such outcome matters much, but I have a bunch of completely unscientific anecdotes regarding their past performance.

Intrade and Obamacare Supreme Court Judgment

Intrade was really bad at predicting Supreme Court judgment, it peaked at about 80%. And Intrade turned out to be completely wrong.

The worst thing about is was not so much wrong prediction - it was how it went up from 65% (2:1) to 80% (4:1), doubling the odds, with zero new information becoming available. It was just Intrade bubble - people were estimating odds higher because they were going higher, unless someone had insider information to Supreme Court justices, which in this case at least we can safely say didn't happen.

It completely baffled Nate Silver as well, who responded with much more reasonable 1:1 odds based on statistical analysis of past information, and turned out to be much closer to being right.

Intrade and Gadaffi

Here's even more baffling one - market that Muamar Gaddafi will no longer be leader of Libya by 31 December 2011.

Not only it didn't go up to 100% after fall of Tripoli, when Gaddafi was reduced to being a mayor of Sirte at most, it actually fell down, because it turned out into market for "what the hell Intrade admins meant by leader of Libya", and it stayed there for days.

It wouldn't be the weirdest thing to ever happen if some rebels with big Intrade investment made sure the market was judged correctly by putting a few bullets in Gaddafi's head.

This shows how difficult it is to run prediction markets for anything will less defined outcomes than winners of Oscars or elections. Uncertainty over interpretation of market rules might be greater than uncertainty over interpretation of outcomes. And unfortunately most important things in life are less clear than election winners, so if prediction markets can't handle them, they're of not that much use.

Republican Primaries

Here's a very interesting case. Intrade was always far more pro-Romney than polls during primaries.

Polls disagreed widely on who was leading Republican primaries - with just about everyone including Herman Cain having their time as the top candidate - except Romney who was always number two. Meanwhile Intrade didn't doubt Romney for a moment, and Intrade was right.

Nate Silver even wrote a post why on we shouldn't underestimate Herman Cain's chances, which looks really silly in retrospect.

Tomorrow

Regardless of tomorrow's winner - be it Nate Silver/Obama or Intrade/Romney - we will learn something important about which way of predicting the future is more reliable - and I'm sure the future will care far more about this than about pretty minimal differences between Obama and Romney.

Post Scriptum

To make things even more clear, as of Tuesday morning UK time, 538 updated its predictions to 92.0% to 8.0% (11.5 : 1).

Intrade and Betfair are completely unmoved. (I don't actually know how often Betfair updates its odds, but Intrade is live).