Plan 9 from Outer Forks

I love IMDB. For one, it might be one of the very few non-personalized recommendation systems that actually work. Scores on IMDB correlate very highly with likelihood of me enjoying a movie, especially if I apply a correction for genres that are overrated (like very old and very long movies) or underrated (like zombie flicks). In any case IMDB is vastly more accurate than professional critics' reviews.

The most drastic example of IMDB and pro critics disagreeing was Transformers: Revenge of the Fallen, which I absolutely loved! It got an okish 6.5 score from IMDB - which is accurate enough, it's a really nice action-packed movie vaguely following Transformers canon, with as much concern for the plot as a typical action movie (that is not terribly much), and perhaps far higher aircraft carrier to robot ratio than you'd expect from the title.

It was also universally panned by the pro critics getting Rotten Tomatoes metascore of just 20%. Seriously, critics? Is it really one of the worst movies in existence? Have you expected anything else than robots meaninglessly fighting each other and destroying valuable stuff? It's almost as if critics panned Transformers 2 to signal their sophisticated taste, not to provide good service to the audience.

Anyway, back to my point. IMDB not only provides great recommendation service, they also make a lot of the underlying data available in convenient formats. Yes, they charge huge money for some of the data, but even the free portion is very useful.

Some voting patterns are very interesting. Here are two examples. First, one of the "so bad it's good" movies - Troll 2. Normal movies have sort-of Gaussian distribution of votes. OK, not really Gaussian but with a fairly definite single peak. Not so with the "so bad it's good" movies - these tend to have mostly 1s and 10s, with perhaps a few 2s and 3s, but amusingly very rarely many 9s and 8s - it's either very low, or a 10!



The second interesting bit is a demographically polarizing movie like Twilight. People of different ages and genders tend to like pretty much the same movies.

Top 100 for men, and Top 100 for women contain virtually the same movies, just slightly rearranged - neither stereotypically feminine chick flicks nor stereotypically masculine action flicks get into top lists, and movies rarely have very different male and female scores, or much different scores by age group. But exceptions do happen, here's one:



Under-18 men scored it almost 2 points lower than under-18 women! I have a perfectly good explanation for the Twilight ratings effect - normally movies are watched by people who like the genre, so only men who like chick flicks and only women who like action movies watch them. So there will
be very little discernible gender-genre bias. But with Twilight, millions of girlfriends worldwide must have forced their boyfriends to watch Twilight against their wills, to what the aforementioned boyfriends reacted in a passive-aggressive way by downvoting Twilight on IMDB. I don't have any way of
testing this theory, but I watched Twilight on my own free will, and liked it a lot.

Anyway, back to "so bad it's good" movies. I really like the genre, but it's difficult to tell the "so bad it's good" movies from straightforwardly bad movies. So like a good hacker I am, I decided to grab IMDB's database, and find out movies. Here's the list, criteria being - at least 10% of votes are 1s, at least 10% of votes are 10s, ordered by number of votes.

	Votes	10s %	1s %	IMDB score	Title
1	77257	12.6%	11.0%	6.2	The Blair Witch Project
2	69978	24.6%	12.7%	6.1	Twilight
3	68560	30.4%	10.5%	7.6	Fahrenheit 9/11
4	38459	20.0%	19.2%	5.5	Sex and the City
5	33628	15.0%	10.1%	6.0	Halloween
6	24265	31.7%	23.0%	5.1	High School Musical
7	24044	10.6%	10.3%	5.1	The Pink Panther
8	22873	14.9%	12.1%	5.6	House of 1000 Corpses
9	22324	18.6%	27.5%	3.9	Freddy Got Fingered
10	21704	15.3%	12.4%	5.0	White Chicks
11	19898	14.5%	10.7%	7.6	Midnight Express
12	19418	11.0%	26.0%	3.7	Norbit
13	19230	10.6%	43.9%	2.8	Crossroads
14	18410	11.6%	10.3%	6.3	Funny Games U.S.
15	17779	12.2%	10.0%	4.7	Queen of the Damned
16	17606	10.5%	27.9%	3.8	In the Name of the King: A Dungeon Siege Tale
17	17103	25.3%	10.4%	5.7	Kung Pow: Enter the Fist
18	16840	13.6%	54.9%	2.5	You Got Served
19	16743	14.7%	35.0%	2.8	Spice World
20	16706	10.5%	74.0%	1.6	From Justin to Kelly
21	16596	17.2%	10.8%	5.7	Southland Tales
22	16338	16.2%	31.9%	3.6	Plan 9 from Outer Space
23	16191	19.1%	11.8%	5.5	Step Up 2: The Streets
24	16179	15.1%	40.5%	3.8	Get Rich or Die Tryin'
25	15728	22.4%	44.0%	3.7	High School Musical 3: Senior Year
26	15417	12.5%	38.1%	3.4	Little Man
27	15177	15.0%	11.0%	5.6	Alvin and the Chipmunks
28	14938	13.5%	16.0%	3.7	Super Mario Bros.
29	14431	10.8%	14.6%	5.4	Star Wars: The Clone Wars
30	13960	11.0%	10.4%	4.9	DOA: Dead or Alive
31	13812	41.3%	10.2%	8.1	Ikiru
32	13643	23.6%	27.6%	4.5	High School Musical 2
33	13377	35.3%	61.5%	1.3	Jonas Brothers: The 3D Concert Experience
34	12764	11.2%	43.2%	3.3	Dragonball Evolution
35	12625	10.5%	62.6%	2.0	Glitter
36	12596	18.3%	26.7%	4.2	Stomp the Yard
37	12560	17.8%	11.3%	6.1	Cannibal Holocaust
38	12554	11.6%	11.2%	4.8	Caligola
39	12382	13.8%	23.6%	3.8	D-War
40	12168	10.7%	12.7%	4.9	An American Haunting
41	12006	13.0%	11.9%	4.4	Stay Alive
42	11770	14.0%	18.4%	3.5	Grease 2
43	11740	20.3%	29.5%	4.1	Postal
44	11554	29.3%	10.4%	8.3	Le salaire de la peur
45	11383	14.9%	13.9%	4.6	Honey
46	11257	12.2%	10.4%	5.0	Just My Luck
47	11150	10.6%	24.7%	3.7	Prom Night
48	11054	11.4%	18.9%	4.5	The Marine
49	10942	11.0%	44.5%	2.4	Kazaam
50	10742	10.9%	77.7%	1.7	Who's Your Caddy?
51	10589	11.7%	16.7%	4.7	Meet Dave
52	10397	19.2%	11.9%	6.1	Sal? o le 120 giornate di Sodoma
53	10378	13.2%	16.6%	3.8	Bio-Dome
54	9774	12.4%	20.6%	4.0	Big Momma's House 2
55	9255	45.0%	10.2%	8.1	La passion de Jeanne d'Arc
56	9234	15.3%	10.9%	5.8	Georgia Rule
57	9145	11.1%	71.9%	1.9	Troll 2
58	9121	13.1%	11.0%	5.1	Strange Wilderness
59	9085	12.2%	16.9%	4.3	Black Christmas
60	8906	12.0%	13.0%	5.3	The Ten
61	8888	17.9%	15.8%	4.7	The Lizzie McGuire Movie
62	8840	12.0%	19.9%	3.3	Universal Soldier: The Return
63	8678	18.4%	45.3%	2.9	Material Girls
64	8654	18.5%	22.8%	3.8	Pok?mon: The First Movie
65	8486	21.4%	12.2%	5.8	Gummo
66	8419	10.1%	17.5%	3.8	On Deadly Ground
67	8378	13.3%	13.4%	5.7	Last Days
68	8210	12.7%	76.4%	1.4	SuperBabies: Baby Geniuses 2
69	8179	20.1%	58.3%	2.9	Hannah Montana: The Movie
70	8103	12.0%	52.7%	3.1	Beverly Hills Chihuahua
71	8045	12.2%	29.0%	3.6	Soul Plane
72	8030	20.3%	13.6%	4.5	Aquamarine
73	8013	41.1%	11.9%	8.1	Idi i smotri
74	7898	14.6%	32.4%	2.7	Hercules in New York
75	7854	23.1%	11.6%	6.4	17 Again
76	7804	20.6%	19.5%	5.2	What the #$*! Do Wenow!?
77	7726	26.4%	10.9%	7.1	The Birth of a Nation
78	7714	13.8%	10.4%	5.0	Annapolis
79	7684	10.7%	23.7%	3.3	Problem Child 2
80	7643	25.6%	15.0%	5.1	Raise Your Voice
81	7605	12.6%	11.2%	5.0	Eight Crazy Nights
82	7588	25.4%	12.4%	8.1	Du rififi chez les hommes
83	7563	11.4%	51.8%	2.7	Hannah Montana/Miley Cyrus: Best of Both Worlds Concert Tour
84	7499	12.6%	20.3%	4.0	Van Wilder 2: The Rise of Taj
85	7417	14.1%	12.6%	6.2	Gerry
86	7237	12.7%	55.4%	2.7	Larry the Cable Guy: Health Inspector
87	7169	11.7%	10.2%	4.4	Superstar
88	7095	31.4%	10.4%	4.9	Good Burger
89	7093	28.6%	11.3%	8.1	Z
90	7093	11.7%	75.1%	1.7	Crossover
91	7043	44.5%	10.2%	8.0	Les enfants du paradis
92	6930	23.9%	18.3%	3.9	Mighty Morphin Power Rangers: The Movie
93	6925	23.2%	13.1%	5.8	Pink Flamingos
94	6915	45.5%	16.8%	5.8	Kurtlar vadisi - Irak
95	6766	13.5%	12.9%	4.4	Yours, Mine and Ours
96	6744	10.5%	10.5%	4.2	Cyborg
97	6714	12.9%	40.7%	3.1	Are We Done Yet?
98	6687	20.9%	57.1%	3.7	Expelled: No Intelligence Allowed
99	6586	27.9%	11.6%	8.1	Tengoku to jigoku
100	6555	11.4%	36.8%	3.3	Zoom

Trivia from the list:

• Transformers 2 almost got to the list by having 9.9% 1 votes.
• The most controversial movie in every way is Jonas Brothers: The 3D Concert Experience with 35.3% 10s, 61.5% 1s, and only 3.2% everything else. It also has huge 5.0 to 1.3 under-18 gender gap, but as you can see even most teenage females aren't huge fans of it. It's also the worst scored movie on the list, with score of just 1.3, what suggests IMDB's filters think most of the 10s are attempt at ballot stuffing, and get thrown away.
• The highest rated controversial movie is Le salaire de la peur with IMDB score of 8.3 and somehow still more 1s (10.4%) than Transformers 2.
• The highest rated controversial movies that seem popular (by number of votes they received) are Fahrenheit 9/11, and Twilight, both of which I liked, and not because of the "so bad it's good effect", and The Blair Witch Project, which I hated with passion for being so unbelievably boring.
• The lowest scored movie from the list I watched was Troll 2 and I liked it because of the "so bad it's good" effect.
• Celebrity movies are huge on the list, like Crossroads, Glitter, Spice World, Hannah Montana: The Movie, and the aforementioned Jonas Brothers' movie. The only one I watched of them was Crossroads, and I genuinely enjoyed it, in semi-ironic way.

Things I hate about Stepmania

It's been such a long time since my last flaming rant. In this episode of taw's flaming flames I focus on Stepmania. And I don't intend to differentiate between Stepmania the software, DDR as such, songs, and everything else - all of it is Fair Game.

First, I hate all the delays. The only part of Stepmania I care about is the one playing the songs and checking my steps, everything else is just an extra that should try its best not to get in the way. Stepmania spectacularly fails at it - there are delay screens and delay animations everywhere, some possible to turn off, most not really. I do not need the "CLEARED" screen, I especially do not need the score spin, and then 5 second delay back to the song selection wheel. Just give me the damn score, and get me back to the selection wheel immediately when I press START! PC is not an arcade machine!

Second, slowness. I mean, the non-deliberate delays. Now Stepmania is fast enough with 50 songs like arcade machines, but real installations will have over 9000 songs - what means a few minutes to start the damn thing. What is it doing during the startup, checking for rickrolls?

A minor related annoyance is the menu system. Configuration options are divided into way too many submenus that don't mean anything. You think they do? Now quickly tell me where are the options to speed up the wheel, and turn off all the delays that can be turned off... Yeah, I fail to see the logic behind it too. And while it makes sense to make the dance mat the primary controller, it would be nice if it was possible to use keyboard sometimes too. For the sake of example, let's say I'm looking for a Zelda song I really like but I don't remember the title. Right now I have to sort by title, go to Z(elda), L(egend of Zelda), T(the Legend of Zelda) and probably ten other places! How difficult would it be to add an option of search by keyboard for those times when I'm looking for something in particular? Remember, there are over 9000 songs, it's not an arcade machine!

Oh, and there entire difficulty levels system - I have so many complaints about it! For one, what does it matter to me if a song is an 8-feeter on standard, or 8-feeter on expert? It's an 8-feeter either way, but there's no way to browse all 8-feeters together. (for every possible value of 8 of course)

And I wouldn't really mind if the feet ratings were replaced with something else - foot ratings are sort of reasonably accurate for official releases, but for fan-made ones they can be horribly wrong. I wonder if we could take this 5-difficulty diagram or something like that, measure what player is good at, and convert that to player-specific difficulty ratings... That would be cool, wouldn't it?

Now something about songs - I really hate songs that do stupid tricks with arrows display. The root of the problem is that arrows move along with the beat timer, not along with wall clock time. This causes so many problems like:

• Low bpm songs have extremely densely packed slowly moving arrows, and are hard to read
• High bpm songs have very quickly moving arrows that don't last on the screen very long and provide very little reaction time
• Songs that change bpm a lot, and you don't know when you have to hit the arrows
• Songs that freeze beat timer and unfreeze it by surprise - unless you know when the unfreeze happens you'll miss the arrows.
  

In all cases you basically need to learn the song's arrows instead of looking at them on screen. These tricks could be fun if used occasionally, but they're overused to the point of being extremely annoying.

But most of all let me dance instead of waiting and I'll be happy enough.

Arduino and Wii Nunchuck to control iTunes

Here's my latest project. It involves way too many technologies for the sole purpose of controlling iTunes with Wii Nunchuck. Signal flow is almost uni-directional, so let's go from step by step from the Nunchuck to iTunes.

Wii Nunchuck and I2C

I2C is a very simple protocol back from the 80s. There are four wires. Vcc (traditionally +5V, but lower voltages are used with I2C too) and ground are obvious. Clock and data lines require some more explanation.

I2C is a bus protocol - so multiple devices can operate on the same wires. Even in the simplest configuration we have a master (Arduino board), and a slave (Wii Nunchuck), both of which can read and write to the same lines. And it doesn't take much science to know that when one device writes 0 while other writes 1 to the same line hilarity ensues, also known as a circuit shorted, everyone dies.

The way I2C and many other protocols solve this problem is by using "open drain" design. In open drain no device is allowed to write 1 to the bus - you can only write 0 (connect to the ground), or leave the connection in high impedance state. The bus is connected to Vcc using a pull-up resistor. This way if any device writes 0, bus will be at 0. If none is writing, it will be at 1. That's somewhat unintuitive at first, if it confuses you just check the Internet for explanation.

Fortunately Arduino has hardware support for I2C over its analog input pins 4 and 5, and library for that (called Wire) is included in the distribution. So it would seem that we don't have anything to do... except it all doesn't work.

Now I'm not 100% sure about that, but here's my guess what happens. Arduino analog input pins can be set to output low, output high, high impedance, and high impedance with pull-up (allegedly 20k). For I2C we're interested in output low, and high impedance with pull-up. However - this built-in pull-up seems too weak, or it's not working for some other reason. Directly connecting clock and data lines to Vcc via 10k pull-up resistors (lower resistance = more pull-up, at least until you fry it) makes it all work. Nice way to spend an entire night, isn't it?

So after cutting the Wii Nunchuck wire, the connections are:

• Nunchuck white (GND) to Arduino GND
• Nunchuck red (VCC) to Arduino 5V
• Nunchuck yellow (clock) to Arduino analog input pin 5
  
• Nunchuck green (data) to Arduino analog input pin 4
• Clock line to VCC via 10k resistor
• Data line to VCC via 10k resistor
  

Speaking with Wii Nunchuck

Now that we have electric signals handled, we need to engage Wii Nunchuck in a meaningful conversation. It has device ID 0x52 (82), and we need to write 0x40 0x00 (64 0) to initialize it, and then request data in packets of six bytes, and write 0x00 (0) to confirm we got it.

Now I try to avoid word like "retarded" on this blog, but Wii Nunchuck data is "encrypted", and we need to "decrypt" it first:

• decrypted = (0x17 XOR encrypted) + 0x17

Seriously, whoever came up with this fully deserves this adjective.

After that it's just a straightforward decoding, and we dump data onto serial interface. Here's full code:

#define CPU_FREQ 16000000L
#define TWI_FREQ 100000L

#include <Wire.h>

int diode = HIGH;

void blink() {
  digitalWrite(13, diode);
  diode = (diode == HIGH) ? LOW : HIGH;
}

void setup() {
  Serial.begin(19200);
  pinMode(13, OUTPUT);
  Wire.begin();
  Wire.beginTransmission(0x52);
  Wire.send(0x40);
  Wire.send(0x00);
  Wire.endTransmission();
}

void send_zero() {
  Wire.beginTransmission(0x52);
  Wire.send(0x00);
  Wire.endTransmission();
}

void loop() {
  int cnt = 0;
  uint8_t outbuf[6];

  Wire.requestFrom(0x52, 6);
  while(Wire.available()) {
    if(cnt < 6) {
      outbuf[cnt] = (0x17 ^ Wire.receive()) + 0x17;
    }
    cnt++;
  }

  int b = ~outbuf[5] & 3;
  int joy_x = outbuf[0]-125;
  int joy_y = outbuf[1]-128;
  int accel_x = (outbuf[2] << 2 | ((outbuf[5] >> 2) & 0x03)) - 500;
  int accel_y = (outbuf[3] << 2 | ((outbuf[5] >> 4) & 0x03)) - 488;
  int accel_z = (outbuf[4] << 2 | ((outbuf[5] >> 6) & 0x03)) - 504;

  Serial.print("B=");
  Serial.print(b);
  Serial.print(" XY=");
  Serial.print(joy_x);
  Serial.print(",");
  Serial.print(joy_y);
  Serial.print(" XYZ=");
  Serial.print(accel_x);
  Serial.print(",");
  Serial.print(accel_y);
  Serial.print(",");
  Serial.print(accel_z);
  Serial.print("\n");
  
  send_zero();

  blink();
  delay(50);
}

Controlling iTunes

iTunes is an outrageously bad music player, but at least it has one redeeming quality of being controllable from command line via AppleScript.

By issuing silly commands like

tell application iTunes
  next track
end tell

we can control its basic functionality, what will be good enough for us.

We also need to get some information from iTunes. It doesn't have "increase/decrease volume" commands, so we need to find current volume, and then set it to higher/lower values. It also doesn't have a single play/pause command, so we need to figure out what state we're in - if we're playing then pause, otherwise play. Some of the nasty code refactored to OSA class to keep ITunes class a bit cleaner, but not by much.

The code doesn't depend on the rest of the project, so you can use it in your own projects, or just look what it's doing and use it directly from command line.

class OSA
  def get(var)
    `osascript -e 'tell application "#{name}" to #{var}'`
  end
  def do!(cmd)
    system 'osascript', '-e', %Q[tell application "#{name}"], '-e', cmd, '-e', 'end tell'
  end
end

class ITunes < OSA
  def name
    'iTunes'
  end
  def get_volume
    get('sound volume as integer').to_i
  end
  def get_state
    get('player state as string').chomp
  end
  def set_volume(v)
    system 'osascript', '-e', %Q[tell application "iTunes" to set sound volume to #{v}]
  end
  def next!
    do! 'next track'
  end
  def prev!
    do! 'previous track'
  end
  def pause!
    do! 'pause'
  end
  def play!
    do! 'play'
  end
  def pause_flip!
    if get_state == 'playing'
      pause!
    else
      play!
    end
  end
  def vol_up!
    set_volume(get_volume + 5)
  end
  def vol_down!
    set_volume(get_volume - 5)
  end
end

Reading data from Arduino

Communication is strictly uni-directional. Arduino writes to virtual serial interface, and we're reading from it as the data arrives. The data is current position of buttons, analog stick, and accelerometer. We're not really interested in any of it directly - we care about button presses and releases, and about stick going up/down/left/right (analog stick is used only as fake D-pad here). So change of state, not state as such.

For buttons it's straight forward. We get a bit for each button, so we just check it. For the stick, there's a small trickery involved. The range on each axis is from about -100 to +100, so we could just set points like +64/-64 from which we count it as a up/down. But what if stick is hold around that level? Sensor is analog (and user's hand is shaking), so if user hold the stick around +64, the reading would go +64, +63, +65, +63, +64, +64, +65 ..., what such naive implementation would treat as plenty of up presses, even though there weren't any.

This is an extremely common problem in electronics, and there's a simple solution - hysteresis. We simply take two slightly different levels for two sides of the same transition. If stick goes above +64 it's up. If it goes below +48 it's neutral. If it's betweer +48 and +64, we just keep it where it was. So unless the user has Parkinsons's and wiggles it a lot, it won't register any spurious clicks.

Other than that the code is pretty straightforward. We throw away first five readings because electronics tend to take some time after power-up to stabilize. And with every reading we set state to what we read, put all transitions into @cur_events instance variable, and yield.

require 'rubygems'
require 'serialport'

class SerialDevice
  def get_device
    while true
      dev = Dir["/dev/cu.usb*"][0]
      return dev if dev
      sleep 1
    end
  end
  def initialize
    @sp = SerialPort.new(get_device, 19200)
  end
end

class Nunchuck < SerialDevice
  attr_reader :cur_events, :b, :sx, :sy, :sxs, :sys
  attr_accessor :ax, :ay, :az
  def initialize
    super
    @b,@sx,@sy,@ax,@ay,@az,@sxs,@sxs = 0,0,0,0,0,0,0,0
  end
  def each_state
    skip = 5
    while line = @sp.readline
      unless line =~ /B=(\d) XY=(\S+),(\S+) XYZ=(\S+),(\S+),(\S+)/
        puts line
        next
      end
      if skip > 0
        skip -= 1
        next
      end
      @cur_events = []
      self.b, self.sx, self.sy, self.ax, self.ay, self.az = [$1,$2,$3,$4,$5,$6].map{|x| x.to_i}
      yield
    end
  end
  def b=(nv)
    @cur_events << :press_lo   if nv & ~@b & 1 != 0
    @cur_events << :release_lo if @b & ~nv & 1 != 0
    @cur_events << :press_hi   if nv & ~@b & 2 != 0
    @cur_events << :release_hi if @b & ~nv & 2 != 0
    @b = nv
  end
  def hysteresis(new_measurement, old_state, *ranges)
    while true
      state, bounds = ranges.shift, ranges.shift
      if bounds.nil? or
        (new_measurement < bounds.first) or
        (new_measurement < bounds.last and old_state <= state)
        return state
      end
    end
  end
  def sxs=(nv)
    @cur_events << :sxs_change if nv != @sxs
    @sxs = nv
  end
  def sys=(nv)
    @cur_events << :sys_change if nv != @sys
    @sys = nv
  end
  def sx=(nv)
    self.sxs = hysteresis(nv, sxs, -1, -64..-48, 0, 48..64, 1)
    @sx = nv
  end
  def sy=(nv)
    self.sys = hysteresis(nv, sys, -1, -64..-48, 0, 48..64, 1)
    @sy = nv
  end
  def raw
    [@b,@sx,@sy,@ax,@ay,@az]
  end
end

All of it together

The final bit of code is trivial. We just instantiate ITunes and Nunchuck objects, and connect them in obvious way:

• Press C to play/pause
• Analog stick right/left for next/previous
• Analog stick up/down to increase/decrease volume

This style of volume control sucks, but if I tried to run two osascript commands on every reading (if stick is above +64 increase volume, if under -64 decrease volume), it would be too slow, at least in this naive implementation, and you'd only see there would be a pretty big lag between moving the stick and iTunes reacting.

i = ITunes.new
n = Nunchuck.new
n.each_state do
  if n.cur_events.include?(:sxs_change)
    if n.sxs == 1
      i.next!
    elsif n.sxs == -1
      i.prev!
    end
  end
  if n.cur_events.include?(:sys_change)
    if n.sys == 1
      i.vol_up!
    elsif n.sys == -1
      i.vol_down!
    end
  end
  if n.cur_events.include?(:press_hi)
    i.pause_flip!
  end
  p(n.raw + n.cur_events)
end

Wicked Cool Ruby Scripts are not so wicked cool

I got a review copy of Wicked Cool Ruby Scripts by Steve Pugh. I had my hopes up, as I quite enjoy reading cookbook-style books on programming - they are digestible in small pieces, what works great with my Internet-induced attention deficit, and they are mines of useful tidbits of programming knowledge.

Unfortunately Wicked Cool Ruby Scripts didn't do it for me. The subtitle says "Useful scripts that solve difficult problems", but most of the scripts were targeting trivial toy problems instead, like playing rock, paper, scissors with a computer (why would anybody do that), or reimplementing grep... not terribly useful. I'd say maybe 20% of the scripts do something useful that isn't a one-liner.

Now that on its own wouldn't be enough to give the book a bad review - I might have written a few Library-of-Congress-fuls of Ruby scripts already, so basics are obviously boring to me, but there are more beginners around than people like me, so beginner books are very useful to them. But there's a second problem that bothers me a lot - it's not said anywhere but the book is clearly targeted at Windows system administrators. The scripts instead of following Unix conventions like input from STDIN and arguments, output to STDOUT, errors to STDERR and so on, ask for all the input interactively or load it from predefined files, dump errors on STDOUT, and save output to predefined files.

For example here's a script which prints all IP addresses between a starting and ending one. They way it's implemented in the book is:

class IP
  # Code here is perfectly fine
end

print "Input Starting IP Address"
start_ip = gets.strip

print "Input Ending IP Address: "
end_ip = gets.strip

i = IP.new(start_ip)

ofile = File.open("ips.txt", "w")
ofile.puts i.succ! until i == end_ip
ofile.close

But that's horrible! This script is only useful for anything when manually operated. The entire point of scripts is that they can be building blocks of bigger scripts!

The proper way would be:

class IP
  # ...
end

raise "Usage: #{$0} start_ip end_ip" unless ARGV.size == 2
start_ip, end_ip = *ARGV

i = IP.new(start_ip)
puts i.succ! until i == end_ip

That is - input from command line arguments, output to stdandard out. Unlike script in the book, this can be used as a building block for something bigger.

I'd say the book was a great idea, but a wasted opportunity. I'd only recommend it for beginner Windows administrators, for everybody else it's either too basic, or teaches some seriously bad practice.

Arduino microcontroller and Ruby to display song lyrics

I got myself an Arduino, Seeeduino version to be more exact, and I'll build a robot based on it. I'm using the word "robot" very vaguely, I just want to make a bunch of fun projects taking advantage of the microcontroller.

The first non-trivial project I did was displaying lyrics for music played from a computer. I'll get to the interesting bits in due time, let's start with the basics.

Lyrics

The first problem was finding lyrics with timing information. I really wanted it to play Still Alive from Portal, but I couldn't find any timed lyrics. And I was far too lazy to find accurate timing myself. Then I remembered - many Stepmania songs already come with timed lyrics. So I wrote a parser to their LRC format. It is much more complicated than you'd expect - we only have two lines of 16 characters each, and even silly songs are much longer than that. So there's quite a bit of logic necessary to break longer lines into pairs of 16-character fragments, and extrapolate their timing. API for this library is very simple: LRC.new(file_handle).each_line{|top_line, bottom_line, time| ...}.

class Line
  attr_reader :text, :start_time, :end_time
  def initialize(text, start_time, end_time)
    @text, @start_time, @end_time = text, start_time, end_time
  end
  def to_s
    "#{@text} [#{@start_time}..#{@end_time}]"
  end
  def text_fragments
    unless @text_fragments
      @text_fragments = []
      text = @text.sub(/\|/, ' ')
      while text != ""
        text.sub!(/\A\s*/, "")
        if text.sub!(/\A(.{1,16}(\b|\Z))/, "")
          @text_fragments << $1
        else text.sub!(/\A(.{1,16})/, "")
          @text_fragments << $1
        end
      end
    end
    @text_fragments
  end
  def text_fragment_pairs
    pairs = ((text_fragments.size+1) / 2)
    pairs = 1 if pairs == 0
    pairs
  end
  def duration
    end_time ? end_time - start_time : 2.0
  end
  def each
    fragments = text_fragments
    time_shift = duration / text_fragment_pairs
    time = start_time
    while fragments.size > 0
      yield(fragments.shift||"", fragments.shift||"", time)
      time += time_shift
    end
  end
end

class Lrc
  def initialize(fh)
    entries = []
    fh.each{|line|
      line.sub!(/\s*\Z/, "")
      line =~ /\A\[(.*?)\](.*?)\Z/ or raise "Cannot parse: #{$1}"
      time, txt = $1, $2
      next unless time =~ /\A(\d+):(\d+\.\d+)\Z/
      entries << [$1.to_i*60+$2.to_f, txt]
    }
    @lines = []
    entries.size.times{|i|
      cur, nxt = entries[i], entries[i+1]||[nil]
      @lines << Line.new(cur[1], cur[0], nxt[0])
    }
  end
  def print!
    @lines.each{|x| puts x}
  end
  def each_line(&blk)
    @lines.each{|line|
      line.each(&blk)
    }
  end
end

Seeeduino board

I'll get back to the computer-side software, let's go to the hardware for a moment. Seeeduino board is awesomely easy to use, it uses mini-USB connection for power (it can take battery power too), for uploading programs, and for emulated serial communication. No extra cabling necessary.



Top pins are digital I/O 0 to 13 (some also work as PWM / analog output, but we're not using this feature here), bottom pins are analog input and power pins. On the left there's USB mini-B, external power in, and three switches - manual/auto reset (set to auto), 5V/3.3V (set to 5V, as we need 5V on output), and USB/external power (there is no external power so it works as a power switch).

There are a few interesting diodes, that you can see blinking on the movie. Red RX and TX are for USB communication - RX is blinking every time we receive another lyrics line, we're not sending anything back here. The green diode is connected to digital pin 13, you can conveniently use it a pulse to verify that your program is actually running and not hanging up somewhere.

LCD display

This is surprisingly complicated piece of equipment. It turns out almost all one and two line character LCD displays in the world follow the same HD44780 standard. Which is quite complex. Fortunately there's a library for Arduino for handling them, whih solves most of our problems.

The first big complication is that it needs 3 different voltages - 5V for electronics, 4V for LCD backlight, and unspecified voltage for liquid crystals which regulated contrast.

We already have 5V of course, that's what USB uses, and what all the TTL uses. Backlight requires 4V plus minus 0.5V according to datasheets, but that's a lie, it's barely visible around 3.5V. I tried to cheat and put some resistor between its positive pole and 5V, but that didn't really work, I would need a very small resistor as its actual power not just reference voltage. I finally figured out how to get 4V with a proper mix of batteries - two 1.2V rechargables, and one 1.5V alkaline, giving it a grand total of, well 3.84V at the moment, I probably need to recharge them. Surprisingly AAA batteries work well enough in AA battery holder without any extra hacks.

I might be wrong, but the third voltage, needed to drive liquid crystals, seems to be just reference voltage, and not actually used to power anything. Setting it too low makes liquid crystals all black, setting it too high makes them all transparent. The idea is to have on ones black, and off ones transparent. It turns out putting 5kohm between ground and contrast pin (achieved by a pair of 10kohm resistors in parallel) works well enough. It seems to drive it to about 0.96V. I'm not sure what's the optimal level, but 1kohm and 10kohm are too extreme for convenient viewing.

The LCD circuit has 8 data pins, only 4 of which are used, R/W pin, which is grounded, so we only ever write there, and two extra control pins.

Data pins are connected to Arduino pins 7-10, control pins to Arduino pins 2 and 12, all for no particular reason, these are just library defaults.

Timing

I first wanted to put timing and lyrics into program driving Arduino, but there are two big timing issues. First, song on computer and lyrics need to start at the same time - what could be achieved either by some Arduino-computer communication, or by me pressing two buttons at the same time. Ugly but possible. The second problem is that Arduino doesn't seem to have any real-time clock, or instruction timer. I might be wrong about that, there wasn't any obvious one in documentation. It has good delay function, but driving LCD requires waiting for it, and we would need to do some nasty guessing how long it took. There's also the third problem that every new song would require uploading new Arduino program.

So I decided instead to send lyrics from computer to Arduino as they are played. It's quite nice because the same program and circuit can be repurposed as a Twitter client or anything else I want.

Arduino program

The program is unbelievably easy. We initialize USB serial line at 9600 baud, initialize LCD, and set pin 13 (driving that green diode) to output mode.

Then if any character is available on serial line - if it's NUL/LF/CR we treat it as clear screen, otherwise we display it. The display thinks it has 40 characters per line, so we can fill it with spaces and we don't need a command to move cursor to the second line. Clearing the screen resets the cursor.

If there's no serial data available we blink the diode and wait 100ms. All the heavy lifting is provided by LCD4Bit and Serial libraries.

And yes - this is C++. I forgot to mention. It doesn't have main() function. As Arduino can run only one program, and it never exits, it has two functions. setup() to start it up, and loop() which is run in a loop automatically. There are some complications here, somehow resetting the serial line causes setup() to rerun, I'm not sure why. Maybe it is a segfault even, it's C++ after all. And LCD doesn't become operational immediately, it takes a second or so and if we write in this time data is going to be ignored. But let's just skip that for now.

#include <LCD4Bit.h>

LCD4Bit lcd = LCD4Bit(2); 

void setup() {
  Serial.begin(9600);
  pinMode(13, OUTPUT);
  lcd.init();
  lcd.printIn("Ready to play");
}

int diode = HIGH;

void loop() {
  int val;
  
  if(Serial.available()) {
    val = Serial.read();
    if(val == 0 || val == 10 || val == 13)
      lcd.clear();
    else
      lcd.print(val);
  } else {
    digitalWrite(13, diode);
    diode = (diode == HIGH) ? LOW : HIGH;
    delay(100);
  }
}

Ruby lyrics driver

Ruby gem serialport provides all our serial communication needs. Just create an object using SerialPort.new("/dev/cu.usbserial-A9009rh4", 9600) (name of serial over USB device will differ), and use #write(data) to write there.

There are just a few more complications, to get good camera shots we need to wait a few seconds between initializing serial communication and starting to send data. 3s would be enough, but it defaults to 15s so I can position the camera etc.

And we want mplayer to start without any delay and without any garbage on the output, so -really-quiet -hardframedrop -nocache and redirecting everything to /dev/null

require 'rubygems'
require 'serialport'
require 'lrc_extract'

class Arduino
  def initialize
    @sp = SerialPort.new(Dir["/dev/cu.usb*"][0], 9600)
  end
  def print(text0, text1)
    @sp.write("\n"+text0+(" " * (40-text0.size))+text1)
  end
  def countdown(i)
    i.times{|j|
      @sp.write("\n#{i-j}")
      sleep 1
    }
    @sp.write("\nGO!")
  end
end

class Song
  def initialize(song_dir)
    @mp3_file = Dir["#{song_dir}/*.mp3"][0]
    @lrc_file = Dir["#{song_dir}/*.lrc"][0]
    raise "No mp3 file in #{song_dir}" unless @mp3_file
    raise "No lrc file in #{song_dir}" unless @lrc_file
    @lrc = Lrc.new(open(@lrc_file))
  end
  def each_line(&blk)
    @lrc.each_line(&blk)
  end
  def fork_mp3_player!
    unless pid = fork
      exec "mplayer -really-quiet -hardframedrop -nocache '#{@mp3_file}' >/dev/null </dev/null 2>/dev/null"
    end
    pid
  end
end

class Timer
  def initialize
    @start_time = Time.now
  end
  def sleep_until(time)
    cur_time = Time.now - @start_time
    to_sleep = time - cur_time
    sleep(to_sleep) if to_sleep > 0
  end
end

a = Arduino.new
song = Song.new(ARGV[0] || "songs/Dance Dance Revolution 6th Mix -Max-/WWW.BLONDE GIRL (MOMO MIX)")
a.countdown((ARGV[1] || 15).to_i)

begin
  pid = song.fork_mp3_player!
  timer = Timer.new
  song.each_line{|text0, text1, line_time|
    timer.sleep_until(line_time)
    puts "#{text0} #{text1}"
    a.print(text0, text1)
  }
  Process.waitpid pid
rescue
  system "kill", "-9", "#{pid}"
  raise
end

What next

If someone has Still Alive lyrics with timing, I'll upload a video of that to Youtube. I thought about Twitter client or something like that, but 32 characters make even 140 seem very very long. The display is also quite slow, so smoothly scrolling letters probably won't work too well. It might be because we use it in write only mode, so instead of waiting for busy flag to become false we simply wait maximum necessary delay. That's just a guess.

Anyway, I have plenty of things to connect to Arduino, if anything interesting ever comes out of it I'll let you know.

Medieval 2 Total War Concentrated Vanilla 0.06

After some playtesting, well playing really, the testing was just a side effect, here's another release of Concentrated Vanilla. Previous posts: 0.02, 0.04.

This release tries its best at making cavalry less overpowered. The problem isn't really how strong knights are - they're supposed to be given right conditions, just how cheap and versatile they are - all knight army beats balanced army of twice the price in almost every situation.

The first thing that needed to be done was making sieges harder.

• Walls and gates are 5x stronger. Towers are not. This leads to many interesting tactics where siege engines are used just as a part of strategy, not for one unit of ballistas to take down the walls of a fortress.
• Towers fire 50% faster, and most importantly the distance needed to activate the tower is now 8x bigger. So a single unit can activate half of the towers in a minor city, and even relatively modest garrison can keep every single tower in a citadel active. The radius used to be almost infinite in previous versions, but I think it's strategically richer this way.
• Missile infantry (but not cavalry) has twice the ammo. This is useful in field battles too, but mostly in siege defenses.
  
• Unlike in previous versions, rams are no longer specially nerfed, gates are strong enough now that it's not really needed.
• To limit the spy spam loophole, spies cost 2x more to recruit and upkeep.
• Not really about sieges, but bodyguards are now 1hp, half size, and all cavalry is 50% more expensive in recruitment and upkeep (per soldier, so bodyguard unit ends up 25% cheaper). This makes faction-specific units much more important, and factions more varied.
  

Campaign balance was significantly changed. Probably the biggest problem with vanilla, well other than the retarded AI, is that a vast compact-shaped empire is the only way to go. So you blitz to get large number of cities, and then by turn 20 nobody can realistically challenge you, unless they band up together for a crusade or something else exceptional like that. I wanted to go for quality not quantity.

• Everybody moves 75% faster on campaign map. This together with stronger settlement defenses means it's much easier to have empire in non-compact shape. It now makes sense to cherry-pick interesting settlements instead of going for the closest ones because quality didn't matter, and to defend them all later by quickly transporting soldiers from your recruitment centres.
• King's purse is 2x bigger. Not a big deal to big countries, but it helps small ones a lot.
• All buildings take 1 turn to build. Except for mines they are all 50% more expensive. This means you can spend your money on developing your existing settlements, instead of just to buying armies and expanding like in vanilla.
• Merchant trade 50% more valuable.
• Settlement trade 50% more valuable.
• Mines are 3x more expensive, and return 3x more money. This all tremendously increases profitability of good and well-developed settlements. In vanilla it was mostly about quantity.
  
• Tax effects on settlement growth 2x bigger, so low taxes are +1%, very high are -2%, not +0.5% / -1% like in vanilla.

And finally:

• Pirates and brigands spawn 4x less often, but stronger by a similar factor. It reduces annoyance significantly without changing balance.
  

So here's version 0.06. You can also grab older versions if you want.

The best MP3 player in the world

My iPod nano ended its life in a hilarious accident involving my cat and large bodies of water. I wish I had recorded it, that would be my first YouTube hit. Anyway, it was time to shop for a new MP3 player. Years ago when I first bought the iPod, there was really very little decent competition. I suspected by this time it would be much greater, so I took some time exploring the alternatives. Google, Amazon, Wikipedia, and product comparison sites were all completely useless, not too surprisingly (especially Wikipedia - it always sucks at any kind of useful product comparisons), so as the last resort I asked on 4chan. Yes, that's right, I based my purchase on advice of chantards!

And I'm happy to say - Anonymous delivered! Sansa Clip is far better than anything offered by Apple in almost every imaginable way. Let's start from the basics. MP3 players are used for two activities - listening to music (usually on shuffle) and listening to audiobooks and podcasts. iPods are completely oblivious to the second use, which is very different from the first, Sansa Clip on the other hand was built with both uses in mind and it makes a huge difference!

• Shuffle mode on Sansa shuffles only music, not audiobooks. On iPods once you had a single audiobook on it, it broke the shuffle mode and you'd get random book chapters mixed with your music. There was no workaround as far as I know.
• Sansa has option to increase speed of audiobook/podcasts. It uses the simplest algorithm which unfortunately changes their pitch, but it's infinitely less annoying than listening to the realy slow page at which most audiobooks and podcasts proceed. Seriously, it's a huge huge huge feature.
  

This two alone would be enough to make avoid iPods forever, but that's far from all. Unlike iPods which follow Appless walled garden philosophy, Sansa Clip is based on open standards.

• To connect Sansa Clip to computer a normal mini-B USB cable is used, the same as used with all digital cameras and other small equipment. So you don't need to carry a cable with you when you want to recharge it at work.
• You can simply drag and drop files to the right directories - like Music / Podcasts / Audiobooks. No proprietary software is needed. I might complain less if iTunes didn't suck half as much as it does, but it's the worst music player and music management program I've ever seen in my life. After getting disconnected from a computer Sansa Clip reads ID3 tags from all files, what takes about 30 seconds on 8GB. An annoyance, but far lesser than iTunes.
  
• In addition to MP3 other useful formats are supported - including OGG Vorbis and FLAC.
  

Sansa Clip also has some really cool extra features. Now to be honest I never cared about either, but they just come out of the box:

• FM radio
• Voice recording

It also has really nice tiny form factor:

• You don't have to keep it in your pocket, just clip it outside
• And in spite of the tiny form it does has a screen. iPod shuffle is such a retarded idea for anybody who ever plays audiobooks or podcasts, something that Apple completely ignores.
  
• Sansa's plastic body seems much more durable, my iPod nano was scratched after first week of use, Sansa Clip doesn't have any scratches
• It's a consequence of multiple design choices, but in the end I can adjust volume with separate volume buttons instead of the whole: take out of my pocket, unlock, adjust volume, lock, put back into my pocket ritual that I had to use with iPod nano. As different songs on shuffle will have different volume, this is something that needs to be done a lot, so it should be fast.
• Battery seems to last a lot longer. And as I said if it's ever low, you can recharge it without any special cable.

And finally:

• It's far cheaper. 8GB Sansa Clip is £48.91 on Amazon, compared to £97.65 for 8GB iPod nano, and £59.00 for iPod shuffle of mere 4GB. In my experience 2GB was annoyingly small for any audiobook use, and 4GB would most likely be rather smallish too. Not that iPod shuffle would be of any use for audiobooks of course.
  
• You most likely don't care, but it came with one free book of your choice from Audible. Oh yeah, it's just a generic promotional URL, they don't really check if you have the player. And it lasts until tomorrow. So in case you want one free audiobook, just grab it there ;-) You have to give them your credit card details, download one audiobook in this outrageous AA format, and then cancel your subscription. You can probably think of a way to convert it to something nice like MP3 if you need to.
  

Now to be honest there are a few small annoyances:

• Display turns itself off when it's not active instead of just turning off backlight. This makes it less useful as an improvised stopwatch than iPod nano.
• Sansa doesn't display current song's total time, only time since it started. Well, now I'm nitpicking.
  
• iPod's wheel was quite convenient for a few things compared to Sansa's buttons. On the other hand it was triggered very easily and so made locking and unlocking iPod all the time a necessity, and all the most common functions work perfectly on Sansa Clip. So I'd say it's more or less even.
  
• iPod nano kept partial logs of what was played, so I could use them to feed my last.fm account with my awesome script. I don't know if Sansa Clip keeps any.

tl;dr version (more respectably known as an abstract) - fuck iPods, get yourself far better and cheaper Sansa Clip, especially if you listen to audiobooks or podcasts.

EDIT: One problem I had with Sansa Clip, to activate it as an Audible device to work with Audible DRM you need Windows-based Audible Manager. Audible says iTunes can activate Sansa Clip as well, but I couldn't get it working. You only need to activate it once, so it's not a huge deal. In case you didn't know it yet - DRM sucks. Oh well, that one was free, I'm probably unsubscribing anyway, I don't really feel like paying for something that's DRMed even if it's pretty cheap.

The Joel Reddit Effect

Back when I still cared I spent some time experimenting with various ways to promote this blog. What to write about, how to write it, where to promote it. Google Analytics was really helpful in providing raw data to work with. Actually it doesn't provide raw data, just some standard queries, raw data would be infinitely more helpful. Anybody knows where to get it? There must be some GA-like service which provides more data, right?

One absolutely brilliant technique I found was submitting links on obscure social news sites. This is extremely counter-intuitive, but you get about as much traffic no matter how popular the site is, so I got more traffic from extremely obscure ones like Joel Reddit than from extremely popular ones like Digg. I call this "The Joel Reddit Effect".

Mathematics

So you submitted your link? Assuming it's relevant and sounds interesting, what determines how many people click on it? Number of readers (by what I mean users who use the site to read, not to promote) obviously. But also fraction of readers who will see your submission. On very small site, let's say on Zeppelin Reddit (if there is one), all readers will see your submission. These sites are really tiny and we don't care much about them. On even moderately big sites, there are too many submissions for everyone to see so it depends on number of other submissions, and on how fair the site is to your submission.

• Impact ~ Readers · Fraction of readers who see your submission
• Fraction of readers who see your submission ~ Fairness / Submissions
• Submissions ~ Readers
• Impact ~ Readers · Fairness / Readers ~ Fairness

So surprise surprise - past really tiny sites, total number of readers has nothing to do with impact. What's the Fairness factor then? On most small sites nobody is SEOing, so all submissions are pretty much equal. You need a cool title to get any clicks of course, but that's pretty obvious. On big sites like Digg clicques and SEOers monopolize the main page, so chance of normal interesting content getting to the main site are extremely slim. In the end you should expect more traffic from Joel Reddit (high Fairness) than from Digg (very low Fairness).

How to increase your traffic? Submit to as many obscure social news sites as you can.
For example I submitted post about Aumann's theorem to Psychology Reddit as it seemed to me most appropriate, and Redditors there found it quite interesting. And I've found out that llimllib submitted to to Cogsci Reddit, as he consider it more appropriate. In the end I got twice as much traffic. I often see traffic from sites I've never heard about - someone decided to submit my posts, and the traffic followed.

This model is only really applicable to Reddit/Digg-like sites; Delicious, Stumble Upon etc. follow completely different rules.

How Robin Hanson increased my conviction that healthcare works

Divide asked in comments to my post "Which crackpot cult to join" why I didn't make any snarky remarks about Eliezer and the rest of the Overcoming Bias crew. Well, here they come.

One of the most peculiar beliefs shared by the Overcoming Bias crew and most of the readers (as estimated by the comments) but hardly anyone else is "Aumann's agreement theorem", which informally says:

If two people are perfect Bayesian rationalists, they cannot agree to disagree.

Like most Bayesian theorems it's mathematically flawless, the problems start when you try to apply it to the actual world. The bastardized version which is really popular on Overcoming Bias is:

After discussion between two non-perfect Bayesian rationalists, their views should be expected to be closer to convergence than before.

It sounds quite reasonable - the more pro-X person shared some pro-X evidence, the more anti-X person shared some anti-X evidence, and how could knowing some extra pro-X evidence could possibly make you more anti-X than before or vice versa? At worst it will make no difference, unless someone was really horrible at discussing.

Still with me? Great! Now let's apply that to Robin Hanson and effectiveness of healthcare. As you might already know if you're a regular OB reader, Robin Hanson believes that healthcare spending is largely wasted and ineffective, he also blogs about it a lot. So according to bastardized Aumann's theorem after reading some of his posts I should be less convinced about healthcare than before, right? Yet it had the opposite effect on me, and I believe my reaction is rationally Bayesian.

Discussion is not random evidence seeking

The problem with bastardized Aumann is that it models discussion as evidence discovery. If I did random research and found out result more consistent with healthcare not working, this would indeed make me less convinced about efficiency healthcare. But discussion is not unbiased evidence discovery! Here's a model I like a lot better:

• P(healthcare works) = 50% - or any other number, it only matters in which direction it goes
• P(Robin skillful at discussing) = 90% - I have little reason to suspect lack of skill on Robin's part
• P(good evidence against|NOT healthcare works) = 90% - if it doesn't work, there should be some good evidence against it
• P(good evidence against|healthcare works) = 10% - if it works, there will probably be no good evidence against it working, weak evidence will almost certainly exist
• P(Robin's posts convincing|Robin skillful at discussing AND good evidence against) = 90% - if good evidence exists, and Robin is skillful enough, he will most likely use it correctly and his blog posts will most likely be convincing
• P(Robin's posts convincing|NOT Robin skillful at discussing OR NOT good evidence against) = 10% - if good evidence doesn't exist, or alternatively Robin fails at arguing, his blog posts will most likely not be convincing
  

It's just a simple Bayesian network. The only direct observation I have is P(Robin's posts convincing) being false. I find his arguments extremely weak, the kind that I might care a tiny bit about if I found them randomly, but ones you can nit-pick about pretty much anything if you look long enough. Now let's see how it updates my posterior probabilities.

• P(Robin skillful at discussing|NOT Robin's posts convincing) = 83.3% - down from 90%
• P(good evidence against|NOT Robin's posts convincing) = 16.7% - very significantly down from 50%
• P(healthcare works|NOT Robin's posts convincing) = 76.7% - significantly up from 50%
  

So reading failed arguments against healthcare, I believe more in healthcare, contrary to Robin's intentions. If I didn't really believe in Robin's discussion skills (10% instead of 90%), updates would be very small, but still in the same direction:

• P(Robin skillful at discussing|NOT Bad Robin's posts convincing) = 5.8% - down from 10%
• P(good evidence against|NOT Bad Robin's posts convincing) = 47.7% - slightly down from 50%
• P(healthcare works|NOT Bad Robin's posts convincing) = 51.9% - very slightly up from 50%
  

I'm not really going to discuss particulars of the healthcare case, but the general heuristic is:

If someone who's good at convincing tries to convince you about X and fails, it is a good evidence against X.

This effect is only strong when you have reasons to believe convincing arguments would be used. If you expect appeal to emotion or other low value arguments (like when a politician speaks to uneducated voters), and get it, it's really low evidence (but still against). If you expect to get strongest arguments available, but get something very weak, that's some very good evidence against.

Medieval 2 Total War Concentrated Vanilla 0.04

Here's another release of Concentrated Vanilla minimod.

A few iterations of playtesting went into it. Sieges should now hopefully be more balanced and a lot more fun. All changes related to sieges and battles:

• Towers and are always manned as long as there's any unit behind the walls they're on. In multi-walled settlements you need to be within proper walls.
  
• Rams are at 50% attack and health.
  
• Walls and gates have 4x the strength. Towers have only 50% extra strength, but because they're always active you will want to actively destroy them instead of just scaring the AI away from them to deactivate.
  
• Towers have 50% faster fire rate.
• All missile infantry has twice the ammo, so they can defend settlement easier, and are much more effective skirmishers against heavy infantry not supported by cavalry.
  
• Bodyguards are half the size and 1hp only. Having cheap and extremely strong heavy cavalry that early basically made all factions the same, nerfing bodyguards makes factions a lot more different again.
  

It all increases value of artillery (for siege attack), heavy infantry (for siege attack and defense), missile infantry (for siege defense and skirmish), light cavalry (as cheap anti-skirmisher force), and actual heavy cavalry units (because you cannot rely on bodyguards). Light infantry can be used more effectively as cheap but effective settlement defense, and using spies to open gates and protect against other spied doing the same are more important. Missile cavalry doubles as anti-skirmisher light cavalry and cheap skirmish troops. Bodyguards are massively nerfed. This all makes battles more interesting, especially sieges.

Campaign changes are mostly to make campaign more dynamic, make money important again, and reduce micromanagement:

• Everybody moves 75% faster. That's about the right value for the vanilla map. I wanted to make ships 100% faster but land units only 50%, unfortunately it's not possible the way M2TW is implemented.
• Taxes have twice the effect on population growth. Low = +1%, high = -1%, very high = -2%. No longer is "as high as possible without getting riots" the best tax strategy.
• Resources are 50% more valuable, so trade, merchant trade, and mines are worth more.
  
• All buildings take 1 turn to build. It's now about money.
• All buildings except for mines are 20% more expensive.
• Mines are 3x more valuable, and cost 80% more money. It makes them hugely important.
  
• Rebels and pirates spawn 4x less often, it seems to make them a lot stronger when they actually spawn.
  

It really makes vanilla a lot more dynamic - because units are faster you no longer have to attack the closest settlement mindlessly, you can invade Venice from Egypt if you want to. As resources and especially mines are now a lot more valuable, and big cities can be developed with money, importance of particular cities is massive. In vanilla a settlement is a settlement, as long as it's close enough to ship armies there, in Concentrated Vanilla a good city is worth starting war for, no matter how far it is from you. And they're also much harder to take, especially well-manned bigger cities, Fortresses, and Citadels, making it all far more interesting.

Here's download link to the most recent version 0.04 . You can also get older versions if you want.