The best kittens, technology, and video games blog in the world.

Monday, October 20, 2008

Put your variables on diet

Svanspervot on a Chair by Steffe from flickr (CC-NC-SA)

There are all kinds of categorization schemes for programming languages, by paradigms or checklists of supported features. Categorizations criteria tend to be highly subjective (e.g. "builtin support for regular expressions"), useless (e.g. "significant indentation"), or both (e.g. "does it have a standard").

I want to propose a new categorization - objective, easy to evaluate, and at the same time exposing something very deep about programming languages.

I will divide languages into:

  • thin variable languages - where variables refer to data.
  • fat variable languages - where variables contain data. Variables can also contain references to data, but there's a distinction between direct and indirect access.


This division is very old. Assembly language is obviously a fat variable language, even though its variable system is very simple - registers and memory locations contain stuff directly, or contain references (memory addresses) to stuff. As languages need to be compiled to assembly plenty of high performance languages follow this road. Fortran and C variables are just assembly variables plus types. C++ didn't break up with it, it made container variables much fatter and much more complicated - RAII, copy constructors, assignment operators, and all the related mess. Java in spite of superficial similarity to C++ is definitely a thin variable language.

Thin variable languages are also very old. The original Lisp was the first language with thin variables, and all Lisp dialects, just like all ML and Haskell dialects, are thin variable languages. I don't think a single seriously functional language uses fat variables.

All object-oriented languages, most popular of them being Smalltalk, Ruby, Javascript, and (let's be charitable) Java - use thin variables too. There are some fat languages like C++ and Perl with objects bolted on top of them, but they are definitely not object-oriented, they just support some limited object functionality.

Scripting languages are interesting. Old languages like Unix shell have very fat variables, even though all variables are simple strings. Perl and PHP continue this tradition, but Python and Ruby are soundly in the thin variable camp.

Observations


Having categorized all popular languages let's do some observations.

  • All (pure and impure, strict and lazy) functional languages are thin variable.
  • All honestly object-oriented languages are thin variable.
  • Thin variable languages and garbage collected languages are very closely related categories. There are some reference counted languages in both camps (Perl, PHP on thick side, Python on thin side), but there seem to be no thin language with manual memory allocation or thick language with full GC.
  • All segfaulting languages (assembly, C, C++) use fat variables, but many fat variable languages are non-segfaulting (Fortran, bash, Perl, PHP).
  • Dynamic typing is on both sides (Perl/PHP vs Lisp/Ruby).
  • Explicit static typing is also on both sides (C/C++ vs Java).
  • Implicit static typing is only no the thin side (ML/Haskell) and is actually quite popular there.
  • Almost all thin languages have closures. A few languages like Python and Java have less than full closures, in form of named inner functions or anonymous inner classes. In both cases it's a syntactic not semantic limitation.
  • Almost no thick language has closures. A big exception is Perl, which has full closures.
  • Lexical and global scope exists on both sides, in almost every language.
  • Dynamic scope is unusual, but is supported on both thick (Perl), and thin (some Lisp dialects including the original Lisp, Emacs Lisp, and Common Lisp, but not Scheme) side.
  • Rich literal notation is supported (Perl vs Python/Ruby/Lisp/ML/etc.) and not supported (C/C++ vs Java) on both sides.
  • Macros exists only on the thin side (Lisps, Dylan, Nemerle). There doesn't seem to be any obvious reason for it.


I could go on. It actually surprises me how many semantic differences follow the thin vs fat divide, with Perl and Java being the biggest outliers (and also their derivatives like PHP and C#). These outliers are very interesting. Java's lack of power is definitely syntactic not semantic and there is plenty of JVM languages which are little more than fully compatible alternative syntaxes for Java with more expressive power. Nothing like that ever happened to popular fat variable languages like C/C++, which fail for semantic not syntactic reasons.

The biggest outlier on the fat side in Perl. While Perl was able to get almost 100% score on supported features checklist, it seems to be an evolutionary dead end. Every new thin variable language steals ideas from Perl, but Perl 6 effort was never able to transform the language, and Perl programmers have been leaving for Ruby and Python for years now.

If you're writing a new language today, and every programmer should do that, just forget about fat variables. They have one big advantage of allowing explicit memory management, what can still result in more memory efficient programs, but that's about it. Expressiveness of fat variable languages have been pushed to the limits by Perl, and it seems it cannot be pushed any further. Thin side is already far ahead, with Ruby, Scheme, Haskell and all the small research languages you've never heard of.

Monday, October 06, 2008

Rome Total War strategy guide

cat&stone by Ray Chang from flickr (CC-NC-SA)
I'm not sure why but all strategy guides I've seen on the Internet suggest gameplay very different from how I've been playing. It's quite likely that on higher difficulty the strategy I'm proposing doesn't work any more, so you should listen to them instead of me.

In a way of full disclosure I play with fog of war turned off (~ toggle_fow), Follow AI movement turned off (in menu when you start), and battle camera movement restriction turned off (~ toggle_restrictcam). I don't think it affects difficulty at all, it mostly makes interface nicer.

You already know map of Europe, and turning off FOW doesn't reveal enemy units or buildings, so you still need some spies to learn that. Battle camera restrictions make giving orders to your units in battle much harder, especially when the battle gets bigger or you get some reinforcements. It's just user interface pain in the ass, so I got rid of it. I also played with low unit size not to fry my GPU, but it seemed to handle it quite nicely, so maybe I'll up it to medium next time.

Economy

I think the most confusing part of Rome: Total War interface is city income. Real city income is revenue minus garrison cost. Income displayed by the game is revenue minus total cost of your army divided proportionally to city population. This leads to bullshit numbers, as revenue is not growing proportionally to population. So totally ignore city income figures.

You want as much revenue and as little spending as possible. Two most important revenue sources are sea trade, and land trade. Build best ports you can, and paved roads (or even highways) as early as possible, and get trade right with everyone. Trade-enhancing buildings like traders, market etc. are also very useful. Trade revenues grow only slowly with city size, so you basically want to have as many cities as you can, even if they're quite small, as long you they can build paved roads or port (that's level 2).

The second biggest source of income is farming and mining. Because it doesn't depend on city size at all, it's better to build basic farming and mining everywhere you can before you build advanced farming/mining. For 4000 you can build two mines in two cities, producing 400/turn. If you built mines+mines upgrade in one city instead it would cost you 5500 and produce only 350/turn. It's quite similar with farms - it's cheaper to have a little everywhere than to concentrate them all in one place.

Your next major source of income are wonders. Hanging Gardens give you +20% to farming income, unfortunately it's on the edge of the map. Colossus of Rhodes is even better as gives you +40% to naval trade income and is often barely defended by very weak Greek Cities faction.

Taxes aren't worth that much. There's some tricky equation to make them grow much slower than population, so big cities produce maybe ten times less in taxes than you'd expect. And in any case tax income is going to be much lower than trade income, and as higher tax rates reduce growth and public order I normally set all tax rates to Low by default. In smaller settlements it's important for the sake of growth, in larger settlements for the sake of public order. If you don't want to grow the settlement any more and it has high public order you could set tax rates higher, but it's usually not worth the bother.

Public order

It's important to keep your public order high, otherwise citizens can rebel damaging population, buildings, and your soldiers, brigands will spawn more often, and the whole city may even rebel throwing your garrison away. If a big city with a lot of military buildings rebels they will have a very respectable army and will be a major pain in the ass to retake.

Public order is determined by many factors. First is distance to capital, with very distant settlements getting even as much as -80%. So move your capital close to center of your empire every now and then and try to conquer in one nice block, not random settlements all over the map. Starting positions of Carthage, Greek Cities and Parthia are pretty bad for this reason. It will also make your empire easier to defend, another reason to try having solid empire. There's even a Javascript app to determine best location for your capital.

Another is culture penalty. I'm not exactly sure how it's calculated, but it's based on buildings from foreign cultures. There are six cultures - Barbarian (Britons, Dacia, Gauls, Germans, Scythia, Spain), Carthaginian (Carthage, Numidia), Eastern (Armenia, Parthia, Pontus), Egyptian (Egypt), Greek (Greek Cities, Macedon, Seleucid, Thrace), and Roman (Brutii, Julii, Scipii, Senate), and conquering within your culture should give you no penalty.

Government buildings and religious buildings having the biggest effect, so the first thing you want to do to reduce it is destroying any religious building they might have and building your own. Then upgrade all buildings you can, especially government building, as upgrading removes culture penalty for them. If you cannot upgrade, you can sometimes destroy and rebuild, what's going to cost you some money but it will work. Unfortunately you cannot destroy many buildings, especially the important government buildings, so you might need to live with some culture penalty forever.

The next important factor is unrest. Unrest starts high when you conquer a city, and then goes down by 5% until reaching the base level, but only if public order is at least 75%. So if you can get a conquered city to 75% by temporary measures like large garrison and good governor then just by staying in the city for a few turns the unrest will disappear. On the other hand if you cannot even reach 75% you will have pretty much permanent problems with the city. Well, not really permanent as rioting kills of citizens, what solves your problem slowly, but if you think the city is going to be a problem, just enslave it, and by the time it grows back it will have no unrest. Basic unrest rates are 0% for most cities but can be 15% for many and even 30% for a few like Jerusalem.

The next factor is squalor, based on city size. Big cities simply have high squalor, and it makes people unhappy. That's why when you're conquering a new city, with high distance to capital, high culture penalty, high unrest, and high squalor, it will be totally ungovernable. Better to enslave or even exterminate its population to reduce squalor, then quickly raze their temples and build your own to reduce culture penalty, keep large garrison in it for a few turns to reduce unrest, and keep capital close to center of your empire to make distance to capital penalties manageable.

To increase public order you can build buildings, especially religious buildings which usually give you +10% increase per level. Some give you only +5% and some other bonus, so you need to decide if the extra bonus is worth losing 5-25% of public order (depending on city size, unfortunately you cannot change your mind easily later, you'd need to raze the temple completely and start from shrine). Depending on culture there are some other buildings like execution square, odeon, tavern etc.

Wonders are also helpful here. Statue of Zeus at Olympia gives you permanent bonus to all your settlements, capturing any wonder gives you a high bonus for a few turns but it wears off too quickly to be worth relying on.

Influence rating of the governor improves public unrest, but you won't have enough governors with high influence for all of your empire. I never did it, but maybe you could try a shitty governor and building an academy for him to get better stats. Governors can also get useful retinue from religious building.

A very important way of improving public order is garrison. The most important fact is that the bonus is based on ratio of total number of troops to total population. So 30 peasants are worth as much as 30 cataphracts. But everything including Town Guard is significantly more expensive per person than peasants - so seriously use only peasants as a garrison. Using serious combat troops for garrison in all your cities is going to cost you a lot of money, and you cannot even use them for crushing rebellions, as then you'd need another way of keeping your public order up.

Also - only use garrisons in small to medium settlements. In a small settlement, one stack of peasants can give you +80% garrison bonus (you can never go higher than that) and will cost you 100 denari/turn. In a large settlement, one stack of peasants will give you less than +5% and sitll cost you 100 denari/turn. Every other method costs the same no matter the settlement size. So it simply makes no sense to use garrison as a way of keeping large settlements happy, unless you positively have to. Much more effective ways are low taxes, religious buildings, governor influence, games, wonders etc. Oh and having absolutely no troops in a settlement gives it extra -15%, not to mention making it super easy to lose, so keep at least some peasants in all of them.

Tactics

Most strategy guides talk a lot about advanced unit types, but unless you play very slowly you will only get them in the end game, when you're so strong that it doesn't really matter that much.

A very important unit type are peasants, which you should use exclusively for garrisoning, but never for battles. Peasant garrisons are half the upkeep costs of the weakest battle units, so use them. If you have some spare peasants you can take them with your army so they can serve as an occupying force for the captured settlements, while your real forces go forward. Or maybe if someone needs to hold battering rams and you have pure-cavalry army, but that's already getting somewhat dubious.

One of the nicest things about Rome: Total War is richness of unit types and tactics you can develop with different combinations of unit types against different combinations of enemy unit types.

Many tactics can be formed around phalanges. First you take a wall of any units capable of forming a phalanx, which most factions can get almost immediately. Then you make either your opponent run into the wall of spears, or attack your opponent with this wall. If you set guard mode on your units will try to keep the wall of spears, if you set it to off they will break it to attack the enemy. Either can be useful depending on circumstances.

Even a phalanx formed with very cheap units can kill most of enemies. Non-phalanx infantry, and light cavalry will simply get massacred. With heavy cavalry, elephant, and chariots it's not that simple. Depending on your and their strength they might be killed by your phalanx (instant win) or break it (instant lose), but often they will simply get entangled in it without getting killed. Now it's time to attack, either with adjacent phalanx unit, or even better with your cavalry attacking from a side or a back. You will have some loses in your cheap phalanx, but you'll kill a much more expensive heavy cavalry unit this way, for a major win.

To fight with other phalanx you simply form a pushing match. If you have stronger phalanx, especially one with longer spears, you can outstab the enemy. If not, just attack the enemy phalanx with some cavalry while they cannot move.

The main problem with phalanges is that they are very vulnerable from flanks and back, so you want to keep some cavalry, or at least some non-phalanx infantry on both wings. Not only will it protect your phalanx, they can attack and destroy any enemy captured by it.

If enemy has some cavalry, you might want to take all your cavalry, charge into his, and try to destroy it before the infantry battle begins. Cavalry has very high charge bonuses so by properly maneuvering you can do disproportionate damage even without very strong units. If they rout, don't follow but go back to wings of your formation - you won't catch horses with horses, and your cavalry is too important to waste just yet. Even if the enemy reforms they won't have much morale left and you can rerout them later with a simple charge.

It's important to have cavalry superiority, even though the cavalry won't win the battle for you. If you charge at a phalanx, or even straightforward melee infantry, you're going to suffer heavy loses. Cavalry is strong because of high charge bonus, if they're entangled they suddenly become very vulnerable. There are some units you can safely charge at - missile units like archers/bowmen, peltasts/velites/skirmishers etc.; very light infantry like rebel peasants; other cavalry units, including generals (who are weak without their bonus charge); and later artillery units. Charging at anything else means high loses even with relatively heavy cavalry, and is outright suicidal for light cavalry. Even if some unit is weak, you can still suffer if there's a strong unit in its proximity. So obviously bowmen shouldn't be left without protection.

Another important thing you can do with cavalry is killing off routing infantry units. The difference between defeated army suffering 30% loses refilled in the first visit to nearby city, and 90% loses that are impossible to refill comes from your cavalry. Normally infantry cannot catch routing infantry, and cavalry cannot catch routing cavalry, so the only kind of massacre you will be able to do is cavalry massacring routing infantry. Even the cheapest light cavalry is very important just for this purpose.

If you have cavalry superiority you can get some missile units, or later some artillery. They are very weak against cavalry, and it's not too easy to protect them using just infantry after the chaos of the battle starts. Your tactics depends on range of your missile units, so usually it makes no sense at all to have multiple kinds of missile units in the same army. Either take all bowmen, or all peltasts, or all onagers, or some other consistent army.

Missile units cause free damage to your opponents. They can try to stand it out until you run out of ammo and suffer heavy casualties, or else they'll be forced to attack you on your terms. If the enemy has no cavalry and no missile units (or you just killed them with your cavalry), you can even send your missile units in skirmish mode way in front of your main units to pepper them with arrows and javelins. Normally however you want to keep them close to your units, either behind your main infantry line, or just in front and then making them fall back when the attack finally happens.

Missile troops alone are not going to win the battle. But with enough damage the enemy will be losing morale and will rout more easily. After some of the enemy starts routing the battle is pretty much won, and just use your cavalry to kill off what's left. Missile units are also quite useful at finishing off routing units.

There are a few more tactics you can try. Romans have strong infantry units that throw javelins before attacking. It's like an ok missile troop and a good melee troop in one unit. If you stand against enemy phalanx with Roman infantry try splitting your troops and attacking from flanks. It might work just fine.

Chariots are like heavy cavalry except you can have them very early in the right faction like Egypt and Seleucids. They have even lower defense than normal cavalry, so try charging through enemy unit to the other site, not on enemy unit. Also, never let enemy missile units fire at your chariots, or anybody to attack a standing chariot. It takes some practice to learn how to use chariots properly, but it's fully worth the effort.

If you have mixed heavy and light cavalry and want to charge a strong unit, charge with your heavy cavalry first, immediately followed by the light cavalry. This reduces your loses a lot. On the initial charge will still be capable of defending itself, so a heavy charge is needed. Just afterwards it will be completely disorganized so even light cavalry will be able to make a kill.

Sieges

You're not fighting for the sake of fighting, you're fighting to conquer, so many battles will be sieges. The good news is that in early game most settlements will at most wooden wall. Just make one battering ram, and here we go. Easy way to win a siege is having missile units and cavalry. Take battering ram and missile units to the gates, and fire at anybody standing next to the gate. Because unlike in a field battle they're not going to charge at you, if you're patient you can empty all your ammo on them and inflict severe loses, no need to hurry with the ram.

Once it's done take your cavalry and charge at everybody. Hopefully there aren't any strong phalanges in the vicinity of the gate now that you peppered them with arrows. Everything else will just rout when charged. It's important to kill them off now, otherwise they'll run towards the town square and reform there. As you go toward the town square just charge on every unit you find on the streets. AI tends not to offer any organized resistance on the streets, so it's going to be an easy kill. You may find some serious resistance in the town square, hopefully by this time you inflicted sufficient damage that it's just a matter of finishing everyone off.

In my experience for settlements without at least stone walls, AI defending army is a hell lot weaker than the same army would be on a field.

It's much much worse if they have stone walls and some missile units to man the walls. Because they're higher and protected by walls they're easily ten times as deadly as your missile units shooting at them. Add to that fire coming from towers, and reinforced games slowing down ramming a lot, and you can lose a quarter of your army before even coming inside.

You can obviously starve them, forcing them to attack you, but that's going to be very slow. Fortunately there's a much better way. First you need at least one hole in the walls. In can be gate opened by your spy, or a sap point, or hole made by your onagers if you already have them (onagers come late in the game, but so do stone walls). Then charge with all your cavalry, the more and heavier the better, and go towards the city center as soon as possible. By not wasting time on the gates and moving all your units at an instant you won't get fired upon that much. If you take the city center the enemy has 3 minutes to take it back or they lose and all die, so they'll have to go off their comfy wall posts and fight on the street, and they really suck at it.

If you make more holes you can choose one that will take you away from the walls as soon as possible, and that's the least defended. If you have some artillery you can even try to shoot at units protecting the hole to make it easier for your cavalry to charge over them. Unfortunately bowmen won't do, as enemy bowmen have better range while on the walls.

If this siege tactic won't work, well starving them is always an option.

I wouldn't depend on city defenses too much. Walls delay the enemy, but you still need enough field army to defeat him. Otherwise he can just starve you off.

Strategy

In Rome: Total War there's a very smooth transition between high level strategy and battle tactics, with both level affecting each other. Your strategic choices will have major impact on your battles, and your battles will have major strategic implications.

Unfortunately diplomacy is pretty much broken, and all AIs will attack you sooner rather than later, so you cannot go to the very high level and try forging one alliance against another alliance. Sadly it's like that in almost every strategy game I can think of. At least everybody against the player is better than everybody pussying out from every fight and letting you do whatever you want like in Colonization. So don't think too much about diplomacy.

You should have some idea on how you're going to play. In my last game with Egyptians I took all my armies, bought all the mercenaries I could, and expanded in three directions at once, west to Siwa and Cyrene, east to Petra, and Bostra, and north-west to Kydonia, and Sparta. Initial battles were tough, but this way I had a decent empire before anybody could even react. This plan was very ambitious, and it's quite likely that it might not have worked on higher difficulty levels, but it's a good idea to have some plan before starting to play.

It's important to move early, as you get some decent army to start with, often with units that you won't be able to make for a long time, while everybody's armies are still quite small and many settlements are barely defended, especially the rebel ones.

It's plenty of fun to start a war, but before you do think how you're going to end it. Can you take all enemy settlements quickly? Or at least all that are nearby, if the enemy is all over the place like Greek Cities, Carthage, and Parthia.

If you don't think of it, you might get into a long trench warfare that someone else will use to backstab you. In my last game I wasn't thinking enough about this and at one point I had wars with Parthia, Armenia, Seleucids, Pontus, Greek Cities, Macedon, Thrace, and all four Roman factions.

Your armies are investment. Don't keep them in secure cities. Actually don't keep them in cities at all - peasants should patrol cities, all your armies should be in the field fighting. Upkeep is very high compared to initial cost. Typically cost is about three turns worth of upkeep. So if you have a unit that's going to spend four or more turns doing nothing it's cheaper to disband it and make a new one when it's needed. You cannot make them as fast as you'd like to and you'd lose all experience, so maybe the threshold is higher than four, but armies should fight.

If you need some troops fast, and are far from your lands you might want to get some mercenaries, but normally cheaper and stronger units are available in your cities. Exception are units that you cannot produce but would like to, like Sarmatian Cavalry, and Createan Archers, depending on your faction.

Bribing is even more dubious practice. It costs more to bribe than to recruit an army (even mercenary army), defeat whoever you were thinking of bribing, and disband just afterwards. Bribing should be thought of as a really special case situation. Maybe if you want to bribe generals, because you cannot normally recruit them. But you can recruit a unit, take it into a fight, and you have a chance of getting "man of the hour" free general. I got plenty of generals this way.

Navy is expensive. You can do two things with Navy. In early game you want to transport armies from one coast to the other in a single turn (or otherwise as quickly as possible), so they are never vulnerable. There's no point fighting at this point, unless someone is trying to blockade you and you really have no choice.

The other thing is total domination of a sea. If you have plenty of money you can buy enough ships to take total control of a large body of water, like "whole Mediterranean sea east of Sicily". Just buy loads of ships, attack everywhere it in, then you can blockade everything, and safely transport all your troops.

There's little reason to go between the two extremely. Semi-strong navy that won't protect your transports is just throwing money away. If your navy isn't useful any more, just disband it.

When you're waging a war on a strategic level it's important to achieve local superiority of force. It doesn't matter who's got more units on the entire map. What matters is who's got more and better units in the place of battle. So go completely out of proportion, committing all your armies you can to the small part of the front where you want to achieve your objectives, like conquering enemy city, or destroying his not very strong armies. If AI has weakly protected settlement, take your big army and take it. If it has medium sized army in the field, crush it with army twice the size. If it took all its strongest units and heads towards your capital, raid his territory and force it to go back home.

You will be much more mobile if you have paved roads, a bit of navy, and no sense of fear. It's even more important to have all your battle troops on the front, not back home patrolling streets.

The biggest factor however is economy strong enough that you can easily raise new armies and replace your loses while the opponent cannot, even if they manage to win a few big battles, because that's how Rome became great!

Monday, August 25, 2008

Black hat SEO on Amazon

What happens after mom spends a weekend in Mexico by Malingering from flickr (CC-NC-ND)

Amazon started as a bookstore, then after a brief period of being a patent troll it became an generic Internet shop, now it seems to be turning more and more into a shopping search engine, with thousands of sellers using it as an intermediary between themselves and the customers. I'd really enjoy the fight for shopping search engines between Amazon-style thick intermediaries and Google Products-style thin intermediaries. For now I must say Amazon is much more useful than Google Products, but maybe some day with some open standard shopping metadata the fight will be more even.

And just like every search engine Amazon starts to see the problem of black hat search engine optimization. Some time ago I was looking for earphones extension cord on Amazon, and the first results was one by Belkin with cost of £0.01. That's a pretty good price except there was a shipping cost of £4.50 per item, so they were basically abusing the search results screen.

Right now it seems Belkin isn't doing this anymore. I guess someone at Amazon noticed the abuse and wrist slapped them. Amazon as a thick intermediary has plenty of power over the sellers to control abusive behavior. It would be much more difficult for Google Products to enforce good behavior. Here's my prediction:

If thin intermediaries for shopping like Google Products ever become widespread, abusing behavior like hiding shipping costs, transaction costs, taxes and other fees will become as common as it is now in the airline industry.

Sunday, August 03, 2008

Highly innovative YCombinator startups

Lolcat by taw based on Blogging for Cats by Vicki's Pics from flickr (CC-NC-SA)

A couple of weeks ago Paul Graham wrote a list of startups Y Combinator wants to fund. That's an interesting list, but I'd rather like to know what kind of startups they actually funded, and even better what kind of startups that they funded became successful. Every step on the road from a plan to a success can change the list quite significantly.

While I was somewhat interested in it, up till now I was also too lazy to do the research. Fortunately a recent thread on Reddit did it for me and provided an official list of Y Combinator startups that are considered "successful", for some definition of "successful". I'm not sure if "getting bought" is really the apex of success - it basically means you lose all control over your startup and work at a huge corporation again, exactly what you were trying to escape. At least unless they pay you enough money to buy a small island and retire, but press releases rarely mention how much money the founders got out of the deal.

So here's my list of Y Combinator successful startups with a brief explanations of what they're all about. The descriptions might be less that 100% accurate, but that's the impression I got from 5 minutes of interacting with their websites, and if an user doesn't get what your website is about in 5 minutes then fix your website, don't blame the user.

  • Infogami. Wiki-style website builder. I don't see what's the big deal.
  • Reddit. Just like Digg except without all the Digg users, so at least at first comments were of higher quality. Now that it got bigger it's just like Digg, except that you don't have to waste your precious time writing descriptions of your submissions and have more time to read how Ron Paul used Lisp to impeach Bush. I guess I shouldn't be so cynical as I'm using it as my main source of the news and one third of my readers come from Reddit, but cynicism is one of Terms & Conditions you need to agree to when you start a blog.
  • ClickFacts. Online ads fraud detection. I don't know how well it works, but it looks the most innovative of them all. By the way isn't it kinda ironic that geeks do ad-supported startups and block all ads at the same time?
  • Zenter. Web clone of PowerPoint, website died after Google bought them.
  • Auctomactic. Another user interface for eBay.
  • Parakey. Some sort of "Web operating system". They were bought by Facebook, but their website is dead, and it's not obvious if they actually built anything beyond buzzwords.
  • Anywhere.FM. Website where you can upload MP3s you own, play on your other computer, and then pay RIAA for it. So much for "A cure for the disease of which the RIAA is a symptom".
  • Omnisio. YouTube clone with a bunch of extra nifty features for cutting and annotating videos. They have nice screencasts. Too bad you cannot annotate Colbert with it, that would be awesome.
  • Xobni. Microsoft Outlook plugin for social networking. Kinda client side version of what Trampoline Systems is doing on server side. The obvious problem with pure client side solution like that is that you probably already know about your direct contacts, and you won't learn anything about your second-level connections without access to your first-level connections' mailboxes. So I'm not sure how much value there is it in, not to mention it only supports one bad email client on one bad operating system.
  • Scribd. You thought PDFs are bad? Now you can upload them to Scribd and make them even worse by displaying them in a shitty Flash applet. Scribd has a distinction of being the Y Combinator startup with highest chances of making the Web even worse than it is now. I seriously hope they never succeed.
  • Wufoo. Online builder for web forms. Nifty little app.
  • VirtualMin. Just like WebMin except on the Web. I think the main problem with most shared hosting/DNS registrars s not lack of GUIs, but too many crappy GUIs and limitted control over direct shell access or some sort of API.
  • SnapTalent. Use targeted online ads for recruitment. I like the idea, I just don't think it's likely to work, as contrary to the popular opinion most smart people do not work at Google. They probably don't do it, but it made me see a fresh wave of startups approaching, all based on a:visited security hole. a:visited is a fundamental problem with the web security architecture and isn't going to be fixed anytime soon, as people like marked visited links and CSS too much, and the only simple way to fix it is getting rid of either one of them. So from now on every website has access to your full history, and I know that you've visited: but not: Digg Reddit YouTube RedTube XTube YouPorn Facebook Orkut Adult Friendfinder. (it doesn't require JavaScript or cookies to track users, CSS background images are good enough). Architectural problems like that are fascinating. Even when you think you've covered every possible security issue, somebody comes up with an attack that cannot be fixed except with major rearchitecture of the Web. My favourite example is the Epilepsy Foundation attack (Encyclopedia Dramatica is for things Wikipedia doesn't have balls to write about). I don't think anybody expected that an animated GIF could be more than an annoyance and actually cause people seizures. No way to fix it now without breaking the Web, just like there's no way to fix a:visited attack. I'm getting sidetracked here, but my dear readers seem to like it when I'm getting sidetracked.
  • Justin.TV. Another YouTube clone. My first impression is that it's a lot less interesting than Omnisio.
  • Weebly. Another website creator.
  • Loopt. Let friends spy on you with their mobiles. Some people might find it usefull, but I don't get the whole mobile thing. Skype, email and IM are vastly superior for communicating with people than phone calls and SMS texting, cost a fraction of them, and most of the time people do have Internet access so they don't need mobiles. I just don't get it.
  • RescueTime. A service which records your browsing habits to sell to advertisers and CIA, I mean to help you avoid procrastination. If they convince people to install it they have the chance to be the new Google and earn billions on ads.

So most of them seem follow pattern of "just like X only Y", where Y more often than not equals "online". Don't get me wrong, I'm not saying that's a bad thing - most of the progress in the human history follows the "just like X only Y" pattern, and with sufficient number of "only Y"s you can create something that creators of X never dreamed of.

It's kinda proving that Paul Graham was right saying the idea doesn't really matter, as in most of these startups it was good execution not brilliant idea what made them successful. So go ahead and start your startup.

EDIT: As some people on Hacker News noticed, I was wrong, and justin.tv is for live streaming with webcams, not for recording YouTube-style videos using webcams as I guessed from looking at their website and reading their FAQ. The point of the article was not a detailed review of every startup on the list, but getting a general idea of what kind of startups YCombinator funds. Sorry if I haven't been clear about that. My conclusion that most of the startups follow the "just like X only Y" still follows, in case of justin.tv for X="Youtube" and Y="live not prerecorded".

Sunday, July 20, 2008

Making me think about usability

That's Funny by ehpien from flickr (CC-NC-ND)

I've just finished reading "Don't Make Me Think" by Steve Krug, which claims to be some sort of a web usability classic.

What I liked most about it was the chapter about user testing. The short summary is - test early, test often, 3 randomly selected people with one hour per person is the best cost to benefit ratio for a round of testing, don't bother with detailed written reports, just fix the worst problems and do another round later.

Other than that I didn't like the book that much. I see how it might have been useful in the Web 1.0 days, but its focus just feels wrong. It seems to think that the most important part of a website is its navigation and so spends most of the pages discussing it, but how relevant is navigation when vast majority of users come from Google, or some other kind of search, or direct links, or RSS, or even bookmarks, and people reaching your content by patiently clicking their way though from your homepage are a tiny minority?

Most blogs, wikis, and web apps don't even have real navigation structure, and they're the part of the Web that's growing fastest. Even Amazon which the book glorifies as The Paragon Of Web Usability seems to have mostly given up hierarchical navigation and now exposes search and personalized recommendations as the most prominent means of getting around.

So the book has some useful content but is hopelessly outdated and unfocused. 3 stars out of 5 if you numbers.

Usability



As the review made a rather short blog post I'm going to conclude it with somewhat controversial remarks on Web usability. By the way if what you're saying doesn't cause any controversy then
  • you've just discovered some new universal truth (only happens in maths),
  • or you're just repeating some nicely sounding but ultimately meaningless slogan like "use the best tool for the job",
  • or most likely - nobody reads your blog.

So controversy and exaggeration to get the point across are good.

First, accessibility is overrated. I've seen so many people performing magic rituals supposed to make their websites "accessible", like anal adherence to "web standards" even to the point of using XHTML, making sure the website works without Javascript and/or CSS, making unique text in headers and so on. Two things they never do are measuring how many users are actually going to use accessibility features, and testing accessibility in realistic settings (like with a screen reader or Braille terminal), and these two things seem to me to be the only ones that would matter.

Accessibility zealots also often claim that:
  • websites done in some "right" way are somehow magically accessible,
  • making a website Google-friendly is somehow the same as making it accessible,
  • and that you're legally required to spend as much effort making your website accessible as necessary.

All of which are bollocks.

I think this misguided thinking is caused by a combination of two reasons. The first is the good old web standards zealotry. Seriously, why people get so emotional over some badly written documents without even a reference implementation or a serious test suite? Even FreeBSD vs Gentoo and Emacs vs vim (by the way correct answers are Ubuntu and TextMate, and 2 spaces) flamewars aren't as heated as the ones about web standards.

The second reason is a certain kind of a reality distortion field. There are many kinds of web sites on the Internet. The kind that used to be most popular are "text-based content sites", where users come to read static stuff. The reality distortion field makes people think this kind of websites is still dominant, but it's not. Looking at Alexa's top list it's difficult to find a single text content site other than Wikipedia. Everything is some sort of a web app, or video site, or social networking site, or anything but text content.

Now I mostly agree on one point - making text content site reasonably "accessible" is not very difficult, and to some extend correlates with things you need to do anyway to support Google, mobiles and so on. I will still laugh at people who think they can get "accessible" without genuine testing (if accessibility checklists worked, why won't they write IE6 accessibility checklist, wouldn't that be so much easier?). But the problem is something else - only reality distortion field of epic proportions can make you think most websites are still text content.

In a way it's similar to DBAs being deluded that most SQL databases serve as integration point between multiple independent codebases (and so need stored procedures, triggers, complex constraint enforcement and business logic on database side etc.) when in reality 99% of them are little more than persistent store for a single app and these features are more harmful than helpful.

So how about the vast majority of web sites that are not just about text content? Oh sure, you can make them accessible. It's just going to be long and painful process with very little payoff, and which won't ever end in a full experience parity unless your application is very very simple. And you know what - screw the blind people! The largest discriminated group are people who don't speak English as their first language. It's about 95% of all people, a lot more than 0.6% who are blind. Why don't they care adequate attention? It's an insane misallocation of resources if you care about the blind but don't even have a German version of your website (and versions in at least 50 other languages). Blind people are more or less as important as Welsh localization, and you're probably not caring much about that, are you?

Forget about experience parity, no mater what technology. Vision is the most important of human senses and the only way to make non-visual interface work as well as a visual interface is by horribly dumbing down the visual one, hurting 99.4% of your users who can see. Try getting somewhere with a spoken Wikipedia article about London instead of a map. Or reading long lists of numbers instead of a graph. Or playing Portal with a screen reader. Or for more relevant example using "basic html" version of Gmail instead of the real thing.

My second point, video websites should not autostart their videos. Haven't they heard of tabbed browsing? To open multiple tabs with YouTube videos you need to:
  • open the tabs,
  • go to first tab,
  • wait until the video partially loads and starts playing,
  • click pause (you cannot click pause earlier),
  • then do the same for the all other tabs,
  • and finally go to the tab you want, rewind the video and click play.

Couldn't they at least detect they're in an inactive tab somehow? Firefox seems to be sending onfocus when a new page is opened in a new tab or when you first switch to an inactive tab - not autostarting until you get onfocus sounds like a good start. Making pause button work before the video loads would also help a lot. Or are there perhaps some Firefox plugins or Greasemonkey scripts to get rid of video autostarting? The only thing it's good for is rickrolling people.

The third thing, stop using lowest common denominator fixed width layout. CSS lets you do many things that weren't possible with tables, but some of them are just stupid. More and more people have widescreen these days. Widescreen means wide but short. OS and browser usually take a lot of space on top and bottom of the screen, but not on the left and right, making screen aspect ratios even more extreme. So there's lot of space on the screen but it's mostly horizontal, and what do so many designers do? Completely ignore half of the available space, and then fill the rest with ads and other crap so the useful content is taking a tiny fraction of what it could take. I know making "design" work in multiple sizes is harder, but design isn't what users are interested in. They came to your website for the content, not to revere your mad design skills.

And the last thing, CSS compatibility with IE6 is a waste of time. JS compatibility became an almost non-issue thanks to jQuery and other JS frameworks. So why not use a JavaScript solution like ie7-js and be mostly done with IE6 CSS too? Some people won't like the suggestion of depending on JavaScript, but IE6 does run JavaScript just fine, and no other browser (Google spider, mobiles, text browsers etc.) is going to be affected in any way, so what's the big deal?

OK, if JavaScript dependency on ie7-js still makes you feel uneasy, couldn't we at least somehow use for automated testing? Run the website with and without it, get a list of differences and suggestions based on how ie7-js fixed them. Seems like a lot better way than manual debugging. But I'm not a Windows guy, so don't expect me to code that.

The New Law of Demeter

Baby cat by fofurasfelinas from flickr (CC-NC-ND)
Once upon a time I've stumbled upon The Law of Demeter, which says that you can only call methods on:

  • self
  • self's fields
  • current method's parameters
  • objects you created
  • global objects

It's often shortened to never use two dots in a row. The implication of this law is that if you legitimately obtained order object, then order.customer and order.customer_name are valid, but order.customer.name is not.

At first I interpretted this law literally and (just like you did a moment ago) reached the obvious conclussion that the idea is total bollocks - writing (or even worse autogenerating) thousands of proxy methods like def customer_name; customer.name; end or even delegate :name, :to => :customer is not going to improve quality of your code at all.

It sounded like thousands of those little silly rules like "Use factories instead of constructors" that Java code monkeys make up all the time so they can think "If only everybody adhered to these rules, Java coding would be just fine" and keep blaming other people for their problems instead of blaming themselves for being so daft and sticking with Java. The same Java monkeys don't even adhere to their rules, and so Java coding isn't fine, but even if they did adhere it wouldn't help as you cannot fix technical problems on cultural level. So I forgot about the whole thing and moved along - another stupid rule for people who are afraid of the code.

Of course if this was how the story ended I wouldn't even bother blogging about it. On the contrary, I had a sudden realization - sure, the Law of Demeter has many obvious counterexamples of perfectly valid code that's illegal according to it, and would be made much worse if it was enforced. But most of them go away once it's subtly reinterpretted. There's a loophole for "objects you created" which doesn't have to mean "object you created by ClassName.new" - every object that was created at your request and the producer doesn't hold any reference to it any more can be considered to be "created by you" (old objects whose ownership was relinquished to you should also qualify, but this is a very rare case and I'm not going to talk about it any more). Let's call this interpretation "The New Law of Demeter" and apply it to an exaggerated example:
message = hash.keys.sort_by(&:abs).map(&:to_s).join(" ")

Naively interpretting the Law of Demeter it should not only be illegal but sent straight to the Guantanamo Bay - there are 4 explicit and 2 implicit dots in it, all in a single line.

But this is perfectly sensible piece of code. It does something useful and it's definitely far saner than hash.keys_sorted_by_absolute_value_stringified_and_joined_by_spaces, hash.keys_sorted_stringified_and_joined(&:abs, " ") or any other Demeter transmogrification I can think of.

But why is it really illegal? Every single dot creates a fresh object - Hash#keys does so, and so do Array#sort_by, Numeric#abs, Array#map, Numeric#to_s, and Array#join. The code doesn't violate the New Law of Demeter in any way!

The Law of Demeter is much more permissive when interpretted this way, but I wouldn't stop here yet. I really think order.customer.name is perfectly sensible even if customer is owned by the order. What counterargument some proponents of the Law make? Mostly that it makes customer's name harder to mock if you ever want to make order return customer name not tied to any genuine customer. But why should Order#customer be limited to returning real Customer objects? How about making it return read-only snapshots of current customer? This is exactly how order.customer.XXX is used vast majority of the time. As snapshots don't need any write behaviour, it's very easy to return a snapshot with overridden #name, and as a customer snapshot is a fresh object you've just created, you are allowed to call any methods on it you want.

So let's pretend that every time you do order.customer you really mean order.customer_snapshot, and code like this becomes legal:
puts order.customer.address.postcode

The New Law of Demeter lets you do a lot more than the old, including legalizing almost all sensible view code, but it still delegalizes a lot of dubious code. For example code like this is definitely not legal, as it expects read-write objects, not fresh snapshots:
order.customer.first_name = "Bob"

This is also not legal, as snapshots should be fresh objects and not be auto-updatable:
customer = order.customer
order.add(item)
puts customer.balance # item included or not?

but it can easily be transformed into this legal code:
order.add(item)
customer = order.customer
puts customer.balance # item definitely included

Reinterpretted this way, the New Law of Demeter is really useful at detecting code smell. Unfortunately it also becomes much more difficult to apply, as you need to look not only at the syntax but also at the sema an object to which the original creator still holds a reference; and does the method affect state or does it have snapshot access semantics and so on. I'd love to see a Demeter lint which would try to check compliance with The New Law of Demeter in a meaningful way.

Monday, July 14, 2008

All the Firefox extensions you need

I love my tail! by play4smee from flickr (CC-NC)

Here's the third edition of my insanely popular (who am I kidding, people only come here for angry rants and Ruby tricks) series on all the Firefox plugins you need to turn it into a useful browser.

The Firefox world has changed a lot since the first (May 2006) and second (May 2007) editions of this list - and now with Firefox 3 released an update is very much in order.

The Web the way it's supposed to be


Internet out of the box is crap. It's full of ads and sites which require registration, bookmarks disappear when you switch computers, and Google Images links to tiny thumbnails instead of images it's supposed to. All these annoyances can be cured by aggressive Firefox plugin therapy.

First install Adblock Plus. Filterset.G Updater isn't recommended any more, so you can skip this step. Adblock Plus blocks vast majority of ads of all kinds (flash, banner, text, css pseudo-popups etc.) without disrupting too many websites (in fact crappy ads are much more likely to disrupt a website than Adblock Plus). The reason it works so well is that on almost all websites ads are provided by a third party external request (image, iframe, script etc.), so it's very easy to tell the difference between website content and ad content. Technology used by Adblock Plus wouldn't really work against other kinds of unwanted content like malware, spam, porn etc., because there's no first party-third party separation in their case. It also won't that well work against ads on search results pages, but these tend to be less obnoxious. The only website of any use that I found not working with Adblock Plus was TubeJP. If you need to fix TubeJP or any other website, try adding "@@tubejp.co.uk" exception to your Adblock Plus's list (@@ means exception) and reloading the page. It should work. Or you could completely disable Adblock Plus from Adblock Plus Preferences, and reenable it when you're done.

Once you've done that let's move to the next problem - websites which require registration. I'm getting an impression that they are fewer in number these days, but in case you find one it's good to have BugMeNot extension installed. Just right click on the form, select "Login with BugMeNot", and the annoyance hopefully goes away. Or not if they're good at deleting shared accounts, as YouTube seems to be. If it doesn't work you can always register with a disposable email address and crappy random data. They usually only check that the email address is routable and you selected that you're older than 18, so happily tell them you're Albert van der Putin, a 95 year old gold farmer from Thailand, living at 1337 Eric Cartman Street, New Buenos Aires, post code 12345 or whatever. It's in their best interest to let you do that, as this kind of crap is easy to filter. If they actually try to verify that what you're filling in makes sense then just google some realistically looking address and phone number and use them to make their lives more difficult when they try to analyze their user data.

Now let's make your bookmarks work. You probably already have a del.icio.us account. If not, create one before you go on reading this post. Then you could test your new del.icio.us account by bookmarking this blog for instance. del.icio.us Firefox extension disables browser-side bookmarks and replaces them with server-side bookmarks that you can access from everywhere and share with others. To bookmark something press Control-D, and most of the form will be prefilled for you - URL, page title, and even suggestions for keywords based on how others tagged that page. It's just an insanely fast way of creating del.icio.us bookmarks. For accessing bookmarks you can use methods provided by the plugin but personally I always type http://del.icio.us/USERNAME/TAG in URL bar instead. Much faster if you know what you're looking for.

The next plugin requires some work to get it right, but as you'll spend significant portion of your online time with Google services you might want to use it anyway. It's CustomizeGoogle and it's used to (surprise) customize various Google services. There's a nice two-minute screencast showing some of the basic options, and there's a lot more of them.

Some of the things you might want to enable are:
  • Disable ads on services where they don't make much sense. You can leave them on Google Search but disable on Google Images etc., any way you wish.
  • Switch sensitive services like Google Mail to SSL.
  • Make Google Images results point directly to images instead of the thumbnail+frames page.
  • Make search results AJAX-stream. Especially important for Google Images.
  • Add links to other search engines on search results page. I mostly use it to search Wikipedia.
  • Hide Google Mail spam counters and invite box, which are simply annoying.
  • Filter your search results.
  • Enable Google Suggest.
  • and much more.

Google won the search engine wars, the web mail wars, the image search wars, the map search wars, and for that matter almost any wars it got into, so we're spending a lot of time with Google things that it'd really be a horrible waste not to customize them a bit.

Making Firefox a better browser


The Web just became a better place, so let's apply a few fixes to the browser. First let's get rid of the downloads window, which is an ugly remnant of the pre-tabs days. Opera uses a downloads tab which is a far better solution than a window, but we can do even better than that with a Download Statusbar. If Firefox developers won't incorporate something like that in Firefox 4 that's a definite proof they're really a sleeper cell of al-Redmont and they should be ie6boarded until they start to cooperate.

Crash recovery (aka session management) was greatly improved in Firefox 3, but it's still better to use Session Manager which saves sessions also on exit, so you won't have to killall -9 firefox as a crude "Save Session" button.

Just these two extensions plus the ones I mentioned in the previous section are probably sufficient for most users. To add some finishing touches you may want to install ColorfulTabs for color-coded tab labels (switch to URL-based colors and fading 7 for best results), Fission for Safari-style progress bar integrated with URL bar, Tabs Open Relative for new tabs that open next to the current tab instead of the end of the tab list, and ReloadEvery for web site autoreload functionality like in Opera.

User Agent Switcher lets you pretend you're using some browser other than Firefox. Fortunately web sites requiring Internet Explorer are pretty much thing of the past. The only one I've had problems with in the last year was one supporting Firefox 2 but not Firefox 3 beta, what was really stupid, but they've fixed it already.

Unfortunately at the time of writing this post TinyURL creator for Firefox. didn't support Firefox 3. That's a shame as it was a very useful plugin.

Web development


If you do any web development at all, and as you're reading this blog there's a high chance you do, you absolutely positively need to install Firebug. It's more important for web development than Ruby on Rails, TextMate, jQuery, and Selenium taken together. Of course that's no reason not to use Ruby on Rails, TextMate, jQuery and Selenium together with Firebug. Here are some screencasts to get you started.

In addition to Firebug there are a few small plugins you might want to install, like Web Developer which provides a right click menu with some extras not included in Firebug, like manipulating cookies, disabling Javascript and so on. Nothing really big, but having all these small nitfy functions in one place is useful.

If you're using Windows you may want to use IE Tab. This of course doesn't give you Firebug for IE, and unless it lets you run multiple versions of IE simultaneously (does it, I have no way of testing that?) it's not even that useful for compatibily testing.

XPath Checker helps you write XPath expressions for page elements and therefore prepare some test cases for view layer unit testing somewhat more easily.

Web 2.0 Integration


CustomizeGoogle, del.icio.us, and BugMeNot are technically about website integration but I don't really think of them this way - they're just too fundamental to the way the Web works.

If you use Twitter like I do you will probably want to install Twitterfor extension, which is very simple and gets Twitter integration done right.

If you have too much time to waste you will want to install StumbleUpon plugin. Just vote up or down a few pages and you'll start getting your own personalized reading list. By selecting the right categories you can even use it for personalized lolcats or porn or lolcat porn.

If you have a Gmail account, Gmail Notifier is very useful for getting notifications.

Total Conversion


Some mods are standalone applications living inside Firefox, for things like IRC, BitTorrent, FTP and so on. They're usually not as good as the best standalone applications, but are fully crossplatform, so if you're switching your systems often you may want to take a look at them anyway. The only one I've used extensively is ChatZilla which seems to be the best cross-platform IRC client out there.

Non on the list


I've heard that Greasemonkey is absolutely awesome, but I never quite got to trying it out.

NoScript seems to be quite popular, but it didn't get on the list, as I really don't see a point of using it for web browsing (what next, NoCSS, NoHTTP1.1 ?), and for compatibility testing Web Developer extension already provides this functionality. By the way it doesn't provide you as much privacy as you think, plain CSS can leak your browsing habits quickly enough.

One plugin that you should definitely disable are "Ubuntu Firefox Modifications". It's automatically enabled by default on Ubuntu and the only thing it seems to do is breaking right click functionality on many websites.

Final words


That's the list of Firefox plugins I'm using. Of course your needs are different, so your list won't be exactly the same. I hope you've found some suggestions in this blog post and I'd love to hear back from you on how you're using your Firefox.

As Firefox is changing quite rapidly I'll probably post an updated list in a year or so, perhaps incorporating some of your suggestions.

Sunday, June 15, 2008

How to create value by blogging

He's so cute when he's angry! by rockygirl05 from flickr (CC-NC)

I'm hardly blogging these days, so I though I'd at least open a Twitter account - and here it is, and some stuff I've done recently like London Barcamp 4 and getting OLPC XO-1 laptop weren't blogged at all, just twitted. This is all caused by a certain problem with blogging - people really like to read long blog posts. Just look at the "Popular posts" sidebar - it's long post after another. People even like insanely long blog posts like those by Steve Yegge. But long posts take too much time and energy, so most of the posts written by me and other bloggers are pretty short. So I somehow thought that writing stort blog posts isn't really worthwhile, and tweeted instead.

Today I thought maybe it's time to verify it, so for some hard data I collected statistics from Google Analytics (unique page views) and del.icio.us (number of bookmarks except of mine), and divided my 202 blog posts into ten buckets depending on their plain text size.

And indeed - it seems that almost nobody cares about the short blog posts and the longer the post is the more people read it. The difference is even more pronounced when counting del.icio.us bookmarks. I feel that the number of del.icio.us bookmarks is a much better indicator of post's "value" than page views, as page view is generated before the reader even seen the post, while bookmark is generated only after it was read and decided to be valuable. Search good vs experience good.

BucketPostsAverage sizeAverage page viewsAverage bookmarksAverage page views per kBAverage bookmarks per kB
12016742177620.01261.4
2206890151311.02241.6
321459010364.92451.1
42034564963.81491.1
52028033142.31140.9
62021293011.11460.5
72016564021.92381.1
82111882230.21770.2
9207892761.23991.7
10204211430.23840.5


This confirms my beliefs and disproves the commonly held ADHD theory of blog readers which states that most blog readers have very short attention spans and would much rather look at the kittens. It seems that to the contrary, reader really love long posts. At least my readers. You'll still be getting kittens of course, my blog would look quite empty without them.

On the other hand a completely different picture arises when page views per kB and bookmarks per kB are measured. Bookmarks per kB is pretty flat, while page views per kB is going down fast. So if kBs of text are a good measure of blogger's effort then the best way of generating value is writing tons of stort posts.

I'm so undecided. Is it better to write fewer long posts, many of which would be big hits (relative to the blog popularity of course, this isn't I CAN HAS CHEEZBURGER), or rather many posts which would generate less value per post but more value overall. I'm kinda writing for myself, but I still think if the post would be valuable to the average reader or not before posting it. I should probably simply keep posting instead of thinking too much.

Thursday, June 12, 2008

Bolting Aspect Oriented Programming on top of Python

What's this? by Steffe from flickr (CC-NC-SA)
The bigest difference between native support and bolting things on top of a programming language is that you can only bolt so much before things start to collapse. In C++ even strings, arrays, and hashtables are bolted on - and while they work just fine any interoperability between different libraries using strings, arrays, and hashtables is almost impossible without massive amount of boilerplate code.

In Perl and Python these basic data structures are native and well supported, but the next step of supporting objects is bolted on. So the objects work reasonably well, but metaprogramming with them is very difficult and limited (in Python) or outright impossible in any sane way (in Perl).

Ruby takes a step further and real object-oriented programming is native, so people can bolt other things on top of it like aspect-oriented programming. AOP in Ruby (before_foo, after_bar, alias_method_chain, mixins, magical mixins, many method_missing hacks etc.) works reasonably well, but I wouldn't want to bolt anything on top of that, or the whole thing would fall apart.

This is the problem with bolting stuff on - bolting stuff on in a valid technique (just like design patterns, code generation and other band-aids), and bolted-on stuff like objects in Perl/Python or arrays/strings/hashtables in C++ do work, they're just infinitely less flexible than native types when it comes to further extending.

But I really miss AOP in Python. Multiple inheritance can kinda simulate very weak kind of mixins, but is rather cumbersome to use. I wanted to write a test suite using aspect-oriented mixins, but there were simply so many super(omg, who).made(up, this, syntax) calls that it looked as painful as Java inner classes. So I thought - would it already collapse if I added a very simple AOP support?

It turned out not to be so bad. Here's a distilled example. There's a bunch of classes inheriting from BaseTest. Their setup methods should be called from superclass down to subclass, while their teardown methods should be called from subclass up to superclass. If there are multiple AOP methods on the same level all should be called, in some consistent order (I do alphanumeric, order of definition would be better but Python metaprogramming isn't powerful enough to do that). It's also possible to override parent's AOP methods (you could even compose AOP method override using super if you really needed). Or you could override the whole setup/teardown method if you really needed - this is very flexible.

class BaseTest(object):
def setup(self):
aop_call_down(self, 'setup')
def teardown(self):
aop_call_up(self, 'teardown')

class WidgetMixin(object):
def setup_widget(self):
print "* Setup widget"
def teardown_widget(self):
print "* Teardown widget"

class Foo(BaseTest):
def setup_foo(self):
print "* Setup foo"
def teardown_foo(self):
print "* Teardown foo"

class Bar(WidgetMixin, Foo):
def setup_bar(self):
print "* Setup bar"
def teardown_bar(self):
print "* Teardown bar"

class Blah(Bar):
def setup_blah(self):
print "* Setup blah1"

def setup_widget(self):
print "* Setup widget differently"

def setup_blah2(self):
print "* Setup blah2"

def teardown_blah(self):
print "* Teardown blah1"

def teardown_blah2(self):
print "* Teardown blah2"


The output of a = Bar(); a.setup(); a.teardown() is exactly what we would expect:
* Setup foo
* Setup widget
* Setup bar
* Teardown bar
* Teardown widget
* Teardown foo


The more difficult case of b = Blah(); b.setup(); b.teardown() is also handled correctly - notice that setup of widget mixin was overriden:
* Setup foo
* Setup widget differently
* Setup bar
* Setup blah1
* Setup blah2
* Teardown blah2
* Teardown blah1
* Teardown bar
* Teardown widget
* Teardown foo


The code to call make it possible isn't strikingly beautiful but it's not any worse than some of my Django templatetags.

def aop_call_order(obj, prefix):
already_called = {}
for cls in reversed(obj.__class__.mro()):
for name in sorted(dir(cls)):
if name[0:len(prefix)+1] != prefix + '_':
continue
if not already_called.has_key(name):
yield(name)
already_called[name] = True

def aop_call_up(obj, prefix):
for name in reversed(list(aop_call_order(obj, prefix))):
getattr(obj, name)()

def aop_call_down(obj, prefix):
for name in aop_call_order(obj, prefix):
getattr(obj, name)()


aop_call_order returns a list of methods with names like prefix_* defined in obj's ancestor classes in order of Python's multiple inheritance resolution, falling back to alphabetic if they're on the same layer. Overriding a method in subclass doesn't affect the order, making the "Setup widget differently" trick possible. aop_call_down and aop_call_up methods then call these methods in straight or reverse order.

Of course like all other multilayer bolted-on features, it's going to horribly collapse if you use it together with other metaprogramming feature. If you don't like that - switch to Ruby.

Coming up next - bolting closures on top of Fortran.

Saturday, May 10, 2008

Relax, fuel is cheap

Photo of my cat Cloud, for no particular reason (public domain)
As a proud Cornucopian I'm tired of baseless claims of neo-Malthusian peakniks that cheap oil is over and it's time to eat dirt and die.

If you think I'm exaggerating, and haven't been on the Internet for the last five years or so here's a typical example of a Peaknik Doomsdayer:

Peak Oil. It's bigger than terrorism, global warming or genocide. It's the end of your way of life. [...] Which means if you don't live by your farm, no food for you. There won't be much food anyway. [...] 4 billion people will not survive. [...] So what can you do to prevent peak oil? Nothing. Seriously, nothing.

Fortunately these claims don't withstand scrutiny. Fuel is much more affordable than anytime in the history except for 1990s where it was not only very affordable but insanely cheap.

I hereby present my Fuel Affordability Index, which compares fuel affordability to standard of 100.0 in 1975. Fuel affordability calculates how many miles you can go on average salary. To calculate it you multiply new car fuel efficiency in mpg, gdp per capita and divide by crude oil prices. We can dispute stuff like gdp per capita vs median household income, crude oil prices vs retail gas prices, new car vs average car fuel efficiency and so on but they don't fundamentally affect conclusions so I just took whatever was easiest to find. Data is for USA, mostly because I couldn't find historical fuel efficiencies for any other country. I guess the conclusion would be even stronger for EU as European cars are more energy efficient and strong euro makes crude oil cheaper than in US.

YearFuel AffordabilityGDP per capitaFuel efficiencyCrude oil prices
1975100$19,96213.5$47.63
1976113$20,82614.9$48.36
1977119$21,57015.6$49.88
1978140$22,53116.9$48.17
197997$22,98717.2$71.96
198084$22,66620.0$95.50
1981105$23,01121.4$82.70
1982126$22,35022.2$69.33
1983147$23,14822.1$61.34
1984168$24,59822.4$58.14
1985196$25,38623.0$52.56
1986394$26,02823.7$27.66
1987342$26,66823.8$32.81
1988443$27,51924.1$26.45
1989381$28,22623.7$31.05
1990315$28,43523.3$37.17
1991372$28,01123.4$31.15
1992405$28,55923.1$28.81
1993493$28,94323.5$24.36
1994552$29,74423.3$22.19
1995540$30,13123.4$23.09
1996465$30,88623.3$27.38
1997541$31,89123.4$24.40
1998885$32,83723.4$15.35
1999662$33,90823.0$20.83
2000421$34,77022.9$33.39
2001517$34,70123.0$27.29
2002536$34,93123.1$26.61
2003460$35,47923.2$31.62
2004356$36,43323.1$41.84
2005287$37,20623.5$53.77
2006257$37,92823.3$60.73
2007244$38,34023.4$64.92


Notes: Crude oil prices adjusted to 2007 dollars. GDP per capita in 2000 dollars. Different basis doesn't affect the results as only ratio is taken. Fuel efficiency is combined urban+highway, for all cars except trucks (so if I understand it correctly without SUVs too).

As you can see fuel is much more affordable than in 1970s or early 1980s. So as the civilization in 1970s very much existed, it will continue at current fuel prices, or even at prices significantly higher than current. One thing I expect to start happening about now is further increase in average car fuel economy - as soon as fuel became cheap in mid 1980s cars stopped improving, but hybrids are much better than 23.4 - the popular Toyota Prius has combined mpg of 46 - almost double the current average.

Even if people keep buying the same cars and the economy stays stagnant fuel would have to become 2.44 times as expensive in order to bring fuel affordability back to 1970s level (which if you're old enough to remember, weren't end of civilization). If people start buying hybrids (and they will) and economy grows at 3% a year (and it will) for the next 10 years, even $420 barrel won't reduce fuel affordability to below-1970s level.

Relax, fuel is cheap.