Subtle changes is Web architecture

Today I'd like to discuss some things that has been happening to the Internet lately, and I'm talking about the subtle stuff, not the well known things like AJAX, Comet, and Javascript that doesn't suck.

wget --mirror no longer works

Simple mirroring crawlers like wget --mirror used to work very well, and you could easily get a static snapshot of most websites. The way it worked was extracting and following all links, like <a href=''>, <img src=''>, <style href=''>, <script src=''>, and <link rel='stylesheet' href=''>. When you think about it - there weren't that many kinds of links.

Now it fails for most websites. Links from Javascript, and CSS @import are so prevalent, it's harder to find a website without them than one with them. In theory wget could be patched to get it back to supporting 90% of websites, but unfortunately nobody seems to care, and it's in C, not the most popular language amongst Web developers.

Per-website settings no longer work

It used to be possible to tell browser to treat some website in a special way. If you wanted not to send cookies to Google Search, or to proxy-anonymize your traffic to porn torrent search website, or make Comedy Central think you're watching Colbert Report from US not UK, you'd set it on domain basis. It won't work any more, as too many websites use multiple domains on the same page, serving media files or Javascripts or anything else they feel like serving from different domains. They could even share domains with other unrelated websites, like getting Flash videos from youtube or images from S3. This means per domain settings are no longer a good enough substitute for per-website settings, and no other equivalent is widely available.

With most modern browsers and tabbed browsing it's not even that simple to open different instances of a browser, and switch settings per instance, you'd need to use Firefox for normal browsing, Opera for some extra privacy, and Safari for porn, or something like that.

Email spam is no longer a problem

I don't mean there's no more spam, there's plenty of it - but good web email clients
got just unreasonably good at filtering it into Spam folders, something none of desktop-based solutions of 5 years ago could do. So nobody needs to obfuscate their email address any more. I don't think spammers really cared about obfuscation anyway, do you think they're too dumb to use regular expression like /@|\s*\[\s*at\s*\]\s*/i? Putting your email address in captcha would work a lot better, but pretty much nothing supported it anyway.

Anyway I don't care any more, and I don't see why anybody else would. My email address is Tomasz.Wegrzanowski@gmail.com and you can send more viagra ads to my Spam folder if you want to.

Life of honest bot makers is harder

In just a few short years Captcha infected the entire Internet. They don't give any protection against people who make money on breaking your website - Indian captcha breakers cost something like a dollar per 1000 captchas, and if that didn't work specialized computer algorithms seem to be catching up with human solvers quickly enough.

The main result of captchas is making life harder for honest bot makers like me. Yes, I like screenscrapping sites and extracting fun info from them. Do you think I made the list of most popular blog posts on this sidebar by manually entering data into OpenCalc spreadsheet? No, I just screenscrapped Google Analytics, Delicious, and Blogger to get all relevant data - fortunately they weren't captcha-protected.

People like me cannot really afford to spend time finding and contracting Indian captcha breakers, it only makes sense for high volume for profit operations like spammers. It's ironic that this alleged anti-spam measure hurts spammers very little, and hurts honest bot makers a lot. I guess disabled people are not big fans of it, and neither are normal people who fail half of the captchas already, and will fail 80% soon if the trend in captcha complexity continues.

Sensible ideas get abused a lot

If you want to register or change password you need to enter your new password. To make sure nobody sees it it's invisible, but then it's easy to make a typo. It's not a big deal during login, as you can always retry, but it would be really annoying if you made a typo during registration and couldn't login at all.

So a perfectly sensible idea was to make you enter your password twice. You're unlikely to do the same typo twice, so you're protected against both onlookers and mistakes. So who was the first asshole who thought it's a good idea to make people repeat their email address?

Another thing that really annoys me are websites that require captchas for anything else than potentially spammable actions. Some forums want captchas for search, how stupid is that? It's pure usability breaker, making human action harder at no benefit whatsoever.

While I'm at it, let me whine about Xbox Live registration. They use double-entered invisible password just like any website - except that Xbox360 doesn't have a keyboard, so they use big highlighted virtual keyboard on the screen, so every onlooker sees your password anyway.

Geographical restrictions are harder to bypass

Web used to be full of open proxies, so limiting viewers by country wasnt't really working. Unfortunately most proxies got closed because of spammers, and now it's really hard to access Pandora or Comedy Central from UK IP address. Pandora is the only such service that I care about, if I wanted Colbert Report so badly I could always use Bittorrent. Unfortunately the fascist music industry forced Pandora to introduce this and all other restrictions, and none of Pandora alternatives are even close, and believe me I tried many of them.

It's such a shame because Pandora is the best thing to happen to music since Audio Galaxy, which was also great in the same way of making meaningful music recommendations. I'm surprised Pandora is still alive even in their restricted form, it must really piss off some people at the record industry that some innovation is happening in spite of all their efforts to supress it.

Internet is a much milder place

People used to trick others into clicking links to goatse.cx and other shock sites. Now it's Rick Astley. Even 4chan which not that long ago was full of gore and borderline kiddie porn is fascinated by a cute hyperactive girl from Gaia Online. Torrent sites are no longer about kinky porn - they're mostly about downloading TV series so you can watch them without following TV schedules, just a more convenient version of TiVo.

There are no ads on the Internet

It's funny because pretty much everything on the Internet is ad-supported, yet nobody has to watch ads unless they choose to, by not installing an ad-blocker. Ads started as fairly reasonable static links and banners that nobody cared about. So ad companies in their greed and stupidity moved to animated banners, flash, popups, popunders, and abused users so much that a great backlash emerged, and now all ads are gone, even the most meaningful and least annoying ones like Google text ads.

I like this example of consumers taking control over things. In pretty much everything else big corporations abuse their powers and force consumers into following their ways, while governments passively stand by or even help the abusers. For example banks can charge you extra fees whenever they want to, and it's up to you to challenge them. Who gave them that right? Why cannot consumers charge bank fees for crappy service for symmetry? Such abuses are really common, and consumers rarely stand up against them. I cannot really think of any case other than ad blocking where it worked.

Most of these processes have been happening slowly and quietly in the background, but over years they made Internet into a very different place from what it used to be.

Which crackpot cult to join

I was once young and naive and thought that smart people can think for themselves, and don't believe in crackpot ideas. Some pretended to, but just because that's how they made money. The Internet proved me very very wrong - it turns out that many otherwise smart people believe in the most outrageous things, and actively spread them and argue about them on the Internet.

It's not just that their ideas are weird or wrong. Eliezer Yudkowsky's Friendly AI stuff is almost certainly wrong, but at least it's original and it makes a really great blog read. Most crackpots on the Internet are not only wrong and stupid, but they're the same few kinds of crackpots everywhere. If I had a nickel for every peakoil aynrandist wake-up-sheeple comment thrown at me I'd buy them another Internet so they can leave ours.

I know the basics of crackpottery science. There are memes that infect people's brains, and the particularly outrageous ones like religions also serve as pretty good in-group vs out-group indicators, and maybe mildly wrong memes somehow protects against infection by even worse memes and so on and so on, but couldn't you people at least follow some better cult? Ayn Rand? Peak oil? 9/11? Haskell monads? What's wrong with you!

I know a blog post won't make anyone leave their current cult once they're sucked in, but here's a list of a few better cults, so maybe future generations are at least better crackpots than the current one.

Linus Pauling

Crackpottery spreads better with a good guru, and there are fewer 20th century figures more notable than Linus Pauling. The guy won not just one but two Nobel Prizes, and was one of the first researchers of quantum chemistry, biochemistry, and modern genetics. He was also one of the leading activist for world piece and nuclear disarmament. There's really not much more you can ask from a leader, and he sure beats someone who writes about economy without any clue about either writing or economy, am I right?

So what's the cult? Pauling believed that the key to good health is taking massive amounts of vitamins and other micronutrients, like many grams a day of vitamin C (most scientists believe 100mg is enough to prevent vitamin C deficiency). He lived 93 years, so maybe there's something to it. I mean, Ayn Rand somehow didn't build a great financial empire in spite of her "insight" into economy and human nature, so ten years above average should count for something.

So back to vitamins. Research is pretty consistent at showing they don't work, but they also don't seem to harm you, so why not try taking some?

Pros: Your guru will be far more awesome than anybody else's. It might just have some tiny effect on your health, even if you live just one month longer that's worth something, isn't it? Maybe learning about all the vitamins will make you stop eating so much junk food.

Cons: You might spend a few more bucks on supplements, but they're pretty cheap these days. If you really overdo vitamin C you can get diarrhea, but this will only happen at over 100x the usual amounts. If you overdo some other vitamins you can get into more serious problems, but Pauling didn't advocate taking too much of those, so just follow the guru and you'll be fine.

Richard Dawkins

Richard Dawkins is one of the most eloquent gurus you could possibly follow, and his ideology is pretty simple - there is no God. Ok, that's not all of it - everybody knows there is no God, even the Catholics, the religion is just a fancy social ceremony they follow.

Dawkins' real point is that religion is not only false but also evil. Everything any religious organization does is pure wrongness and suffering, so they should be fought, primarily by writing blog posts on how all religion is evil. For some bonus points you can also buy a flying spaghetti monster t-shirt. Never mind that most of the suffering attributed by Dawkins to religion is simply basic human tendency to hate out-groupers, and in absence of religion humans have a proven track record of killing and otherwise hurting each other based on language, skin color, political ideology, programming language, or even favorite chariot team and I'm totally not making this last one up.

Pros: You save one hour a week by not attending Mass (or equivalent of your choice). You will have more eloquent guru than anybody else. The cool people are atheists much more often than lame people, so if you're openly atheist people might think you're cool too, at least before they come to know you better.

Cons: You won't be able to win any public elections in United States. You will piss off all religious people, and if you're persistent enough even the mildly religious ones. Well, that might as well be considered a pro.

Aubrey de Grey

Not much impressed by popping vitamin pills and blogposting that there is no God? Fear not, there's a cult for you too. Aubrey de Grey intends to cure the entire aging and make people live forever - young and healthy. Well, he's not much of a biological researcher, he's a computer programmer with a funny beard, even funnier than Stallman's.

And you know what - the way biotechnology is progressing it might even work some day. World without aging sounds even better than world without religion doesn't it? So far it's a total failure even on mice, with the longest living mice being simply calorie-restricted but maybe someday.

Pros: You can brag about having the best goal of all crackpots - it's even better than revealing that Mossad was behind 9/11. If it somehow works everyone will know, and you not only get the bragging rights, but you get bragging rights for the next 5 billion years, until Sun dies! Isn't that cool?

Cons: You'll have really creepy guru. It's going to really suck if they find out how to keep people young forever a week after you die.

Raymond Kurzweil

Eternal youth is not enough? Raymond Kurzweil says we're all going to be happy human-AI hybrid living in post-scarcity economy of singularity or whatever. I'm getting buzzword overload from all that. Vaguely conceived "growth" will accelerate, until we grow at infinite number of percents a day, and then singularity happens and everyone lives happily ever after as an upload.

Well, the growth doesn't really seem to be accelerating much. Of Kurzweil's many predictions for 2009 he made in 1999 pretty much none turned out to be right. Just listen to some - "In communications, translate telephone technology is commonly used. This allow you to speak in English, while your Japanese friend hears you in Japanese, and vice-versa", or "The majority of texts is created using continuous speech recognition, or CSR (dictation software). CSRs are very accurate, far more than the human transcriptionists, who were used up until a few years ago", or "Heptic technologies are emerging. They allow people to touch and feel objects and other persons at a distance. These force-feedback devices are wildly used in games and in training simulation systems. Interactive games routinely include all encompassing all visual and auditory environments", or "Jesus returns to judge the world and to establish the Kingdom of God". Oh sorry, the last one is from a different crackpottery, but they're pretty much equally accurate.

Pros: You can play with electronic gadgets all day and still believe you're doing the world a favor. Technology is progressing, so some of this stuff might come true eventually, just don't give any dates in your predictions and stay vague enough so that when something similar happens you can claim you predicted it.

Cons: Your guru gives out too many dates and details in his predictions, so others will make fun of you every time they don't come true. If you want to be hardcore enough you will need to use speech recognition instead of a keyboard, and with the software we have now half of the phrases recognized will be fuck you stupid machine just go to Amazon and order me a damn keyboard.

Richard Stallman

I probably don't have to advertise rms. He believes all software should be free, and to look at a web page he sends email to a demon which runs wget and mails the page back to him. It is very efficient use of his time.

To follow RMS you will write Free Software using Emacs on Debian (at least it's not GNU/Hurd these days), preferably in Lisp, but he's OK with you using C if you really have to. You will not use any proprietary software, like Flash, or any computer games. You will correct everyone who uses phrases like "Open Source" or "Intellectual Property".

Pros: You might actually write some useful software and do something good for humanity. Emacs is not the worst possible editor, and writing some software improves your CV.

Cons: You will forever have to fight with drivers for your wireless card. Everybody will make fun of your guru's beard and hygiene.

Paul Graham

Paul Graham is so good Arc is just Blub for him. He writes web startups in Lisp and sells them to Yahoo during dotcom boom. He also thinks the same trick is going to work even in financial recession - because every time is the best possible time to start a startup.

He's a bit like Ayn Rand in his belief that economic success is the best measure of person's value, but at least he can write, and actually made a successful company and helped a few others. He believes that if you try to make a startup you will spend all your time chasing successive rounds of investment and most likely fail or with any luck sell it to Google with investors getting 95% of the money, and at the same time that it's a totally awesome way of building a company, unlike let's say bootstrapping, which just cannot possibly work.

Pros: It might get you rich, and you can create something cool on the Internet. Your guru will have a pretty decent blog.

Cons: You might have to work 100 hours a week for years, mostly chasing investors not coding, and then investors take all your money, and Yahoo decides to rewrite your beautiful Lisp program in C++ for lulz.

Barack Obama

Barrack Obama is your new bicycle. He brought hope and change into the world of despair, and the new era of world peace, prosperity, and justice is starting on January 20th.

Oh wait, he agrees with Bush and McCain on pretty much everything - he supports occupation of Afghanistan, surge and pretty much indefinite stay in Iraq, unsupervised bailout of banks, car makers and everybody else, Israeli genocide in Gaza, criminalization of marijuana possession, telecom immunity, offshore drilling, embargo against Cuba, and pretty much everything Bush stands for except for stem cell research. But worry not - he flip flips a lot, so he may yet align himself with your favorite views.

Pros: At least he's not Bush, and it's hard to imagine anyone worse than Bush. Maybe he'll fix a few things and you'll blame everything else on Republican opposition, or whatever. And at least stem cell research will be legal.

Cons: You will be making excuses for him for the next 4 to 8 years, and even neocons will be laughing at you. Finally disgruntled, you'll volunteer for Republican candidate in 2016, also campaigning on platform of hope and change, just like every politician since ancient Greece.

Choose your own crackpot cult

World will be a better place if instead of reading the same tired conspiracy theories over and over again, I could learn about a new crackpot cult each day. Wake up sheeple, you can think for yourself! Or if not, at least get more original theories, for example the ones I just presented, but Internet is full of other interesting crackpot cults if none of them interests you.

If you know any other cool crackpot cults people could join please list them in comments section below, just please no more Ayn Rand, ok?

Medieval 2 Total War Concentrated Vanilla minimod

I really enjoy Rome Total War and Medieval 2 Total War games. Yet there's something about me that's never content with what I have, and always wanting to make things better. There's been some things about Medieval 2 Total War that really bothered me - like how small differences between factions were compared to Rome Total War, or how often the game turned into a siegefest.

I tried a few mods but were mostly disappointed with them. Lands to Conquer was only a bit better than vanilla - on the one hand I finally had a chance of playing with late units, on the other hand it didn't really fix most of vanilla gameplay issues. Stainless Steel simply made things a lot worse to me, by slowing everything down, and creating this huge boring map on which nothing ever happens. Maybe other people like it this way, I didn't.

So I decided to take things in my hands, and do some modding myself. First I identified things I wanted to see changed about vanilla:

• Differences between factions are too small. Part of it is because there are too many sieges and units are pretty much the same in a siege. Part of it is because of overpowered generals' bodyguards - you can send them all to the front, and they crush every low-to-mid-tier unit. Even if they die you'll get man of the hour or adoption soon enough.
  
• Settlement defenses are very weak, one unit of Peasants or Militia with a ram is enough to break into even medium-tier settlement. This makes artillery, spies, ladders, and siege engine pretty much useless.
• There's too much micromanagement of agents, and also rebels.
  
• Everything moves too slowly. It takes ages to bring reinforcements to the front lines. It also makes field battles meaningless - AI will keep making units as fast as you're defeating them in field battles, so you have to siege if you want to win a war.
  
• Building things is constrained by time, not by money. Most of the time almost every settlement builds something.
• The only way to build good economy is quick expansion. Small economies have no chance of competing, so you have to expand quickly. After you expand AI has no chance of competing so there's not that much fun in finishing the campaign.
• The game lasts too short before becoming boring. I never even got to Mongols, and I don't think it's even possible to get to New World while still having fun.
  
• Units are supposed to be rock-paper-scissors - cavalry beats archers beat infantry beat cavalry. In reality cavalry charge kills everyone, and you have very good cavalry (bodyguards) immediately, and missile infantry runs out of missiles before killing anyone.
  

Plus I had some complaints about diplomacy and stuff, but they weren't easy to fix, so I let them be for some time (diplomacy is one of the things that mods like LTC actually fix rather well, I need to borrow one of them). So I decided to do some pretty drastic changes. Now I know doubling and halving stuff is an overkill in many cases, but at least I'll know what effects will it have on gameplay. If I adjusted statistics by 20% I would need to spend months in playtesting before I could tell if it's actually any improvement or not. So the changes were:

• Bodyguards nerfed to half their original size and 1 hitpoint per unit (like normal units) instead of 2 (like elephants).
• Missile infantry (but not missile cavalry or siege engines) get twice the ammo.
• Gates twice as strong, ram attack half as powerful, settlement towers fire rate doubled. Rams also have half the hitpoints, but it only matters if you attack them with artillery, not if you attack them with fire arrows.
  
• Everyone moves twice as fast in campaign mode.
• Rebels and pirates twice less common.
• All buildings build in one turn.
• All resources twice as valuable, this doubles mines, trade, and merchant trade income. Cost of mines doubled to compensate.
• Tax rates have 1% per level impact on population growth, not 0.5%.
  

I only did some moderate playtesting, so I don't know all the consequences of this yet. Fortunately as the changes were rather large major consequences can be seen very easily.

First, more field battles. Because sieges are harder you cannot just siege, build one ram, and assault in the next turn. You will likely spend more time building siege equipment, or even waiting the defenders out, and as AI is twice as fast now, many sieges seem to end in AI reinforcement coming to the rescue and a big field battle - exactly the way I wanted. I think even after all these doublings rams are still a bit overpowered and settlements are still a bit too easy to take, so I'm not going to reduce these numbers.

There's less micromanagement. If you want to move your agents somewhere, it takes half the time. Rebels are less frequent (this seems to make them stronger, but I'm not completely positive about this effect), so one annoyance disappears, and when they come you don't have to spend so many turns just going there and then going back.

Even drastic nerfing bodyguards didn't make them too weak. They cannot do a frontal charge on a strong unit and hope to survive without casualties, but they're good enough at protecting the general, and they can still get tripple digit kill rate when charging Peasant Archers and other low-tier mobs.

Doubling ammo of missile troops worked quite well. First, they can do a lot more damage when defending a settlement, especially since it now takes a lot longer to break the gates, so there's plenty of times to pour arrows on the enemy. Unless you skip the gates and use ladders, or siege towers, or spies, or break gates from afar with artillery. Artillery is a lot more useful now, not only relative to rams, but also because unit speeds are much more reasonable. Second, increased ammo turned mid-to-high-tier archers really good skirmishers, as they won't run out of arrows before killing anyone. It doesn't help them that much in battles with full spectrum of units, as ammo is rarely a limiting factor in such cases. Stronger missile infantry balances missile cavalry, which now pays for increased mobility with less ammo; and it makes light cavalry more important - small detachment of light cavalry can protect your infantry-based force against enemy skirmishers, which would otherwise become a much bigger pain in the ass.

All this tweaking made factions feel a lot more different from each other than they did in vanilla, even though they still have the same roosters of units and buildings and starting positions. I'm happy with every tweak I made to battles, and while the exact number can be changed a bit higher or lower, they feel a lot better than vanilla's numbers.

With campaign it's been somewhat more mixed. I really like that buildings can be build very quickly if you have enough cash, and that taxes have serious consequences and are no longer an automatic "as high as possible without a rebellion". Increased movement speed is a great thing, but I think doubling it was too extreme, and it makes map feel really very small. By the way, isn't anybody bothered by how unrealistically slow everyone moves? Columbus crossed the Atlantic in just five weeks, not 20 years as it would take him in vanilla. So maybe I'll make 200% into 175% or something in the next release.

Doubling value of resources wasn't such a great idea. Yes, it changes the balance so resource-rich (like Timbuktu) settlements and those with good access to maritime trade are a lot more valuable now, and trade rights can score you many thousands of florins per turn. Unfortunately everyone is flooded with cash. More money isn't necessarily a bad thing, as bigger armies are fun, but I think it goes too far, and I'll reduce resource values from 200% to 150% of vanilla's.

The game progresses differently now. Instead of fast early expansion, which then slows because everyone attacks you and it takes ages to move reinforcements to the front lines, you have slow early expansion (as sieges are hard), followed by a happy total war with everybody fighting everybody else with big armies, and fast counterattacks. Unfortunately I failed at making it last much longer, so you probably won't see Mongols, much less Timurids or Aztecs.

AI seems to handle these changes quite well. It doesn't seem to lose too many generals. It sucks at economy, and spends all its money on Peasant armies, but that's more or less what it did in vanilla, so I'm not complaining much. AI has a big advantage, because autoresolve completely ignores settlement defenses as long as you have at least a single ram or ladder. This isn't as bad as it sounds, as AI really sucked at sieges back in vanilla, failing to take rebel settlements even with big numerical advantage, so now it's more even. But - autoresolving a siege is considered nasty cheating now! It really goes against everything the mod stands for. Try toggle_fow instead - unlike gamebreaking autoresolve toggle_fow makes the game more fun and saves you from a lot of boring micromanagement.

Here's a link to download Concentrated Vanilla mod at original settings (200% speed, 200% resources). Here's a link to Concentrated Vanilla mod with tweaks that haven't been playtested yet (175% speed, 150% resources and mines cost). Just unzip all files in your Medieval 2 Total War directory and double click on the bat file. The files are pretty big as it seems some files need to be copied even if I don't change them, like campaign start menu.

I'd love to hear your feedback about this minimod. I'd also like to include some good bugfix and diplomacy minimods, so your help with that would be appreciated too.