taw's blog

The best kittens, technology, and video games blog in the world.

Thursday, November 26, 2020

Empires are not about economic exploitation of periphery by the core

Pacificats Floris by Pacificat Ragdolls from flickr (CC-NC-ND)

I recently discovered a fun blog "A Collection of Unmitigated Pedantry", and it's been very enjoyable reading, but one of the posts is so wrong I need to write a rebuttal.

The author in Why Are There No Empires in Age of Empires? comes up with the following thesis:

An empire is a state where the core ruling population exercises control and extracts resources from a periphery which is composed of people other than the core group (linguistically/culturally/ethnically/religiously distinct). So an empire is a state where one set of people (the core) extract resources (typically by force) from another set of people (the periphery).

And so...

And that's how we know that "United States" is really a Puerto Rican Empire, as Puerto Rico is sure sucking up vastly greater flow of resources than it's ever getting back. The ruling Puerto Rican elite is literally not even paying income tax like those exploited mainlanders!

The so called "Russia" is really Great Chechen Empire, they sneakily conquered it by losing the war.

The so called "China" is really Greater Tibetan Empire.

The "United Kingdom" is really Scottish Empire, just look at all the money they're extracting from England!

And the earlier so called "British" Empire was really ran by the Americans - the exploited Brits were paying 10x as much in tax per capita than the ruling Americans.

And during Scramble for Africa, Africans managed to colonize Europe, as sure as hell average European was paying for all that elite map painting, and didn't benefit in any way whatsoever.

And so on.

This is all nonsense

If you look at any actual empires, especially recently, the overwhelming pattern is that burden of maintaining the empire - in terms of both money and manpower - falls disproportionately on core population, and peripheral populations are mostly expected to not cause trouble, and are otherwise net economic drain.

Exceptions happen, and occasionally core areas find some place extremely exploitable, but they're just that - rare exceptions. Like in 1700s' French colonial empire, tiny Guadeloupe was somehow economically worth far more than a quarter of North American continent.

Most of the time, those peripheral taxes are far less than core population pays, and far less than costs of holding those places in the Empire.

Foreign elites

There's a distinct and far more common situation where foreigners conquer some place, displace local elite, and rule as a dynasty or nobility over far more numerous locals.

This would actually fit in the original definition, except as a general rule the conquerors move to their conquered lands, so it feels a bit silly to talk about "Norman Empire" which lost Normandy rather quicky, or "Manchu Empire", which was absolutely ruled from within China, and is generally labelled "China" in history books, or such.

Exceptions proving the rule

Perhaps if one keeps looking, it's possible to find some "empires" where core successfully and sustainably extracts resources from periphery, especially in more ancient times.

Roman Empire is actually halfway there, as it managed to exploit places like Greece and Egypt pretty damn well, but even Romans admitted that their periphery like Britain were a net drain. Eventually the ruling center of Roman Empire moved to its richest province, and the usual pattern was restored.

In any case, such exceptional cases cannot possibly be used as definition of an empire.

Monday, November 16, 2020

Mechs vs Tanks

Miisa by andymiccone from flickr (PUBLIC-DOMAIN)

Sci-fi and anime loves mechs, but there's a widespread belief that mechs are just absolute trash idea, and tanks are superior in every way.

Here's example of such claims: part one, part two.

This is completely wrong.

Cube Square Law

Part of the problem is conflating the idea of a mech, with the idea of a giant mech. Mechs can come in all sizes.

And in fact military is already actively researching mechs! Just among many such projects - DARPA's TALOS project is an bulletproof, weaponized, AI-enhanced exoskeleton - basically a small mech.

It might take a few decades before such mechs become deployed by actual armed forced, but it's ridiculous to call something crazy when prototypes have been in development for years now.

Weapons vs Armor

History of warfare is an endless race between weapons and armor, with either being on top at different times.

During the Middle Ages, armor was king, and a fully armored knight was nearly invulnerable to whatever peasants could throw at them. But just a few short centuries later, no amount of armor humanly possible to carry could stop a musket bullet, so weapons were king, and armor got abandoned completely. Ironically the Peak Armor era was time when armor was already on verge of obsolescence, but it was easier to double down on old strategy instead of rethinking everything.

When tanks were introduced, a big reason they were quite successful was protection offered by their armor. Common infantry weapons just couldn't do anything to them, but of course that didn't last long.

As weapons got better, tank armor got heavier, and fast. First tanks to see combat in World War I had 12mm of armor.

By end of World War II, armor reached 250mm. To still be functional, it had to have far more compact shape.

It wasn't really possible to keep increasing the weight, and modern tanks aren't really much heavier than WW2 tanks, but using advanced materials to increase protection, the race between weapons and armor continued for a while longer.

Passive vs Active Protection

Right now it looks like the weapons are getting ahead. Armor is by no means useless, but in 2006 Lebanon War, Iranian anti-tank guided missiles were able to penetrate the most modern tanks just fine.

Weapons keep getting better, and it doesn't seem like armor has much space left to improve.

All militaries can see that, and they're equipping their tanks and other vehicles with active protection systems, which attempt to shoot down incoming missiles before they hit.

It's tempting to think that active protection is just another phase in evolution of tanks, but that's just wrong.

Tanks are the form they are because that's the only way to have thick armor while keeping weight reasonable.

Active protection has no such limitations. It doesn't matter if you install it on a tank, a light vehicle, or for that matter on an exoskeleton.

If active protection ends up being the winner, then mechs work just as well as tanks. And then liberated from just one form factor, we'll likely see a wide variety of different shapes of military vehicles - some might still look like current tanks, but others might very well look like mechs of various sizes, or something completely different.

And if active protection ends up being a big loser? Then giant mechs make no sense, but then neither do tanks.

Power Considerations

Another issues with mechs of all sizes is difficulty providing enough power and energy storage for them. Right now tanks are just more efficient, and mechs and exoskeletons are not very practical.

Fortunately for mechs, battery and engine technology is improving at very rapid pace, so it's entirely reasonable to expect this issue will go away soon.

There are even ways to recharge from air or space without any physical contact.

Future Prediction

In our world, it's very likely that exoskeletons will become widespread over next few decades, just like drones before them.

It's also very likely that active protection systems will increasingly become main protection. This will likely mean future tanks becoming lighter, as all that heavy armor just won't matter all that much. Wider variety of (mostly autonomous) military vehicles will come into widespread use, with very few relying on heavy armor, and most on active protection, stealth, or just being really cheap and expendable like most types of current drones.

Giant mechs? That's more farfetched for now, but if we have this discussion in a few decades, at time when most armies consist of exoskeleton-clad infantry supported by drone swarms, this really won't sound that far outside realm of possibility.

So if even our world is on verge of having mechs, it's just ridiculous to think there's no place for them in any sci-fi world, where technologies might have developed along different lines, and trade-offs are different?

The "tanks are always superior" crowd is going to sound just as ridiculous as believers in eternal superiority of heavily armored cavalry were a few centuries ago.

Thursday, November 05, 2020

Prediction Markets were Batshit Insane this Election

. by Runs With Scissors from flickr (CC-NC-ND)

The elections are over. The biggest loser are the prediction markets.

As I'm writing this, Arizona has already been called for Biden. And PredictIt has it as 30% chance of Trump.

Trump has no mathematically possible way of winning. PredictIt has it him at 20%.

I talked shit about 538, and I was absolutely right, but even they in the end had Trump at just 10%. Models I'd consider more reasonable like The Economist's had Trump at 3%. Prediction markets had him at completely batshit insane 40%.

And as votes started coming, when Trump did a tiny bit better than expected (but at no point was anywhere near having any chance of winning), Trump peaked at completely ridiculous 72%! It was known long in advantage that election day vote will be a lot more Republican than mail in vote, and states were clear how much of each kind outstanding there is. Prediction markets completely missed that, and had Trump as a favourite just because Republicans mostly went there in person while Democrats mostly sent their votes by mail, and it was too hard for them to understand.

Everything I said turned out to be right. If polls were right, elections would have been very easy Biden victory, and the "normal polling error" benefiting Biden would turn them into a Reagan style landslide.

But not only that - "normal polling error" towards Trump, would still mean easy Biden victory. Even unusually high polling error - what we had this year - didn't bring Trump particularly close to competitive territory. Everything that possible could go right for Republicans were perfectly for them - and Trump lost anyway, because his chances were nonexistent.

Polls weren't great, but they were fine. They predicted Biden to win by about +8%. By the time counting will end, he'll end up closer to +4 or +5%. We won't know for a while, as California takes forever to count their votes, so popular votes always ends up more Democratic than it seems on election day.

It turns out Trump was indeed closer in tipping point states, so Electoral College gave me a few points, but it never managed to overcome that kind of lead. There's no reason to predict which party will be at an advantage in 2024.

Real markets work because they're dominated by sophisticated institutional investors, whose job it is to figure things out, and whose predictions are tested every day. We hoped that small prediction markets dominated by small scale amateurs tested once every 4 years would be just as good, but it turns out they failed utterly.

I'm definitely disappointed as for a while it seemed that we might get great rationalist tool for very little money. It turns out it takes more than that.

EDIT: It's November 13th now, Trump lost nearly two weeks ago, every state has been called long ago, most betting sites paid out on Biden (including one where I bet), and Trump is still 12% on PredictIt. Some people must really hate their own money.

Wednesday, August 26, 2020

Why 538 election model is wrong and Biden has an easy win ahead

#TrumpYourCat by jillccarlson from flickr (CC-BY)

538 releases a model predicting what they think is going to happen every elections, and this time they released a crazy one, where "Biden is only slightly favored to win the election", and Trump has insanely high 30% chance of  second term.

This is in stark contrast to everyone else's predictions. For example The Economist gives Trump just 9% chance of winning, and this is a much more sensible number.

In this post I'll explain why 538 believes elections are quite close, and why it's completely wrong.

It's not a terribly sophisticated analysis, but it's too long for a tweet, so I thought I'd put it here.

If you're interested, I wrote "A Guide to 2020 Elections for non-Americans" a while back, and I dislike both Trump and Biden very much.

Why 538 model believes what it does

Their model is very sophisticated, but its basic assumptions leading to their crazy conclusions are fairly straighforward:

  • Biden is far ahead in polls, however:
  • there's still a lot of time until elections
  • things are likely to change until then
  • especially since pandemic year is full of events
  • Electoral College benefits Trump, so he will likely win even if he losses popular vote narrowly
  • the economy went to hell earlier this year, but they think it will improve, and that will somehow benefit Trump

Why 538 is completely wrong - not much time

The only thing they got right is that Biden is indeed far ahead in the polls. They're wrong about everything else.

The elections are a lot closer than they think. In normal times, most people vote on Election Day. This time due to the pandemic a lot more people will vote by mail early, and mail voting opens about a month before Election Day, depending on the state. So for a lot of voters, time until elections is only half of what they think. And people might be voting even earlier as there's a lot of talk about Post Office having trouble processing votes in time.

Even if Biden says the dumbest thing he ever said one week until elections, or Trump single-handedly invents coronavirus vaccine, enough votes will be cast by then that it probably won't matter.

Why 538 is completely wrong - pandemic prevents most campaigning

People don't change political beliefs based on clocks ticking, they do based on events and campaigning. This time due to pandemic there's going to be a lot less campaiging, and people's attention will be largely on other issues.

It's not even certain if the usual debates will happen or not. And even if they do, only the first of the three is supposed to happen before mail voting starts.

Why 538 is completely wrong - people already made up their mind

If you compare Trump's approval ratings with all past presidents, Trump's are the least volatile.

People love Trump, or hate Trump, and there's a lot fewer people in between that could be convinced by the campaigns. So no matter how good Trump's campaign is, it will likely not matter. And likewise, while Biden will almost certainly win, he's also very unlikely to win by 1984 style landslide.

Why 538 is completely wrong - Electoral College advantage can go either way

In 2000 and 2016 elections, Democrats won the popular vote, but Republicans won the presidency. While this might give an impression that Electoral College system inherently benefits Republicans (the way Senate arguably does), if you look back both parties benefited from the system about equally often.

Right now it might look like 2020 map will be similar to 2016 map, and seems to benefit Trump. This thinking is completely mistaken.

If very little changes, Trump will lose terribly, and while he might indeed lose Electoral College by less than popular vote, it will be completely inconsequential.

For elections to be even competitive, fairly big changes must happen. But while it's possible, there's no reason to expect that they'll uniformly apply to all states (in proportion to their "swinginess").

For example if Biden says something that will offend a million Californians and makes them stay at home or vote third party, this will significantly reduce his popular vote share, but it won't matter for Electoral College one bit, since California will go Democratic regardless.

Trump needs not only a really big swing - he needs a swing that's either eerily uniform, or happen in just the right states.

In reality, if a major swing happens, it throws all assumptions about Electoral College advantage out of the window. Even if much bigger swing than could be reasonably expected happens, and Trump somehow manages to reach vote share similar to what he got in 2016 (losing popular vote by 2%), it's actually still quite unlikely that this swing will leave his Electoral College advantage intact, and get him another victory.

Why 538 is completely wrong - the economy will not favor Trump

A popular way to predict elections is by looking at "economic fundamentals". The thinking goes that whenever economy is doing poorly, people blame whoever is in charge for it, and vote for someone else. This is generally true, and after every major economic crisis like 1929 or 2008, governments all across the world got a lot less popular, regardless of which part of the political spectrum they were on, and how much or how little they had to do with causes of the crisis. There's no reason to expect 2020 to be any different.

538 really hates fundamentals models, and for a good reason. While bad "economy" affects people's voting preferences, there's no obvious way to pick a single number for what is the "economy". You can use unemployment, or GDP, or disposable income, or stock market, or so many other indicators, of combinations of indicators. So in year were economy is doing so-so, you can't really use those models to predict much.

Instead of stopping there, 538 somehow decided to ask experts where the economy will go just before elections, and since most expect some recovery from the worst of the lockdowns, they count this as benefiting Trump. While there's a room for reasonable disagreement how to model this, there's no way in hell 2020 economy could possibly benefit Trump, people's memories are longer than a few weeks, and US is still and will likely be in state of mass unemployment.

Are 2020 Elections a foregone conclusion?

Democrats are overwhelmingly likely to win the House - they'll win the popular vote, and there's not even a possibility of Electoral College stopping them. Republican House is about as likely as Nicola Sturgeon becoming the next Prime Minister.

Biden is extremely likely to win presidency - things would need to go really damn well for Trump.

Senate is another matter. Only 1/3 of it is reelected each time, and Republicans do have a meaningful structural advantage there, so it seems fairly close to even who's going to win. And it probably matters more than anything else this year.

If Democrats control the Senate, they'll be able to abolish filibuster, pass pretty much whatever they want, and then possibly also pack Supreme Court with ten new liberal judges that will pronounce all of that Constitutional. They probably won't go that far, but in principle they could.

In Democrats keep the House, and Republicans keep the Senate, then it will most likely be four more years of gridlock regardless of who's the president. The parties hate each other so much, they won't compromise on anything beyond the most routine operations of the government. Biden's policies will be different than Trump's, but presidential policies don't matter as much as people thing, an the next president in 2024 might very well reverse most of them.

And yes, I obviously put money where my mouth is.

Friday, August 14, 2020

Trying out Minecraft again after 10 years

Dolphin who followed me to my base, jumped onto the sand, and died tragically.

Some time later its buddy made the same bad life choice.

Like everyone else, I played Minecraft really long time ago. I recently discovered that the game is still receiving regular updates after all those years. Thanks to corona, I have a bit more free time than usual, so I decided to give it another go!

Here's my thoughts. Little of this will be new to you if you played recently.

My favourite changes

I really like the new villager system. This is possibly the biggest change - trading with villagers at first seems pointless, as they don't sell anything useful, but it turns out they level up, and offer more and more interesting items - while stuff they buy you can build semi-automated system to collect.

It's also possible to train villagers to choose new profession, breed villagers, and they have some protection with iron golems they summon, and some enemies with pillagers.

Treasure ships and treasure maps found in them are great. You'll randomly run into them as you explore, and they have nice stuff inside, and map where to get even more nice stuff. I really didn't love how I got two copies of same treasure map once, but such small side quests are great fun.

Boats no longer take damage when sailing around! Looks like they changed that back in 2012, that makes boats a great transportation system, and canals for them are quite easy to build.

The game now shows you recipes when you open crafting table, and that's a great change! No need to alt tab for rarely used things.

The game needs a damn map!

The game now has maps, and I thought that's great until I tried actually using them, and omg they're truly terrible.

This was one thing that finally convinced me to go through the hassle of installing Forge modding system (make sure you get one that matches your Minecraft's version), and then both Xaero's map mods (world map, minimap). It seems like such a small thing, but honestly it makes bigger difference for player enjoyment than everything they changed in Minecraft over last 10 years put together.

Game still hates the player sometimes

The game has its share of "fuck you player" moments. Falling into lava and losing all you stuff is a classic, but the game has more.

Like when you get hit by a Wither Skeleton, that not only damages you when hit, and poisons you so you keep losing your health (which is far enough so far...), but it also hides your health bar, which is just Minecraft devs being assholes.

For another "fuck you player moment", I was just wandering around exploring, setting up temporary bed whenever night fell and then taking it with me afterwards. I was in the boat during daytime, and somehow tridents started flying my way from the water, and I died out of nowhere. I lost all my stuff as usual, but most annoyingly game decided to spawn me at 0, 0 instead of my base or last sleeping place since last bed I slept in was no longer there.

The problem was that in my seed, there were no villages within over a 1000 blocks from my spawn point, and the closest one I finally found was abandoned, so I finally setup my base at about -2000, -1000 or so next to the first inhabited village I found. So by spawning at 0, 0 I wasted like 15 minutes of my life walking there. Just because the devs hate me.

And then there's all the other stuff, like all the sand updating because I placed a torch, resulting in sudden fall into a ravine.

There are games where mechanics like that make sense, but Minecraft is mainly about exploring and building, and last thing it should do is punishing exploration.

Other things that suck

Inexplicably there's still no builtin modding system, and you need to install third party mod manager to even download and install mods. That is truly baffling.

Combat in Minecraft was always bad, and it's still bad. Turning it on Easy doesn't seem to do anything. At some point they changed combat system, and before I watched some youtube tutorials explaining how it really works, it seemed just ridiculous.

Placing item when building a bridge is really obnoxious. And it looks like other versions of Minecraft fixed it, but somehow Java version still uses the old and really tedious system.

With single exception of crafting table recipes, the game is really bad at explaining anything. There's so many things that are completely unintuitive, and which are not explained, and not realistically discoverable. It's as if the game expects players to watch youtube tutorials for half of its mechanics. And sometimes I watched a tutorial, tried something, and it absolutely didn't work, because Bedrock and Java are subtly different, or they updated things at some point, and it no longer works.

I always felt like player's inventory was too small, and it wanted me to keep running back to base too often. It makes sense that I can only carry so much, but why can't my boat or minecart have some extra inventory space, that I'd then bring home with me? Big Boat With Extra Chest - as well as Big Boat With Space For An Extra Animal would be great additions to the game.

Things I didn't care much for

I really didn't care for the Nether and End changes - they're far too combat oriented, and combat in Minecraft is just not very fun, before or after the big combat update.

It feels like there's far too many blocks that just don't do anything different. There's naturally occurring stone in like 10 colors now, and a lot of cosmetic variation that feels largely pointless.

There's a lot of new mobs too, which seem to do nothing interesting.

I don't hate such changes, and I guess they add some extra flavour.

Is it worth playing again?

To be honest compared with other games that had 10+ years of updates, Minecraft changed surprisingly little. Something like Factorio or Crusader Kings II are such drastically different games than they were at release, but Minecraft now and 10 years ago, pretty much the same thing.

I might be worth giving it a go. Then again, if you never played Factorio, you should try that first instead. There, your buildings actually do things.

Sunday, May 03, 2020

Fun New Conspiracy Theories

p1240692 by generalising from flickr (CC-SA)

For your entertainment. There's no hard evidence for any of them, and I'm definitely like actually endorsing any of them, but they all sound quite plausible, right?

Corbyn was a Tory plant

UK is not even close to being a democracy.

It's also trying its best to be a Tory single party state. Starting the timer when the last Liberal Party government ended in 1922, and ending it when Tony Blair was elected with New Labour in 1997, Tories were in government 68% of the time - despite having actual majority of the vote only once.

They achieve this by combination of heavy gerrymandering, radicalising the Labour Party, and splitting the opposition.

UK redistricting is technically performed by "independent" Boundary commissions, but rules under which they're operating resulted in very consistent Tory bias.

Cheating a few percent through gerrymandering is however not enough. Tory Deep State figured out a far better way - by radicalizing Labour Party in ensures moderates are too scared to vote for them, and most of the time manages to split the vote between Labour and Liberals/LibDems/SDP/such.

Political science predicts that in FPTP systems both parties will drift to where the center of the electorate is. Somehow UK is the only exception to this pattern.

All those Tory shenanigans were ended when Tony Blair created New Labour - which was exactly what the political science predicted. Modern pro-European center-left party without any of the openly Communist baggage.

New Labour was mortal enemy of the Tory One Party State project, so they did their best to undermine it. The 2008 Great Recession let them get back to power as part of a coalition government, but that was living on borrowed time.

So what did they do? Tory Deep State dug up Comrade Corbyn, and managed to put him in charge of Labour Party to destroy it. Once in charge, Tory plant Comrade Corbyn somehow managed to get as many open Communists and open Anti-Semites as possible to join to make Labour completely unacceptable to an average voter.

Under control of Tory puppet-masters, Labour adopted the most extremist plans possible on basically every issue. It did not take pro-European position, in spite of overwhelming majority of Labour voters being pro-European.

Normally when a party loses elections, its leader goes, but somehow the Tory puppet managed to stay and lose elections after elections, and only got kicked out this year. Damage Tories managed to do to Labour Party was so extensive, it's not clear if Keir Starmer will ever be able to fix it.

Boris faked coronavirus infection

It is very well established that Boris is willing to lie about literally anything to get the power, and doesn't care about how much damage his actions cause.

So why do people suddenly believe that he wouldn't lie about getting coronavirus and needing ICU?

Let's get some facts.

Boris allegedly got moved to ICU on April 7th. According to data I could find for two days later, only 3883 people in all of UK got COVID-19 ICU.

There's 0.005% chance that a random person would need COVID-19 ICU.

Also, only 20% of patients admitted to ICU for COVID19 recovered.

So the chance of Boris getting COVID-19, needing ICU at this point, and promptly recovering, is about 0.0001%. Do you seriously believe that the chance that habitual liar Boris Johnson lying about this is less than 0.0001%? If so, I have a great EU trade deal to sell you.

And it would be far from the first case of head of state lying about their health.  It sounds totally implausible, but Roosevelt managed to hide his paralysis from the whole public and the press cooperated with that!

Secret Service actively interfered with photographers who tried to take pictures of Roosevelt in a wheelchair or being moved about by others. The Secret Service commonly destroyed photographs they caught being taken in this manner
The list of cases of politicians hiding their illness which only comes out after their death is very long, including nearly half of recent US presidents.

How many faked instead of hiding their illness? That's harder to tell, as there's usually no advantage to do so, but it's sure as hell higher than 0.0001%.

In all likelihood Boris was either totally fine, or just got common cold, and requested ICU admittance to fake COVID-19, as he thought that would be politically beneficial to him.

Democratic governors are destroying their states' economies on purpose

"Fundamentals" models of election forecasting have been accused of overfitting, but their basic message is still true - when economy is doing awfully, people vote against the ruling party. The Great Recession in most countries showed drastic drop in support for whoever happened to be in the government - left wing, right wing, center, or extreme, regardless of how responsible they were, and how well they were doing.

None of that mattered, voters always turned away from the ruling party - either towards the opposition, or towards new usually extremist parties.

Notably there was no such backlash in any subnational elections. Republican states stayed relatively Republican, and Democratic states stayed relatively Democratic, there was no grand flipping.

During corona crisis, Democratic governors have generally issued far more strict lockdown orders than Republican governors, pretty much regardless of how affected their states are. Outside New York, most states aren't particularly affected, and none of that is even close to rational policy.

So here's a fun conspiracy theory to think about - what if they're trying to damage their economies on purpose, because they know Trump is going to get blamed for it anyway?

During normal times it's not something they could get away with, but this is a great unique opportunity to do just that.

Notice this theory doesn't work backwards - if you try to flip that and speculate that Republican governors are trying to make things better to help Trump, well can't blame people for trying to make things better.

Wednesday, December 18, 2019

Challenges for November 2019 SecTalks London

Selfie by SethBahl from flickr (CC-NC)


October 2019 London SecTalks CTF was jointly won by me and Laurence, so we wrote challenges for November together.

There was a total of 9 challenges, 5 written by me and 4 by Laurence. It was a bit much, as one of the challenges got zero solves until after the event. In spite of very generous hints given halfway through the event.

Challenge files and code used to generate them are available on github.

There are no answers below, but some serious hints which might make it too easy.

For previous rounds, see posts about September 2017November 2017May 2018July 2018October 2018February 2019April 2019, and July 2019 CTFs.

Rockstar (5 points)

The puzzle was a rock song / program in Rockstar programming language. Just running the program would print ASCII codes, which could be easily turned into a flag.

Think Outside The Box (10 points)

The challenge is a BMP image file with flag hidden by incorrect image size. Changing the header reveals the image.

Dots (15 points)

This was Unicode Braille, but dots and no-dots were simply 1s and 0s to be decoded, ignoring any "Braille" structure. To my surprise nobody solved it even when given basically full explanations of what they're supposed to do.

CRC32 MAC (20 points)

A signed cookie server, which uses CRC32 with secret key as MAC. CRC32 is extremely unsuitable in this role.

Decryption Engine (25 points)

A small Python program which uses default password based on current date, and encoded file. Assuming password wasn't passed, the space of possible keys is tiny.

Salted MAC (30 points)

A signed cookie server, which uses SipHash with secret key, and then salt just to be extra sure.

In principle this should be difficult to break. The problem is a trivial implementation mistake.

PHPain (40 points)

It was a separately hosted PHP application vulnerable to recently published CVE.

Poly-DIY MAC (50 points)

Another signed cookie server, using a simplified variant of Poly1305. The variant is reasonable, however the server isn't very careful about nonce handling, and such schemes tend to fail catastrophically when nonce is reused.

GeoCaptcha (60 points)

It's a web "Captcha" which asks the user to locate 20 major cities on the map, with any mistake failing the Captcha.

You can play it here (for a while).

It's worth so many points mostly due to its very high entertainment value, not due to difficulty.

I was curious how people would approach it - only 2 players solved it with code (as robots), 6 solved it with tedious manual process (as humans). So I guess it works as a Captcha.

Thursday, October 10, 2019

How to configure OSX 10.14 Mojave for software development

Kitten by Magnus Bråth from flickr (CC-BY)

With every new Macbook, I'm updating the guide, previous version is here.

And I get it that it's ironic that I'm posting this just as 10.15 came out, but I started the setup a few days ago, and in general it's a poor idea to update OS as soon as it comes out.

Basics

  • Go to Settings > Security > FireVault, turn on FireVault. This will restart your computer.
  • Install some sensible browser like Chrome or Firefox.
  • Afterwards either sign up into your account on which you hopefully have your ad blocker setup, or install some. Most popular seems to be uBlock Origin these days, but pretty much any of them will do just fine.
  • Install whichever cloud sync service you're using like Dropbox etc. And start syncing your stuff.
  • Install iTerm2 for sensible terminal emulator.
  • Clean up all crap from dock. Other than Launchpad and System Settings, everything else should be gone. Add iTerm2, your browser, and your text editor, and any application you wish to install there instead of stock Apple crap.

Editor

Install some sensible text editor. These days most people use Visual Studio Code.

If that's your choice, run it, open Command Palette, and choose: "Shell Command: Install 'code' command in PATH".

Settings

Like every other operating system, OSX has a lot of bad default settings. Here are some obvious fixes:
  • Settings > Mouse > increase speed
  • Settings > Keyboard > Key Repeat > Fast
  • Settings > Keyboard > Delay Until Repeat > Short
  • Settings > Sound > Disable "Play user interface sound effects"
  • Settings > Sound > Alert volume > 0% (for Terminal ping)
  • Settings > Sound > Show volume in menu bar
  • Settings > Trackpad > Scroll & Zoom > Disable "Scroll direction: natural"
  • Settings > Displays > Arrangement > drag and drop your external monitors into desired order
  • Settings > Displays > Max out brightness
  • Settings > Displays > Turn off "Automatically adjust brightness"
  • Settings > Dock > enable "Automatically hide and show the Dock"
  • Menu bar > Battery icon in task bar > Enable "Show Percentage"
  • Menu bar > Time icon in task bar > Open Date & Time Preferences > Use a 24-hour clock
  • Menu bar > Time icon in task bar > Open Date & Time Preferences > Show date
  • iTerm > Preferences... > Profiles > Terminal > Unlimited Scrollback
Press Ctrl-Up arrow, add a few desktops (or "spaces" as they were used to know), then go to Settings > Keyboard > Shortcuts > Mission Control - and enable their keyboard shortcuts Ctrl-1 to Ctrl-6 or however many you have there.

Open Screenshot app, choose options, then disable "Show Floating Thumbnail".

Drivers

OSX already includes drivers for laptop itself, but you might need some for peripheral hardware.

In particular, external PC keyboard need a tweak to work properly, as left and left Windows keys are in reverse order from Mac keyboard.

Go to Settings, Keyboard, Modifier Keys..., and swap positions of Option and Command keys. Feel free to change functionality of Caps Lock key as well, it's a huge easily accessible key with no useful function people love to remap, usually to extra Control.

If you need any special keyboard layouts, get them too.

Standard paths

OSX renames a lot of directories. While in theory scripts could just use env vars to find proper paths, it's more reliable to symlink all the things:

  sudo ln -s /Volumes /mnt
  sudo ln -s /Volumes /media
  sudo mv /home /home-old
  sudo ln -s /Users /home

Development tools

You'll need a package manager, and the only one anyone uses is homebrew, MacPorts and the rest being basically dead. You need to tell homebrew to not spy on you with brew analytics off command.

You'll need Xcode. Fortunately homebrew installer does it for you automatically. If you don't use homebrew for some reason, you can install Xcode manually by running xcode-select --install from command line.

Create new SSH key pair

Before you do that, name your computer something memorable with sudo scutil --set HostName your_host_name command.

Open Terminal and run ssh-keygen to create ~/.ssh/id_rsa, then upload the generated key to any place that needs to know about it like githubbitbucket, or whatever else you use.

Checkout your dotfiles

Hopefully you're storing your dotfiles somewhere. If it's a git repository, or your Dropbox account, get them now and symlink them all properly.

If there are any other repositories you might need, checkout them too.

Install homebrew packages

Your list might vary. Here'sa few obvious suggestions:

brew install rbenv ruby-build mc wget p7zip unrar trash git htop bash zsh youtube-dl jq imagemagick coreutils bash-completion zsh-completion

Then enable all services you installed, unless you want to start them manually:

  ln -sfv /usr/local/opt/*/*.plist ~/Library/LaunchAgents/

And install non-system ruby, so you can install gems without sudo. Currently latest is:

  rbenv install 2.7.0-preview1
  rbenv global 2.7.0-preview1

To make that actually work, you need to make sure ~/.rbenv/shims is in your $PATH. If you type rbenv init, it will tell you what to do.

Due to OSX limitations you'll need to run sudo htop if you want to use htop.

Install gems

Again, your list my vary. These days most of the software will have its own Gemfile so long list of gems are generally unnecessary. But some global utilities are still useful:

gem install bundler rak pry pry-rescue

Sane bash or zsh

bash version shipped with OSX is ancient and BSD utilities are all awful.

You have two choices. Either install modern version of bash from homebrew, or use zsh from homebrow.

First, edit /etc/shells as admin and add the following lines at the end of it:
/usr/local/bin/bash
/usr/local/bin/zsh

Then set it as your shell, with whichever one you prefer:
  chsh -s /usr/local/bin/bash $USER
  chsh -s /usr/local/bin/zsh $USER

Either way, run to touch ~/.hushlogin to prevent some worthless spam on every open terminal tab.

If zsh is your choice, the first time you start a new terminal, it will ask you some questions on how to configure it.

Coreutils

This is optional step. OSX coreutils are awful, and generally GNU versions are a lot better, but if you switch there could be occasional minor incompatibilities.

If you want to do so, add GNU coreutils to your PATH:

  export PATH="/usr/local/opt/coreutils/libexec/gnubin:$PATH"
  export MANPATH="/usr/local/opt/coreutils/libexec/gnuman:$MANPATH"

Better window manager controls

Sadly OSX window manager is extremely dubious for keyboard use. Fortunately programs to make it usable exist. I recommend installing HyperSwitch for sane alt-tab window switching, and Spectacle for managing windows.

I previously used HyperSwitch, but it no longer works with latest version of OSX. Spectacle shortcuts are different than HyperSwitch, and I like HyperSwitch's more, so I adjusted shortcuts to match.

You'll need to give them necessary access. To do so:
  • Settings > Security & Privacy > Privacy > Allow the apps below to control your computer > enable HyperSwitch and Spectacle
Also open its preferences, and set it to run in the background, and start at login.

All other software

There's a lot of other software you might want. The most obvious one is the VLC media player.

You might also want some kind of Git UI program, like GitUp or (another for of) GitX.

Enjoy

Once you go through this list, and successfully get everything going, I'd recommend modifying it to your liking and reposting your version on your blog. Everybody's needs are different, so guide like this is just a starting point.

Monday, September 09, 2019

Technologies that inspired me

Cat's bored by SethBahl from flickr (CC-NC)

Nearly every day I run into another language, framework, or program. Most of them are just plain awful, and the ones which aren't are usually nothing special.

Every now and then I run into a true gem. Here's a list of such great technologies, in approximately the order I found them.

Perl

Perl was absolutely amazing compared with every language which came before it. Other languages were motivated by what worked for computers, or by some silly mathematical theory, or sometimes like with Java by outright hostility to programmers.

Perl tried to be as much like a natural language as possible. If what you wrote made some kind of sense, it was going to try to run it. And most of the time it worked.

Unfortunately Perl supported too much weird stuff, and it caused the language to collapse under its own weight, but the most influential languages of the 1990s like Ruby, Javascript, and PHP were very heavily inspired by Perl. And as next generation of languages was very heavily inspired by those, the legacy of Perl lives on.

Perl tried to double down on its philosophy with Perl 6, but without first dropping failed ideas it was never really going to work.

Regular Expressions

Text is nearly as common data type as numbers, but most pre-Perl languages didn't treat text data as first class. In many old languages text is literally an array of small numbers, or such other nonsense. Even today Erlang / Elixir do literally that - "Hello" is the same as [72, 101, 108, 108, 111].

Regular expressions are to text what arithmetic is to numbers, and let you extract or modify text with tiny amount of highly readable and highly performant code. They have weird reputation of unreadability, but one line of dense regular expression is infinitely easier to understand than 100 lines of text processing code it replaces.

It's key skill for every programmer, no matter what's the domain. If you don't know regular expressions, drop everything and do some regexp crosswords.

There used to be different variants of regular expressions, but these days basically every language uses some variant of "Perl-Compatible Regular Expressions". Except Unix grep command, which still uses stupid 1970s regexps for some crazy reason.

Wikis

Wiki was a revolutionary idea - just put a bunch of linked pages online, and let anyone who wants edit them. It sounds like a recipe for disaster, but it worked remarkably well. Wikipedia was the greatest success story of wikis, but there are all kinds of wikis for all kinds of content. And as a general rule, they're a lot more reliable than non-editable sites.

Ruby

Ruby took everything that made Perl great, everything that made Lisp great, everything that made Smalltalk great, then it threw away their bad parts. And then it added more magic on top.

It is the high point of programming langugae design, unmatched up to this day. Implementation issues aside, I'd rather use Ruby as it existed 20 years ago than any language that came out since. It's just baffling how far ahead of time it was.

Of course it could be even better, but that's true of anything.

I hope someday a language comes that's better than Ruby, but so far it doesn't look like anyone's even really trying. The best anyone's trying to do is to make their languages as close to Ruby as possible given various constraints.

Ruby on Rails

Ruby on Rails is another miracle technology. When it came out, it was completely unmatched as the best way to write web backends. Today, after the web changed beyond recognition, it's still completely unmatched as the best way to write web backends. It's rare for any technology to have such staying power.

To always stay on top, Rails evolved very aggressively, and Rails upgrades on large codebases are notoriously difficult, but then again, the choice is between upgrading Rails 4 to Rails 5, or writing your code in some shitty framework that's going to die, and fully rewriting it into another shitty framework that also won't last.

It's quite baffling that so far no real competitor to Rails emerged. The best everyone else can do is to copy as much as they can from Rails, and even that is usually really poor.

The main reason is that Ruby is completely unmatched at making DSLs than any other language, and Rails is very DSL-heavy. But other aspects of Rails success like everything working seamlessly out of the box (while being individually replaceable) without needing to duct tape a bunch of disparate components should be replicable, and yet nobody even tries.

Why so many frameworks come out without such basic things like comprehensive testing system, or database migration system, or sensible security defaults? I suspect their authors just don't get Rails, and people who get Rails use Rails.

TextMate

Once upon a time, there were thousands of editors and they were all 💩. It was a running joke that every programmer writes their own editor, and their own IRC client (and later their own Javascript framework).

And just as with infinite monkeys randomly typing on infinite number of keyboards, one of such attempt miraculously produced TextMate - the first editor which was not 💩.

Multiple cursors, command palette, file palette, extension system, nested file types, integration with modern UIs, sensible keybindings making use of what's on modern keyboards - TextMate was far ahead of everything else.

TextMate itself didn't last, but Sublime Text, Atom, VS Code, and some others are essentially TextMate clones, and they completely dominate the editor world.

The only non-TextMate editors still in use are people using IDEs like IntelliJ or Android Studio due to being forced to use programmer-hostile languages like Java. And people who use Vim because they still haven't figured out how to quit it.

jQuery

Javascript used to be awful beyond belief. Well, it's still not great, but back then doing even the most trivial dynamic interactions, like changing some class based on an user action was a page of code, and then another page of different code if you wanted to support more than one browser.

Then jQuery came, and with a few $("selector").action() you could write complex UI interactions with very little code. It was even bigger difference when it came to jQuery AJAX vs browser APIs for AJAX. The gap was as enormous as between regular expressions and manipulating text by shuffling arrays of characters around.

Gradually browser APIs became more compatible, more expressive, and less buggy, they added a lot of features to close the gap with jQuery, so now it's sort of viable to write short programs directly with browser APIs. And while jQuery is amazing at progressively enhancing website functionality, it's a stretch if you want to make Single Page Applications.

So is it the end of jQuery era? Not even close! jQuery is still vastly more popular than all frontend frameworks put together - 80% of top million websites use jQuery, while all the hipster frameworks put together are still just a rounding error. Currently most popular React at 1.6% is behind 20 different jQuery plugins.

And beyond jQuery itself, D3 is basically jQuery for data visualization. Most acceptance testing frameworks from Capybara to Cypress use jQuery-style for writing tests. Single Page Applications are useful, but most of the Web is better off not being an SPA, and good jQuery skills will never go to waste.

It might be unpopular opinion now, but I feel the best way people should learn frontend is still be HTML, CSS, jQuery, then ES6+ Javascript, and only then some SPA framework like React.

Amazon Web Services

I remember bad the old days when setting up any server required making a request to a grumpy sysadmin (or even worse - a grumpy DBA) and waiting potentially weeks for them to find some time for it. Even more fun if that required more physical hardware, and budget requests for those. Meanwhile, there was nothing you could do.

Then came Amazon and decided that making billions selling everything wasn't enough so it opened a small side business of getting rid of grumpy sysadmins. Want to start a new server? A few clicks and it's done by the time your coffee finished brewing. And they kept adding new services beyond the initial EC2 and S3, to simplify your life more and more.

I feels silly to even mention how great it is, as AWS and other Clouds won so utterly that most people barely remember the pre-Cloud days, and those who do would rather forget them.

pry

I didn't love debuggers, and I usually employed print-based debugging as my first choice, strace and friends as my second choice, and only used interactive debuggers if all else failed.

pry was something else. A hybrid of a debugger and REPL, with full power of Ruby metaprogramming and introspection, it was on a completely different level than anything I've used before.

Adding binding.pry became my first not the last resort. And then there's pry-rescue and other plugins to make things even more awesome.

I'm still annoyed that pry isn't just part of Ruby yet, and that bundler blocks pry unless you explicitly add it to every single Gemfile just in case.
Sure there's finally binding.irb, but irb is not even close to pry. Seriously, get your act together guys and get pry into Ruby standard library.

MongoDB document model

Relational databases are just a bunch of damn spreadsheets, and forcing your program's object oriented (or otherwise richly structured) data into spreadsheets for storage is invariably painful. Object-Relational Mapping tries to make it bearable, with limited degree of success.

There were many attempts to create alternative database models, but they were either specific to one programming language, or used something like XML which was just as awkward as spreadsheets.

Then came MongoDB, made everything JSON, and it was glorious. Oh and depending on your application it could easily be orders of magnitude faster than relational databases.

Unfortunately while MongoDB JSON-based document model was amazing, and its performance was generally outstanding, everything else about MongoDB was less so. Doing even relatively simple queries in MongoDB is stupidly painful compared with SQL, so you end up doing it all in application. Its performance-first defaults got a lot of people into trouble as well.

I'm surprised that even a decade later nobody managed to build anything that united JSON-based (as opposed to spreadsheet-based) document model with some rich query language that can at least match SQL.

Spreadsheet-based databases nowadays support JSON data a little bit, NoSQL databases nowadays have a bit richer query languages, but the gap between those two is still enormous.

Capybara

TDD lived in this weird state, where backend code was decently tested, but frontend testing was minimal and mostly manual.

Even today most tests for JS programs are fairly low value efforts, like "component renders without crashing" tests and snapshot tests.

Capybara was the first practical way to do end to end tests of web apps with dynamic frontend and dynamic backend as a single system.

Nothing better than Capybara was created so far. There's a bunch of fairly poor Capybara clones in other languages, which might be just about good enough. Cypress might be developing into the first real alternative, but it's very frontend-based and integration with dynamic backends is its secondary consideration.

Continuous Integration servers

I wasted too much time running tests manually, or not bothering to run them because the change I was doing was trivial, and the tests took too long.

There is an answer to that - CI servers. Most companies have them. But people coding for themselves, or coding Open Source software are still usually not using them.

Semaphore CI and many other sites let you easily setup CI for free for your Open Source project, and you should do that.

Chrome Developer Tools

The story started in the olden days with Firebug, which provided just console.log and a few other very simple functions. Or possibly even with the View Source button (which some people wanted to delete from browsers to "protect the source code", #truestory).

Nowadays every browser comes with amazing debugging and development tools, and they keep getting better every year.

It's one of few places where I feel Javascript has a big lead over other languages. pry might be as good as possible on terminal, but it's still limited to a terminal. Chrome Developer Tools have full power of the browser UI, and they fully use it.

Dropbox

I have multiple computers. Also multiple phones and tablets. Sharing content between them used to be such a mess - connecting and disconnecting external disks, or manually copying files by cables or rsync. Dropbox made it seem that the same file is magically present in every location. It just works, with very few issues.

Unfortunately Dropbox for phones is quite bad, so if anyone solves this better, I'm interested.

Weirdly many Dropbox alternatives like Google Drive are simply horrendous.

Slack

Just like everyone was writing their editor, everyone was writig their IRC client. Or some other chat system which was basically IRC clone.

They were all solving the wrong problem. The issue wasn't that IRC needed better UI, the issue was that IRC was based on fundamentally wrong assumptions (broadcast to whoever happens to be connected at time) that made it completely unsalvageable.

Slack fixed those assumptions, and it completely revolutionized workplace communication, not just in tech.

All post-Slack chat systems like Discord use basically Slack model.

Electron

Electron is the first time in history of computing where we have cross-platform desktop apps with decent UI. There's performance cost, as Electron apps bundle whole browser, and are infamous for their memory use, but there's no alternative, so let's just throw more hardware at the problem, and restart every now and then if it gets too bad.

There were a few earlier attempts like with Java, but the UX was just attrocious. I cringe whenever I have to use some Java-based GUI software.

Z3

Normal programming answers questions like x = 2+2 perfectly fine, but it is shockingly bad at dealing with questions like 2+x = 4. Z3 is an amazing system for answering just such questions.

Solving sudoku, hacking CTFs, proving hardware design? Z3 has you covered.

I liked it so much that I wrote a Ruby gem for it. It's much nicer than official APIs.

Docker

I initially dismissed Docker as just another VM. It turns out that such lightweight micro-VMs have a lot more uses, and just being able to install and run a lot of images neatly solves a lot of issues like CI, software with weird dependency, security isolation and so on.

Imba

Javascript used to have a million frameworks. They were generally called "MVC frameworks" back then, a name you don't hear much anymore, as it turns out MVC is a poor fit to what they were trying to do.

At certain point very-much-non-MVC React pretty much won, by being decent at solving smaller problem (views), not even trying to deal with the hard problem (model and network layers / state management), and so having a lot easier learning curve and fewer gotchas. Acceptable performance and being pushed hard by Facebook also helped.

I even liked React before I found Imba. Imba however, is on a completely different level. It's the least known technology on this list, so I'll need to write a separate post about it, but you'll get orders of magnitude higher performance at fraction of code size, all while being a lot simpler than React.

It's a tiny project with very few development resources, so it feels like it really needs someone to push it to some sort of Imba 2 fully embracing post-ES6 world. But even in its current state, it's years ahead of any alternative.

Saturday, August 10, 2019

Kingdoms Grand Campaign Mod - Medieval 2 Total War mod highlight

18/8.2014 - kitten royalty by julochka from flickr (CC-NC)

Medieval 2, released back in 2006, was the peak of Total War series. One of the great things about it is abundance of mods, and this post is about one such mod.

Kingdoms Grand Campaign Mod

The idea behind Kingdoms Grand Campaign Mod is taking content from minicampaigns from Kingdoms expansion pack, and integrating them into one big campaign.

Campaign map is adjusted somewhat, and in addition to vanilla factions, you get:
  • Antioch from Crusades campaign
  • Ireland from Britannia campaign
  • Jerusalem from Crusades campaign
  • Kiev from Teutonic campaign
  • Lithuania from Teutonic campaign
  • Mongols from Teutonic campaign (they're in vanilla as unplayable horde)
  • Norway from Britannia and Teutonic campaign
  • Wales from Britannia campaign
For reasons I don't understand, Portugal is no longer a playable faction.

Kingdoms Features

The mod brings in extra units from Kingdoms campaign, like Mangonel, which is the only pre-gunpowder artillery unit worth using in battles.

Forts are now permanent stone buildings, and cost a lot more money.

Hanseatic League and potential Lithuania conversion events from Teutonic campaign are in the game.

Merchants are actually worth something.

Game balance

Medieval 2 is very easy for experienced player, so the mod does a few things to make it more difficult.

There are a lot fewer rebel settlements, and they tend to have a lot bigger garrison, so you can't just grab enormous areas of land without any opposition during your first few turns.

Time scale is set to 2 turns per year. All buildings build a lot more slowly.

Units provided by buildings are a lot different - high tier building no longer allow recruitment of low tier units, possibly to help AI or to clean up the UI.

Castles are massively nerfed because base castle buildings no longer provide any recruitment options, and you actually need to build stables and barracks (which also build really slowly). On the other hand, a lot of castle units get free upkeep as long as they're castle garrison.

Pope is as bitchy as ever, and due to a lot fewer rebels, he's going to be excommunicating people left and right a lot more than in vanilla. I never liked that part of the game, and I wished they toned it down.

AI gets a lot of free money on hard and very hard. On any difficulty levels, if it goes into severe enough debt, it gets a lot of money to get it out of debt.

And then there's garrison script.

Garrison script

Unlike later Total War games, Medieval 2 settlements have no automatic garrison, and AI often leaves them with minimal defences. That allows player to really easily destroy all factions.

As AI isn't really fixable, to counter that, a lot of mods add garrison script which spawn a lot of troops when garrison is besieged. I'm not convinced this is a great idea - I hated it when I didn't know where garrisons are going to happen and how big.

Once I generated a list of which faction would spawn what kind of units where, it was actually decent. So here's the script. Once it triggers for a settlement there's 10 turn timer before it can trigger again. Only AI gets free troops. Some factions have script for settlements that don't belong to them at start of campaign.

Antioch:
  • Antioch (14) - 1 Knights Hospitaller, 2 Antioch Militia, 3 Armenians of Celicia, 3 Dismounted Knights of Antioch, 5 Hospitaller Sergeant
  • Krak_de_Chevaliers (12) - 2 Dismounted Knights of Antioch, 2 Knights Hospitaller, 3 Armenians of Celicia, 5 Hospitaller Sergeant
Aztecs:
  • Huaxtepec (14) - 1 Coyote Priests, 2 Aztec Archers, 2 Jaguar Warriors, 3 Arrow Warriors, 6 Cuahchiqueh
  • Tenochtitlan (14) - 1 Coyote Priests, 2 Aztec Archers, 2 Jaguar Warriors, 3 Arrow Warriors, 6 Cuahchiqueh
  • Tixtla (14) - 1 Coyote Priests, 2 Aztec Archers, 2 Jaguar Warriors, 3 Arrow Warriors, 6 Cuahchiqueh
  • Tollan (14) - 1 Coyote Priests, 2 Aztec Archers, 2 Jaguar Warriors, 3 Arrow Warriors, 6 Cuahchiqueh
Byzantium:
  • Constantinople (16) - 1 Greek Firethrower, 2 SE Town Militia, 4 Byzantine Guard Archers, 4 Varangian Guard, 5 Byzantine Infantry
  • Corinth (11) - 1 Byzantine Lancers, 3 Byzantine Spearmen, 3 Trebizond Archers, 4 Dismounted Byzantine Lancers
  • Smyrna (11) - 1 Byzantine Lancers, 3 Byzantine Spearmen, 3 Trebizond Archers, 4 Dismounted Byzantine Lancers
  • Thessalonica (10) - 1 Byzantine Lancers, 2 Pronoia Infantry, 3 Trebizond Archers, 4 Byzantine Infantry
  • Trebizond (11) - 1 Byzantine Lancers, 3 Byzantine Spearmen, 3 Trebizond Archers, 4 Dismounted Byzantine Lancers
Denmark:
  • Arhus (14) - 1 Feudal Knights, 2 Town Militia, 3 Norse Archers, 3 Sami Axemen, 5 Norse Swordsmen
  • Hamburg (11) - 1 Huscarls, 2 Sami Axemen, 3 Norse Archers, 5 Dismounted Huscarls
  • Lund (11) - 1 Huscarls, 2 Sami Axemen, 3 Norse Archers, 5 Dismounted Huscarls
Egypt:
  • Alexandria (11) - 1 Mamluks, 2 Hashishim, 3 Desert Archers, 5 Saracen Militia
  • Cairo (15) - 1 Mamluks, 2 al Ashair, 3 Tabardariyya, 4 Desert Archers, 5 Saracen Militia
  • Jerusalem (12) - 1 Mamluks, 3 Desert Archers, 3 Hashishim, 5 Saracen Militia
England:
  • Caen (11) - 1 Feudal Knights, 2 Armored Swordsmen, 3 Longbowmen, 5 Armored Sergeants
  • Dublin (11) - 1 Mailed Knights, 2 Armored Sergeants, 3 Archer Militia, 5 Spear Militia
  • Gloucester (11) - 1 Mailed Knights, 2 Armored Sergeants, 3 Longbowmen, 5 Spear Militia
  • London (14) - 1 Feudal Knights, 2 Town Militia, 3 Armored Swordsmen, 3 Longbowmen, 5 Spear Militia
  • Nottingham (11) - 1 Feudal Knights, 2 Armored Swordsmen, 3 Longbowmen, 5 Armored Sergeants
  • York (11) - 1 Mailed Knights, 2 Armored Sergeants, 3 Longbowmen, 5 Spear Militia
France:
  • Angers (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Peasant Crossbowmen, 5 Armored Sergeants
  • Caen (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Peasant Crossbowmen, 5 Armored Sergeants
  • Marseille (10) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Crossbowmen, 4 Spear Militia
  • Paris (14) - 1 Feudal Knights, 2 Town Militia, 3 Dismounted Feudal Knights, 3 Scots Guard, 5 Spear Militia
  • Toulouse (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Peasant Crossbowmen, 5 Armored Sergeants
HRE:
  • Bologna (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Crossbow Militia, 5 Spear Militia
  • Frankfurt (14) - 1 Teutonic Knights, 2 Town Militia, 3 Crossbow Militia, 3 Dismounted Feudal Knights, 5 Spear Militia
  • Hamburg (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Peasant Crossbowmen, 5 Armored Sergeants
  • Innsbruck (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Peasant Crossbowmen, 5 Armored Sergeants
Hungary:
  • Bran (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Bosnian Archers, 5 Pavise Spearmen
  • Budapest (14) - 1 Feudal Knights, 2 EE Town Militia, 3 Bosnian Archers, 3 Dismounted Feudal Knights, 5 EE Spear Militia
Ireland:
  • Cork (14) - 1 Ridire, 2 Ceitherne, 3 Galloglaich, 3 Saighdeoir, 5 Ostmen
  • Donegal (11) - 1 Ridire, 2 Galloglaich, 3 Saighdeoir, 5 Cliathairi
  • Galway (11) - 1 Ridire, 2 Galloglaich, 3 Saighdeoir, 5 Cliathairi
Jerusalem:
  • Acre (11) - 1 Knights Templar, 2 Dismounted Knights Of Jerusalem, 3 Maronites of Lebanon, 5 Templar Sergeant
  • Jerusalem (14) - 1 Knights Templar, 2 Pisan and Geonese sailors, 3 Dismounted Knights Of Jerusalem, 3 Maronites of Lebanon, 5 Templar Sergeant
Kievan Rus:
  • Kiev (11) - 1 EE Cavalry Militia, 2 Berdiche Axemen, 3 Dismounted Dvor, 5 EE Spear Militia
  • Moscow (11) - 1 EE Cavalry Militia, 2 Dismounted Boyar Sons, 3 Dismounted Dvor, 5 EE Spear Militia
  • Novgorod (11) - 1 EE Cavalry Militia, 2 Dismounted Boyar Sons, 3 Dismounted Dvor, 5 EE Spear Militia
  • Smolensk (11) - 1 EE Cavalry Militia, 2 Dismounted Boyar Sons, 3 Dismounted Dvor, 5 EE Spear Militia
Lithuania:
  • Minsk (11) - 1 Szlachta, 2 Samogitian Axemen, 3 Baltic Archers, 5 Dismounted Szlachta
  • Riga (12) - 1 Dievas Guard, 3 Baltic Archers, 3 Samogitian Axemen, 5 EE Spear Militia
  • Vilnius (14) - 1 Dievas Guard, 2 EE Town Militia, 3 Baltic Archers, 3 Samogitian Axemen, 5 EE Spear Militia
Milan:
  • Bologna (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Genoese Crossbow Militia, 5 Italian Spear Militia
  • Genoa (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Genoese Crossbow Militia, 5 Italian Spear Militia
  • Milan (14) - 1 Feudal Knights, 2 Italian Militia, 3 Dismounted Feudal Knights, 3 Genoese Crossbow Militia, 5 Italian Spear Militia
  • Rome (14) - 1 Feudal Knights, 2 Italian Militia, 3 Dismounted Feudal Knights, 3 Genoese Crossbow Militia, 5 Italian Spear Militia
Mongols:
  • Bulgar (13) - 1 Khan's Guard, 1 Naffatun, 3 Dismounted Heavy Archers, 3 Dismounted Heavy Lancers, 5 ME Halberd Militia
  • Constantinople (12) - 1 Khan's Guard, 1 Naffatun, 2 Dismounted Heavy Lancers, 3 Dismounted Heavy Archers, 5 ME Spear Militia
  • Kiev (12) - 1 Khan's Guard, 1 Naffatun, 2 Dismounted Heavy Lancers, 3 Dismounted Heavy Archers, 5 ME Spear Militia
  • Moscow (12) - 1 Khan's Guard, 1 Naffatun, 2 Dismounted Heavy Lancers, 3 Dismounted Heavy Archers, 5 ME Spear Militia
  • Novgorod (12) - 1 Khan's Guard, 1 Naffatun, 2 Dismounted Heavy Lancers, 3 Dismounted Heavy Archers, 5 ME Spear Militia
  • Sarkel (12) - 1 Naffatun, 2 Mongol Heavy Archers, 2 Mongol Heavy Lancers, 3 Dismounted Heavy Archers, 4 Dismounted Heavy Lancers
  • Smolensk (12) - 1 Naffatun, 2 Mongol Heavy Archers, 2 Mongol Heavy Lancers, 3 Dismounted Heavy Archers, 4 Dismounted Heavy Lancers
  • Trebizond (12) - 1 Naffatun, 2 Mongol Heavy Archers, 2 Mongol Heavy Lancers, 3 Dismounted Heavy Archers, 4 Dismounted Heavy Lancers
  • Viatka (12) - 1 Naffatun, 2 Mongol Heavy Archers, 2 Mongol Heavy Lancers, 3 Dismounted Heavy Archers, 4 Dismounted Heavy Lancers
  • Vilnius (12) - 1 Khan's Guard, 1 Naffatun, 2 Dismounted Heavy Lancers, 3 Dismounted Heavy Archers, 5 ME Spear Militia
  • Yerevan (11) - 1 Khan's Guard, 1 Naffatun, 2 Dismounted Heavy Lancers, 3 Dismounted Heavy Archers, 4 ME Spear Militia
Moors:
  • Algiers (11) - 1 Arab Cavalry, 2 Dismounted Arab Cavalry, 3 Desert Archers, 5 Lamtuna Spearmen
  • Cordoba (14) - 1 Arab Cavalry, 2 ME Town Militia, 3 Desert Archers, 3 Hashishim, 5 ME Spear Militia
  • Granada (11) - 1 Arab Cavalry, 2 Dismounted Arab Cavalry, 3 Desert Archers, 5 Lamtuna Spearmen
  • Lisbon (11) - 1 Arab Cavalry, 2 Dismounted Arab Cavalry, 3 Desert Archers, 5 Lamtuna Spearmen
  • Marrakesh (11) - 1 Arab Cavalry, 2 ME Town Militia, 3 Desert Archers, 5 ME Spear Militia
  • Valencia (11) - 1 Arab Cavalry, 2 Dismounted Arab Cavalry, 3 Desert Archers, 5 Lamtuna Spearmen
Norway:
  • Agder (11) - 1 Huscarls, 2 Dismounted Feudal Knights, 3 Norse Archers, 5 Spear Militia
  • Oslo (14) - 1 Huscarls, 2 Town Militia, 3 Dismounted Feudal Knights, 3 Norse Archers, 5 Norse Swordsmen
  • Wick (11) - 1 Huscarls, 2 Dismounted Feudal Knights, 3 Norse Archers, 5 Norse Swordsmen
Novgorod:
  • Novgorod (11) - 1 EE Cavalry Militia, 2 Dismounted Boyar Sons, 3 Dismounted Dvor, 5 EE Spear Militia
Papal States:
  • Rome (17) - 2 Italian Militia, 3 Dismounted Feudal Knights, 4 Italian Spear Militia, 4 Papal Guard, 4 Pavise Crossbow Militia
Poland:
  • Danzig (11) - 1 Polish Knights, 2 Dismounted Polish Knights, 3 Lithuanian Archers, 5 EE Spear Militia
  • Krakow (14) - 1 Polish Knights, 2 EE Town Militia, 3 Dismounted Polish Knights, 3 Lithuanian Archers, 5 EE Spear Militia
  • Poznan (11) - 1 Polish Knights, 2 Dismounted Polish Knights, 3 Lithuanian Archers, 5 EE Spear Militia
Portugal:
  • Lisbon (14) - 1 Feudal Knights, 2 Town Militia, 3 Crossbow Militia, 3 Dismounted Feudal Knights, 5 Swordsmen Militia
  • Oporto (10) - 2 Dismounted Feudal Knights, 3 Crossbow Militia, 5 Swordsmen Militia
Scotland:
  • Edinburgh (14) - 1 Feudal Knights, 2 Town Militia, 3 Dismounted Feudal Knights, 3 Highland Archers, 5 Spear Militia
  • Wick (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Highland Archers, 5 Spear Militia
Sicily:
  • Naples (14) - 1 Norman Knights, 2 Italian Militia, 3 Dismounted Norman Knights, 3 Sicilian Muslim Archers, 5 Italian Spear Militia
  • Palermo (11) - 1 Norman Knights, 2 Dismounted Norman Knights, 3 Sicilian Muslim Archers, 5 Armored Sergeants
  • Rome (14) - 1 Norman Knights, 2 Italian Militia, 3 Dismounted Norman Knights, 3 Sicilian Muslim Archers, 5 Italian Spear Militia
Spain:
  • Leon (14) - 1 Knights of Santiago, 2 Town Militia, 3 Crossbow Militia, 3 Dismounted Feudal Knights, 5 Swordsmen Militia
  • Lisbon (14) - 1 Feudal Knights, 2 Town Militia, 3 Crossbow Militia, 3 Dismounted Feudal Knights, 5 Swordsmen Militia
  • Toledo (11) - 1 Knights of Santiago, 2 Dismounted Feudal Knights, 3 Peasant Crossbowmen, 5 Swordsmen Militia
  • Valencia (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Peasant Crossbowmen, 5 Swordsmen Militia
Teutonic Order:
  • Danzig (12) - 1 Christ Knights, 3 Order Spearmen, 3 Prussian Archers, 5 Sword Brethren
  • Konigsberg (12) - 1 Christ Knights, 3 Order Spearmen, 3 Prussian Archers, 5 Sword Brethren
  • Marienburg (14) - 1 Christ Knights, 2 Dismounted Ritterbruder, 2 Order Militia, 3 Prussian Archers, 6 Order Spearmen
  • Riga (12) - 1 Christ Knights, 3 Order Spearmen, 3 Prussian Archers, 5 Sword Brethren
Turks:
  • Adana (11) - 1 Turkish Horse Archers, 2 Hashishim, 3 Turkish Archers, 5 Dismounted Hasham
  • Antioch (13) - 2 Kurdish Auxiliaries, 3 Hashishim, 3 Janissary Archers, 5 Saracen Militia
  • Baghdad (15) - 2 Kurdish Auxiliaries, 2 ME Town Militia, 3 Janissary Archers, 3 Janissary Heavy Inf, 5 Saracen Militia
  • Constantinople (12) - 2 Hashishim, 2 Kurdish Auxiliaries, 3 Janissary Archers, 5 Saracen Militia
  • Iconium (11) - 1 Turkish Horse Archers, 2 Hashishim, 3 Turkish Archers, 5 Dismounted Hasham
  • Yerevan (11) - 2 Hashishim, 2 Kurdish Auxiliaries, 3 Turkish Archers, 4 Saracen Militia
Venice:
  • Bologna (11) - 1 Feudal Knights, 2 Venetian Heavy Infantry, 3 Venetian Archers, 5 Italian Spear Militia
  • Ragusa (12) - 2 Italian Militia, 2 Venetian Heavy Infantry, 3 Venetian Archers, 5 Italian Spear Militia
  • Rome (13) - 2 Italian Militia, 3 Venetian Archers, 3 Venetian Heavy Infantry, 5 Italian Spear Militia
  • Venice (13) - 2 Italian Militia, 3 Venetian Archers, 3 Venetian Heavy Infantry, 5 Italian Spear Militia
Wales:
  • Caernarvon (15) - 2 Welsh Militiamen, 2 Welsh Skirmishers, 3 Rhyfelwyr, 3 Saethwyr, 5 Meirionnydd Spearmen
  • Cardiff (12) - 2 Gwent Raiders, 2 Welsh Skirmishers, 3 Saethwyr, 5 Spear Militia

Free Upkeep Units


These are only free if settlement has right building to recruit them. Some are also free in forts, I think these are castle units. Generally all militia units (including cavalry militia) and castle infantry are upkeep free while garrisoned. Castle cavalry, siege units, and boats are not free.
  • Ahdath - turks
  • Al Haqa Infantry - egypt
  • Alamanoi - byzantium, slave
  • Almughavars - spain, portugal
  • Antioch Militia - antioch
  • Archer Militia - england, slave
  • Armored Sergeants - england, france, hre, milan, venice, papal_states, sicily, slave
  • Aventuros - portugal
  • Aztec Spearmen - aztecs, slave
  • Aztec Warriors - aztecs, slave
  • Baltic Archers - lithuania
  • Bill Militia - england
  • Burgher Pikemen - teutonic_order, slave
  • Byzantine Spearmen - byzantium, slave
  • Ceitherne - ireland, slave
  • Cliathairi - ireland
  • Croat Axemen - hungary, slave
  • Crossbow Militia - france, hre, denmark, spain, portugal, teutonic_order, norway, slave
  • Deisi Javelinmen - ireland
  • Dismounted Byzantine Lancers - byzantium
  • Dismounted Christian Guard - moors
  • Dismounted Conquistadores - spain, portugal
  • Dismounted Druchima - russia, novgorod
  • Dismounted Feudal Knights - england, scotland, france, hre, denmark, spain, portugal, milan, venice, papal_states, hungary, norway
  • Dismounted Ghulams - egypt, slave
  • Dismounted Halbbruder - teutonic_order
  • Dismounted Hasham - turks
  • Dismounted Heavy Lancers - mongols, timurids
  • Dismounted Knights Of Jerusalem - jerusalem
  • Dismounted Knights of Antioch - antioch
  • Dismounted Light Lancers - mongols
  • Dismounted Norman Knights - sicily
  • Dismounted Polish Knights - poland
  • Dismounted Polish Nobles - poland
  • Dismounted Szlachta - lithuania
  • Dismounted Tartar Lancers - russia, lithuania
  • EE Archer Militia - russia, novgorod, slave
  • EE Cavalry Militia - russia, novgorod
  • EE Crossbow Militia - poland, russia, novgorod, lithuania, slave
  • EE Spear Militia - poland, russia, novgorod, hungary, lithuania, slave
  • EE Spearmen - poland, russia, novgorod
  • EE Town Militia - poland, hungary, lithuania
  • Estonian Rebels - lithuania, slave
  • Followers of Perkunas - lithuania
  • Frankish Axemen - jerusalem, slave
  • Frankish Swordsmen - antioch, slave
  • Galloglaich - ireland, slave
  • Genoese Crossbow Militia - milan
  • Gotland Footmen - denmark, norway
  • Greek Militia Cavalry - byzantium
  • Gwent Raiders - wales, slave
  • Halberd Militia - hre, papal_states, sicily, poland, hungary, slave
  • Heavy Bill Militia - england
  • Heavy Billmen - england
  • Heavy Pike Militia - scotland
  • Helwyr - wales
  • Highlanders - scotland, slave
  • Hospitaller Sergeant - antioch
  • Italian Cavalry Militia - milan, venice, papal_states, sicily
  • Italian Militia - milan, venice, papal_states, sicily, slave
  • Italian Spear Militia - milan, venice, papal_states, sicily, slave
  • Jaguar Warriors - aztecs, slave
  • Kurdish Javelinmen - egypt
  • L Cavalry Militia - lithuania
  • Landschneckt Pikemen - slave, hre, milan
  • Levy Spearmen - england, slave
  • ME Archer Militia - egypt
  • ME Crossbow Militia - moors
  • ME Halberd Militia - egypt, turks, mongols, timurids
  • ME Levy Spearmen - mongols, slave
  • ME Peasant Archers - egypt, turks, mongols, timurids, slave
  • ME Spear Militia - moors, egypt, turks, mongols, timurids, slave
  • ME Town Militia - moors, turks, mongols, timurids, slave
  • Meirionnydd Spearmen - wales
  • Merchant Cavalry Militia - england, scotland, france, hre, denmark, spain, portugal, poland, hungary, ireland, norway, teutonic_order, wales
  • Mongol Foot Archers - mongols
  • Mongol Infantry - mongols
  • Morgannwg Spearmen - wales, slave
  • Native Archers - slave
  • Native Warriors - slave
  • New World Cuirassers - england, france
  • Noble Pikemen - scotland
  • Noble Swordsmen - scotland
  • Norse Swordsmen - denmark, norway, slave
  • Nubian Spearmen - moors, egypt, slave
  • Obudshaer - denmark, norway
  • Order Militia - teutonic_order
  • Order Spearmen - teutonic_order, slave
  • Ostmen - ireland
  • Partisan Militia - france
  • Pavise Crossbow Militia - venice, papal_states, sicily, hungary, slave
  • Peasant Archers - england, scotland, france, hre, denmark, spain, portugal, milan, venice, papal_states, slave, norway, teutonic_order
  • Pike Militia - france, hre, spain, portugal, milan, venice, papal_states, sicily, ireland
  • Pikemen - france, england
  • Pronoia Infantry - byzantium
  • S Archer Militia - byzantium, slave
  • SE Spear Militia - byzantium, slave
  • SE Town Militia - byzantium, slave
  • Saracen Militia - egypt, turks, slave
  • Scots Pike Militia - scotland, slave
  • Sergeant Spearmen - france, hre, milan, venice, papal_states, sicily, slave
  • Spear Militia - england, scotland, france, hre, denmark, spain, portugal, slave, norway, wales
  • Sudovian Tribesmen - lithuania, slave
  • Swiss Guard - papal_states
  • Swiss Pikemen - slave, venice, sicily
  • Sword Brethren - teutonic_order
  • Sword and Buckler Men - spain, sicily
  • Swordsmen Militia - spain, portugal
  • Swordstaff Militia - denmark, norway
  • Syrian Militia - jerusalem, antioch
  • Templar Sergeant - jerusalem
  • Tercio Pikemen - spain
  • Town Militia - england, scotland, france, hre, denmark, spain, portugal, norway, slave
  • Tripolitan Squires - jerusalem
  • Turkish Crossbowmen - turks
  • Ulster Swordsmen - ireland
  • Urban Militia - moors
  • Viking Raiders - denmark, norway, slave
  • Welsh Militiamen - wales
  • Woodsmen - poland, russia, novgorod, slave
  • Zweihander - hre, milan
  • al Ashair - egypt

Overall Impressions

The mod is pretty decent - just takes the best out of Medieval 2 without doing anything drastic.

Unfortunately I had a lot of crashes in game menu.

Fortunately once in game, crashes were not too common. I'd still manually save before and after every battle, but it was rarely necessary. Sadly vanilla crashes are not unheard of either.

If you want to give Medieval 2 another go, and want just a bit more variety, highly recommended.