taw's blog

The best kittens, technology, and video games blog in the world.

Wednesday, August 26, 2020

Why 538 election model is wrong and Biden has an easy win ahead

#TrumpYourCat by jillccarlson from flickr (CC-BY)

538 releases a model predicting what they think is going to happen every elections, and this time they released a crazy one, where "Biden is only slightly favored to win the election", and Trump has insanely high 30% chance of  second term.

This is in stark contrast to everyone else's predictions. For example The Economist gives Trump just 9% chance of winning, and this is a much more sensible number.

In this post I'll explain why 538 believes elections are quite close, and why it's completely wrong.

It's not a terribly sophisticated analysis, but it's too long for a tweet, so I thought I'd put it here.

If you're interested, I wrote "A Guide to 2020 Elections for non-Americans" a while back, and I dislike both Trump and Biden very much.

Why 538 model believes what it does

Their model is very sophisticated, but its basic assumptions leading to their crazy conclusions are fairly straighforward:

  • Biden is far ahead in polls, however:
  • there's still a lot of time until elections
  • things are likely to change until then
  • especially since pandemic year is full of events
  • Electoral College benefits Trump, so he will likely win even if he losses popular vote narrowly
  • the economy went to hell earlier this year, but they think it will improve, and that will somehow benefit Trump

Why 538 is completely wrong - not much time

The only thing they got right is that Biden is indeed far ahead in the polls. They're wrong about everything else.

The elections are a lot closer than they think. In normal times, most people vote on Election Day. This time due to the pandemic a lot more people will vote by mail early, and mail voting opens about a month before Election Day, depending on the state. So for a lot of voters, time until elections is only half of what they think. And people might be voting even earlier as there's a lot of talk about Post Office having trouble processing votes in time.

Even if Biden says the dumbest thing he ever said one week until elections, or Trump single-handedly invents coronavirus vaccine, enough votes will be cast by then that it probably won't matter.

Why 538 is completely wrong - pandemic prevents most campaigning

People don't change political beliefs based on clocks ticking, they do based on events and campaigning. This time due to pandemic there's going to be a lot less campaiging, and people's attention will be largely on other issues.

It's not even certain if the usual debates will happen or not. And even if they do, only the first of the three is supposed to happen before mail voting starts.

Why 538 is completely wrong - people already made up their mind

If you compare Trump's approval ratings with all past presidents, Trump's are the least volatile.

People love Trump, or hate Trump, and there's a lot fewer people in between that could be convinced by the campaigns. So no matter how good Trump's campaign is, it will likely not matter. And likewise, while Biden will almost certainly win, he's also very unlikely to win by 1984 style landslide.

Why 538 is completely wrong - Electoral College advantage can go either way

In 2000 and 2016 elections, Democrats won the popular vote, but Republicans won the presidency. While this might give an impression that Electoral College system inherently benefits Republicans (the way Senate arguably does), if you look back both parties benefited from the system about equally often.

Right now it might look like 2020 map will be similar to 2016 map, and seems to benefit Trump. This thinking is completely mistaken.

If very little changes, Trump will lose terribly, and while he might indeed lose Electoral College by less than popular vote, it will be completely inconsequential.

For elections to be even competitive, fairly big changes must happen. But while it's possible, there's no reason to expect that they'll uniformly apply to all states (in proportion to their "swinginess").

For example if Biden says something that will offend a million Californians and makes them stay at home or vote third party, this will significantly reduce his popular vote share, but it won't matter for Electoral College one bit, since California will go Democratic regardless.

Trump needs not only a really big swing - he needs a swing that's either eerily uniform, or happen in just the right states.

In reality, if a major swing happens, it throws all assumptions about Electoral College advantage out of the window. Even if much bigger swing than could be reasonably expected happens, and Trump somehow manages to reach vote share similar to what he got in 2016 (losing popular vote by 2%), it's actually still quite unlikely that this swing will leave his Electoral College advantage intact, and get him another victory.

Why 538 is completely wrong - the economy will not favor Trump

A popular way to predict elections is by looking at "economic fundamentals". The thinking goes that whenever economy is doing poorly, people blame whoever is in charge for it, and vote for someone else. This is generally true, and after every major economic crisis like 1929 or 2008, governments all across the world got a lot less popular, regardless of which part of the political spectrum they were on, and how much or how little they had to do with causes of the crisis. There's no reason to expect 2020 to be any different.

538 really hates fundamentals models, and for a good reason. While bad "economy" affects people's voting preferences, there's no obvious way to pick a single number for what is the "economy". You can use unemployment, or GDP, or disposable income, or stock market, or so many other indicators, of combinations of indicators. So in year were economy is doing so-so, you can't really use those models to predict much.

Instead of stopping there, 538 somehow decided to ask experts where the economy will go just before elections, and since most expect some recovery from the worst of the lockdowns, they count this as benefiting Trump. While there's a room for reasonable disagreement how to model this, there's no way in hell 2020 economy could possibly benefit Trump, people's memories are longer than a few weeks, and US is still and will likely be in state of mass unemployment.

Are 2020 Elections a foregone conclusion?

Democrats are overwhelmingly likely to win the House - they'll win the popular vote, and there's not even a possibility of Electoral College stopping them. Republican House is about as likely as Nicola Sturgeon becoming the next Prime Minister.

Biden is extremely likely to win presidency - things would need to go really damn well for Trump.

Senate is another matter. Only 1/3 of it is reelected each time, and Republicans do have a meaningful structural advantage there, so it seems fairly close to even who's going to win. And it probably matters more than anything else this year.

If Democrats control the Senate, they'll be able to abolish filibuster, pass pretty much whatever they want, and then possibly also pack Supreme Court with ten new liberal judges that will pronounce all of that Constitutional. They probably won't go that far, but in principle they could.

In Democrats keep the House, and Republicans keep the Senate, then it will most likely be four more years of gridlock regardless of who's the president. The parties hate each other so much, they won't compromise on anything beyond the most routine operations of the government. Biden's policies will be different than Trump's, but presidential policies don't matter as much as people thing, an the next president in 2024 might very well reverse most of them.

And yes, I obviously put money where my mouth is.

Friday, August 14, 2020

Trying out Minecraft again after 10 years

Dolphin who followed me to my base, jumped onto the sand, and died tragically.

Some time later its buddy made the same bad life choice.

Like everyone else, I played Minecraft really long time ago. I recently discovered that the game is still receiving regular updates after all those years. Thanks to corona, I have a bit more free time than usual, so I decided to give it another go!

Here's my thoughts. Little of this will be new to you if you played recently.

My favourite changes

I really like the new villager system. This is possibly the biggest change - trading with villagers at first seems pointless, as they don't sell anything useful, but it turns out they level up, and offer more and more interesting items - while stuff they buy you can build semi-automated system to collect.

It's also possible to train villagers to choose new profession, breed villagers, and they have some protection with iron golems they summon, and some enemies with pillagers.

Treasure ships and treasure maps found in them are great. You'll randomly run into them as you explore, and they have nice stuff inside, and map where to get even more nice stuff. I really didn't love how I got two copies of same treasure map once, but such small side quests are great fun.

Boats no longer take damage when sailing around! Looks like they changed that back in 2012, that makes boats a great transportation system, and canals for them are quite easy to build.

The game now shows you recipes when you open crafting table, and that's a great change! No need to alt tab for rarely used things.

The game needs a damn map!

The game now has maps, and I thought that's great until I tried actually using them, and omg they're truly terrible.

This was one thing that finally convinced me to go through the hassle of installing Forge modding system (make sure you get one that matches your Minecraft's version), and then both Xaero's map mods (world map, minimap). It seems like such a small thing, but honestly it makes bigger difference for player enjoyment than everything they changed in Minecraft over last 10 years put together.

Game still hates the player sometimes

The game has its share of "fuck you player" moments. Falling into lava and losing all you stuff is a classic, but the game has more.

Like when you get hit by a Wither Skeleton, that not only damages you when hit, and poisons you so you keep losing your health (which is far enough so far...), but it also hides your health bar, which is just Minecraft devs being assholes.

For another "fuck you player moment", I was just wandering around exploring, setting up temporary bed whenever night fell and then taking it with me afterwards. I was in the boat during daytime, and somehow tridents started flying my way from the water, and I died out of nowhere. I lost all my stuff as usual, but most annoyingly game decided to spawn me at 0, 0 instead of my base or last sleeping place since last bed I slept in was no longer there.

The problem was that in my seed, there were no villages within over a 1000 blocks from my spawn point, and the closest one I finally found was abandoned, so I finally setup my base at about -2000, -1000 or so next to the first inhabited village I found. So by spawning at 0, 0 I wasted like 15 minutes of my life walking there. Just because the devs hate me.

And then there's all the other stuff, like all the sand updating because I placed a torch, resulting in sudden fall into a ravine.

There are games where mechanics like that make sense, but Minecraft is mainly about exploring and building, and last thing it should do is punishing exploration.

Other things that suck

Inexplicably there's still no builtin modding system, and you need to install third party mod manager to even download and install mods. That is truly baffling.

Combat in Minecraft was always bad, and it's still bad. Turning it on Easy doesn't seem to do anything. At some point they changed combat system, and before I watched some youtube tutorials explaining how it really works, it seemed just ridiculous.

Placing item when building a bridge is really obnoxious. And it looks like other versions of Minecraft fixed it, but somehow Java version still uses the old and really tedious system.

With single exception of crafting table recipes, the game is really bad at explaining anything. There's so many things that are completely unintuitive, and which are not explained, and not realistically discoverable. It's as if the game expects players to watch youtube tutorials for half of its mechanics. And sometimes I watched a tutorial, tried something, and it absolutely didn't work, because Bedrock and Java are subtly different, or they updated things at some point, and it no longer works.

I always felt like player's inventory was too small, and it wanted me to keep running back to base too often. It makes sense that I can only carry so much, but why can't my boat or minecart have some extra inventory space, that I'd then bring home with me? Big Boat With Extra Chest - as well as Big Boat With Space For An Extra Animal would be great additions to the game.

Things I didn't care much for

I really didn't care for the Nether and End changes - they're far too combat oriented, and combat in Minecraft is just not very fun, before or after the big combat update.

It feels like there's far too many blocks that just don't do anything different. There's naturally occurring stone in like 10 colors now, and a lot of cosmetic variation that feels largely pointless.

There's a lot of new mobs too, which seem to do nothing interesting.

I don't hate such changes, and I guess they add some extra flavour.

Is it worth playing again?

To be honest compared with other games that had 10+ years of updates, Minecraft changed surprisingly little. Something like Factorio or Crusader Kings II are such drastically different games than they were at release, but Minecraft now and 10 years ago, pretty much the same thing.

I might be worth giving it a go. Then again, if you never played Factorio, you should try that first instead. There, your buildings actually do things.

Sunday, May 03, 2020

Fun New Conspiracy Theories

p1240692 by generalising from flickr (CC-SA)

For your entertainment. There's no hard evidence for any of them, and I'm definitely like actually endorsing any of them, but they all sound quite plausible, right?

Corbyn was a Tory plant

UK is not even close to being a democracy.

It's also trying its best to be a Tory single party state. Starting the timer when the last Liberal Party government ended in 1922, and ending it when Tony Blair was elected with New Labour in 1997, Tories were in government 68% of the time - despite having actual majority of the vote only once.

They achieve this by combination of heavy gerrymandering, radicalising the Labour Party, and splitting the opposition.

UK redistricting is technically performed by "independent" Boundary commissions, but rules under which they're operating resulted in very consistent Tory bias.

Cheating a few percent through gerrymandering is however not enough. Tory Deep State figured out a far better way - by radicalizing Labour Party in ensures moderates are too scared to vote for them, and most of the time manages to split the vote between Labour and Liberals/LibDems/SDP/such.

Political science predicts that in FPTP systems both parties will drift to where the center of the electorate is. Somehow UK is the only exception to this pattern.

All those Tory shenanigans were ended when Tony Blair created New Labour - which was exactly what the political science predicted. Modern pro-European center-left party without any of the openly Communist baggage.

New Labour was mortal enemy of the Tory One Party State project, so they did their best to undermine it. The 2008 Great Recession let them get back to power as part of a coalition government, but that was living on borrowed time.

So what did they do? Tory Deep State dug up Comrade Corbyn, and managed to put him in charge of Labour Party to destroy it. Once in charge, Tory plant Comrade Corbyn somehow managed to get as many open Communists and open Anti-Semites as possible to join to make Labour completely unacceptable to an average voter.

Under control of Tory puppet-masters, Labour adopted the most extremist plans possible on basically every issue. It did not take pro-European position, in spite of overwhelming majority of Labour voters being pro-European.

Normally when a party loses elections, its leader goes, but somehow the Tory puppet managed to stay and lose elections after elections, and only got kicked out this year. Damage Tories managed to do to Labour Party was so extensive, it's not clear if Keir Starmer will ever be able to fix it.

Boris faked coronavirus infection

It is very well established that Boris is willing to lie about literally anything to get the power, and doesn't care about how much damage his actions cause.

So why do people suddenly believe that he wouldn't lie about getting coronavirus and needing ICU?

Let's get some facts.

Boris allegedly got moved to ICU on April 7th. According to data I could find for two days later, only 3883 people in all of UK got COVID-19 ICU.

There's 0.005% chance that a random person would need COVID-19 ICU.

Also, only 20% of patients admitted to ICU for COVID19 recovered.

So the chance of Boris getting COVID-19, needing ICU at this point, and promptly recovering, is about 0.0001%. Do you seriously believe that the chance that habitual liar Boris Johnson lying about this is less than 0.0001%? If so, I have a great EU trade deal to sell you.

And it would be far from the first case of head of state lying about their health.  It sounds totally implausible, but Roosevelt managed to hide his paralysis from the whole public and the press cooperated with that!

Secret Service actively interfered with photographers who tried to take pictures of Roosevelt in a wheelchair or being moved about by others. The Secret Service commonly destroyed photographs they caught being taken in this manner
The list of cases of politicians hiding their illness which only comes out after their death is very long, including nearly half of recent US presidents.

How many faked instead of hiding their illness? That's harder to tell, as there's usually no advantage to do so, but it's sure as hell higher than 0.0001%.

In all likelihood Boris was either totally fine, or just got common cold, and requested ICU admittance to fake COVID-19, as he thought that would be politically beneficial to him.

Democratic governors are destroying their states' economies on purpose

"Fundamentals" models of election forecasting have been accused of overfitting, but their basic message is still true - when economy is doing awfully, people vote against the ruling party. The Great Recession in most countries showed drastic drop in support for whoever happened to be in the government - left wing, right wing, center, or extreme, regardless of how responsible they were, and how well they were doing.

None of that mattered, voters always turned away from the ruling party - either towards the opposition, or towards new usually extremist parties.

Notably there was no such backlash in any subnational elections. Republican states stayed relatively Republican, and Democratic states stayed relatively Democratic, there was no grand flipping.

During corona crisis, Democratic governors have generally issued far more strict lockdown orders than Republican governors, pretty much regardless of how affected their states are. Outside New York, most states aren't particularly affected, and none of that is even close to rational policy.

So here's a fun conspiracy theory to think about - what if they're trying to damage their economies on purpose, because they know Trump is going to get blamed for it anyway?

During normal times it's not something they could get away with, but this is a great unique opportunity to do just that.

Notice this theory doesn't work backwards - if you try to flip that and speculate that Republican governors are trying to make things better to help Trump, well can't blame people for trying to make things better.

Wednesday, December 18, 2019

Challenges for November 2019 SecTalks London

Selfie by SethBahl from flickr (CC-NC)

October 2019 London SecTalks CTF was jointly won by me and Laurence, so we wrote challenges for November together.

There was a total of 9 challenges, 5 written by me and 4 by Laurence. It was a bit much, as one of the challenges got zero solves until after the event. In spite of very generous hints given halfway through the event.

Challenge files and code used to generate them are available on github.

There are no answers below, but some serious hints which might make it too easy.

For previous rounds, see posts about September 2017November 2017May 2018July 2018October 2018February 2019April 2019, and July 2019 CTFs.

Rockstar (5 points)

The puzzle was a rock song / program in Rockstar programming language. Just running the program would print ASCII codes, which could be easily turned into a flag.

Think Outside The Box (10 points)

The challenge is a BMP image file with flag hidden by incorrect image size. Changing the header reveals the image.

Dots (15 points)

This was Unicode Braille, but dots and no-dots were simply 1s and 0s to be decoded, ignoring any "Braille" structure. To my surprise nobody solved it even when given basically full explanations of what they're supposed to do.

CRC32 MAC (20 points)

A signed cookie server, which uses CRC32 with secret key as MAC. CRC32 is extremely unsuitable in this role.

Decryption Engine (25 points)

A small Python program which uses default password based on current date, and encoded file. Assuming password wasn't passed, the space of possible keys is tiny.

Salted MAC (30 points)

A signed cookie server, which uses SipHash with secret key, and then salt just to be extra sure.

In principle this should be difficult to break. The problem is a trivial implementation mistake.

PHPain (40 points)

It was a separately hosted PHP application vulnerable to recently published CVE.

Poly-DIY MAC (50 points)

Another signed cookie server, using a simplified variant of Poly1305. The variant is reasonable, however the server isn't very careful about nonce handling, and such schemes tend to fail catastrophically when nonce is reused.

GeoCaptcha (60 points)

It's a web "Captcha" which asks the user to locate 20 major cities on the map, with any mistake failing the Captcha.

You can play it here (for a while).

It's worth so many points mostly due to its very high entertainment value, not due to difficulty.

I was curious how people would approach it - only 2 players solved it with code (as robots), 6 solved it with tedious manual process (as humans). So I guess it works as a Captcha.

Thursday, October 10, 2019

How to configure OSX 10.14 Mojave for software development

Kitten by Magnus Bråth from flickr (CC-BY)

With every new Macbook, I'm updating the guide, previous version is here.

And I get it that it's ironic that I'm posting this just as 10.15 came out, but I started the setup a few days ago, and in general it's a poor idea to update OS as soon as it comes out.


  • Go to Settings > Security > FireVault, turn on FireVault. This will restart your computer.
  • Install some sensible browser like Chrome or Firefox.
  • Afterwards either sign up into your account on which you hopefully have your ad blocker setup, or install some. Most popular seems to be uBlock Origin these days, but pretty much any of them will do just fine.
  • Install whichever cloud sync service you're using like Dropbox etc. And start syncing your stuff.
  • Install iTerm2 for sensible terminal emulator.
  • Clean up all crap from dock. Other than Launchpad and System Settings, everything else should be gone. Add iTerm2, your browser, and your text editor, and any application you wish to install there instead of stock Apple crap.


Install some sensible text editor. These days most people use Visual Studio Code.

If that's your choice, run it, open Command Palette, and choose: "Shell Command: Install 'code' command in PATH".


Like every other operating system, OSX has a lot of bad default settings. Here are some obvious fixes:
  • Settings > Mouse > increase speed
  • Settings > Keyboard > Key Repeat > Fast
  • Settings > Keyboard > Delay Until Repeat > Short
  • Settings > Sound > Disable "Play user interface sound effects"
  • Settings > Sound > Alert volume > 0% (for Terminal ping)
  • Settings > Sound > Show volume in menu bar
  • Settings > Trackpad > Scroll & Zoom > Disable "Scroll direction: natural"
  • Settings > Displays > Arrangement > drag and drop your external monitors into desired order
  • Settings > Displays > Max out brightness
  • Settings > Displays > Turn off "Automatically adjust brightness"
  • Settings > Dock > enable "Automatically hide and show the Dock"
  • Menu bar > Battery icon in task bar > Enable "Show Percentage"
  • Menu bar > Time icon in task bar > Open Date & Time Preferences > Use a 24-hour clock
  • Menu bar > Time icon in task bar > Open Date & Time Preferences > Show date
  • iTerm > Preferences... > Profiles > Terminal > Unlimited Scrollback
Press Ctrl-Up arrow, add a few desktops (or "spaces" as they were used to know), then go to Settings > Keyboard > Shortcuts > Mission Control - and enable their keyboard shortcuts Ctrl-1 to Ctrl-6 or however many you have there.

Open Screenshot app, choose options, then disable "Show Floating Thumbnail".


OSX already includes drivers for laptop itself, but you might need some for peripheral hardware.

In particular, external PC keyboard need a tweak to work properly, as left and left Windows keys are in reverse order from Mac keyboard.

Go to Settings, Keyboard, Modifier Keys..., and swap positions of Option and Command keys. Feel free to change functionality of Caps Lock key as well, it's a huge easily accessible key with no useful function people love to remap, usually to extra Control.

If you need any special keyboard layouts, get them too.

Standard paths

OSX renames a lot of directories. While in theory scripts could just use env vars to find proper paths, it's more reliable to symlink all the things:

  sudo ln -s /Volumes /mnt
  sudo ln -s /Volumes /media
  sudo mv /home /home-old
  sudo ln -s /Users /home

Development tools

You'll need a package manager, and the only one anyone uses is homebrew, MacPorts and the rest being basically dead. You need to tell homebrew to not spy on you with brew analytics off command.

You'll need Xcode. Fortunately homebrew installer does it for you automatically. If you don't use homebrew for some reason, you can install Xcode manually by running xcode-select --install from command line.

Create new SSH key pair

Before you do that, name your computer something memorable with sudo scutil --set HostName your_host_name command.

Open Terminal and run ssh-keygen to create ~/.ssh/id_rsa, then upload the generated key to any place that needs to know about it like githubbitbucket, or whatever else you use.

Checkout your dotfiles

Hopefully you're storing your dotfiles somewhere. If it's a git repository, or your Dropbox account, get them now and symlink them all properly.

If there are any other repositories you might need, checkout them too.

Install homebrew packages

Your list might vary. Here'sa few obvious suggestions:

brew install rbenv ruby-build mc wget p7zip unrar trash git htop bash zsh youtube-dl jq imagemagick coreutils bash-completion zsh-completion

Then enable all services you installed, unless you want to start them manually:

  ln -sfv /usr/local/opt/*/*.plist ~/Library/LaunchAgents/

And install non-system ruby, so you can install gems without sudo. Currently latest is:

  rbenv install 2.7.0-preview1
  rbenv global 2.7.0-preview1

To make that actually work, you need to make sure ~/.rbenv/shims is in your $PATH. If you type rbenv init, it will tell you what to do.

Due to OSX limitations you'll need to run sudo htop if you want to use htop.

Install gems

Again, your list my vary. These days most of the software will have its own Gemfile so long list of gems are generally unnecessary. But some global utilities are still useful:

gem install bundler rak pry pry-rescue

Sane bash or zsh

bash version shipped with OSX is ancient and BSD utilities are all awful.

You have two choices. Either install modern version of bash from homebrew, or use zsh from homebrow.

First, edit /etc/shells as admin and add the following lines at the end of it:

Then set it as your shell, with whichever one you prefer:
  chsh -s /usr/local/bin/bash $USER
  chsh -s /usr/local/bin/zsh $USER

Either way, run to touch ~/.hushlogin to prevent some worthless spam on every open terminal tab.

If zsh is your choice, the first time you start a new terminal, it will ask you some questions on how to configure it.


This is optional step. OSX coreutils are awful, and generally GNU versions are a lot better, but if you switch there could be occasional minor incompatibilities.

If you want to do so, add GNU coreutils to your PATH:

  export PATH="/usr/local/opt/coreutils/libexec/gnubin:$PATH"
  export MANPATH="/usr/local/opt/coreutils/libexec/gnuman:$MANPATH"

Better window manager controls

Sadly OSX window manager is extremely dubious for keyboard use. Fortunately programs to make it usable exist. I recommend installing HyperSwitch for sane alt-tab window switching, and Spectacle for managing windows.

I previously used HyperSwitch, but it no longer works with latest version of OSX. Spectacle shortcuts are different than HyperSwitch, and I like HyperSwitch's more, so I adjusted shortcuts to match.

You'll need to give them necessary access. To do so:
  • Settings > Security & Privacy > Privacy > Allow the apps below to control your computer > enable HyperSwitch and Spectacle
Also open its preferences, and set it to run in the background, and start at login.

All other software

There's a lot of other software you might want. The most obvious one is the VLC media player.

You might also want some kind of Git UI program, like GitUp or (another for of) GitX.


Once you go through this list, and successfully get everything going, I'd recommend modifying it to your liking and reposting your version on your blog. Everybody's needs are different, so guide like this is just a starting point.

Monday, September 09, 2019

Technologies that inspired me

Cat's bored by SethBahl from flickr (CC-NC)

Nearly every day I run into another language, framework, or program. Most of them are just plain awful, and the ones which aren't are usually nothing special.

Every now and then I run into a true gem. Here's a list of such great technologies, in approximately the order I found them.


Perl was absolutely amazing compared with every language which came before it. Other languages were motivated by what worked for computers, or by some silly mathematical theory, or sometimes like with Java by outright hostility to programmers.

Perl tried to be as much like a natural language as possible. If what you wrote made some kind of sense, it was going to try to run it. And most of the time it worked.

Unfortunately Perl supported too much weird stuff, and it caused the language to collapse under its own weight, but the most influential languages of the 1990s like Ruby, Javascript, and PHP were very heavily inspired by Perl. And as next generation of languages was very heavily inspired by those, the legacy of Perl lives on.

Perl tried to double down on its philosophy with Perl 6, but without first dropping failed ideas it was never really going to work.

Regular Expressions

Text is nearly as common data type as numbers, but most pre-Perl languages didn't treat text data as first class. In many old languages text is literally an array of small numbers, or such other nonsense. Even today Erlang / Elixir do literally that - "Hello" is the same as [72, 101, 108, 108, 111].

Regular expressions are to text what arithmetic is to numbers, and let you extract or modify text with tiny amount of highly readable and highly performant code. They have weird reputation of unreadability, but one line of dense regular expression is infinitely easier to understand than 100 lines of text processing code it replaces.

It's key skill for every programmer, no matter what's the domain. If you don't know regular expressions, drop everything and do some regexp crosswords.

There used to be different variants of regular expressions, but these days basically every language uses some variant of "Perl-Compatible Regular Expressions". Except Unix grep command, which still uses stupid 1970s regexps for some crazy reason.


Wiki was a revolutionary idea - just put a bunch of linked pages online, and let anyone who wants edit them. It sounds like a recipe for disaster, but it worked remarkably well. Wikipedia was the greatest success story of wikis, but there are all kinds of wikis for all kinds of content. And as a general rule, they're a lot more reliable than non-editable sites.


Ruby took everything that made Perl great, everything that made Lisp great, everything that made Smalltalk great, then it threw away their bad parts. And then it added more magic on top.

It is the high point of programming langugae design, unmatched up to this day. Implementation issues aside, I'd rather use Ruby as it existed 20 years ago than any language that came out since. It's just baffling how far ahead of time it was.

Of course it could be even better, but that's true of anything.

I hope someday a language comes that's better than Ruby, but so far it doesn't look like anyone's even really trying. The best anyone's trying to do is to make their languages as close to Ruby as possible given various constraints.

Ruby on Rails

Ruby on Rails is another miracle technology. When it came out, it was completely unmatched as the best way to write web backends. Today, after the web changed beyond recognition, it's still completely unmatched as the best way to write web backends. It's rare for any technology to have such staying power.

To always stay on top, Rails evolved very aggressively, and Rails upgrades on large codebases are notoriously difficult, but then again, the choice is between upgrading Rails 4 to Rails 5, or writing your code in some shitty framework that's going to die, and fully rewriting it into another shitty framework that also won't last.

It's quite baffling that so far no real competitor to Rails emerged. The best everyone else can do is to copy as much as they can from Rails, and even that is usually really poor.

The main reason is that Ruby is completely unmatched at making DSLs than any other language, and Rails is very DSL-heavy. But other aspects of Rails success like everything working seamlessly out of the box (while being individually replaceable) without needing to duct tape a bunch of disparate components should be replicable, and yet nobody even tries.

Why so many frameworks come out without such basic things like comprehensive testing system, or database migration system, or sensible security defaults? I suspect their authors just don't get Rails, and people who get Rails use Rails.


Once upon a time, there were thousands of editors and they were all 💩. It was a running joke that every programmer writes their own editor, and their own IRC client (and later their own Javascript framework).

And just as with infinite monkeys randomly typing on infinite number of keyboards, one of such attempt miraculously produced TextMate - the first editor which was not 💩.

Multiple cursors, command palette, file palette, extension system, nested file types, integration with modern UIs, sensible keybindings making use of what's on modern keyboards - TextMate was far ahead of everything else.

TextMate itself didn't last, but Sublime Text, Atom, VS Code, and some others are essentially TextMate clones, and they completely dominate the editor world.

The only non-TextMate editors still in use are people using IDEs like IntelliJ or Android Studio due to being forced to use programmer-hostile languages like Java. And people who use Vim because they still haven't figured out how to quit it.


Javascript used to be awful beyond belief. Well, it's still not great, but back then doing even the most trivial dynamic interactions, like changing some class based on an user action was a page of code, and then another page of different code if you wanted to support more than one browser.

Then jQuery came, and with a few $("selector").action() you could write complex UI interactions with very little code. It was even bigger difference when it came to jQuery AJAX vs browser APIs for AJAX. The gap was as enormous as between regular expressions and manipulating text by shuffling arrays of characters around.

Gradually browser APIs became more compatible, more expressive, and less buggy, they added a lot of features to close the gap with jQuery, so now it's sort of viable to write short programs directly with browser APIs. And while jQuery is amazing at progressively enhancing website functionality, it's a stretch if you want to make Single Page Applications.

So is it the end of jQuery era? Not even close! jQuery is still vastly more popular than all frontend frameworks put together - 80% of top million websites use jQuery, while all the hipster frameworks put together are still just a rounding error. Currently most popular React at 1.6% is behind 20 different jQuery plugins.

And beyond jQuery itself, D3 is basically jQuery for data visualization. Most acceptance testing frameworks from Capybara to Cypress use jQuery-style for writing tests. Single Page Applications are useful, but most of the Web is better off not being an SPA, and good jQuery skills will never go to waste.

It might be unpopular opinion now, but I feel the best way people should learn frontend is still be HTML, CSS, jQuery, then ES6+ Javascript, and only then some SPA framework like React.

Amazon Web Services

I remember bad the old days when setting up any server required making a request to a grumpy sysadmin (or even worse - a grumpy DBA) and waiting potentially weeks for them to find some time for it. Even more fun if that required more physical hardware, and budget requests for those. Meanwhile, there was nothing you could do.

Then came Amazon and decided that making billions selling everything wasn't enough so it opened a small side business of getting rid of grumpy sysadmins. Want to start a new server? A few clicks and it's done by the time your coffee finished brewing. And they kept adding new services beyond the initial EC2 and S3, to simplify your life more and more.

I feels silly to even mention how great it is, as AWS and other Clouds won so utterly that most people barely remember the pre-Cloud days, and those who do would rather forget them.


I didn't love debuggers, and I usually employed print-based debugging as my first choice, strace and friends as my second choice, and only used interactive debuggers if all else failed.

pry was something else. A hybrid of a debugger and REPL, with full power of Ruby metaprogramming and introspection, it was on a completely different level than anything I've used before.

Adding binding.pry became my first not the last resort. And then there's pry-rescue and other plugins to make things even more awesome.

I'm still annoyed that pry isn't just part of Ruby yet, and that bundler blocks pry unless you explicitly add it to every single Gemfile just in case.
Sure there's finally binding.irb, but irb is not even close to pry. Seriously, get your act together guys and get pry into Ruby standard library.

MongoDB document model

Relational databases are just a bunch of damn spreadsheets, and forcing your program's object oriented (or otherwise richly structured) data into spreadsheets for storage is invariably painful. Object-Relational Mapping tries to make it bearable, with limited degree of success.

There were many attempts to create alternative database models, but they were either specific to one programming language, or used something like XML which was just as awkward as spreadsheets.

Then came MongoDB, made everything JSON, and it was glorious. Oh and depending on your application it could easily be orders of magnitude faster than relational databases.

Unfortunately while MongoDB JSON-based document model was amazing, and its performance was generally outstanding, everything else about MongoDB was less so. Doing even relatively simple queries in MongoDB is stupidly painful compared with SQL, so you end up doing it all in application. Its performance-first defaults got a lot of people into trouble as well.

I'm surprised that even a decade later nobody managed to build anything that united JSON-based (as opposed to spreadsheet-based) document model with some rich query language that can at least match SQL.

Spreadsheet-based databases nowadays support JSON data a little bit, NoSQL databases nowadays have a bit richer query languages, but the gap between those two is still enormous.


TDD lived in this weird state, where backend code was decently tested, but frontend testing was minimal and mostly manual.

Even today most tests for JS programs are fairly low value efforts, like "component renders without crashing" tests and snapshot tests.

Capybara was the first practical way to do end to end tests of web apps with dynamic frontend and dynamic backend as a single system.

Nothing better than Capybara was created so far. There's a bunch of fairly poor Capybara clones in other languages, which might be just about good enough. Cypress might be developing into the first real alternative, but it's very frontend-based and integration with dynamic backends is its secondary consideration.

Continuous Integration servers

I wasted too much time running tests manually, or not bothering to run them because the change I was doing was trivial, and the tests took too long.

There is an answer to that - CI servers. Most companies have them. But people coding for themselves, or coding Open Source software are still usually not using them.

Semaphore CI and many other sites let you easily setup CI for free for your Open Source project, and you should do that.

Chrome Developer Tools

The story started in the olden days with Firebug, which provided just console.log and a few other very simple functions. Or possibly even with the View Source button (which some people wanted to delete from browsers to "protect the source code", #truestory).

Nowadays every browser comes with amazing debugging and development tools, and they keep getting better every year.

It's one of few places where I feel Javascript has a big lead over other languages. pry might be as good as possible on terminal, but it's still limited to a terminal. Chrome Developer Tools have full power of the browser UI, and they fully use it.


I have multiple computers. Also multiple phones and tablets. Sharing content between them used to be such a mess - connecting and disconnecting external disks, or manually copying files by cables or rsync. Dropbox made it seem that the same file is magically present in every location. It just works, with very few issues.

Unfortunately Dropbox for phones is quite bad, so if anyone solves this better, I'm interested.

Weirdly many Dropbox alternatives like Google Drive are simply horrendous.


Just like everyone was writing their editor, everyone was writig their IRC client. Or some other chat system which was basically IRC clone.

They were all solving the wrong problem. The issue wasn't that IRC needed better UI, the issue was that IRC was based on fundamentally wrong assumptions (broadcast to whoever happens to be connected at time) that made it completely unsalvageable.

Slack fixed those assumptions, and it completely revolutionized workplace communication, not just in tech.

All post-Slack chat systems like Discord use basically Slack model.


Electron is the first time in history of computing where we have cross-platform desktop apps with decent UI. There's performance cost, as Electron apps bundle whole browser, and are infamous for their memory use, but there's no alternative, so let's just throw more hardware at the problem, and restart every now and then if it gets too bad.

There were a few earlier attempts like with Java, but the UX was just attrocious. I cringe whenever I have to use some Java-based GUI software.


Normal programming answers questions like x = 2+2 perfectly fine, but it is shockingly bad at dealing with questions like 2+x = 4. Z3 is an amazing system for answering just such questions.

Solving sudoku, hacking CTFs, proving hardware design? Z3 has you covered.

I liked it so much that I wrote a Ruby gem for it. It's much nicer than official APIs.


I initially dismissed Docker as just another VM. It turns out that such lightweight micro-VMs have a lot more uses, and just being able to install and run a lot of images neatly solves a lot of issues like CI, software with weird dependency, security isolation and so on.


Javascript used to have a million frameworks. They were generally called "MVC frameworks" back then, a name you don't hear much anymore, as it turns out MVC is a poor fit to what they were trying to do.

At certain point very-much-non-MVC React pretty much won, by being decent at solving smaller problem (views), not even trying to deal with the hard problem (model and network layers / state management), and so having a lot easier learning curve and fewer gotchas. Acceptable performance and being pushed hard by Facebook also helped.

I even liked React before I found Imba. Imba however, is on a completely different level. It's the least known technology on this list, so I'll need to write a separate post about it, but you'll get orders of magnitude higher performance at fraction of code size, all while being a lot simpler than React.

It's a tiny project with very few development resources, so it feels like it really needs someone to push it to some sort of Imba 2 fully embracing post-ES6 world. But even in its current state, it's years ahead of any alternative.

Saturday, August 10, 2019

Kingdoms Grand Campaign Mod - Medieval 2 Total War mod highlight

18/8.2014 - kitten royalty by julochka from flickr (CC-NC)

Medieval 2, released back in 2006, was the peak of Total War series. One of the great things about it is abundance of mods, and this post is about one such mod.

Kingdoms Grand Campaign Mod

The idea behind Kingdoms Grand Campaign Mod is taking content from minicampaigns from Kingdoms expansion pack, and integrating them into one big campaign.

Campaign map is adjusted somewhat, and in addition to vanilla factions, you get:
  • Antioch from Crusades campaign
  • Ireland from Britannia campaign
  • Jerusalem from Crusades campaign
  • Kiev from Teutonic campaign
  • Lithuania from Teutonic campaign
  • Mongols from Teutonic campaign (they're in vanilla as unplayable horde)
  • Norway from Britannia and Teutonic campaign
  • Wales from Britannia campaign
For reasons I don't understand, Portugal is no longer a playable faction.

Kingdoms Features

The mod brings in extra units from Kingdoms campaign, like Mangonel, which is the only pre-gunpowder artillery unit worth using in battles.

Forts are now permanent stone buildings, and cost a lot more money.

Hanseatic League and potential Lithuania conversion events from Teutonic campaign are in the game.

Merchants are actually worth something.

Game balance

Medieval 2 is very easy for experienced player, so the mod does a few things to make it more difficult.

There are a lot fewer rebel settlements, and they tend to have a lot bigger garrison, so you can't just grab enormous areas of land without any opposition during your first few turns.

Time scale is set to 2 turns per year. All buildings build a lot more slowly.

Units provided by buildings are a lot different - high tier building no longer allow recruitment of low tier units, possibly to help AI or to clean up the UI.

Castles are massively nerfed because base castle buildings no longer provide any recruitment options, and you actually need to build stables and barracks (which also build really slowly). On the other hand, a lot of castle units get free upkeep as long as they're castle garrison.

Pope is as bitchy as ever, and due to a lot fewer rebels, he's going to be excommunicating people left and right a lot more than in vanilla. I never liked that part of the game, and I wished they toned it down.

AI gets a lot of free money on hard and very hard. On any difficulty levels, if it goes into severe enough debt, it gets a lot of money to get it out of debt.

And then there's garrison script.

Garrison script

Unlike later Total War games, Medieval 2 settlements have no automatic garrison, and AI often leaves them with minimal defences. That allows player to really easily destroy all factions.

As AI isn't really fixable, to counter that, a lot of mods add garrison script which spawn a lot of troops when garrison is besieged. I'm not convinced this is a great idea - I hated it when I didn't know where garrisons are going to happen and how big.

Once I generated a list of which faction would spawn what kind of units where, it was actually decent. So here's the script. Once it triggers for a settlement there's 10 turn timer before it can trigger again. Only AI gets free troops. Some factions have script for settlements that don't belong to them at start of campaign.

  • Antioch (14) - 1 Knights Hospitaller, 2 Antioch Militia, 3 Armenians of Celicia, 3 Dismounted Knights of Antioch, 5 Hospitaller Sergeant
  • Krak_de_Chevaliers (12) - 2 Dismounted Knights of Antioch, 2 Knights Hospitaller, 3 Armenians of Celicia, 5 Hospitaller Sergeant
  • Huaxtepec (14) - 1 Coyote Priests, 2 Aztec Archers, 2 Jaguar Warriors, 3 Arrow Warriors, 6 Cuahchiqueh
  • Tenochtitlan (14) - 1 Coyote Priests, 2 Aztec Archers, 2 Jaguar Warriors, 3 Arrow Warriors, 6 Cuahchiqueh
  • Tixtla (14) - 1 Coyote Priests, 2 Aztec Archers, 2 Jaguar Warriors, 3 Arrow Warriors, 6 Cuahchiqueh
  • Tollan (14) - 1 Coyote Priests, 2 Aztec Archers, 2 Jaguar Warriors, 3 Arrow Warriors, 6 Cuahchiqueh
  • Constantinople (16) - 1 Greek Firethrower, 2 SE Town Militia, 4 Byzantine Guard Archers, 4 Varangian Guard, 5 Byzantine Infantry
  • Corinth (11) - 1 Byzantine Lancers, 3 Byzantine Spearmen, 3 Trebizond Archers, 4 Dismounted Byzantine Lancers
  • Smyrna (11) - 1 Byzantine Lancers, 3 Byzantine Spearmen, 3 Trebizond Archers, 4 Dismounted Byzantine Lancers
  • Thessalonica (10) - 1 Byzantine Lancers, 2 Pronoia Infantry, 3 Trebizond Archers, 4 Byzantine Infantry
  • Trebizond (11) - 1 Byzantine Lancers, 3 Byzantine Spearmen, 3 Trebizond Archers, 4 Dismounted Byzantine Lancers
  • Arhus (14) - 1 Feudal Knights, 2 Town Militia, 3 Norse Archers, 3 Sami Axemen, 5 Norse Swordsmen
  • Hamburg (11) - 1 Huscarls, 2 Sami Axemen, 3 Norse Archers, 5 Dismounted Huscarls
  • Lund (11) - 1 Huscarls, 2 Sami Axemen, 3 Norse Archers, 5 Dismounted Huscarls
  • Alexandria (11) - 1 Mamluks, 2 Hashishim, 3 Desert Archers, 5 Saracen Militia
  • Cairo (15) - 1 Mamluks, 2 al Ashair, 3 Tabardariyya, 4 Desert Archers, 5 Saracen Militia
  • Jerusalem (12) - 1 Mamluks, 3 Desert Archers, 3 Hashishim, 5 Saracen Militia
  • Caen (11) - 1 Feudal Knights, 2 Armored Swordsmen, 3 Longbowmen, 5 Armored Sergeants
  • Dublin (11) - 1 Mailed Knights, 2 Armored Sergeants, 3 Archer Militia, 5 Spear Militia
  • Gloucester (11) - 1 Mailed Knights, 2 Armored Sergeants, 3 Longbowmen, 5 Spear Militia
  • London (14) - 1 Feudal Knights, 2 Town Militia, 3 Armored Swordsmen, 3 Longbowmen, 5 Spear Militia
  • Nottingham (11) - 1 Feudal Knights, 2 Armored Swordsmen, 3 Longbowmen, 5 Armored Sergeants
  • York (11) - 1 Mailed Knights, 2 Armored Sergeants, 3 Longbowmen, 5 Spear Militia
  • Angers (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Peasant Crossbowmen, 5 Armored Sergeants
  • Caen (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Peasant Crossbowmen, 5 Armored Sergeants
  • Marseille (10) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Crossbowmen, 4 Spear Militia
  • Paris (14) - 1 Feudal Knights, 2 Town Militia, 3 Dismounted Feudal Knights, 3 Scots Guard, 5 Spear Militia
  • Toulouse (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Peasant Crossbowmen, 5 Armored Sergeants
  • Bologna (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Crossbow Militia, 5 Spear Militia
  • Frankfurt (14) - 1 Teutonic Knights, 2 Town Militia, 3 Crossbow Militia, 3 Dismounted Feudal Knights, 5 Spear Militia
  • Hamburg (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Peasant Crossbowmen, 5 Armored Sergeants
  • Innsbruck (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Peasant Crossbowmen, 5 Armored Sergeants
  • Bran (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Bosnian Archers, 5 Pavise Spearmen
  • Budapest (14) - 1 Feudal Knights, 2 EE Town Militia, 3 Bosnian Archers, 3 Dismounted Feudal Knights, 5 EE Spear Militia
  • Cork (14) - 1 Ridire, 2 Ceitherne, 3 Galloglaich, 3 Saighdeoir, 5 Ostmen
  • Donegal (11) - 1 Ridire, 2 Galloglaich, 3 Saighdeoir, 5 Cliathairi
  • Galway (11) - 1 Ridire, 2 Galloglaich, 3 Saighdeoir, 5 Cliathairi
  • Acre (11) - 1 Knights Templar, 2 Dismounted Knights Of Jerusalem, 3 Maronites of Lebanon, 5 Templar Sergeant
  • Jerusalem (14) - 1 Knights Templar, 2 Pisan and Geonese sailors, 3 Dismounted Knights Of Jerusalem, 3 Maronites of Lebanon, 5 Templar Sergeant
Kievan Rus:
  • Kiev (11) - 1 EE Cavalry Militia, 2 Berdiche Axemen, 3 Dismounted Dvor, 5 EE Spear Militia
  • Moscow (11) - 1 EE Cavalry Militia, 2 Dismounted Boyar Sons, 3 Dismounted Dvor, 5 EE Spear Militia
  • Novgorod (11) - 1 EE Cavalry Militia, 2 Dismounted Boyar Sons, 3 Dismounted Dvor, 5 EE Spear Militia
  • Smolensk (11) - 1 EE Cavalry Militia, 2 Dismounted Boyar Sons, 3 Dismounted Dvor, 5 EE Spear Militia
  • Minsk (11) - 1 Szlachta, 2 Samogitian Axemen, 3 Baltic Archers, 5 Dismounted Szlachta
  • Riga (12) - 1 Dievas Guard, 3 Baltic Archers, 3 Samogitian Axemen, 5 EE Spear Militia
  • Vilnius (14) - 1 Dievas Guard, 2 EE Town Militia, 3 Baltic Archers, 3 Samogitian Axemen, 5 EE Spear Militia
  • Bologna (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Genoese Crossbow Militia, 5 Italian Spear Militia
  • Genoa (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Genoese Crossbow Militia, 5 Italian Spear Militia
  • Milan (14) - 1 Feudal Knights, 2 Italian Militia, 3 Dismounted Feudal Knights, 3 Genoese Crossbow Militia, 5 Italian Spear Militia
  • Rome (14) - 1 Feudal Knights, 2 Italian Militia, 3 Dismounted Feudal Knights, 3 Genoese Crossbow Militia, 5 Italian Spear Militia
  • Bulgar (13) - 1 Khan's Guard, 1 Naffatun, 3 Dismounted Heavy Archers, 3 Dismounted Heavy Lancers, 5 ME Halberd Militia
  • Constantinople (12) - 1 Khan's Guard, 1 Naffatun, 2 Dismounted Heavy Lancers, 3 Dismounted Heavy Archers, 5 ME Spear Militia
  • Kiev (12) - 1 Khan's Guard, 1 Naffatun, 2 Dismounted Heavy Lancers, 3 Dismounted Heavy Archers, 5 ME Spear Militia
  • Moscow (12) - 1 Khan's Guard, 1 Naffatun, 2 Dismounted Heavy Lancers, 3 Dismounted Heavy Archers, 5 ME Spear Militia
  • Novgorod (12) - 1 Khan's Guard, 1 Naffatun, 2 Dismounted Heavy Lancers, 3 Dismounted Heavy Archers, 5 ME Spear Militia
  • Sarkel (12) - 1 Naffatun, 2 Mongol Heavy Archers, 2 Mongol Heavy Lancers, 3 Dismounted Heavy Archers, 4 Dismounted Heavy Lancers
  • Smolensk (12) - 1 Naffatun, 2 Mongol Heavy Archers, 2 Mongol Heavy Lancers, 3 Dismounted Heavy Archers, 4 Dismounted Heavy Lancers
  • Trebizond (12) - 1 Naffatun, 2 Mongol Heavy Archers, 2 Mongol Heavy Lancers, 3 Dismounted Heavy Archers, 4 Dismounted Heavy Lancers
  • Viatka (12) - 1 Naffatun, 2 Mongol Heavy Archers, 2 Mongol Heavy Lancers, 3 Dismounted Heavy Archers, 4 Dismounted Heavy Lancers
  • Vilnius (12) - 1 Khan's Guard, 1 Naffatun, 2 Dismounted Heavy Lancers, 3 Dismounted Heavy Archers, 5 ME Spear Militia
  • Yerevan (11) - 1 Khan's Guard, 1 Naffatun, 2 Dismounted Heavy Lancers, 3 Dismounted Heavy Archers, 4 ME Spear Militia
  • Algiers (11) - 1 Arab Cavalry, 2 Dismounted Arab Cavalry, 3 Desert Archers, 5 Lamtuna Spearmen
  • Cordoba (14) - 1 Arab Cavalry, 2 ME Town Militia, 3 Desert Archers, 3 Hashishim, 5 ME Spear Militia
  • Granada (11) - 1 Arab Cavalry, 2 Dismounted Arab Cavalry, 3 Desert Archers, 5 Lamtuna Spearmen
  • Lisbon (11) - 1 Arab Cavalry, 2 Dismounted Arab Cavalry, 3 Desert Archers, 5 Lamtuna Spearmen
  • Marrakesh (11) - 1 Arab Cavalry, 2 ME Town Militia, 3 Desert Archers, 5 ME Spear Militia
  • Valencia (11) - 1 Arab Cavalry, 2 Dismounted Arab Cavalry, 3 Desert Archers, 5 Lamtuna Spearmen
  • Agder (11) - 1 Huscarls, 2 Dismounted Feudal Knights, 3 Norse Archers, 5 Spear Militia
  • Oslo (14) - 1 Huscarls, 2 Town Militia, 3 Dismounted Feudal Knights, 3 Norse Archers, 5 Norse Swordsmen
  • Wick (11) - 1 Huscarls, 2 Dismounted Feudal Knights, 3 Norse Archers, 5 Norse Swordsmen
  • Novgorod (11) - 1 EE Cavalry Militia, 2 Dismounted Boyar Sons, 3 Dismounted Dvor, 5 EE Spear Militia
Papal States:
  • Rome (17) - 2 Italian Militia, 3 Dismounted Feudal Knights, 4 Italian Spear Militia, 4 Papal Guard, 4 Pavise Crossbow Militia
  • Danzig (11) - 1 Polish Knights, 2 Dismounted Polish Knights, 3 Lithuanian Archers, 5 EE Spear Militia
  • Krakow (14) - 1 Polish Knights, 2 EE Town Militia, 3 Dismounted Polish Knights, 3 Lithuanian Archers, 5 EE Spear Militia
  • Poznan (11) - 1 Polish Knights, 2 Dismounted Polish Knights, 3 Lithuanian Archers, 5 EE Spear Militia
  • Lisbon (14) - 1 Feudal Knights, 2 Town Militia, 3 Crossbow Militia, 3 Dismounted Feudal Knights, 5 Swordsmen Militia
  • Oporto (10) - 2 Dismounted Feudal Knights, 3 Crossbow Militia, 5 Swordsmen Militia
  • Edinburgh (14) - 1 Feudal Knights, 2 Town Militia, 3 Dismounted Feudal Knights, 3 Highland Archers, 5 Spear Militia
  • Wick (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Highland Archers, 5 Spear Militia
  • Naples (14) - 1 Norman Knights, 2 Italian Militia, 3 Dismounted Norman Knights, 3 Sicilian Muslim Archers, 5 Italian Spear Militia
  • Palermo (11) - 1 Norman Knights, 2 Dismounted Norman Knights, 3 Sicilian Muslim Archers, 5 Armored Sergeants
  • Rome (14) - 1 Norman Knights, 2 Italian Militia, 3 Dismounted Norman Knights, 3 Sicilian Muslim Archers, 5 Italian Spear Militia
  • Leon (14) - 1 Knights of Santiago, 2 Town Militia, 3 Crossbow Militia, 3 Dismounted Feudal Knights, 5 Swordsmen Militia
  • Lisbon (14) - 1 Feudal Knights, 2 Town Militia, 3 Crossbow Militia, 3 Dismounted Feudal Knights, 5 Swordsmen Militia
  • Toledo (11) - 1 Knights of Santiago, 2 Dismounted Feudal Knights, 3 Peasant Crossbowmen, 5 Swordsmen Militia
  • Valencia (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Peasant Crossbowmen, 5 Swordsmen Militia
Teutonic Order:
  • Danzig (12) - 1 Christ Knights, 3 Order Spearmen, 3 Prussian Archers, 5 Sword Brethren
  • Konigsberg (12) - 1 Christ Knights, 3 Order Spearmen, 3 Prussian Archers, 5 Sword Brethren
  • Marienburg (14) - 1 Christ Knights, 2 Dismounted Ritterbruder, 2 Order Militia, 3 Prussian Archers, 6 Order Spearmen
  • Riga (12) - 1 Christ Knights, 3 Order Spearmen, 3 Prussian Archers, 5 Sword Brethren
  • Adana (11) - 1 Turkish Horse Archers, 2 Hashishim, 3 Turkish Archers, 5 Dismounted Hasham
  • Antioch (13) - 2 Kurdish Auxiliaries, 3 Hashishim, 3 Janissary Archers, 5 Saracen Militia
  • Baghdad (15) - 2 Kurdish Auxiliaries, 2 ME Town Militia, 3 Janissary Archers, 3 Janissary Heavy Inf, 5 Saracen Militia
  • Constantinople (12) - 2 Hashishim, 2 Kurdish Auxiliaries, 3 Janissary Archers, 5 Saracen Militia
  • Iconium (11) - 1 Turkish Horse Archers, 2 Hashishim, 3 Turkish Archers, 5 Dismounted Hasham
  • Yerevan (11) - 2 Hashishim, 2 Kurdish Auxiliaries, 3 Turkish Archers, 4 Saracen Militia
  • Bologna (11) - 1 Feudal Knights, 2 Venetian Heavy Infantry, 3 Venetian Archers, 5 Italian Spear Militia
  • Ragusa (12) - 2 Italian Militia, 2 Venetian Heavy Infantry, 3 Venetian Archers, 5 Italian Spear Militia
  • Rome (13) - 2 Italian Militia, 3 Venetian Archers, 3 Venetian Heavy Infantry, 5 Italian Spear Militia
  • Venice (13) - 2 Italian Militia, 3 Venetian Archers, 3 Venetian Heavy Infantry, 5 Italian Spear Militia
  • Caernarvon (15) - 2 Welsh Militiamen, 2 Welsh Skirmishers, 3 Rhyfelwyr, 3 Saethwyr, 5 Meirionnydd Spearmen
  • Cardiff (12) - 2 Gwent Raiders, 2 Welsh Skirmishers, 3 Saethwyr, 5 Spear Militia

Free Upkeep Units

These are only free if settlement has right building to recruit them. Some are also free in forts, I think these are castle units. Generally all militia units (including cavalry militia) and castle infantry are upkeep free while garrisoned. Castle cavalry, siege units, and boats are not free.
  • Ahdath - turks
  • Al Haqa Infantry - egypt
  • Alamanoi - byzantium, slave
  • Almughavars - spain, portugal
  • Antioch Militia - antioch
  • Archer Militia - england, slave
  • Armored Sergeants - england, france, hre, milan, venice, papal_states, sicily, slave
  • Aventuros - portugal
  • Aztec Spearmen - aztecs, slave
  • Aztec Warriors - aztecs, slave
  • Baltic Archers - lithuania
  • Bill Militia - england
  • Burgher Pikemen - teutonic_order, slave
  • Byzantine Spearmen - byzantium, slave
  • Ceitherne - ireland, slave
  • Cliathairi - ireland
  • Croat Axemen - hungary, slave
  • Crossbow Militia - france, hre, denmark, spain, portugal, teutonic_order, norway, slave
  • Deisi Javelinmen - ireland
  • Dismounted Byzantine Lancers - byzantium
  • Dismounted Christian Guard - moors
  • Dismounted Conquistadores - spain, portugal
  • Dismounted Druchima - russia, novgorod
  • Dismounted Feudal Knights - england, scotland, france, hre, denmark, spain, portugal, milan, venice, papal_states, hungary, norway
  • Dismounted Ghulams - egypt, slave
  • Dismounted Halbbruder - teutonic_order
  • Dismounted Hasham - turks
  • Dismounted Heavy Lancers - mongols, timurids
  • Dismounted Knights Of Jerusalem - jerusalem
  • Dismounted Knights of Antioch - antioch
  • Dismounted Light Lancers - mongols
  • Dismounted Norman Knights - sicily
  • Dismounted Polish Knights - poland
  • Dismounted Polish Nobles - poland
  • Dismounted Szlachta - lithuania
  • Dismounted Tartar Lancers - russia, lithuania
  • EE Archer Militia - russia, novgorod, slave
  • EE Cavalry Militia - russia, novgorod
  • EE Crossbow Militia - poland, russia, novgorod, lithuania, slave
  • EE Spear Militia - poland, russia, novgorod, hungary, lithuania, slave
  • EE Spearmen - poland, russia, novgorod
  • EE Town Militia - poland, hungary, lithuania
  • Estonian Rebels - lithuania, slave
  • Followers of Perkunas - lithuania
  • Frankish Axemen - jerusalem, slave
  • Frankish Swordsmen - antioch, slave
  • Galloglaich - ireland, slave
  • Genoese Crossbow Militia - milan
  • Gotland Footmen - denmark, norway
  • Greek Militia Cavalry - byzantium
  • Gwent Raiders - wales, slave
  • Halberd Militia - hre, papal_states, sicily, poland, hungary, slave
  • Heavy Bill Militia - england
  • Heavy Billmen - england
  • Heavy Pike Militia - scotland
  • Helwyr - wales
  • Highlanders - scotland, slave
  • Hospitaller Sergeant - antioch
  • Italian Cavalry Militia - milan, venice, papal_states, sicily
  • Italian Militia - milan, venice, papal_states, sicily, slave
  • Italian Spear Militia - milan, venice, papal_states, sicily, slave
  • Jaguar Warriors - aztecs, slave
  • Kurdish Javelinmen - egypt
  • L Cavalry Militia - lithuania
  • Landschneckt Pikemen - slave, hre, milan
  • Levy Spearmen - england, slave
  • ME Archer Militia - egypt
  • ME Crossbow Militia - moors
  • ME Halberd Militia - egypt, turks, mongols, timurids
  • ME Levy Spearmen - mongols, slave
  • ME Peasant Archers - egypt, turks, mongols, timurids, slave
  • ME Spear Militia - moors, egypt, turks, mongols, timurids, slave
  • ME Town Militia - moors, turks, mongols, timurids, slave
  • Meirionnydd Spearmen - wales
  • Merchant Cavalry Militia - england, scotland, france, hre, denmark, spain, portugal, poland, hungary, ireland, norway, teutonic_order, wales
  • Mongol Foot Archers - mongols
  • Mongol Infantry - mongols
  • Morgannwg Spearmen - wales, slave
  • Native Archers - slave
  • Native Warriors - slave
  • New World Cuirassers - england, france
  • Noble Pikemen - scotland
  • Noble Swordsmen - scotland
  • Norse Swordsmen - denmark, norway, slave
  • Nubian Spearmen - moors, egypt, slave
  • Obudshaer - denmark, norway
  • Order Militia - teutonic_order
  • Order Spearmen - teutonic_order, slave
  • Ostmen - ireland
  • Partisan Militia - france
  • Pavise Crossbow Militia - venice, papal_states, sicily, hungary, slave
  • Peasant Archers - england, scotland, france, hre, denmark, spain, portugal, milan, venice, papal_states, slave, norway, teutonic_order
  • Pike Militia - france, hre, spain, portugal, milan, venice, papal_states, sicily, ireland
  • Pikemen - france, england
  • Pronoia Infantry - byzantium
  • S Archer Militia - byzantium, slave
  • SE Spear Militia - byzantium, slave
  • SE Town Militia - byzantium, slave
  • Saracen Militia - egypt, turks, slave
  • Scots Pike Militia - scotland, slave
  • Sergeant Spearmen - france, hre, milan, venice, papal_states, sicily, slave
  • Spear Militia - england, scotland, france, hre, denmark, spain, portugal, slave, norway, wales
  • Sudovian Tribesmen - lithuania, slave
  • Swiss Guard - papal_states
  • Swiss Pikemen - slave, venice, sicily
  • Sword Brethren - teutonic_order
  • Sword and Buckler Men - spain, sicily
  • Swordsmen Militia - spain, portugal
  • Swordstaff Militia - denmark, norway
  • Syrian Militia - jerusalem, antioch
  • Templar Sergeant - jerusalem
  • Tercio Pikemen - spain
  • Town Militia - england, scotland, france, hre, denmark, spain, portugal, norway, slave
  • Tripolitan Squires - jerusalem
  • Turkish Crossbowmen - turks
  • Ulster Swordsmen - ireland
  • Urban Militia - moors
  • Viking Raiders - denmark, norway, slave
  • Welsh Militiamen - wales
  • Woodsmen - poland, russia, novgorod, slave
  • Zweihander - hre, milan
  • al Ashair - egypt

Overall Impressions

The mod is pretty decent - just takes the best out of Medieval 2 without doing anything drastic.

Unfortunately I had a lot of crashes in game menu.

Fortunately once in game, crashes were not too common. I'd still manually save before and after every battle, but it was rarely necessary. Sadly vanilla crashes are not unheard of either.

If you want to give Medieval 2 another go, and want just a bit more variety, highly recommended.

Monday, July 29, 2019

Challenges for July 2019 SecTalks London

Dewey by angela n. from flickr (CC-BY)

Another CTF, another victory. I won the June 2019 London SecTalks CTF and it was up to me to write challenges for July.

There were 12 challenges, theme of the challenges being Hacker-Archeology. It turns out that was too much, as only 10/12 challenges got even one solve, and nobody got more than 4. Even with very generous hint drops during the event.

Challenge files and code used to generate them are available on github.

There are no answers below, but some serious hints which might make it too easy.

For previous rounds, see posts about September 2017November 2017May 2018July 2018, October 2018February 2019, and April 2019 CTFs.

SHAR (5 points)

Self-extracting Shell archive nested 8 levels deep. It only works on Linux, OSX shell can't extract Linux-created SHAR files, showing what a dumb format this is.

XBM (10 points)

A weird way to encode image into C headers. While totally obsolete, a lot of tools still support it.

Maya (15 points)

An image with a sequence of Maya numerals, each encoding ASCII symbols. It seems people were confused by the fact that multi-digit Maya numerals are stacked vertically.

PCX (20 points)

PCX file with flag on it, but both foreground and background colors having same RGB color, so color palette would need to be adjusted to actually see it.

ECB (25 points)

A signed cookie server which would only sign cookies without admin=yes, and it would only give you the flag if you sent it signed cookie with admin=yes.

It's a classic cryptography attack on ECB mode, rearranging blocks within or between cookies.

SED (30 points)

SED is an obsolete programming language for text stream processing, and the challenge was a simple flag validation script which was just a sequence of regexp replace rules.

Nobody noticed that, but that SED script was also totally valid Perl 5 script.

CBC (35 points)

A signed cookie server which would only sign cookies without admin=yes, and it would only give you the flag if you sent it signed cookie with admin=yes.

It's a classic bit flipping attack on CBC.

MD4 (40 points)

A signed cookie server which would only sign cookies without admin=yes, and it would only give you the flag if you sent it signed cookie with admin=yes.

It's a classic length extension attack.

Midi (45 points)

A midi file with flag encoded in Morse code, played on an Ocarina instrument suggestively named "Morse Ocarina".

It could be done either by hand, or by converting note lengths in Midi format to dots and dashes. I think everybody ended up doing it by hand.

DOS (50 points)

A very small COM file flag validator. It was extremely simple, but a lot of tooling like Ghidra has trouble with COM files, as they're too old to be relevant.

It got zero solves, which was fairly surprising, as flag validator is really simple:

# Initialize counter in BX
  mov bx, 0xd7ab

# Get ASCII code of next character into AL
  mov ah, 0x1
  int 0x21

# Add AX to the counter, check if correct
  add bx, ax
  cmp bx, 0xd911
  jnz near 0x19d

# Repeat for next character

Differences between constants being compared are 256 + ASCII code of each letter (first being 0x166 or 256 + "f").

Hieroglyphs (55 points)

Monoalphabetic cipher encoded into Egyptian hieroglyphs. The text was very long English text with spaces removed.

It's really simple for anyone who ever did monoalphabetic cipher breaking through statistical analysis, so that many points mostly being potentially quite time consuming, but it didn't even take people that much time.

Perl 6 (60 points)

In remote past Perl 6 used to be the language of the future. That future never came.

The challenge is a flag validator in Perl 6 aggressively using many unusual Perl 6 operations. I'm not really surprised that nobody succeeded at solving it.

Friday, July 19, 2019

Some Thoughts on Stepmania

The Dancing XiaoChou 3 by qchen from flickr (CC-NC-ND)

Back in 2009 I even wrote a rant about Stepmania, and here's another one.

In my younger days I did a lot of Dance Dance Revolution, and every now and then I come back to it.

I'd like to play with some modern songs. Supposedly official games exist, but they have like 50 songs per game, plus a few "$5 for 3 songs" DLCs, and mostly only work on some weird-ass consoles.

Basically the only option for it is using Stepmania, and downloading some user-made songpacks.

So what's wrong with all this

The first problem is that music is totally free online on youtube and such, but Big Copyright would never allow a healthy ecosystem of dance games, because they just love the whale exploitation model of "$5 for 3 songs", and the market is too small for someone to force them to be reasonable.

Stepmania tries to avoid any direct entanglements with all those copyright issues, and unfortunately that means it avoids actually trying to solve the problems.

User-made song packs have mixed quality

So I got a bunch of random songpacks, basically keyword matching artists I might like.

The songpacks are basically whatever the author decided to throw there, so I'm generally only interested in very small portion of each, but let's say I keep them all, as at this stage I don't know if those songs are any good or not.

Most of user-made content is decent enough, but it's far too common that there's bullshit songs with ratings like 12-20, which are presumably meant for the keyboard, or maybe arcade machines with safety rail, since it would be unsafe to even try on a soft dance mat, regardless of one's skill level.

There's plenty of songs which are poorly synced. There's plenty of songs which have very questionable ratings, and are actually a lot harder than other songs at same ratings.

This wouldn't be a huge deal if there was a way to filter that out easily, but there isn't.

Stepmania UI is atrocious, especially when you have a lot of songs

All right let's say I have a few thousand songs now. Stepmania will take forever to actually start, like literally over 15 minutes. It seems latest version and SSD finally made it tolerable, but seriously, just checking that a few thousand tiny files didn't change shouldn't take this kind of crazy time.

The next problem is how to actually choose those songs. Stepmania decided to copy dance mat only UI from arcade machines, without any keyboard backup. So instead of taking 1 second to type song title or artist name, it takes literal minutes to scroll through thousands of songs to get there, even at highest speed.

The UI has other issues - like it seems that I end up triggering song options about 1 in 10 times when trying to just start a song, and it registers it as Start button being pressed twice for whichever reason.

There's a lot of weird combos to control the UI, but other than "difficulty up", "difficulty down", and "change sort order" I have no idea what they are, and there's nothing intuitive about it. There should just be goddamn menus and keyboard controls for those rarely used functionality.

Once song starts playing it's pretty much fine.

Ideas for solutions

So it seems I have the same complaints today I had a decade ago.

Anyway, let's talk possible solutions.

These days it's just easier to write cross-platform games in something civilized, with Electron or whatever. Stepmania isn't really a terribly complicated program, so if anyone felt like it, they'd probably have something kinda working in a few days, and it would probably have better performance and usability even at such early point.

Much more interesting is automated step files filtering and analysis. Step files are literally asking for a neural network analysis to flag broken ones, figure out correct difficulty, and so on. Prototypes to just generate step files outright exist, and filtering/analysis should be a lot easier. It just needs to be takes out of research paper and given to users.

A far more interesting idea is just taking songs from youtube or whenever, and doing everything automatically from that point, but that's quite questionable. Stepmania songs are generally ~100s remixes of ~200s songs, so that would already be a major difference. The whole step file process might be too computationally expensive.

How would it work in practice

I feel first step would be writing some parsers to take step files in variety of kinda documented custom formats and export them as some json. It's probably going to be quite tedious, but nothing difficult.

Then figuring out how to interact with dance mats. Most of them are just USB HID devices, so it shouldn't be too hard. In principle Electron supports WebUSB, so dance mat support should be quite straightforward, at least when everything goes right.

With these two, getting simple Stepmania-like program with Electron shouldn't be hard, and that could be a platform for all those crazy ideas.

I'm not hating here

Stepmania is still one of the most successful Open Source games ever. I just think something much better is possible.

EDIT - USB APIs in browser

Well, I tried to use WebUSB API and Gamepad API in Chrome, both supposedly supported, and they don't see my dance mat on Windows or on OSX.

It's possible I'd have more luck with Electron.

EDIT - Stepmania keyboard shortcuts

And it turns out Stepmania actually added some keyboard navigation recently. So if I have songs sorted by artist I can press Control-M to go to M etc. That's fine to reach Katy Perry, maybe less so for Avril Lavigne. Still, even that little change cuts scrolling time by more than half.

Saturday, July 13, 2019

Eric Swalwell is wrong and Nancy Pelosi is right

donkey tuft by Tarnie from flickr (CC-NC-ND)

I bet you did not see this one coming.

Eric Swalwell was one of 20 candidates for Democratic presidential nomination. The only notable moment of his whole candidacy was the "pass the torch" moment during debates, where he used Joe Biden's own words to basically attack Joe Biden for being too old. I strongly believe that there should be mandatory retirement age for politicians, so he had a point there.

Anyway, Eric Swalwell's campaign was going nowhere, he gave up before even second round of debates, and after giving up he gave exit interview to 538 politics podcast, and that's what this is about.

Extremist Drift

Debates were notorious for how far left most candidates went compared with mainstream Democrats of just a few years ago, like Barrack Obama, Hillary Clinton, or Nancy Pelosi. Positions like abolition of borders, free healthcare for illegal immigrants, abolition of private health insurance, forced busing, racial reparations and other extremist positions had far more support that one would have guessed.

In a way the winner of the debates was Donald Trump. His approval rate improved after Democratic debates, and is currently at -7.5%. It's still negative, but it's far better than his the usual -10% to -20% range, and if this extremist drift continues, Democrats might manage to scare off all potential moderate voters.

Swalwell's Argument

During exit interview, Eric Swallwell was asked about that.

His response was that the most important feature of a candidate is "authenticity", and that going "far to the left" is absolutely fine, and no risk whatsoever in general election.
Democrats tiptoe around the issues that are perceived as unpopular, whereas Republicans have no problem leading with very unpopular issues [...] and they don't pay a price at the ballot box.
He believes that Trump's victory proves it.

He also openly advocates violating the Constitution in the same statement, but let's not get there.

It's bullshit

US economy is doing better than it did in living memory. Unemployment rate is as low as it last was in the 1960s.
Inflation has been low and stable. Stock market is at unprecedented heights.
Tax cuts mean most working people have a lot of extra money personally (except for rich people in high tax states). All the secondary metrics like wages growth, gas prices, healthcare access, and so on are doing just fine.

In foreign policy, no new wars were started, for the first time in it's hard to tell how many presidencies. ISIS which spread during Obama's term and overran multiple countries was swiftly crushed. There weren't any major terrorist attacks, or other foreign disasters.

Fundamentals models don't have an amazing track record at predicting elections, but it's a pretty safe prediction that with everything an average person cares about going so much better than basically ever, whoever presides over that should be super popular and crush any challenges 1984 style, right?

Well, that's what would have happened if president Marco Rubio or Mitt Romney was presiding over it.

It's Trump specifically being such a turd that makes these elections a 50:50 thing.

How Trump won

Trump was the most unpopular presidential candidate in recorded history. Fotunately for him, it just so happened that his opponent Hillary Clinton was the second most unpopular presidential candidate in recorder history. She was so unpopular she lost to a totally unknown black half-Kenyan guy with name "Barrack Hussein Obama", in spite of DNC establishment doing all they could to force her though. She was so unpopular she nearly lost to a senile openly socialist Jew who wasn't even in the party, and only managed to somehow got through thanks to DNC establishment forcing her candidacy even harder. She was so unpopular she lost to Donald Trump.

In such Giant Douche vs Turd Sandwich elections Trump just so happened to have better ran campaigns, and barely squeezed the victory.

He did not become any more popular since then. As an aside, Hillary Clinton is now even more unpopular than Trump, but fortunately for Democrats she's not running for the third time.

Trump only managed to win in 2016 because his opponent was so unpopular, and the economy was still only slowly recovering.

In 2020 he has far easier job - no sitting president could ever lose with fundamentals doing so well, unless he's literally a Turd Sandwich.

Why bother with Eric Swalwell?

Eric Swalwell might be out, but other candidates for Democratic nomination seem to think the same way as him. They're trying to score points with extremists in party base, and hope that somehow it will work out.

It could work against Turd Sandwich, but not if they end up picking a Giant Douche again.

The idea that extremist will somehow increase turnout among the base is total nonsense - highly politicized people will vote anyway, and you're far more likely to increase turnout of opponent's base this way - as Trump did during 2018 midterm, with his shittiness really motivating Democrats to go vote, regardless of who was their local candidate.

It's not like this is a novel strategy. Nancy Pelosi, the most successful Democratic politician, has been successfully doing just that - marginalizing the extremists in her party while pushing hard for what's realistically achievable. Especially in country whose political system is designed for gridlock, focusing on popular parts of your party's agenda in alliance with moderates is the only way you can actually achieve anything.

The successful strategy is attacking your opponents where they're weak, not responding to Trump's Wall with abolishing ICE, abolishing borders, and free citizenship to anyone who jumps where the border used to be.

It is still 50:50

Many things can happen before the elections. The economy could crash. Trump could start WW3. One of many Trump scandals might end up discovering some real evidence of crimes resulting in impeachment, not decades old hearsay that convince only partisans. Any of those would shift elections far more than Democratic debates.

If none of that happens, and the elections is still 50:50, it will really matter if Democratic Party follows Nancy Pelosi, and picks a successful moderate, or follows Eric Swalwell and tries to outdo itself in extremist appeals and chooses a Giant Douche while handing over second term to the Turd Sandwich.

Far Left is losing everywhere

It's not US specific issue. Traditional center left parties (more or less analogous to the US Democratic Party) got weakened by the Great Recession, resulting in brief resurgence of the far left.

This resurgence is crashing now. UK Labour was lost every elections since it went extremist, and in some polls is now 4th with 18% support. In two countries worst affected by the crisis where far left actually got power it already lost it, Podemos completely crashed in Spain. Syriza more narrowly lost in Greece.

In most other Western countries, all this bickering on the left just weakened it, and let either center right or populist right take over. Many Western countries like Poland and Israel nowadays have 50 Shades of Right elections.

Seriously guys, just listen to Nancy, she knows what's best for you.