Challenges for November 2019 SecTalks London

October 2019 London SecTalks CTF was jointly won by me and Laurence, so we wrote challenges for November together.

There was a total of 9 challenges, 5 written by me and 4 by Laurence. It was a bit much, as one of the challenges got zero solves until after the event. In spite of very generous hints given halfway through the event.

Challenge files and code used to generate them are available on github.

There are no answers below, but some serious hints which might make it too easy.

For previous rounds, see posts about September 2017, November 2017, May 2018, July 2018, October 2018, February 2019, April 2019, and July 2019 CTFs.

Rockstar (5 points)

The puzzle was a rock song / program in Rockstar programming language. Just running the program would print ASCII codes, which could be easily turned into a flag.

Think Outside The Box (10 points)

The challenge is a BMP image file with flag hidden by incorrect image size. Changing the header reveals the image.

Dots (15 points)

This was Unicode Braille, but dots and no-dots were simply 1s and 0s to be decoded, ignoring any "Braille" structure. To my surprise nobody solved it even when given basically full explanations of what they're supposed to do.

CRC32 MAC (20 points)

A signed cookie server, which uses CRC32 with secret key as MAC. CRC32 is extremely unsuitable in this role.

Decryption Engine (25 points)

A small Python program which uses default password based on current date, and encoded file. Assuming password wasn't passed, the space of possible keys is tiny.

Salted MAC (30 points)

A signed cookie server, which uses SipHash with secret key, and then salt just to be extra sure.

In principle this should be difficult to break. The problem is a trivial implementation mistake.

PHPain (40 points)

It was a separately hosted PHP application vulnerable to recently published CVE.

Poly-DIY MAC (50 points)

Another signed cookie server, using a simplified variant of Poly1305. The variant is reasonable, however the server isn't very careful about nonce handling, and such schemes tend to fail catastrophically when nonce is reused.

GeoCaptcha (60 points)

It's a web "Captcha" which asks the user to locate 20 major cities on the map, with any mistake failing the Captcha.

You can play it here (for a while).

It's worth so many points mostly due to its very high entertainment value, not due to difficulty.

I was curious how people would approach it - only 2 players solved it with code (as robots), 6 solved it with tedious manual process (as humans). So I guess it works as a Captcha.

How to configure OSX 10.14 Mojave for software development

With every new Macbook, I'm updating the guide, previous version is here.

And I get it that it's ironic that I'm posting this just as 10.15 came out, but I started the setup a few days ago, and in general it's a poor idea to update OS as soon as it comes out.

Basics

• Go to Settings > Security > FireVault, turn on FireVault. This will restart your computer.
• Install some sensible browser like Chrome or Firefox.
• Afterwards either sign up into your account on which you hopefully have your ad blocker setup, or install some. Most popular seems to be uBlock Origin these days, but pretty much any of them will do just fine.
• Install whichever cloud sync service you're using like Dropbox etc. And start syncing your stuff.
• Install iTerm2 for sensible terminal emulator.
• Clean up all crap from dock. Other than Launchpad and System Settings, everything else should be gone. Add iTerm2, your browser, and your text editor, and any application you wish to install there instead of stock Apple crap.

Editor

Install some sensible text editor. These days most people use Visual Studio Code.

If that's your choice, run it, open Command Palette, and choose: "Shell Command: Install 'code' command in PATH".

Settings

Like every other operating system, OSX has a lot of bad default settings. Here are some obvious fixes:

• Settings > Mouse > increase speed
• Settings > Keyboard > Key Repeat > Fast
• Settings > Keyboard > Delay Until Repeat > Short
• Settings > Sound > Disable "Play user interface sound effects"
• Settings > Sound > Alert volume > 0% (for Terminal ping)
• Settings > Sound > Show volume in menu bar
• Settings > Trackpad > Scroll & Zoom > Disable "Scroll direction: natural"
• Settings > Displays > Arrangement > drag and drop your external monitors into desired order
• Settings > Displays > Max out brightness
• Settings > Displays > Turn off "Automatically adjust brightness"
• Settings > Dock > enable "Automatically hide and show the Dock"
• Menu bar > Battery icon in task bar > Enable "Show Percentage"
• Menu bar > Time icon in task bar > Open Date & Time Preferences > Use a 24-hour clock
• Menu bar > Time icon in task bar > Open Date & Time Preferences > Show date
• iTerm > Preferences... > Profiles > Terminal > Unlimited Scrollback

Press Ctrl-Up arrow, add a few desktops (or "spaces" as they were used to know), then go to Settings > Keyboard > Shortcuts > Mission Control - and enable their keyboard shortcuts Ctrl-1 to Ctrl-6 or however many you have there.

Open Screenshot app, choose options, then disable "Show Floating Thumbnail".

Drivers

OSX already includes drivers for laptop itself, but you might need some for peripheral hardware.

In particular, external PC keyboard need a tweak to work properly, as left and left Windows keys are in reverse order from Mac keyboard.

Go to Settings, Keyboard, Modifier Keys..., and swap positions of Option and Command keys. Feel free to change functionality of Caps Lock key as well, it's a huge easily accessible key with no useful function people love to remap, usually to extra Control.

If you need any special keyboard layouts, get them too.

Standard paths

OSX renames a lot of directories. While in theory scripts could just use env vars to find proper paths, it's more reliable to symlink all the things:

  sudo ln -s /Volumes /mnt
  sudo ln -s /Volumes /media
  sudo mv /home /home-old
  sudo ln -s /Users /home

Development tools

You'll need a package manager, and the only one anyone uses is homebrew, MacPorts and the rest being basically dead. You need to tell homebrew to not spy on you with brew analytics off command.

You'll need Xcode. Fortunately homebrew installer does it for you automatically. If you don't use homebrew for some reason, you can install Xcode manually by running xcode-select --install from command line.

Create new SSH key pair

Before you do that, name your computer something memorable with sudo scutil --set HostName your_host_name command.

Open Terminal and run ssh-keygen to create ~/.ssh/id_rsa, then upload the generated key to any place that needs to know about it like github, bitbucket, or whatever else you use.

Checkout your dotfiles

Hopefully you're storing your dotfiles somewhere. If it's a git repository, or your Dropbox account, get them now and symlink them all properly.

If there are any other repositories you might need, checkout them too.

Install homebrew packages

Your list might vary. Here'sa few obvious suggestions:

brew install rbenv ruby-build mc wget p7zip unrar trash git htop bash zsh youtube-dl jq imagemagick coreutils bash-completion zsh-completion

Then enable all services you installed, unless you want to start them manually:

  ln -sfv /usr/local/opt/*/*.plist ~/Library/LaunchAgents/

And install non-system ruby, so you can install gems without sudo. Currently latest is:

  rbenv install 2.7.0-preview1
  rbenv global 2.7.0-preview1

To make that actually work, you need to make sure ~/.rbenv/shims is in your $PATH. If you type rbenv init, it will tell you what to do.

Due to OSX limitations you'll need to run sudo htop if you want to use htop.

Install gems

Again, your list my vary. These days most of the software will have its own Gemfile so long list of gems are generally unnecessary. But some global utilities are still useful:

gem install bundler rak pry pry-rescue

Sane bash or zsh

bash version shipped with OSX is ancient and BSD utilities are all awful.

You have two choices. Either install modern version of bash from homebrew, or use zsh from homebrow.

First, edit /etc/shells as admin and add the following lines at the end of it:
/usr/local/bin/bash
/usr/local/bin/zsh

Then set it as your shell, with whichever one you prefer:
  chsh -s /usr/local/bin/bash $USER
  chsh -s /usr/local/bin/zsh $USER

Either way, run to touch ~/.hushlogin to prevent some worthless spam on every open terminal tab.

If zsh is your choice, the first time you start a new terminal, it will ask you some questions on how to configure it.

Coreutils

This is optional step. OSX coreutils are awful, and generally GNU versions are a lot better, but if you switch there could be occasional minor incompatibilities.

If you want to do so, add GNU coreutils to your PATH:

  export PATH="/usr/local/opt/coreutils/libexec/gnubin:$PATH"
  export MANPATH="/usr/local/opt/coreutils/libexec/gnuman:$MANPATH"

Better window manager controls

Sadly OSX window manager is extremely dubious for keyboard use. Fortunately programs to make it usable exist. I recommend installing HyperSwitch for sane alt-tab window switching, and Spectacle for managing windows.

I previously used HyperSwitch, but it no longer works with latest version of OSX. Spectacle shortcuts are different than HyperSwitch, and I like HyperSwitch's more, so I adjusted shortcuts to match.

You'll need to give them necessary access. To do so:

• Settings > Security & Privacy > Privacy > Allow the apps below to control your computer > enable HyperSwitch and Spectacle

Also open its preferences, and set it to run in the background, and start at login.

All other software

There's a lot of other software you might want. The most obvious one is the VLC media player.

You might also want some kind of Git UI program, like GitUp or (another for of) GitX.

Enjoy

Once you go through this list, and successfully get everything going, I'd recommend modifying it to your liking and reposting your version on your blog. Everybody's needs are different, so guide like this is just a starting point.

Technologies that inspired me

Nearly every day I run into another language, framework, or program. Most of them are just plain awful, and the ones which aren't are usually nothing special.

Every now and then I run into a true gem. Here's a list of such great technologies, in approximately the order I found them.

Perl

Perl was absolutely amazing compared with every language which came before it. Other languages were motivated by what worked for computers, or by some silly mathematical theory, or sometimes like with Java by outright hostility to programmers.

Perl tried to be as much like a natural language as possible. If what you wrote made some kind of sense, it was going to try to run it. And most of the time it worked.

Unfortunately Perl supported too much weird stuff, and it caused the language to collapse under its own weight, but the most influential languages of the 1990s like Ruby, Javascript, and PHP were very heavily inspired by Perl. And as next generation of languages was very heavily inspired by those, the legacy of Perl lives on.

Perl tried to double down on its philosophy with Perl 6, but without first dropping failed ideas it was never really going to work.

Regular Expressions

Text is nearly as common data type as numbers, but most pre-Perl languages didn't treat text data as first class. In many old languages text is literally an array of small numbers, or such other nonsense. Even today Erlang / Elixir do literally that - "Hello" is the same as [72, 101, 108, 108, 111].

Regular expressions are to text what arithmetic is to numbers, and let you extract or modify text with tiny amount of highly readable and highly performant code. They have weird reputation of unreadability, but one line of dense regular expression is infinitely easier to understand than 100 lines of text processing code it replaces.

It's key skill for every programmer, no matter what's the domain. If you don't know regular expressions, drop everything and do some regexp crosswords.

There used to be different variants of regular expressions, but these days basically every language uses some variant of "Perl-Compatible Regular Expressions". Except Unix grep command, which still uses stupid 1970s regexps for some crazy reason.

Wikis

Wiki was a revolutionary idea - just put a bunch of linked pages online, and let anyone who wants edit them. It sounds like a recipe for disaster, but it worked remarkably well. Wikipedia was the greatest success story of wikis, but there are all kinds of wikis for all kinds of content. And as a general rule, they're a lot more reliable than non-editable sites.

Ruby

Ruby took everything that made Perl great, everything that made Lisp great, everything that made Smalltalk great, then it threw away their bad parts. And then it added more magic on top.

It is the high point of programming langugae design, unmatched up to this day. Implementation issues aside, I'd rather use Ruby as it existed 20 years ago than any language that came out since. It's just baffling how far ahead of time it was.

Of course it could be even better, but that's true of anything.

I hope someday a language comes that's better than Ruby, but so far it doesn't look like anyone's even really trying. The best anyone's trying to do is to make their languages as close to Ruby as possible given various constraints.

Ruby on Rails

Ruby on Rails is another miracle technology. When it came out, it was completely unmatched as the best way to write web backends. Today, after the web changed beyond recognition, it's still completely unmatched as the best way to write web backends. It's rare for any technology to have such staying power.

To always stay on top, Rails evolved very aggressively, and Rails upgrades on large codebases are notoriously difficult, but then again, the choice is between upgrading Rails 4 to Rails 5, or writing your code in some shitty framework that's going to die, and fully rewriting it into another shitty framework that also won't last.

It's quite baffling that so far no real competitor to Rails emerged. The best everyone else can do is to copy as much as they can from Rails, and even that is usually really poor.

The main reason is that Ruby is completely unmatched at making DSLs than any other language, and Rails is very DSL-heavy. But other aspects of Rails success like everything working seamlessly out of the box (while being individually replaceable) without needing to duct tape a bunch of disparate components should be replicable, and yet nobody even tries.

Why so many frameworks come out without such basic things like comprehensive testing system, or database migration system, or sensible security defaults? I suspect their authors just don't get Rails, and people who get Rails use Rails.

TextMate

Once upon a time, there were thousands of editors and they were all 💩. It was a running joke that every programmer writes their own editor, and their own IRC client (and later their own Javascript framework).

And just as with infinite monkeys randomly typing on infinite number of keyboards, one of such attempt miraculously produced TextMate - the first editor which was not 💩.

Multiple cursors, command palette, file palette, extension system, nested file types, integration with modern UIs, sensible keybindings making use of what's on modern keyboards - TextMate was far ahead of everything else.

TextMate itself didn't last, but Sublime Text, Atom, VS Code, and some others are essentially TextMate clones, and they completely dominate the editor world.

The only non-TextMate editors still in use are people using IDEs like IntelliJ or Android Studio due to being forced to use programmer-hostile languages like Java. And people who use Vim because they still haven't figured out how to quit it.

jQuery

Javascript used to be awful beyond belief. Well, it's still not great, but back then doing even the most trivial dynamic interactions, like changing some class based on an user action was a page of code, and then another page of different code if you wanted to support more than one browser.

Then jQuery came, and with a few $("selector").action() you could write complex UI interactions with very little code. It was even bigger difference when it came to jQuery AJAX vs browser APIs for AJAX. The gap was as enormous as between regular expressions and manipulating text by shuffling arrays of characters around.

Gradually browser APIs became more compatible, more expressive, and less buggy, they added a lot of features to close the gap with jQuery, so now it's sort of viable to write short programs directly with browser APIs. And while jQuery is amazing at progressively enhancing website functionality, it's a stretch if you want to make Single Page Applications.

So is it the end of jQuery era? Not even close! jQuery is still vastly more popular than all frontend frameworks put together - 80% of top million websites use jQuery, while all the hipster frameworks put together are still just a rounding error. Currently most popular React at 1.6% is behind 20 different jQuery plugins.

And beyond jQuery itself, D3 is basically jQuery for data visualization. Most acceptance testing frameworks from Capybara to Cypress use jQuery-style for writing tests. Single Page Applications are useful, but most of the Web is better off not being an SPA, and good jQuery skills will never go to waste.

It might be unpopular opinion now, but I feel the best way people should learn frontend is still be HTML, CSS, jQuery, then ES6+ Javascript, and only then some SPA framework like React.

Amazon Web Services

I remember bad the old days when setting up any server required making a request to a grumpy sysadmin (or even worse - a grumpy DBA) and waiting potentially weeks for them to find some time for it. Even more fun if that required more physical hardware, and budget requests for those. Meanwhile, there was nothing you could do.

Then came Amazon and decided that making billions selling everything wasn't enough so it opened a small side business of getting rid of grumpy sysadmins. Want to start a new server? A few clicks and it's done by the time your coffee finished brewing. And they kept adding new services beyond the initial EC2 and S3, to simplify your life more and more.

I feels silly to even mention how great it is, as AWS and other Clouds won so utterly that most people barely remember the pre-Cloud days, and those who do would rather forget them.

pry

I didn't love debuggers, and I usually employed print-based debugging as my first choice, strace and friends as my second choice, and only used interactive debuggers if all else failed.

pry was something else. A hybrid of a debugger and REPL, with full power of Ruby metaprogramming and introspection, it was on a completely different level than anything I've used before.

Adding binding.pry became my first not the last resort. And then there's pry-rescue and other plugins to make things even more awesome.

I'm still annoyed that pry isn't just part of Ruby yet, and that bundler blocks pry unless you explicitly add it to every single Gemfile just in case.
Sure there's finally binding.irb, but irb is not even close to pry. Seriously, get your act together guys and get pry into Ruby standard library.

MongoDB document model

Relational databases are just a bunch of damn spreadsheets, and forcing your program's object oriented (or otherwise richly structured) data into spreadsheets for storage is invariably painful. Object-Relational Mapping tries to make it bearable, with limited degree of success.

There were many attempts to create alternative database models, but they were either specific to one programming language, or used something like XML which was just as awkward as spreadsheets.

Then came MongoDB, made everything JSON, and it was glorious. Oh and depending on your application it could easily be orders of magnitude faster than relational databases.

Unfortunately while MongoDB JSON-based document model was amazing, and its performance was generally outstanding, everything else about MongoDB was less so. Doing even relatively simple queries in MongoDB is stupidly painful compared with SQL, so you end up doing it all in application. Its performance-first defaults got a lot of people into trouble as well.

I'm surprised that even a decade later nobody managed to build anything that united JSON-based (as opposed to spreadsheet-based) document model with some rich query language that can at least match SQL.

Spreadsheet-based databases nowadays support JSON data a little bit, NoSQL databases nowadays have a bit richer query languages, but the gap between those two is still enormous.

Capybara

TDD lived in this weird state, where backend code was decently tested, but frontend testing was minimal and mostly manual.

Even today most tests for JS programs are fairly low value efforts, like "component renders without crashing" tests and snapshot tests.

Capybara was the first practical way to do end to end tests of web apps with dynamic frontend and dynamic backend as a single system.

Nothing better than Capybara was created so far. There's a bunch of fairly poor Capybara clones in other languages, which might be just about good enough. Cypress might be developing into the first real alternative, but it's very frontend-based and integration with dynamic backends is its secondary consideration.

Continuous Integration servers

I wasted too much time running tests manually, or not bothering to run them because the change I was doing was trivial, and the tests took too long.

There is an answer to that - CI servers. Most companies have them. But people coding for themselves, or coding Open Source software are still usually not using them.

Semaphore CI and many other sites let you easily setup CI for free for your Open Source project, and you should do that.

Chrome Developer Tools

The story started in the olden days with Firebug, which provided just console.log and a few other very simple functions. Or possibly even with the View Source button (which some people wanted to delete from browsers to "protect the source code", #truestory).

Nowadays every browser comes with amazing debugging and development tools, and they keep getting better every year.

It's one of few places where I feel Javascript has a big lead over other languages. pry might be as good as possible on terminal, but it's still limited to a terminal. Chrome Developer Tools have full power of the browser UI, and they fully use it.

Dropbox

I have multiple computers. Also multiple phones and tablets. Sharing content between them used to be such a mess - connecting and disconnecting external disks, or manually copying files by cables or rsync. Dropbox made it seem that the same file is magically present in every location. It just works, with very few issues.

Unfortunately Dropbox for phones is quite bad, so if anyone solves this better, I'm interested.

Weirdly many Dropbox alternatives like Google Drive are simply horrendous.

Slack

Just like everyone was writing their editor, everyone was writig their IRC client. Or some other chat system which was basically IRC clone.

They were all solving the wrong problem. The issue wasn't that IRC needed better UI, the issue was that IRC was based on fundamentally wrong assumptions (broadcast to whoever happens to be connected at time) that made it completely unsalvageable.

Slack fixed those assumptions, and it completely revolutionized workplace communication, not just in tech.

All post-Slack chat systems like Discord use basically Slack model.

Electron

Electron is the first time in history of computing where we have cross-platform desktop apps with decent UI. There's performance cost, as Electron apps bundle whole browser, and are infamous for their memory use, but there's no alternative, so let's just throw more hardware at the problem, and restart every now and then if it gets too bad.

There were a few earlier attempts like with Java, but the UX was just attrocious. I cringe whenever I have to use some Java-based GUI software.

Z3

Normal programming answers questions like x = 2+2 perfectly fine, but it is shockingly bad at dealing with questions like 2+x = 4. Z3 is an amazing system for answering just such questions.

Solving sudoku, hacking CTFs, proving hardware design? Z3 has you covered.

I liked it so much that I wrote a Ruby gem for it. It's much nicer than official APIs.

Docker

I initially dismissed Docker as just another VM. It turns out that such lightweight micro-VMs have a lot more uses, and just being able to install and run a lot of images neatly solves a lot of issues like CI, software with weird dependency, security isolation and so on.

Imba

Javascript used to have a million frameworks. They were generally called "MVC frameworks" back then, a name you don't hear much anymore, as it turns out MVC is a poor fit to what they were trying to do.

At certain point very-much-non-MVC React pretty much won, by being decent at solving smaller problem (views), not even trying to deal with the hard problem (model and network layers / state management), and so having a lot easier learning curve and fewer gotchas. Acceptable performance and being pushed hard by Facebook also helped.

I even liked React before I found Imba. Imba however, is on a completely different level. It's the least known technology on this list, so I'll need to write a separate post about it, but you'll get orders of magnitude higher performance at fraction of code size, all while being a lot simpler than React.

It's a tiny project with very few development resources, so it feels like it really needs someone to push it to some sort of Imba 2 fully embracing post-ES6 world. But even in its current state, it's years ahead of any alternative.

Kingdoms Grand Campaign Mod - Medieval 2 Total War mod highlight

Medieval 2, released back in 2006, was the peak of Total War series. One of the great things about it is abundance of mods, and this post is about one such mod.

Kingdoms Grand Campaign Mod

The idea behind Kingdoms Grand Campaign Mod is taking content from minicampaigns from Kingdoms expansion pack, and integrating them into one big campaign.

Campaign map is adjusted somewhat, and in addition to vanilla factions, you get:

• Antioch from Crusades campaign
• Ireland from Britannia campaign
• Jerusalem from Crusades campaign
• Kiev from Teutonic campaign
• Lithuania from Teutonic campaign
• Mongols from Teutonic campaign (they're in vanilla as unplayable horde)
• Norway from Britannia and Teutonic campaign
• Wales from Britannia campaign

For reasons I don't understand, Portugal is no longer a playable faction.

Kingdoms Features

The mod brings in extra units from Kingdoms campaign, like Mangonel, which is the only pre-gunpowder artillery unit worth using in battles.

Forts are now permanent stone buildings, and cost a lot more money.

Hanseatic League and potential Lithuania conversion events from Teutonic campaign are in the game.

Merchants are actually worth something.

Game balance

Medieval 2 is very easy for experienced player, so the mod does a few things to make it more difficult.

There are a lot fewer rebel settlements, and they tend to have a lot bigger garrison, so you can't just grab enormous areas of land without any opposition during your first few turns.

Time scale is set to 2 turns per year. All buildings build a lot more slowly.

Units provided by buildings are a lot different - high tier building no longer allow recruitment of low tier units, possibly to help AI or to clean up the UI.

Castles are massively nerfed because base castle buildings no longer provide any recruitment options, and you actually need to build stables and barracks (which also build really slowly). On the other hand, a lot of castle units get free upkeep as long as they're castle garrison.

Pope is as bitchy as ever, and due to a lot fewer rebels, he's going to be excommunicating people left and right a lot more than in vanilla. I never liked that part of the game, and I wished they toned it down.

AI gets a lot of free money on hard and very hard. On any difficulty levels, if it goes into severe enough debt, it gets a lot of money to get it out of debt.

And then there's garrison script.

Garrison script

Unlike later Total War games, Medieval 2 settlements have no automatic garrison, and AI often leaves them with minimal defences. That allows player to really easily destroy all factions.

As AI isn't really fixable, to counter that, a lot of mods add garrison script which spawn a lot of troops when garrison is besieged. I'm not convinced this is a great idea - I hated it when I didn't know where garrisons are going to happen and how big.

Once I generated a list of which faction would spawn what kind of units where, it was actually decent. So here's the script. Once it triggers for a settlement there's 10 turn timer before it can trigger again. Only AI gets free troops. Some factions have script for settlements that don't belong to them at start of campaign.

Antioch:

• Antioch (14) - 1 Knights Hospitaller, 2 Antioch Militia, 3 Armenians of Celicia, 3 Dismounted Knights of Antioch, 5 Hospitaller Sergeant
• Krak_de_Chevaliers (12) - 2 Dismounted Knights of Antioch, 2 Knights Hospitaller, 3 Armenians of Celicia, 5 Hospitaller Sergeant

Aztecs:

• Huaxtepec (14) - 1 Coyote Priests, 2 Aztec Archers, 2 Jaguar Warriors, 3 Arrow Warriors, 6 Cuahchiqueh
• Tenochtitlan (14) - 1 Coyote Priests, 2 Aztec Archers, 2 Jaguar Warriors, 3 Arrow Warriors, 6 Cuahchiqueh
• Tixtla (14) - 1 Coyote Priests, 2 Aztec Archers, 2 Jaguar Warriors, 3 Arrow Warriors, 6 Cuahchiqueh
• Tollan (14) - 1 Coyote Priests, 2 Aztec Archers, 2 Jaguar Warriors, 3 Arrow Warriors, 6 Cuahchiqueh

Byzantium:

• Constantinople (16) - 1 Greek Firethrower, 2 SE Town Militia, 4 Byzantine Guard Archers, 4 Varangian Guard, 5 Byzantine Infantry
• Corinth (11) - 1 Byzantine Lancers, 3 Byzantine Spearmen, 3 Trebizond Archers, 4 Dismounted Byzantine Lancers
• Smyrna (11) - 1 Byzantine Lancers, 3 Byzantine Spearmen, 3 Trebizond Archers, 4 Dismounted Byzantine Lancers
• Thessalonica (10) - 1 Byzantine Lancers, 2 Pronoia Infantry, 3 Trebizond Archers, 4 Byzantine Infantry
• Trebizond (11) - 1 Byzantine Lancers, 3 Byzantine Spearmen, 3 Trebizond Archers, 4 Dismounted Byzantine Lancers

Denmark:

• Arhus (14) - 1 Feudal Knights, 2 Town Militia, 3 Norse Archers, 3 Sami Axemen, 5 Norse Swordsmen
• Hamburg (11) - 1 Huscarls, 2 Sami Axemen, 3 Norse Archers, 5 Dismounted Huscarls
• Lund (11) - 1 Huscarls, 2 Sami Axemen, 3 Norse Archers, 5 Dismounted Huscarls

Egypt:

• Alexandria (11) - 1 Mamluks, 2 Hashishim, 3 Desert Archers, 5 Saracen Militia
• Cairo (15) - 1 Mamluks, 2 al Ashair, 3 Tabardariyya, 4 Desert Archers, 5 Saracen Militia
• Jerusalem (12) - 1 Mamluks, 3 Desert Archers, 3 Hashishim, 5 Saracen Militia

England:

• Caen (11) - 1 Feudal Knights, 2 Armored Swordsmen, 3 Longbowmen, 5 Armored Sergeants
• Dublin (11) - 1 Mailed Knights, 2 Armored Sergeants, 3 Archer Militia, 5 Spear Militia
• Gloucester (11) - 1 Mailed Knights, 2 Armored Sergeants, 3 Longbowmen, 5 Spear Militia
• London (14) - 1 Feudal Knights, 2 Town Militia, 3 Armored Swordsmen, 3 Longbowmen, 5 Spear Militia
• Nottingham (11) - 1 Feudal Knights, 2 Armored Swordsmen, 3 Longbowmen, 5 Armored Sergeants
• York (11) - 1 Mailed Knights, 2 Armored Sergeants, 3 Longbowmen, 5 Spear Militia

France:

• Angers (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Peasant Crossbowmen, 5 Armored Sergeants
• Caen (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Peasant Crossbowmen, 5 Armored Sergeants
• Marseille (10) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Crossbowmen, 4 Spear Militia
• Paris (14) - 1 Feudal Knights, 2 Town Militia, 3 Dismounted Feudal Knights, 3 Scots Guard, 5 Spear Militia
• Toulouse (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Peasant Crossbowmen, 5 Armored Sergeants

HRE:

• Bologna (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Crossbow Militia, 5 Spear Militia
• Frankfurt (14) - 1 Teutonic Knights, 2 Town Militia, 3 Crossbow Militia, 3 Dismounted Feudal Knights, 5 Spear Militia
• Hamburg (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Peasant Crossbowmen, 5 Armored Sergeants
• Innsbruck (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Peasant Crossbowmen, 5 Armored Sergeants

Hungary:

• Bran (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Bosnian Archers, 5 Pavise Spearmen
• Budapest (14) - 1 Feudal Knights, 2 EE Town Militia, 3 Bosnian Archers, 3 Dismounted Feudal Knights, 5 EE Spear Militia

Ireland:

• Cork (14) - 1 Ridire, 2 Ceitherne, 3 Galloglaich, 3 Saighdeoir, 5 Ostmen
• Donegal (11) - 1 Ridire, 2 Galloglaich, 3 Saighdeoir, 5 Cliathairi
• Galway (11) - 1 Ridire, 2 Galloglaich, 3 Saighdeoir, 5 Cliathairi

Jerusalem:

• Acre (11) - 1 Knights Templar, 2 Dismounted Knights Of Jerusalem, 3 Maronites of Lebanon, 5 Templar Sergeant
• Jerusalem (14) - 1 Knights Templar, 2 Pisan and Geonese sailors, 3 Dismounted Knights Of Jerusalem, 3 Maronites of Lebanon, 5 Templar Sergeant

Kievan Rus:

• Kiev (11) - 1 EE Cavalry Militia, 2 Berdiche Axemen, 3 Dismounted Dvor, 5 EE Spear Militia
• Moscow (11) - 1 EE Cavalry Militia, 2 Dismounted Boyar Sons, 3 Dismounted Dvor, 5 EE Spear Militia
• Novgorod (11) - 1 EE Cavalry Militia, 2 Dismounted Boyar Sons, 3 Dismounted Dvor, 5 EE Spear Militia
• Smolensk (11) - 1 EE Cavalry Militia, 2 Dismounted Boyar Sons, 3 Dismounted Dvor, 5 EE Spear Militia

Lithuania:

• Minsk (11) - 1 Szlachta, 2 Samogitian Axemen, 3 Baltic Archers, 5 Dismounted Szlachta
• Riga (12) - 1 Dievas Guard, 3 Baltic Archers, 3 Samogitian Axemen, 5 EE Spear Militia
• Vilnius (14) - 1 Dievas Guard, 2 EE Town Militia, 3 Baltic Archers, 3 Samogitian Axemen, 5 EE Spear Militia

Milan:

• Bologna (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Genoese Crossbow Militia, 5 Italian Spear Militia
• Genoa (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Genoese Crossbow Militia, 5 Italian Spear Militia
• Milan (14) - 1 Feudal Knights, 2 Italian Militia, 3 Dismounted Feudal Knights, 3 Genoese Crossbow Militia, 5 Italian Spear Militia
• Rome (14) - 1 Feudal Knights, 2 Italian Militia, 3 Dismounted Feudal Knights, 3 Genoese Crossbow Militia, 5 Italian Spear Militia

Mongols:

• Bulgar (13) - 1 Khan's Guard, 1 Naffatun, 3 Dismounted Heavy Archers, 3 Dismounted Heavy Lancers, 5 ME Halberd Militia
• Constantinople (12) - 1 Khan's Guard, 1 Naffatun, 2 Dismounted Heavy Lancers, 3 Dismounted Heavy Archers, 5 ME Spear Militia
• Kiev (12) - 1 Khan's Guard, 1 Naffatun, 2 Dismounted Heavy Lancers, 3 Dismounted Heavy Archers, 5 ME Spear Militia
• Moscow (12) - 1 Khan's Guard, 1 Naffatun, 2 Dismounted Heavy Lancers, 3 Dismounted Heavy Archers, 5 ME Spear Militia
• Novgorod (12) - 1 Khan's Guard, 1 Naffatun, 2 Dismounted Heavy Lancers, 3 Dismounted Heavy Archers, 5 ME Spear Militia
• Sarkel (12) - 1 Naffatun, 2 Mongol Heavy Archers, 2 Mongol Heavy Lancers, 3 Dismounted Heavy Archers, 4 Dismounted Heavy Lancers
• Smolensk (12) - 1 Naffatun, 2 Mongol Heavy Archers, 2 Mongol Heavy Lancers, 3 Dismounted Heavy Archers, 4 Dismounted Heavy Lancers
• Trebizond (12) - 1 Naffatun, 2 Mongol Heavy Archers, 2 Mongol Heavy Lancers, 3 Dismounted Heavy Archers, 4 Dismounted Heavy Lancers
• Viatka (12) - 1 Naffatun, 2 Mongol Heavy Archers, 2 Mongol Heavy Lancers, 3 Dismounted Heavy Archers, 4 Dismounted Heavy Lancers
• Vilnius (12) - 1 Khan's Guard, 1 Naffatun, 2 Dismounted Heavy Lancers, 3 Dismounted Heavy Archers, 5 ME Spear Militia
• Yerevan (11) - 1 Khan's Guard, 1 Naffatun, 2 Dismounted Heavy Lancers, 3 Dismounted Heavy Archers, 4 ME Spear Militia

Moors:

• Algiers (11) - 1 Arab Cavalry, 2 Dismounted Arab Cavalry, 3 Desert Archers, 5 Lamtuna Spearmen
• Cordoba (14) - 1 Arab Cavalry, 2 ME Town Militia, 3 Desert Archers, 3 Hashishim, 5 ME Spear Militia
• Granada (11) - 1 Arab Cavalry, 2 Dismounted Arab Cavalry, 3 Desert Archers, 5 Lamtuna Spearmen
• Lisbon (11) - 1 Arab Cavalry, 2 Dismounted Arab Cavalry, 3 Desert Archers, 5 Lamtuna Spearmen
• Marrakesh (11) - 1 Arab Cavalry, 2 ME Town Militia, 3 Desert Archers, 5 ME Spear Militia
• Valencia (11) - 1 Arab Cavalry, 2 Dismounted Arab Cavalry, 3 Desert Archers, 5 Lamtuna Spearmen

Norway:

• Agder (11) - 1 Huscarls, 2 Dismounted Feudal Knights, 3 Norse Archers, 5 Spear Militia
• Oslo (14) - 1 Huscarls, 2 Town Militia, 3 Dismounted Feudal Knights, 3 Norse Archers, 5 Norse Swordsmen
• Wick (11) - 1 Huscarls, 2 Dismounted Feudal Knights, 3 Norse Archers, 5 Norse Swordsmen

Novgorod:

• Novgorod (11) - 1 EE Cavalry Militia, 2 Dismounted Boyar Sons, 3 Dismounted Dvor, 5 EE Spear Militia

Papal States:

• Rome (17) - 2 Italian Militia, 3 Dismounted Feudal Knights, 4 Italian Spear Militia, 4 Papal Guard, 4 Pavise Crossbow Militia

Poland:

• Danzig (11) - 1 Polish Knights, 2 Dismounted Polish Knights, 3 Lithuanian Archers, 5 EE Spear Militia
• Krakow (14) - 1 Polish Knights, 2 EE Town Militia, 3 Dismounted Polish Knights, 3 Lithuanian Archers, 5 EE Spear Militia
• Poznan (11) - 1 Polish Knights, 2 Dismounted Polish Knights, 3 Lithuanian Archers, 5 EE Spear Militia

Portugal:

• Lisbon (14) - 1 Feudal Knights, 2 Town Militia, 3 Crossbow Militia, 3 Dismounted Feudal Knights, 5 Swordsmen Militia
• Oporto (10) - 2 Dismounted Feudal Knights, 3 Crossbow Militia, 5 Swordsmen Militia

Scotland:

• Edinburgh (14) - 1 Feudal Knights, 2 Town Militia, 3 Dismounted Feudal Knights, 3 Highland Archers, 5 Spear Militia
• Wick (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Highland Archers, 5 Spear Militia

Sicily:

• Naples (14) - 1 Norman Knights, 2 Italian Militia, 3 Dismounted Norman Knights, 3 Sicilian Muslim Archers, 5 Italian Spear Militia
• Palermo (11) - 1 Norman Knights, 2 Dismounted Norman Knights, 3 Sicilian Muslim Archers, 5 Armored Sergeants
• Rome (14) - 1 Norman Knights, 2 Italian Militia, 3 Dismounted Norman Knights, 3 Sicilian Muslim Archers, 5 Italian Spear Militia

Spain:

• Leon (14) - 1 Knights of Santiago, 2 Town Militia, 3 Crossbow Militia, 3 Dismounted Feudal Knights, 5 Swordsmen Militia
• Lisbon (14) - 1 Feudal Knights, 2 Town Militia, 3 Crossbow Militia, 3 Dismounted Feudal Knights, 5 Swordsmen Militia
• Toledo (11) - 1 Knights of Santiago, 2 Dismounted Feudal Knights, 3 Peasant Crossbowmen, 5 Swordsmen Militia
• Valencia (11) - 1 Feudal Knights, 2 Dismounted Feudal Knights, 3 Peasant Crossbowmen, 5 Swordsmen Militia

Teutonic Order:

• Danzig (12) - 1 Christ Knights, 3 Order Spearmen, 3 Prussian Archers, 5 Sword Brethren
• Konigsberg (12) - 1 Christ Knights, 3 Order Spearmen, 3 Prussian Archers, 5 Sword Brethren
• Marienburg (14) - 1 Christ Knights, 2 Dismounted Ritterbruder, 2 Order Militia, 3 Prussian Archers, 6 Order Spearmen
• Riga (12) - 1 Christ Knights, 3 Order Spearmen, 3 Prussian Archers, 5 Sword Brethren

Turks:

• Adana (11) - 1 Turkish Horse Archers, 2 Hashishim, 3 Turkish Archers, 5 Dismounted Hasham
• Antioch (13) - 2 Kurdish Auxiliaries, 3 Hashishim, 3 Janissary Archers, 5 Saracen Militia
• Baghdad (15) - 2 Kurdish Auxiliaries, 2 ME Town Militia, 3 Janissary Archers, 3 Janissary Heavy Inf, 5 Saracen Militia
• Constantinople (12) - 2 Hashishim, 2 Kurdish Auxiliaries, 3 Janissary Archers, 5 Saracen Militia
• Iconium (11) - 1 Turkish Horse Archers, 2 Hashishim, 3 Turkish Archers, 5 Dismounted Hasham
• Yerevan (11) - 2 Hashishim, 2 Kurdish Auxiliaries, 3 Turkish Archers, 4 Saracen Militia

Venice:

• Bologna (11) - 1 Feudal Knights, 2 Venetian Heavy Infantry, 3 Venetian Archers, 5 Italian Spear Militia
• Ragusa (12) - 2 Italian Militia, 2 Venetian Heavy Infantry, 3 Venetian Archers, 5 Italian Spear Militia
• Rome (13) - 2 Italian Militia, 3 Venetian Archers, 3 Venetian Heavy Infantry, 5 Italian Spear Militia
• Venice (13) - 2 Italian Militia, 3 Venetian Archers, 3 Venetian Heavy Infantry, 5 Italian Spear Militia

Wales:

• Caernarvon (15) - 2 Welsh Militiamen, 2 Welsh Skirmishers, 3 Rhyfelwyr, 3 Saethwyr, 5 Meirionnydd Spearmen
• Cardiff (12) - 2 Gwent Raiders, 2 Welsh Skirmishers, 3 Saethwyr, 5 Spear Militia

Free Upkeep Units

These are only free if settlement has right building to recruit them. Some are also free in forts, I think these are castle units. Generally all militia units (including cavalry militia) and castle infantry are upkeep free while garrisoned. Castle cavalry, siege units, and boats are not free.

• Ahdath - turks
• Al Haqa Infantry - egypt
• Alamanoi - byzantium, slave
• Almughavars - spain, portugal
• Antioch Militia - antioch
• Archer Militia - england, slave
• Armored Sergeants - england, france, hre, milan, venice, papal_states, sicily, slave
• Aventuros - portugal
• Aztec Spearmen - aztecs, slave
• Aztec Warriors - aztecs, slave
• Baltic Archers - lithuania
• Bill Militia - england
• Burgher Pikemen - teutonic_order, slave
• Byzantine Spearmen - byzantium, slave
• Ceitherne - ireland, slave
• Cliathairi - ireland
• Croat Axemen - hungary, slave
• Crossbow Militia - france, hre, denmark, spain, portugal, teutonic_order, norway, slave
• Deisi Javelinmen - ireland
• Dismounted Byzantine Lancers - byzantium
• Dismounted Christian Guard - moors
• Dismounted Conquistadores - spain, portugal
• Dismounted Druchima - russia, novgorod
• Dismounted Feudal Knights - england, scotland, france, hre, denmark, spain, portugal, milan, venice, papal_states, hungary, norway
• Dismounted Ghulams - egypt, slave
• Dismounted Halbbruder - teutonic_order
• Dismounted Hasham - turks
• Dismounted Heavy Lancers - mongols, timurids
• Dismounted Knights Of Jerusalem - jerusalem
• Dismounted Knights of Antioch - antioch
• Dismounted Light Lancers - mongols
• Dismounted Norman Knights - sicily
• Dismounted Polish Knights - poland
• Dismounted Polish Nobles - poland
• Dismounted Szlachta - lithuania
• Dismounted Tartar Lancers - russia, lithuania
• EE Archer Militia - russia, novgorod, slave
• EE Cavalry Militia - russia, novgorod
• EE Crossbow Militia - poland, russia, novgorod, lithuania, slave
• EE Spear Militia - poland, russia, novgorod, hungary, lithuania, slave
• EE Spearmen - poland, russia, novgorod
• EE Town Militia - poland, hungary, lithuania
• Estonian Rebels - lithuania, slave
• Followers of Perkunas - lithuania
• Frankish Axemen - jerusalem, slave
• Frankish Swordsmen - antioch, slave
• Galloglaich - ireland, slave
• Genoese Crossbow Militia - milan
• Gotland Footmen - denmark, norway
• Greek Militia Cavalry - byzantium
• Gwent Raiders - wales, slave
• Halberd Militia - hre, papal_states, sicily, poland, hungary, slave
• Heavy Bill Militia - england
• Heavy Billmen - england
• Heavy Pike Militia - scotland
• Helwyr - wales
• Highlanders - scotland, slave
• Hospitaller Sergeant - antioch
• Italian Cavalry Militia - milan, venice, papal_states, sicily
• Italian Militia - milan, venice, papal_states, sicily, slave
• Italian Spear Militia - milan, venice, papal_states, sicily, slave
• Jaguar Warriors - aztecs, slave
• Kurdish Javelinmen - egypt
• L Cavalry Militia - lithuania
• Landschneckt Pikemen - slave, hre, milan
• Levy Spearmen - england, slave
• ME Archer Militia - egypt
• ME Crossbow Militia - moors
• ME Halberd Militia - egypt, turks, mongols, timurids
• ME Levy Spearmen - mongols, slave
• ME Peasant Archers - egypt, turks, mongols, timurids, slave
• ME Spear Militia - moors, egypt, turks, mongols, timurids, slave
• ME Town Militia - moors, turks, mongols, timurids, slave
• Meirionnydd Spearmen - wales
• Merchant Cavalry Militia - england, scotland, france, hre, denmark, spain, portugal, poland, hungary, ireland, norway, teutonic_order, wales
• Mongol Foot Archers - mongols
• Mongol Infantry - mongols
• Morgannwg Spearmen - wales, slave
• Native Archers - slave
• Native Warriors - slave
• New World Cuirassers - england, france
• Noble Pikemen - scotland
• Noble Swordsmen - scotland
• Norse Swordsmen - denmark, norway, slave
• Nubian Spearmen - moors, egypt, slave
• Obudshaer - denmark, norway
• Order Militia - teutonic_order
• Order Spearmen - teutonic_order, slave
• Ostmen - ireland
• Partisan Militia - france
• Pavise Crossbow Militia - venice, papal_states, sicily, hungary, slave
• Peasant Archers - england, scotland, france, hre, denmark, spain, portugal, milan, venice, papal_states, slave, norway, teutonic_order
• Pike Militia - france, hre, spain, portugal, milan, venice, papal_states, sicily, ireland
• Pikemen - france, england
• Pronoia Infantry - byzantium
• S Archer Militia - byzantium, slave
• SE Spear Militia - byzantium, slave
• SE Town Militia - byzantium, slave
• Saracen Militia - egypt, turks, slave
• Scots Pike Militia - scotland, slave
• Sergeant Spearmen - france, hre, milan, venice, papal_states, sicily, slave
• Spear Militia - england, scotland, france, hre, denmark, spain, portugal, slave, norway, wales
• Sudovian Tribesmen - lithuania, slave
• Swiss Guard - papal_states
• Swiss Pikemen - slave, venice, sicily
• Sword Brethren - teutonic_order
• Sword and Buckler Men - spain, sicily
• Swordsmen Militia - spain, portugal
• Swordstaff Militia - denmark, norway
• Syrian Militia - jerusalem, antioch
• Templar Sergeant - jerusalem
• Tercio Pikemen - spain
• Town Militia - england, scotland, france, hre, denmark, spain, portugal, norway, slave
• Tripolitan Squires - jerusalem
• Turkish Crossbowmen - turks
• Ulster Swordsmen - ireland
• Urban Militia - moors
• Viking Raiders - denmark, norway, slave
• Welsh Militiamen - wales
• Woodsmen - poland, russia, novgorod, slave
• Zweihander - hre, milan
• al Ashair - egypt

Overall Impressions

The mod is pretty decent - just takes the best out of Medieval 2 without doing anything drastic.

Unfortunately I had a lot of crashes in game menu.

Fortunately once in game, crashes were not too common. I'd still manually save before and after every battle, but it was rarely necessary. Sadly vanilla crashes are not unheard of either.

If you want to give Medieval 2 another go, and want just a bit more variety, highly recommended.

Challenges for July 2019 SecTalks London

Another CTF, another victory. I won the June 2019 London SecTalks CTF and it was up to me to write challenges for July.

There were 12 challenges, theme of the challenges being Hacker-Archeology. It turns out that was too much, as only 10/12 challenges got even one solve, and nobody got more than 4. Even with very generous hint drops during the event.

Challenge files and code used to generate them are available on github.

There are no answers below, but some serious hints which might make it too easy.

For previous rounds, see posts about September 2017, November 2017, May 2018, July 2018, October 2018, February 2019, and April 2019 CTFs.

SHAR (5 points)

Self-extracting Shell archive nested 8 levels deep. It only works on Linux, OSX shell can't extract Linux-created SHAR files, showing what a dumb format this is.

XBM (10 points)

A weird way to encode image into C headers. While totally obsolete, a lot of tools still support it.

Maya (15 points)

An image with a sequence of Maya numerals, each encoding ASCII symbols. It seems people were confused by the fact that multi-digit Maya numerals are stacked vertically.

PCX (20 points)

PCX file with flag on it, but both foreground and background colors having same RGB color, so color palette would need to be adjusted to actually see it.

ECB (25 points)

A signed cookie server which would only sign cookies without admin=yes, and it would only give you the flag if you sent it signed cookie with admin=yes.

It's a classic cryptography attack on ECB mode, rearranging blocks within or between cookies.

SED (30 points)

SED is an obsolete programming language for text stream processing, and the challenge was a simple flag validation script which was just a sequence of regexp replace rules.

Nobody noticed that, but that SED script was also totally valid Perl 5 script.

CBC (35 points)

A signed cookie server which would only sign cookies without admin=yes, and it would only give you the flag if you sent it signed cookie with admin=yes.

It's a classic bit flipping attack on CBC.

MD4 (40 points)

A signed cookie server which would only sign cookies without admin=yes, and it would only give you the flag if you sent it signed cookie with admin=yes.

It's a classic length extension attack.

Midi (45 points)

A midi file with flag encoded in Morse code, played on an Ocarina instrument suggestively named "Morse Ocarina".

It could be done either by hand, or by converting note lengths in Midi format to dots and dashes. I think everybody ended up doing it by hand.

DOS (50 points)

A very small COM file flag validator. It was extremely simple, but a lot of tooling like Ghidra has trouble with COM files, as they're too old to be relevant.

It got zero solves, which was fairly surprising, as flag validator is really simple:

# Initialize counter in BX
  mov bx, 0xd7ab

# Get ASCII code of next character into AL
  mov ah, 0x1
  int 0x21

# Add AX to the counter, check if correct
  add bx, ax
  cmp bx, 0xd911
  jnz near 0x19d

# Repeat for next character

Differences between constants being compared are 256 + ASCII code of each letter (first being 0x166 or 256 + "f").

Hieroglyphs (55 points)

Monoalphabetic cipher encoded into Egyptian hieroglyphs. The text was very long English text with spaces removed.

It's really simple for anyone who ever did monoalphabetic cipher breaking through statistical analysis, so that many points mostly being potentially quite time consuming, but it didn't even take people that much time.

Perl 6 (60 points)

In remote past Perl 6 used to be the language of the future. That future never came.

The challenge is a flag validator in Perl 6 aggressively using many unusual Perl 6 operations. I'm not really surprised that nobody succeeded at solving it.

Some Thoughts on Stepmania

Back in 2009 I even wrote a rant about Stepmania, and here's another one.

In my younger days I did a lot of Dance Dance Revolution, and every now and then I come back to it.

I'd like to play with some modern songs. Supposedly official games exist, but they have like 50 songs per game, plus a few "$5 for 3 songs" DLCs, and mostly only work on some weird-ass consoles.

Basically the only option for it is using Stepmania, and downloading some user-made songpacks.

So what's wrong with all this

The first problem is that music is totally free online on youtube and such, but Big Copyright would never allow a healthy ecosystem of dance games, because they just love the whale exploitation model of "$5 for 3 songs", and the market is too small for someone to force them to be reasonable.

Stepmania tries to avoid any direct entanglements with all those copyright issues, and unfortunately that means it avoids actually trying to solve the problems.

User-made song packs have mixed quality

So I got a bunch of random songpacks, basically keyword matching artists I might like.

The songpacks are basically whatever the author decided to throw there, so I'm generally only interested in very small portion of each, but let's say I keep them all, as at this stage I don't know if those songs are any good or not.

Most of user-made content is decent enough, but it's far too common that there's bullshit songs with ratings like 12-20, which are presumably meant for the keyboard, or maybe arcade machines with safety rail, since it would be unsafe to even try on a soft dance mat, regardless of one's skill level.

There's plenty of songs which are poorly synced. There's plenty of songs which have very questionable ratings, and are actually a lot harder than other songs at same ratings.

This wouldn't be a huge deal if there was a way to filter that out easily, but there isn't.

Stepmania UI is atrocious, especially when you have a lot of songs

All right let's say I have a few thousand songs now. Stepmania will take forever to actually start, like literally over 15 minutes. It seems latest version and SSD finally made it tolerable, but seriously, just checking that a few thousand tiny files didn't change shouldn't take this kind of crazy time.

The next problem is how to actually choose those songs. Stepmania decided to copy dance mat only UI from arcade machines, without any keyboard backup. So instead of taking 1 second to type song title or artist name, it takes literal minutes to scroll through thousands of songs to get there, even at highest speed.

The UI has other issues - like it seems that I end up triggering song options about 1 in 10 times when trying to just start a song, and it registers it as Start button being pressed twice for whichever reason.

There's a lot of weird combos to control the UI, but other than "difficulty up", "difficulty down", and "change sort order" I have no idea what they are, and there's nothing intuitive about it. There should just be goddamn menus and keyboard controls for those rarely used functionality.

Once song starts playing it's pretty much fine.

Ideas for solutions

So it seems I have the same complaints today I had a decade ago.

Anyway, let's talk possible solutions.

These days it's just easier to write cross-platform games in something civilized, with Electron or whatever. Stepmania isn't really a terribly complicated program, so if anyone felt like it, they'd probably have something kinda working in a few days, and it would probably have better performance and usability even at such early point.

Much more interesting is automated step files filtering and analysis. Step files are literally asking for a neural network analysis to flag broken ones, figure out correct difficulty, and so on. Prototypes to just generate step files outright exist, and filtering/analysis should be a lot easier. It just needs to be takes out of research paper and given to users.

A far more interesting idea is just taking songs from youtube or whenever, and doing everything automatically from that point, but that's quite questionable. Stepmania songs are generally ~100s remixes of ~200s songs, so that would already be a major difference. The whole step file process might be too computationally expensive.

How would it work in practice

I feel first step would be writing some parsers to take step files in variety of kinda documented custom formats and export them as some json. It's probably going to be quite tedious, but nothing difficult.

Then figuring out how to interact with dance mats. Most of them are just USB HID devices, so it shouldn't be too hard. In principle Electron supports WebUSB, so dance mat support should be quite straightforward, at least when everything goes right.

With these two, getting simple Stepmania-like program with Electron shouldn't be hard, and that could be a platform for all those crazy ideas.

I'm not hating here

Stepmania is still one of the most successful Open Source games ever. I just think something much better is possible.

EDIT - USB APIs in browser

Well, I tried to use WebUSB API and Gamepad API in Chrome, both supposedly supported, and they don't see my dance mat on Windows or on OSX.

It's possible I'd have more luck with Electron.

EDIT - Stepmania keyboard shortcuts

And it turns out Stepmania actually added some keyboard navigation recently. So if I have songs sorted by artist I can press Control-M to go to M etc. That's fine to reach Katy Perry, maybe less so for Avril Lavigne. Still, even that little change cuts scrolling time by more than half.

Eric Swalwell is wrong and Nancy Pelosi is right

I bet you did not see this one coming.

Eric Swalwell was one of 20 candidates for Democratic presidential nomination. The only notable moment of his whole candidacy was the "pass the torch" moment during debates, where he used Joe Biden's own words to basically attack Joe Biden for being too old. I strongly believe that there should be mandatory retirement age for politicians, so he had a point there.

Anyway, Eric Swalwell's campaign was going nowhere, he gave up before even second round of debates, and after giving up he gave exit interview to 538 politics podcast, and that's what this is about.

Extremist Drift

Debates were notorious for how far left most candidates went compared with mainstream Democrats of just a few years ago, like Barrack Obama, Hillary Clinton, or Nancy Pelosi. Positions like abolition of borders, free healthcare for illegal immigrants, abolition of private health insurance, forced busing, racial reparations and other extremist positions had far more support that one would have guessed.

In a way the winner of the debates was Donald Trump. His approval rate improved after Democratic debates, and is currently at -7.5%. It's still negative, but it's far better than his the usual -10% to -20% range, and if this extremist drift continues, Democrats might manage to scare off all potential moderate voters.

Swalwell's Argument

During exit interview, Eric Swallwell was asked about that.

His response was that the most important feature of a candidate is "authenticity", and that going "far to the left" is absolutely fine, and no risk whatsoever in general election.

Democrats tiptoe around the issues that are perceived as unpopular, whereas Republicans have no problem leading with very unpopular issues [...] and they don't pay a price at the ballot box.

He believes that Trump's victory proves it.

He also openly advocates violating the Constitution in the same statement, but let's not get there.

It's bullshit

US economy is doing better than it did in living memory. Unemployment rate is as low as it last was in the 1960s.
Inflation has been low and stable. Stock market is at unprecedented heights.
Tax cuts mean most working people have a lot of extra money personally (except for rich people in high tax states). All the secondary metrics like wages growth, gas prices, healthcare access, and so on are doing just fine.

In foreign policy, no new wars were started, for the first time in it's hard to tell how many presidencies. ISIS which spread during Obama's term and overran multiple countries was swiftly crushed. There weren't any major terrorist attacks, or other foreign disasters.

Fundamentals models don't have an amazing track record at predicting elections, but it's a pretty safe prediction that with everything an average person cares about going so much better than basically ever, whoever presides over that should be super popular and crush any challenges 1984 style, right?

Well, that's what would have happened if president Marco Rubio or Mitt Romney was presiding over it.

It's Trump specifically being such a turd that makes these elections a 50:50 thing.

How Trump won

Trump was the most unpopular presidential candidate in recorded history. Fotunately for him, it just so happened that his opponent Hillary Clinton was the second most unpopular presidential candidate in recorder history. She was so unpopular she lost to a totally unknown black half-Kenyan guy with name "Barrack Hussein Obama", in spite of DNC establishment doing all they could to force her though. She was so unpopular she nearly lost to a senile openly socialist Jew who wasn't even in the party, and only managed to somehow got through thanks to DNC establishment forcing her candidacy even harder. She was so unpopular she lost to Donald Trump.

In such Giant Douche vs Turd Sandwich elections Trump just so happened to have better ran campaigns, and barely squeezed the victory.

He did not become any more popular since then. As an aside, Hillary Clinton is now even more unpopular than Trump, but fortunately for Democrats she's not running for the third time.

Trump only managed to win in 2016 because his opponent was so unpopular, and the economy was still only slowly recovering.

In 2020 he has far easier job - no sitting president could ever lose with fundamentals doing so well, unless he's literally a Turd Sandwich.

Why bother with Eric Swalwell?

Eric Swalwell might be out, but other candidates for Democratic nomination seem to think the same way as him. They're trying to score points with extremists in party base, and hope that somehow it will work out.

It could work against Turd Sandwich, but not if they end up picking a Giant Douche again.

The idea that extremist will somehow increase turnout among the base is total nonsense - highly politicized people will vote anyway, and you're far more likely to increase turnout of opponent's base this way - as Trump did during 2018 midterm, with his shittiness really motivating Democrats to go vote, regardless of who was their local candidate.

It's not like this is a novel strategy. Nancy Pelosi, the most successful Democratic politician, has been successfully doing just that - marginalizing the extremists in her party while pushing hard for what's realistically achievable. Especially in country whose political system is designed for gridlock, focusing on popular parts of your party's agenda in alliance with moderates is the only way you can actually achieve anything.

The successful strategy is attacking your opponents where they're weak, not responding to Trump's Wall with abolishing ICE, abolishing borders, and free citizenship to anyone who jumps where the border used to be.

It is still 50:50

Many things can happen before the elections. The economy could crash. Trump could start WW3. One of many Trump scandals might end up discovering some real evidence of crimes resulting in impeachment, not decades old hearsay that convince only partisans. Any of those would shift elections far more than Democratic debates.

If none of that happens, and the elections is still 50:50, it will really matter if Democratic Party follows Nancy Pelosi, and picks a successful moderate, or follows Eric Swalwell and tries to outdo itself in extremist appeals and chooses a Giant Douche while handing over second term to the Turd Sandwich.

Far Left is losing everywhere

It's not US specific issue. Traditional center left parties (more or less analogous to the US Democratic Party) got weakened by the Great Recession, resulting in brief resurgence of the far left.

This resurgence is crashing now. UK Labour was lost every elections since it went extremist, and in some polls is now 4th with 18% support. In two countries worst affected by the crisis where far left actually got power it already lost it, Podemos completely crashed in Spain. Syriza more narrowly lost in Greece.

In most other Western countries, all this bickering on the left just weakened it, and let either center right or populist right take over. Many Western countries like Poland and Israel nowadays have 50 Shades of Right elections.

Seriously guys, just listen to Nancy, she knows what's best for you.

What's wrong with all music streaming services

Music streaming service has three jobs:

• be available
• play music
• discover new music

Be available

First music streaming service that offered decent recommendations was Audiogalaxy It was shut down by the Big Copyright.

A while later, Pandora came out, and it had mindblowingly good music discovery, but Big Copyright forced them to lock out everyone outside US. There's probably some way to access it with VPNs, but the hassle is just too great.

Play music

You'd think this would be trivial, but it's not. As a consequence of watching videos and listening to podcasts at high speed, my baseline speed changed, and I just don't enjoy any media below 140% speed.

That shouldn't be a problem, as browsers can inherently play audio and video at any speed, and there are browser extensions to add speed controls to sites without them.

Unfortunately services like Spotify and Deezer go out of their way to disable that, so I'd be locked to 100% speed at which all the music feels slowed down.

Sure I'm in tiny minority here, but who isn't in tiny minority one way or another when it comes to music?

As far as I know, this leaves me with just two options:

• Download the songs and play them locally
• Youtube

Downloading songs with tcpflow

Downloading songs lets me play whichever way I want, but there's not one shred of discovery there. The easiest way to download songs these days is youtube-dl, but you can use basically any source.

So funny story time! Once upon a time I wrote a music downloader that tapped into network traffic to get music from Pandora by tcpflow network intercepts. The music was just in MP3 files. Some minor complications was metadata being int separate XML HTTP requests, and songs being out of order for buffering purposes.

The whole thing was about 300 lines of ruby, it figured out which song and which metadata match, and then saved and organized all that.

It didn't mass download songs, and it didn't even interact with the website in any way, it was 100% passive and undetectable, it just saved everything I listened to in regular browser.

I needed those files to put them on a hardware MP3 player, as there was no way to listen to internet radio without the internet obviously.

I never mentioned it back then, as I would be way too much hassle for most people to setup, and if it got public they might change the API to make it more difficult somehow.

So I might have been the only person who got their music with tcpflow ever.

I have no idea if that still works. It's unlikely, as everything is routinely encrypted these days, so you'd need either a MITM proxy to strip the SSL, or capture music from a browser plugin, or something like chromedriver.

Youtube

As far as I know, all that leaves Youtube as the only option to me. It is available. It has more music than anything else thanks to all the covers. It plays music at any speed. It even has convenient downloads.

The problem is music discovery. It's not great. For ages it kept suggesting absolute garbage all the time. For example one thing I absolutely can't stand is male vocalists, but it kept doing this to me.

Now it seems it has given up and just plays same songs I already know, and occasionally some new mainstream hit.

Then again, what did I expect? It can't even consistently figure that after Part 4 of some let's play comes Part 5, and not Part 17.

For all the claims about imminent AI takeover, this is miserable. Maybe all those drivers who will lose their jobs to self-driving cars can get new jobs as music recommenders.

Discovering new music

So the last option is separating music discovery from music play.

There's plenty of sites where you put your favorite artists and they recommend some other artists. I don't think recommending based on artists is even a sensible thing, most artists create a range of different music, and good recommendations for someone who really likes this song would be really different from good recommendations for someone who really likes this one.

Of the services I tried, most are overwhelmingly awful. Sage is actually kinda OKish. Like 10% of what it recommends is actually good, and the UI of "click here to go to youtube search for the artist" is not the worst.

So overall I'm not too happy with all this, but it's not like it's possible to just fix it with some code, as Big Copyright would probably ban anything that improved the situation.

Into The Breach - Best Roguelike Ever

A screenshot from the game showing a mission where 4 Vek conveniently lined up for a single laser attack to kill them. Also Spider Boss keeps spawning a lot of spiders eggs each turn, and 3 more Vek are trying to get out of the ground. Best of luck defending that tower.

It's a very information-rich interface.

I wrote the explanation what's a Roguelike and what isn't for a reason. That reason is Into The Breach - possibly the best game I found this year. By the way for some other great game recommendations, check this out.

Overview of a Playthrough

Each playthrough represents a timeline of a fight between humans and Vek (insect aliens).

You start with a time-traveler pilot and a squad of 3 mechs, getting some upgrades on the way. There are 4 corporate islands you try to liberate in whichever order you choose. After liberating 2-4 of them, you can attack the final hive volcano island.

Every corporate island is 4 randomly generated battles (you have a bit of choice which battles to take), and 1 slightly less random final battle. Hive volcano islands is a 2-part special battle.

In other words, it's a Roguelike dungeon of 12-22 levels.

Difficulty is adjusted sensibly, so whichever island you pick first will be easiest, and difficulty of the final hive volcano fight increases based on how long you waited to fight it. You get upgrades (pilot xp, new time traveler pilots, extra mech reactors, new weapons) at about the rate aliens get their upgrades, but it depends on your squad and on how well you play if game gets a bit easier or a bit harder as it progresses.

The whole playthrough takes about 1h (for 2 islands) to 2h (for 4 islands), but it depends on how long you spend thinking. There's no time pressure, except for one achievement.

Metacampaign

Playthroughs are short, but there's a lot of metagame. There's 18 time traveler pilots each with special skill. There's 8 squads of 3 mechs each. You can also play as a random squad (you can reroll any number of times before pressing start), and you can make a custom squad of your choice from unlocked mechs.

You start with just 1 pilot and 1 squad unlocked. There's 3 achievements each for every one of 10 squads, and 25 achievements you can get as anyone. Achievements are currency you can use to unlock more squads. Pilots are unlocked by meeting them in the game - either rescuing them from a time pod, or getting them as a reward for perfectly finishing a corporate island.

Achievements, especially squad specific achievements, are generally great at guiding you towards interesting things you can do, and don't feel like stupid gimmicks or puns at all.

Once you finish, by winning or losing, all surviving pilots time travel to different timelines, and you can follow one of them. In such case, you'll start next campaign with a levelled up pilot. Or you can pick up any other pilot but they'll start at lowest level.

There's also Secret Squad you can get if you get every achievement.

Arguably your starting pilot is the worst of all pilots, but your starting squad is probably second best, so nothing ever feels like "grinding".

Defensive Game

So far it probably sounds pretty whatever, but here it gets interesting. Battles don't need you to kill the Vek. What you need to do is protect buildings connected to the power grid. You can have at most 7 power, and any time a building gets damaged, your power grid takes 1 or 2 hits. If you survive 5 turns all remaining Vek run away.

Each mission has some other objectives too, and you get a bonus for every objective fulfilled. A lot of objectives are also defensive in nature - including the infamous train defence, generally considered the most difficult mission of all, but which mission is easy or hard depends on your squad a lot.

I think that's the first game I've seen which focuses on defence this way. Usually your goal is destroying all enemies, and usually timers go against you, not for you.

Any damaged mechs get automatically repaired for next battle, so taking hits to save buildings or other objectives is expected. If mech gets fully wrecked, it also kills the pilot, so next battle it will start without any of the pilot bonuses, but it's not a huge deal, and you can find replacement pilots as you go.

Fully Deterministic Game

Beyond defensive game, Into The Breach does something even more unique. Your turn is (almost) completely deterministic. Structure of each turn is:

• Vek move and prepare their attacks
• you move and attack with your mechs
• Vek attack and environmental effects happen in predefined order
• more Vek spawn

There's no RNG for movement or attacks - you know exactly how much damage will happen. This leads to some amazing gameplay, as you know where Vek is trying to attack, but many of your attacks push instead (or in addition to) of damaging. So you can push Vek out of the way so it attacks empty space, or other Vek, getting some ridiculous blowouts.

A small exception is that buildings have small random chance of resisting damage. Authors say this was added to avoid player paralysis, as without it pressing "End Turn" sometimes means unavoidable loss, so players would spend forever on their losing turn. The chance is very low (starts at 15%, gets into about 25%-ish eventually), and I'd recommend not relying on it, but maybe it's the only way.

While your one turn is nearly deterministic, the whole battle is not. For most missions, you know where Vek will spawn each turn, but you don't know which kind exactly (every island has a bunch of possible types by some procedural generation). You don't know where Vek are going to move and what they'll try to attack. You have to respond to such challenges a few times in each battle.

Difficulty Level

The game is pretty good at not throwing "RNG says fuck you, so now you lose" moments at you like many other Roguelikes (including FTL by same authors).

If your playthrough is going well, you get more upgrades, so it gets a bit easier. If you barely squeeze by, you get fewer upgrades, and it gets a bit harder. The difference isn't huge - early losses do not generally snowball.

In each battle, if you eliminate Vek very quickly, more Vek will spawn, and if you do it slowly, fewer Vek will spawn. This is pretty much necessary to balance squads which try to kill Vek with squads which try to just get them out of the way.

Game has huge number of mechanics with complex interactions, huge diversity of enemies, a lot of mission types, a lot of different mechs, weapons, and special pilot skills. It's going to take a long time to master this. It's quite good at explaining everything in-game, but even 80h+ in I kept discovering unexpected interactions.

There's "normal" difficulty, which I strongly recommend not playing at. Play on "easy" at least until you get decent and unlock all squads. There's also "hard", which I played once to get that achievement, and barely managed to win. You can unlock all achievements playing on easy on your own pace just fine. Exceptions are achievements for winning campaign on hard, and one for winning two corporate islands with Blitzkrieg squad in under 30 minutes, which were my last two achievements to get.

AI

The Vek are pretty dumb by design. Each of them does something totally reasonable individually, at least it would be if you could just stand still for a moment. They don't coordinate in any way whatsoever, and they like attacking buildings (which you generally need to save) and your mechs (which can often move out of the way) about equally.

Basically if you have 3 mechs, and there's 3 Vek attacking building or other objectives, dealing with them is usually straightforward enough. There will usually be more than 3 Vek at time, and often more than 3 attacking buildings - it's bigger challenge to have one mech deal with more than one Vek with one move. Or sometimes a Vek immobilizes a mech somehow, usually with a web, complicating your life too.

The challenge is mainly in the numbers. Now this is just vague estimate, but let's say Normal has about 1 extra Vek at each time compared with Easy, and Hard has 1 extra Vek compared with Normal. That doesn't look like much, but 3 vs 3 is usually straightforward, while every further Vek requires more and more complex strategy, so difficulty ramps up a lot, and you have lower margin for mistakes.

Graphics

It's all nice retro pixel art. The interface is really good at showing everything that's going on, even when there's a lot going on. There's a few minor issues like tiles affected by multiple effects often show just one (you can mouseover for others), but these are very rare.

tl;dr

10/10 Must Play

It's a strategy with reasonable playthrough time. It's a roguelike without bullshit RNG. It became a classic the moment it got released.

You can get it for £11.39 or regional equivalent.