The Ultimate Fix for JSON format

I previously wrote how JSON is a fake format, which leaves important parts unspecified, so sending JSON from Python to Python, or from JavaScript to JavaScript generally works, but if you mix languages, results are largely unpredictable.

So, should every programming language have its own JSON-like, expressing data types it supports faithfully? That would also be a huge mess, so here's my proposal instead.

There is no code included, but code for it would be fairly straightforward if anyone wanted to go forward with it.

Baseline

All current JSON would still parse under this proposal, but unfortunately all existing ambiguities would still remain.

The proposal is primarily meant for machine to machine communication, but any such format always ends up being written and edited by humans as well. Therefore, before we go any further, we need to include JSON5's obvious fixes. Comments and trailing commas are a must, but there's no downside to just making JSON5 our baseline, and supporting plus and minus infinities, NaNs, and single quoted strings. Technically, this is orthogonal to the rest of the proposal, but if we have an opportunity to fix known issues with JSON, we might just as well do it now.

Extensions mechanism

And now here's the key part - just add one extra construct TypeName(arguments). We don't need hundreds of different syntaxes, this covers everything we might possibly want. Different languages will use a different subset of them, but the important thing is that most of these can be widely shared.

Extensions represent data types semantically, and their names won't necessarily match whatever a given language calls that data type (Time vs Timestamp vs DateTime etc. - just pick whatever's clearest).

Universal extensions

These two types are representable in pretty much every even halfway sane language that exists, and they're badly missing:

Time("iso string") - this is supported by pretty much every language. Everything not in standard timestamp format is an error.

Integer(n) - arbitrary precision integer, native Integer type in Ruby/Python, BigInt in JavaScript. This represents a mathematical object, so for this type Integer(0) and Integer(-0) are defined to be the same. Everything that's not an integer as argument is an error.

Other extensions

Depending on your language, you could have a lot of other useful types, like:

• Date("YYYY-MM-DD")
• Map([[key, val], ...]) - with arbitrary keys, not just strings
• Set([elements])
• Decimal(arbitrary precision number)
• Double(exactly IEEE 754 floating point)
• UUID("hex string")
• Rational(num, den)
• Bytes("base64 string")

There could be even ones that are limited to very few languages, like:

• Symbol("string") - for Ruby symbols, Elixir Atoms etc.
• Tuple([elements]) - for Python tuples
• UInt128(number) - for languages that really need specific precision

Some group of common extension types would likely be agreed on, and these should be generally representable in most languages. We probably won't see round trip invariance - parsing Date(...) might convert it to Time(...), parsing Symbol(...) to a string, parsing Tuple to an array and so on.

Defining new extensions

If you're developing a new programming language with some extra datatypes, or you just have some user-defined types that you really want to represent, you could still use the same format with the same parser. The parsing library could offer hooks to pass whitelisted Type(...) constructors to your hook. These would generally not be interpretable if you send it to another environment, but such data would still parse, and a system could let it stay uninterpreted, or have some universal fallback like converting it to a string.

What should stay out of scope

To keep this format from getting overcomplicated, I'd recommend excluding some things:

• any circular object graphs
• any code-like objects including Regex()

And for obvious security reasons, the parsing library should not just try to parse unknown types by matching classes in the current environment by name and hoping for the best.

Validation

Any such format can have schemas defined, with type and other validations. The only nontrivial part is handling any types that are coerced. For example, if in your Python reader Symbol("foo") and "foo" are going to end up as "foo" anyway, do you want to specify that field as Symbol|string (pre-coercion), or as just string (post-coercion)? I'd lean toward the latter option.

New series - Open Source Adventures

I started a new series last week, Open Source Adventures. It's not terribly focused, it will be some random unrelated Open Source things I wanted to do. The first mini-project is Z3 support for Crystal programming language, but I'll jump to some very different subjects.

Just like with previous two series, you can read it on dev.to or hashnode.

The code for the series, unless it has specific reason to go elsewhere, will go to this repo.

100 Languages Speedrun

I didn't take a long break after finishing the 100-episode daily Electron Adventures series. I already started another one - 100-episode daily "100 Languages Speedrun" series, where I'm trying out a new programming language every day.

It's been going for about a week now, and it's available:

• on dev.to
• on hashnode

I already explained the goals of the series in the first episode, so I'll just repost it below. Enjoy the series!

Time to start a 100 programming languages speedrun. Every day or so, I'll be posting about a different programming language. Not just doing 100 fizzbuzzes, but trying out something that's interesting about each language.

But that's not all, some of the programming languages I will create for purpose of this series. So if you follow along, you'll see not just a lot of different programming languages, but you might also learn a thing or two about how to create your own.

I won't be shy about my opinions, and I might be even exaggerating a bit. Feel free to share your thoughts in the comments.

Episodes will all be independent. Target audience is people who know programming, but don't know a 100 difference languages, so I'll often use some less idiomatic ways of doing things if I think it's clearer for such reader, or if it lets me showcase specific language feature better. For languages where it's not enforced, I'll mostly stick to best-practice cross-language code formatting (2 spaces indentation, double quoted strings, no semicolons etc.), even if that language generally uses something else.

Electron Adventures 100-post series is finished

The Electron Adventures series I've been writing is now over.

The series is available on two platforms, with the same content:

• dev.to
• hashnode

If you want to read the conclusions, episode 99 (dev.to, hashnode) summarizes the technical aspects of the series, and episode 100 (dev.to, hashnode) talks about what it was like to do daily blogging. As it's already all there, I won't be repeating it here.

Or start from the first, or just check whichever episodes look interesting. Some subjects continue over a bunch of episodes, but there's plenty of fresh starts on the way, so you definitely don't need to read it 1 to 100.

I plan to do some more similar series in the future, they'll definitely be announced here, and for now I plan to double-post any such content to dev.to and hashnode, so feel free to follow me on whichever one's more convenient.

It's also possible that I might do a few bonus episodes beyond the 100 someday, as there's a few subjects I couldn't cover for various reasons.

Electron Adventures 50 episodes so far

Two months ago I got an idea of starting a small Electron coding project. And lacking any kind of moderation, I decided to just post daily coding episodes for like a 100 days. I'm halfway there.

There were definitely some technical issues early on, but I got that out of the way.

Since then I blogged at a rate of about 1 post a day - either creating or updating a small Electron program, and then doing a short writeup about it, with code samples and discussion of issues encountered along the way.

When I started, I had a vague idea of where I wanted to head:

• I wanted to try out new code blogging platforms.
• I wanted to collaborate with people. It's something I used to do a lot before the pandemic, but had too few opportunities since.
• I wanted to figure out how I can code Electron in something that's not Javascript - Ruby, Python, basically anything whatsoever. Either purely non-JS, or in some kind of hybrid mode (JS frontend, non-JS backend). All other languages desperately need a good UI system, and I thought this might be worth investigating.
• I wanted to be able to create Windows UIs for my Total War modding tools (and potentially Paradox modding tools too). I did them before with JRuby + Java-based toolkits, but none of that works very well.
• mc more or less broke after software upgrade, and it freezes if I do anything funny, so I wanted to investigate how I can make my own Orthodox File Manager in Electron
• and I might get some Electron and Svelte practice, as I don't use them too often

So far the adventure led me mostly somewhere else:

• I did a few coding sessions with Amanda Cavallaro, but it's actually quite difficult to get someone to join such a big ongoing project for just an episode or two.
• I started coding a file manager. Blogging about every tiny commit is fun, but it takes much longer to blog than to code, so I doubt I'll get even MVP this way.
• I did some coding with Javascript frameworks I don't normally use like Vue or Marko; but zero with non-JS languages so far
• I didn't even try to connect anything with my Total War modding tools
• I definitely got that Electron and Svelte coding practice.

This is fine.

The series is available on two platforms, with the same content:

• dev.to
• hashnode

Each post has about 71 views total (60 views on dev.to and 11 views on hashnode).

Those numbers feel very low to me, as back in the good days a typical post on my blog would get thousands of views, occasionally tens of thousands. My most read post had 170k views.

I also don't know how many views new vs old posts get, so if people would read that for years and it would add up, or if they'd just fade away being barely searchable.

I'm also not sure how much value this series even has to the readers. A 100 post series is not something people really do, ever. Am I expecting people to read from start to finish? To pick it halfway? To just read random a post or two?

A number of the posts are reasonably self-contained, but especially the ones about building the file manager sort of assume some familiarity with existing code. I also didn't really make it obvious which episodes are self-contained and which aren't.

The codebase with each episode's code also has very little interest so far, at 2 stars and zero-activity 3 forks.

I'd probably still be coding something like this anyway, but the main point of writing is having some readers, so if people aren't very interested in this kind of content, I guess I could stop, or do something different.

I'll continue for the rest of the 100 episodes and then write another post about the whole experience.

Oh and I was contacted by multiple different people, who wanted me to write various educational content. Unfortunately I don't really have time for that.

Don't use codepen

Let's talk about websites you shouldn't use  - specifically codepen.

Back in the pre-pandemic days when I was helping people learn frontend programming, I used it a lot for showing various things. So I have 177 such "pens" there, where project is basically three files (HTML+CSS+JS).

Anyway, the why you shouldn't use it part. Codepen deliberately decided to block any way to export your data. Even getting list of your pens is not really possible without logging to Chrome and doing some console script loop. All automated access is deliberately blocked.

The only way to export is to go to 177 pens and click on a lot of buttons to get each individual zip file, and they did their best to block any way to automate this process.

The only way you can get your data - which they don't advertise in any way - is to send them GDPR data export request.

It comes in a fairly annoying format of one big csv file. I wrote a script to convert that data into a more usable form.

Being able to export your data is basic human right on the Internet, and deliberately making it difficult is a reason why you should not use websites like codepen.

Updates Hypocrisy

Tech companies just love forcing updates upon regular users. The idea that someone, somewhere, might be using a version of their software that's a few months old, stops them from sleeping well at night.

This goes all the way from operating systems through big applications all the way down to the tiniest utilities. They will force that update down user's throat, and the most freedom they allow users is to press a 24h snooze button. Some like Microsoft will not even bother asking, and will just reboot user's computer in the middle of whatever they were doing.

And it's not like they make any guarantees about it - if updates break things - and they absolutely will do that - there's usually no way to roll back, and you must be living on a different planet if you imagine you'll get any tech support whatsoever.

So if updates are so important, you'd think at least tech companies would be updating things automatically themselves? Nothing could be further from the truth. They built entire systems like npm's package-lock.json and its equivalents for literally every other programming environment to prevent any updates forever.

Even the idea of operating system updating some shared library dependency is too much, and nowadays everyone bundles all dependency libraries with every application, builds fully static binary, or just puts them in some sort of a fully no-update virtual machine like Docker container.

And it's not just minor packages - tech companies will happily run Python 2 or Java 8 or Debian "stable" a decade after release of their official successors.

So for all that task about importance of updates, this only seems to apply when their costs are borne by someone else.

I believe what tech companies do, not what they say, and I therefore believe that forcing users to update their machines, either by automated updates, or by endless popups without a No button, should be literally illegal. We don't allow manufacturers of physical things to invade your home to "update" your microwave or a book, why should we allow software manufacturers to invade our computers?

Total War UI layout to XML converter

My most recent coding project was decoding UI layout files for all 10 Total War games from Empire to Three Kingdoms and writing converter that translates them to XML and back.

Here's a quick writeup of what I did, and how that went.

UI Layout files

Layout in the games are controlled by UI Layout files. They all helpfully start with a version number header - currently from Version025 to Version129. After that follows top level UI element, and within are nested children UI elements and many other things like UI states, transitions, events, and so on.

Basic building blocks

Basic building blocks of the format were fairly easy to understand, mainly:

• booleans as 00 or 01
• integers as int32
• floats as float32
• colors as BGRA32 (that is - one byte per component, in this order)
• ASCII strings as int16 character count, followed by that many characters
• Unicode strings as int16 character count, followed by that many UTF16 characters
• various data structures had their fields in specific order, without any headers, or delimiters
• for arrays of data structures there was generally int32 element count, then followed by each element in succession, without any headers or delimiters

There were also a few other patterns used less often, like:

• optional fields - either 01 followed by some data structure, or just a 00
• 128-bit uuids (weirdly no specific version, but still market as a uuid in variant bits)
• occasional int8s and int16s
• arrays of elements repeating until some special value like events_end
• 2D arrays of elements prefixed by xsize and ysize
• and so on

Manual decoding with hex editor

Most formats are quite easy to decode with a hex editor. This one wasn't - there were far too many versions, no data structure headers, no separators between data structures, and as pretty much everything was optional, so there were huge blocks of zeroes.

For example a block of 20 zero bytes could be any of:

• 20 booleans false
• 5 floats 0.0
• 5 ints 0
• 10 empty ASCII strings
• 10 empty Unicode strings
• 5 empty nested arrays of some child elements
• or most likely some combinations of all of them

And there were such huge blocks of zeroes everywhere.

Decoding it without tool assist would be just too difficult, especially doing it over and over for every single version.

Original converter

Once upon a time alpaca wrote a Python converter for Napoleon Total War (second game on the engine). I inherited that, and extended it to backwards to Empire and forwards Shogun 2.

Even with all the fixes it had only maybe 90% support for those three games.

The most obvious approach would be fixing remaining issues and extending it further.

Unfortunately that would be very difficult approach.

Internal Representation Pattern

The converter was based on principle of Internal Representation. Every structure has a class. That class  basically has five methods:

• initialize empty data structure with default values
• read from binary file
• write to XML
• read from XML
• write to binary file

This works well enough when there's one version of every structure, and it's fully understood. Unfortunately we have 62 different versions (some numbers between 25 and 129 were skipped), and we have very limited idea how things are represented.

Old converter tried to ignore many of those issues. For example writing to XML was just one hardcoded template string per data structure, so if layout file's version lacked some fields, it would just write default values anyway. Then on converting back it would read them and throw them away. This specific issue was partly limitation of Python, which is bad at DSLs, and this XML output really wanted a DSL.

A bigger problem was that if it didn't work for any reason, I got nothing. I'd get some "reading past end of file" error without any context whatsoever, and actual point where parsing derailed was located long before that crash.

Data gathering

Before I even started, I took latest versions of all 10 Total War games using current engine, extracted all UI layout files and put them as test set.

Analysis tool

Then I wrote analysis tool. The formats were really complicated, but there were some obvious things in them. Especially strings. Basically the analysis tool went over the file and identified every ASCII or Unicode string. Then it printed any undecoded data in nice ASCII + hex format.

That was a good starting point, but there was something I could do next. Not only I could see the strings, it was really easy to guess which string meant what. A string with font name was always followed by some ints controlling text display. A string with shader names by shader variables. Strings with image names were used in a few ways, but some simple heuristics could guess which were they.

So I soon had listings along these lines:

000129-000147 FontNameBlock "Ingame 12, Normal"
000148-000151 LineHeightBlock 2
000152-000155 FontLeadingBlock 1
000156-000159 FontTrailingBlock 255
000160-000174 DataBlock
  ...............  00 00 00 00 00 00 00 00 00 00 00 00 01 00 00
000175-000185 ShaderNameBlock "normal_t0"
000186-000189 ShaderVariableBlock 0.0 (0)
000190-000193 ShaderVariableBlock 0.0 (0)
000194-000197 ShaderVariableBlock 0.0 (0)
000198-000201 ShaderVariableBlock 0.0 (0)
000202-000270 DataBlock
  ........0....... 00 00 00 00 01 00 00 00 30 12 00 09 00 00 00 00
  ................ 00 00 00 00 00 05 00 00 00 04 00 00 bb ff be ff
  ................ 00 00 00 00 00 00 00 01 01 00 00 00 00 00 00 00
  ................ 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  .....            00 00 00 00 00
000271-000282 EventListBlock []
000283-000294 DataBlock
  ........ .<.     00 00 00 00 01 00 00 00 20 b3 3c 0b
000295-000314 StringBlock "government_screens"
000315-000346 DataBlock
  H............... 48 01 00 00 8e 00 00 00 01 01 00 01 00 00 00 00
  ................ 00 ff ff ff ff 00 00 00 00 05 00 00 00 00 00 00
000347-000421 ImageListBlock 1 elements:
  000351-000421 ImageBlockGen1 id=163829448 xsize=256 ysize=256 path="data\\UI\\Campaign UI\\Skins\\fill 2 leather 256 tile.tga" unknown=4294967295
000422-000433 DataBlock
  ............     00 00 00 00 00 00 00 00 01 00 00 00
000434-000437 StateIDBlock 162986096
000438-000447 StateNameBlock "NewState"
000448-000451 XSizeBlock 624
000452-000455 YSizeBlock 720
000456-000484 DataBlock
  ................ 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00
  .............    01 00 00 00 01 00 00 00 00 00 00 00 00
000485-000503 FontNameBlock "Ingame 12, Normal"
000504-000507 LineHeightBlock 2
000508-000511 FontLeadingBlock 1
000512-000515 FontTrailingBlock -16777216
000516-000530 DataBlock
  ...............  00 00 00 00 00 00 00 00 00 00 00 00 01 00 00
000531-000541 ShaderNameBlock "normal_t0"
000542-000545 ShaderVariableBlock 0.0 (0)
000546-000549 ShaderVariableBlock 0.0 (0)
000550-000553 ShaderVariableBlock 0.0 (0)
000554-000557 ShaderVariableBlock 0.0 (0)
000558-000565 DataBlock
  ........         00 00 00 00 01 00 00 00
000566-000589 ImageUseBlock id=163829448 xofs=0 yofs=0 xsize=624 ysize=720 bgra=bgra(255,255,255,255)
000590-000626 DataBlock
  ................ 01 00 00 00 00 00 00 01 01 00 00 00 00 00 00 00
  ................ 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  .....            00 00 00 00 00
000627-000693 EventListBlock ["OnUpdatePulse", "OnUpdatePulse", "OnDock", "DockHudRelative"]
000694-000705 DataBlock
  ............     0a 00 00 00 0e 00 00 00 e8 db b8 09

And you can probably already notice huge blocks of zeros I mentioned before - even after some zeros are not shown as decoded from context.

Direct Conversion Pattern

Now that I wasn't going completely blindly, I started writing a converter. In Ruby, as there was a lot of DSLing to do. But mostly it was based on a completely different principle - Direct Conversion.

Direct Conversion doesn't bother with any classes, or internal representations. It has methods such as (not actual code, just the general idea):

def convert_int
  value = get(4).unpack1("V")
  puts "<i>#{ value }</i>"
end
  
def convert_string
  size = get(2).unpack1("v")
  str = get(size)
  puts "<s>#{ str.xml_escape }</s>"
end

def convert_color
  b, g, r, a = get(4).unpack("CCCC")
  puts "<color>"
  puts "  <byte>#{b}</byte><!-- blue -->"
  puts "  <byte>#{g}</byte><!-- green -->"
  puts "  <byte>#{r}</byte><!-- red -->"
  puts "  <byte>#{a}</byte><!-- alpha -->"
  puts "</color>"
end

But bigger methods can be composed from smaller ones (also not actual code):

def output(str, comment=nil)
  print "  " * indent
  print str
  print "<!-- #{comment} -->" if comment
  print "\n"
end

def convert_int(comment=nil)
  output "<i>#{ get_int }</i>", comment
end

def convert_color
  tag "color" do
    convert_byte "blue"
    convert_byte "green"
    convert_byte "red"
    convert_byte "alpha"
  end
end

Advantages of Direct Conversion

Nice thing about this is that conversion back doesn't need to have any idea whatsoever what tags like color even are - other that most basic data types like strings, ints, floats, and booleans, the converter from XML back to binary needs nearly zero awareness of what those formats are.

So instead of describing every data structure 5 times, we do it just once. And any version specific logic can be handled by a single if @version >= 74 or such.

But there's more. Since we never need to construct any internal representation, if conversion crashes, the converter will give us full context of the error!

  <model>
    <s>composite_scene/porthole/troy_advisor_test.csc</s><!-- mesh path? -->
    <s>standard_advisor</s><!-- mesh name? -->
    <!-- some model data or anim header or sth -->
    <data size="1">
      01
    </data>
    <i>0</i><!-- 00:00:00:00 --><!-- anim count or something? -->
    <s></s><!-- anim name? -->
    <s></s><!-- anim path? -->
    <!-- rest of anim stuff or sth -->
    <data size="4">
      00 80 3f 00
    </data>
    <!-- 2900 - end of model data -->
  </model>
</models>
<no /><!-- end of uientry flag 5B? -->
<no /><!-- end of uientry flag 6B? -->
<error msg="Invalid boolean value: got 63" version="121">
  Data before fail:
  ne/porthole/troy 6e 65 2f 70 6f 72 74 68 6f 6c 65 2f 74 72 6f 79
  _advisor_test.cs 5f 61 64 76 69 73 6f 72 5f 74 65 73 74 2e 63 73
  c..standard_advi 63 10 00 73 74 61 6e 64 61 72 64 5f 61 64 76 69
  sor...........?. 73 6f 72 01 00 00 00 00 00 00 00 00 00 80 3f 00
  Data from fail 2900:
  ..?...?.....9..p 00 00 3f 00 00 00 3f 00 00 98 d1 bd 39 10 00 70
  ortrait_minspec. 6f 72 74 72 61 69 74 5f 6d 69 6e 73 70 65 63 00
  ................ 00 00 00 00 00 00 00 00 00 00 00 00 00 01 01 00
  ................ 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00

Then all I need to do is look back from point of the crash to the last definitely correctly decoded part (in this case those two strings look perfectly fine). Then find where is the first definitely incorrectly decoded part (in this case 00 80 3f is clearly last 3 bytes of a float, so it was off by one at this point already).

Then I can adjust that specific data structure's method. I don't even need to guess what that extra data is. If I see five zeroes I don't have decoding for, I just tell the converter to expect five zero bytes.

Then if some other file has non-zeros at that position, I'll get nice exception like "Zero data expected, got 05 00 00 00 00", then I can pretty clearly see that first four bytes are an int32 - and the last remaining one is likely a boolean (but I'd still leave is as undecoded zero for now).

Debug mode

At some point I implemented a small modification to direct conversion process. There's debug flag to control printing of various extra information like structure offsets, hex values of ints and floats and so on.

Converter first converts binary to XML with debug flag off. If that process crashes - it turns debug flag on, and starts all over. This way normal XML isn't polluted by too much extra information useful only for debugging the converter, but in case of crash I get tons of extra information.

First three games

The first three games were easy enough. I already had a mostly working decoder, so I used it as a starting point, and used procedure described here to fix any issues.

Initially I thought about backporting fixes to the old converter, but I quickly gave up on this idea when I discovered just how extensive the changes would need to be.

In any case I got converter working far better than the old one without any major difficulty.

Next seven games

This is where my plan run into first problems. Starting from a working converter for version X and adding support for version X+1 is easy:

• run conversion anyway, ignoring that version is wrong
• identify where exactly it crashes (based on  <error> tags and my analysis tool)
• try to fix those crashes, gated by some if @version >= x+1 checks

Unfortunately first three games used versions 25 to 54, then next seven games used version 74 to 129. So I had a 20 version gap with nothing in between, and really I looked like I'd need to decode from pretty much from scratch.

Cpecific's decoder

I'm sure I'd be able to figure out the decoding, but I found unexpected help. It turns out Cpecific wrote a PHP-based UI layout decoder. It doesn't actually convert anything - just prints JSON-style output describing contents of various UI layout files.

I tried to run it on a bunch of files, and it seemed to have 80%-ish support for newer 7 games, similar to how well old decoder supported the older 3 games.

The main weakness of Cpecific's decoder is that it doesn't actually convert anything - so you're expected to do hex editing, and then check in the decoder that results are what you expected. Not exactly an ideal workflow, but it super beats hex editing blind.

I also couldn't fully trust its decoding, and it crashed on many files, but it was definitely a huge help at crossing the gap between Version054 and Version074, and once I crossed it, it was easy going to do one version at a time.

I also used it to annotate some fields with comments on what they could likely be.

I don't plan to do any further development of old converter, but in case Cpecific wants to continue with his, at some point I should write down a list of issues I found and their fixes.

Warhammer III

A new Total War game is coming out soon, so the converter will likely need an update. I don't expect this to be difficult - Rome 2 was the last time they did major format update.

Challenges for November 2019 SecTalks London

October 2019 London SecTalks CTF was jointly won by me and Laurence, so we wrote challenges for November together.

There was a total of 9 challenges, 5 written by me and 4 by Laurence. It was a bit much, as one of the challenges got zero solves until after the event. In spite of very generous hints given halfway through the event.

Challenge files and code used to generate them are available on github.

There are no answers below, but some serious hints which might make it too easy.

For previous rounds, see posts about September 2017, November 2017, May 2018, July 2018, October 2018, February 2019, April 2019, and July 2019 CTFs.

Rockstar (5 points)

The puzzle was a rock song / program in Rockstar programming language. Just running the program would print ASCII codes, which could be easily turned into a flag.

Think Outside The Box (10 points)

The challenge is a BMP image file with flag hidden by incorrect image size. Changing the header reveals the image.

Dots (15 points)

This was Unicode Braille, but dots and no-dots were simply 1s and 0s to be decoded, ignoring any "Braille" structure. To my surprise nobody solved it even when given basically full explanations of what they're supposed to do.

CRC32 MAC (20 points)

A signed cookie server, which uses CRC32 with secret key as MAC. CRC32 is extremely unsuitable in this role.

Decryption Engine (25 points)

A small Python program which uses default password based on current date, and encoded file. Assuming password wasn't passed, the space of possible keys is tiny.

Salted MAC (30 points)

A signed cookie server, which uses SipHash with secret key, and then salt just to be extra sure.

In principle this should be difficult to break. The problem is a trivial implementation mistake.

PHPain (40 points)

It was a separately hosted PHP application vulnerable to recently published CVE.

Poly-DIY MAC (50 points)

Another signed cookie server, using a simplified variant of Poly1305. The variant is reasonable, however the server isn't very careful about nonce handling, and such schemes tend to fail catastrophically when nonce is reused.

GeoCaptcha (60 points)

It's a web "Captcha" which asks the user to locate 20 major cities on the map, with any mistake failing the Captcha.

You can play it here (for a while).

It's worth so many points mostly due to its very high entertainment value, not due to difficulty.

I was curious how people would approach it - only 2 players solved it with code (as robots), 6 solved it with tedious manual process (as humans). So I guess it works as a Captcha.

Technologies that inspired me

Nearly every day I run into another language, framework, or program. Most of them are just plain awful, and the ones which aren't are usually nothing special.

Every now and then I run into a true gem. Here's a list of such great technologies, in approximately the order I found them.

Perl

Perl was absolutely amazing compared with every language which came before it. Other languages were motivated by what worked for computers, or by some silly mathematical theory, or sometimes like with Java by outright hostility to programmers.

Perl tried to be as much like a natural language as possible. If what you wrote made some kind of sense, it was going to try to run it. And most of the time it worked.

Unfortunately Perl supported too much weird stuff, and it caused the language to collapse under its own weight, but the most influential languages of the 1990s like Ruby, Javascript, and PHP were very heavily inspired by Perl. And as next generation of languages was very heavily inspired by those, the legacy of Perl lives on.

Perl tried to double down on its philosophy with Perl 6, but without first dropping failed ideas it was never really going to work.

Regular Expressions

Text is nearly as common data type as numbers, but most pre-Perl languages didn't treat text data as first class. In many old languages text is literally an array of small numbers, or such other nonsense. Even today Erlang / Elixir do literally that - "Hello" is the same as [72, 101, 108, 108, 111].

Regular expressions are to text what arithmetic is to numbers, and let you extract or modify text with tiny amount of highly readable and highly performant code. They have weird reputation of unreadability, but one line of dense regular expression is infinitely easier to understand than 100 lines of text processing code it replaces.

It's key skill for every programmer, no matter what's the domain. If you don't know regular expressions, drop everything and do some regexp crosswords.

There used to be different variants of regular expressions, but these days basically every language uses some variant of "Perl-Compatible Regular Expressions". Except Unix grep command, which still uses stupid 1970s regexps for some crazy reason.

Wikis

Wiki was a revolutionary idea - just put a bunch of linked pages online, and let anyone who wants edit them. It sounds like a recipe for disaster, but it worked remarkably well. Wikipedia was the greatest success story of wikis, but there are all kinds of wikis for all kinds of content. And as a general rule, they're a lot more reliable than non-editable sites.

Ruby

Ruby took everything that made Perl great, everything that made Lisp great, everything that made Smalltalk great, then it threw away their bad parts. And then it added more magic on top.

It is the high point of programming langugae design, unmatched up to this day. Implementation issues aside, I'd rather use Ruby as it existed 20 years ago than any language that came out since. It's just baffling how far ahead of time it was.

Of course it could be even better, but that's true of anything.

I hope someday a language comes that's better than Ruby, but so far it doesn't look like anyone's even really trying. The best anyone's trying to do is to make their languages as close to Ruby as possible given various constraints.

Ruby on Rails

Ruby on Rails is another miracle technology. When it came out, it was completely unmatched as the best way to write web backends. Today, after the web changed beyond recognition, it's still completely unmatched as the best way to write web backends. It's rare for any technology to have such staying power.

To always stay on top, Rails evolved very aggressively, and Rails upgrades on large codebases are notoriously difficult, but then again, the choice is between upgrading Rails 4 to Rails 5, or writing your code in some shitty framework that's going to die, and fully rewriting it into another shitty framework that also won't last.

It's quite baffling that so far no real competitor to Rails emerged. The best everyone else can do is to copy as much as they can from Rails, and even that is usually really poor.

The main reason is that Ruby is completely unmatched at making DSLs than any other language, and Rails is very DSL-heavy. But other aspects of Rails success like everything working seamlessly out of the box (while being individually replaceable) without needing to duct tape a bunch of disparate components should be replicable, and yet nobody even tries.

Why so many frameworks come out without such basic things like comprehensive testing system, or database migration system, or sensible security defaults? I suspect their authors just don't get Rails, and people who get Rails use Rails.

TextMate

Once upon a time, there were thousands of editors and they were all 💩. It was a running joke that every programmer writes their own editor, and their own IRC client (and later their own Javascript framework).

And just as with infinite monkeys randomly typing on infinite number of keyboards, one of such attempt miraculously produced TextMate - the first editor which was not 💩.

Multiple cursors, command palette, file palette, extension system, nested file types, integration with modern UIs, sensible keybindings making use of what's on modern keyboards - TextMate was far ahead of everything else.

TextMate itself didn't last, but Sublime Text, Atom, VS Code, and some others are essentially TextMate clones, and they completely dominate the editor world.

The only non-TextMate editors still in use are people using IDEs like IntelliJ or Android Studio due to being forced to use programmer-hostile languages like Java. And people who use Vim because they still haven't figured out how to quit it.

jQuery

Javascript used to be awful beyond belief. Well, it's still not great, but back then doing even the most trivial dynamic interactions, like changing some class based on an user action was a page of code, and then another page of different code if you wanted to support more than one browser.

Then jQuery came, and with a few $("selector").action() you could write complex UI interactions with very little code. It was even bigger difference when it came to jQuery AJAX vs browser APIs for AJAX. The gap was as enormous as between regular expressions and manipulating text by shuffling arrays of characters around.

Gradually browser APIs became more compatible, more expressive, and less buggy, they added a lot of features to close the gap with jQuery, so now it's sort of viable to write short programs directly with browser APIs. And while jQuery is amazing at progressively enhancing website functionality, it's a stretch if you want to make Single Page Applications.

So is it the end of jQuery era? Not even close! jQuery is still vastly more popular than all frontend frameworks put together - 80% of top million websites use jQuery, while all the hipster frameworks put together are still just a rounding error. Currently most popular React at 1.6% is behind 20 different jQuery plugins.

And beyond jQuery itself, D3 is basically jQuery for data visualization. Most acceptance testing frameworks from Capybara to Cypress use jQuery-style for writing tests. Single Page Applications are useful, but most of the Web is better off not being an SPA, and good jQuery skills will never go to waste.

It might be unpopular opinion now, but I feel the best way people should learn frontend is still be HTML, CSS, jQuery, then ES6+ Javascript, and only then some SPA framework like React.

Amazon Web Services

I remember bad the old days when setting up any server required making a request to a grumpy sysadmin (or even worse - a grumpy DBA) and waiting potentially weeks for them to find some time for it. Even more fun if that required more physical hardware, and budget requests for those. Meanwhile, there was nothing you could do.

Then came Amazon and decided that making billions selling everything wasn't enough so it opened a small side business of getting rid of grumpy sysadmins. Want to start a new server? A few clicks and it's done by the time your coffee finished brewing. And they kept adding new services beyond the initial EC2 and S3, to simplify your life more and more.

I feels silly to even mention how great it is, as AWS and other Clouds won so utterly that most people barely remember the pre-Cloud days, and those who do would rather forget them.

pry

I didn't love debuggers, and I usually employed print-based debugging as my first choice, strace and friends as my second choice, and only used interactive debuggers if all else failed.

pry was something else. A hybrid of a debugger and REPL, with full power of Ruby metaprogramming and introspection, it was on a completely different level than anything I've used before.

Adding binding.pry became my first not the last resort. And then there's pry-rescue and other plugins to make things even more awesome.

I'm still annoyed that pry isn't just part of Ruby yet, and that bundler blocks pry unless you explicitly add it to every single Gemfile just in case.
Sure there's finally binding.irb, but irb is not even close to pry. Seriously, get your act together guys and get pry into Ruby standard library.

MongoDB document model

Relational databases are just a bunch of damn spreadsheets, and forcing your program's object oriented (or otherwise richly structured) data into spreadsheets for storage is invariably painful. Object-Relational Mapping tries to make it bearable, with limited degree of success.

There were many attempts to create alternative database models, but they were either specific to one programming language, or used something like XML which was just as awkward as spreadsheets.

Then came MongoDB, made everything JSON, and it was glorious. Oh and depending on your application it could easily be orders of magnitude faster than relational databases.

Unfortunately while MongoDB JSON-based document model was amazing, and its performance was generally outstanding, everything else about MongoDB was less so. Doing even relatively simple queries in MongoDB is stupidly painful compared with SQL, so you end up doing it all in application. Its performance-first defaults got a lot of people into trouble as well.

I'm surprised that even a decade later nobody managed to build anything that united JSON-based (as opposed to spreadsheet-based) document model with some rich query language that can at least match SQL.

Spreadsheet-based databases nowadays support JSON data a little bit, NoSQL databases nowadays have a bit richer query languages, but the gap between those two is still enormous.

Capybara

TDD lived in this weird state, where backend code was decently tested, but frontend testing was minimal and mostly manual.

Even today most tests for JS programs are fairly low value efforts, like "component renders without crashing" tests and snapshot tests.

Capybara was the first practical way to do end to end tests of web apps with dynamic frontend and dynamic backend as a single system.

Nothing better than Capybara was created so far. There's a bunch of fairly poor Capybara clones in other languages, which might be just about good enough. Cypress might be developing into the first real alternative, but it's very frontend-based and integration with dynamic backends is its secondary consideration.

Continuous Integration servers

I wasted too much time running tests manually, or not bothering to run them because the change I was doing was trivial, and the tests took too long.

There is an answer to that - CI servers. Most companies have them. But people coding for themselves, or coding Open Source software are still usually not using them.

Semaphore CI and many other sites let you easily setup CI for free for your Open Source project, and you should do that.

Chrome Developer Tools

The story started in the olden days with Firebug, which provided just console.log and a few other very simple functions. Or possibly even with the View Source button (which some people wanted to delete from browsers to "protect the source code", #truestory).

Nowadays every browser comes with amazing debugging and development tools, and they keep getting better every year.

It's one of few places where I feel Javascript has a big lead over other languages. pry might be as good as possible on terminal, but it's still limited to a terminal. Chrome Developer Tools have full power of the browser UI, and they fully use it.

Dropbox

I have multiple computers. Also multiple phones and tablets. Sharing content between them used to be such a mess - connecting and disconnecting external disks, or manually copying files by cables or rsync. Dropbox made it seem that the same file is magically present in every location. It just works, with very few issues.

Unfortunately Dropbox for phones is quite bad, so if anyone solves this better, I'm interested.

Weirdly many Dropbox alternatives like Google Drive are simply horrendous.

Slack

Just like everyone was writing their editor, everyone was writig their IRC client. Or some other chat system which was basically IRC clone.

They were all solving the wrong problem. The issue wasn't that IRC needed better UI, the issue was that IRC was based on fundamentally wrong assumptions (broadcast to whoever happens to be connected at time) that made it completely unsalvageable.

Slack fixed those assumptions, and it completely revolutionized workplace communication, not just in tech.

All post-Slack chat systems like Discord use basically Slack model.

Electron

Electron is the first time in history of computing where we have cross-platform desktop apps with decent UI. There's performance cost, as Electron apps bundle whole browser, and are infamous for their memory use, but there's no alternative, so let's just throw more hardware at the problem, and restart every now and then if it gets too bad.

There were a few earlier attempts like with Java, but the UX was just attrocious. I cringe whenever I have to use some Java-based GUI software.

Z3

Normal programming answers questions like x = 2+2 perfectly fine, but it is shockingly bad at dealing with questions like 2+x = 4. Z3 is an amazing system for answering just such questions.

Solving sudoku, hacking CTFs, proving hardware design? Z3 has you covered.

I liked it so much that I wrote a Ruby gem for it. It's much nicer than official APIs.

Docker

I initially dismissed Docker as just another VM. It turns out that such lightweight micro-VMs have a lot more uses, and just being able to install and run a lot of images neatly solves a lot of issues like CI, software with weird dependency, security isolation and so on.

Imba

Javascript used to have a million frameworks. They were generally called "MVC frameworks" back then, a name you don't hear much anymore, as it turns out MVC is a poor fit to what they were trying to do.

At certain point very-much-non-MVC React pretty much won, by being decent at solving smaller problem (views), not even trying to deal with the hard problem (model and network layers / state management), and so having a lot easier learning curve and fewer gotchas. Acceptable performance and being pushed hard by Facebook also helped.

I even liked React before I found Imba. Imba however, is on a completely different level. It's the least known technology on this list, so I'll need to write a separate post about it, but you'll get orders of magnitude higher performance at fraction of code size, all while being a lot simpler than React.

It's a tiny project with very few development resources, so it feels like it really needs someone to push it to some sort of Imba 2 fully embracing post-ES6 world. But even in its current state, it's years ahead of any alternative.

Challenges for July 2019 SecTalks London

Another CTF, another victory. I won the June 2019 London SecTalks CTF and it was up to me to write challenges for July.

There were 12 challenges, theme of the challenges being Hacker-Archeology. It turns out that was too much, as only 10/12 challenges got even one solve, and nobody got more than 4. Even with very generous hint drops during the event.

Challenge files and code used to generate them are available on github.

There are no answers below, but some serious hints which might make it too easy.

For previous rounds, see posts about September 2017, November 2017, May 2018, July 2018, October 2018, February 2019, and April 2019 CTFs.

SHAR (5 points)

Self-extracting Shell archive nested 8 levels deep. It only works on Linux, OSX shell can't extract Linux-created SHAR files, showing what a dumb format this is.

XBM (10 points)

A weird way to encode image into C headers. While totally obsolete, a lot of tools still support it.

Maya (15 points)

An image with a sequence of Maya numerals, each encoding ASCII symbols. It seems people were confused by the fact that multi-digit Maya numerals are stacked vertically.

PCX (20 points)

PCX file with flag on it, but both foreground and background colors having same RGB color, so color palette would need to be adjusted to actually see it.

ECB (25 points)

A signed cookie server which would only sign cookies without admin=yes, and it would only give you the flag if you sent it signed cookie with admin=yes.

It's a classic cryptography attack on ECB mode, rearranging blocks within or between cookies.

SED (30 points)

SED is an obsolete programming language for text stream processing, and the challenge was a simple flag validation script which was just a sequence of regexp replace rules.

Nobody noticed that, but that SED script was also totally valid Perl 5 script.

CBC (35 points)

A signed cookie server which would only sign cookies without admin=yes, and it would only give you the flag if you sent it signed cookie with admin=yes.

It's a classic bit flipping attack on CBC.

MD4 (40 points)

A signed cookie server which would only sign cookies without admin=yes, and it would only give you the flag if you sent it signed cookie with admin=yes.

It's a classic length extension attack.

Midi (45 points)

A midi file with flag encoded in Morse code, played on an Ocarina instrument suggestively named "Morse Ocarina".

It could be done either by hand, or by converting note lengths in Midi format to dots and dashes. I think everybody ended up doing it by hand.

DOS (50 points)

A very small COM file flag validator. It was extremely simple, but a lot of tooling like Ghidra has trouble with COM files, as they're too old to be relevant.

It got zero solves, which was fairly surprising, as flag validator is really simple:

# Initialize counter in BX
  mov bx, 0xd7ab

# Get ASCII code of next character into AL
  mov ah, 0x1
  int 0x21

# Add AX to the counter, check if correct
  add bx, ax
  cmp bx, 0xd911
  jnz near 0x19d

# Repeat for next character

Differences between constants being compared are 256 + ASCII code of each letter (first being 0x166 or 256 + "f").

Hieroglyphs (55 points)

Monoalphabetic cipher encoded into Egyptian hieroglyphs. The text was very long English text with spaces removed.

It's really simple for anyone who ever did monoalphabetic cipher breaking through statistical analysis, so that many points mostly being potentially quite time consuming, but it didn't even take people that much time.

Perl 6 (60 points)

In remote past Perl 6 used to be the language of the future. That future never came.

The challenge is a flag validator in Perl 6 aggressively using many unusual Perl 6 operations. I'm not really surprised that nobody succeeded at solving it.

Challenges for April 2019 SecTalks London

I just keep winning them, and by convention the winner organizes the next one, so I just ran another round of London SecTalks CTF.

There were 10 challenges. The winner got only 6, but each challenge was solved by at least one person during the event. So maybe it was just a bit too much, but it's better to have too many than too few challenges.

Challenge files and code used to generate them are available on github.

There are no answers below, but some serious hints which might make it too easy.

For previous rounds, see posts about September 2017, November 2017, May 2018, July 2018, October 2018, and February 2019 CTFs.

EBCDIC (5 points)

It was a simple challenge for beginners. The flag was encoded using EBCDIC. It could even be done manually.

CSS (10 points)

This was possibly the most original challenge, the flag was encoded in a pure CSS maze, where you'd need to mouseover each correct letter in order to get the challenge.

CSS wasn't too obfuscated, so that was the easiest way to solve it.

CTR (15 points)

A small server which encrypted your message with AES-CTR. There was also encrypted flag.

The problem was that IV was not generated every time, and there's a standard attack for that.

sayflag (20 points)

This was a small MP3 file, encrypted with a XOR key. The file did not have ID3 tag, so there's not enough known plaintext for the usual attack.

The hint suggested one alternative approach: MP3 compression works like JPG, but block artifacts would be fatal, so compressed blocks overlap a bit. To avoid special treatment for the last block, encoders add some silence at the end of the file (also sometimes at the start), so that part usually has very low entropy. No ID3 here.

With this hint it should be fairly easy to adapt the attack.

Rust (25 points)

A simple flag validator, this time written in Rust to make reverse engineering a bit more challenging.

RSA Signatures v1 (30 points)

This was a fun double challenge. The challenge was "textbook RSA" signature server without any padding or hashing. It would verify and sign any "safe" message, and the task was to give it "unsafe" one containing admin=yes.

There were two versions of this challenge, and diffing them revealed some extra checks which were missing from the first version, which could be very easily exploited.

Codage Hexadecimal (35 points)

French plaintext got hex-encoded, but with digits mixed up. It was tiny bit harder than previous English challenge, as there were was a bit of Unicode with accented characters.

from_past_import_flag (40 points)

A program that would generate a flag, but only if you run it in 2005. To avoid being fooled by changing laptop clock it double checks with NTP server, but NTP is completely unsecured, so it's very easy to redirect that traffic to a local NTP server.

Or just attack the file with gdb.

RSA Signatures v2 (45 points)

In the second part of textbook RSA signatures challenge, bounds check issues were fixed, so a proper RSA signature malleability attack was needed.

Chifferment par Substitution (50 points)

Substitution cipher, this time on a French text, and without most punctuation. It's not difficult, it's just fairly time consuming.

Challenges for February 2019 SecTalks London

I ran another round of London SecTalks CTF.

There were 12 regular challenges, and 1 super-hard bonus challenge. Only 10/13 got at least one solve during the event, so maybe difficulty or number of challenges were a bit too high.

Challenge files and code used to generate them available on github.

There are no answers below, but some serious hints which might make it too easy.

For previous rounds, see posts about September 2017, November 2017, May 2018, July 2018, and October 2018 CTFs.

from_future_import_flag (5 points)

It was a simple program which could tell you the flag if you run it in year 2020.

Censored SVG (10 points)

A simple HTML file with SVG image on it. The image contains flag covered with some censoring square. SVG image is base64 encoded for one small layer.

Hello RISC-V (15 points)

A simple binary which will give you the flag if you run it. It only runs on RISC-V, so you'll need to figure out a way to do so.

Crystal (20 points)

Password validator binary compiled from Crystal. Binary contains a lot of extra stuff, but validating function shouldn't look all that strange.

Powerpoint (25 points)

PowerPoint PPTX file encrypted with XOR. Flag is on the slides.

RISC-V Verify (30 points)

RISC-V binary validating password. It might be easier to run static analysis, but you should still understand basics of RISC-V assembly. Or you could run gdb on RISC-V emulator.

OCaml (35 points)

Password validator binary compiled from OCaml. OCaml uses unusual integer representation, so it's extra layer of confusion.

It had zero solves during the event.

LOLHEX (40 points)

English text encoded by hex encoding, but which of 16 digits is which is scrambled. Simple statistical analysis should do here.

Imba (45 points)

Password validator written in Imba. It will tell you if you get your flag right. It can be solved by static analysis (data driving logic is easy to see), or in-browser, possibly by by attaching DOM breakpoints.

It had zero solves during the event.

Catzip (50 points)

For mandatory meme challenge. It's a zip format made by gzipping something, then turning it into a PNG containing cat emojis. There's two cat emojis, 8 per row, so you can probably guess where it's going.

FPGA (55 points)

A slight variant of challenge from previous time. It's a small circuit, and you need to find inputs which give positive result. This time circuit has NAND, AND, NOR, OR, and XOR gates.

It's highly advisable to use Z3 or similar for it.

Monoalphabetische Chiffrierung (60 points)

German text encoded with monoalphabetic cipher, and then turned into Unicode Fraktur.

It's actually very easy, as punctuation, capitalization, digits etc. were not really scrambled.

LOL64 (100 points)

For super hard bonus challenge, it's English text encoded by Base64 variant, but which of 64 digits is which is scrambled.

Understandably, it had zero solves during the event.

It's definitely solvable in principle, but I'm not really sure how to write such solver. It can be seen as breaking a key of 64 6-bit parts, so 512 bits total. Knowing that it's all ASCII and simple statistical methods get a lot of those bits, but from that point on it would take some creative statistical analysis. Big difficulty is just size of the key, and how a single error in decoding can mess up all following statistical analysis, and cost a lot of time.

I'm sure there are also some ways to circumvent this whole process, and find exact encoded text from some simple statistical properties, but I didn't try it this way as it's less fun.

finder-sort

I was playing a bit with Electron, building a small image viewer, as Xee is weirdly crashy nowadays. A small aspect of any image viewer is sorting images to show them in order. As far as I know there's no programming language with any special support for sorting file names, and default ASCII sort is just atrocious - going cat1.jpg,  cat10.jpg,  cat2.jpg etc.

It's not uncommon to find file manager or image viewer which uses this completely unsuitable sort order.

OSX Finder is notable for very much not doing so, and using human-friendly ordering. Exact details are unfortunately not properly documented.

I created something which works more or less the same, and published it as npm package finder-sort.

There could be some differences, especially for non-ASCII locales. In the end, it's trying to solve the same problem, not necessarily match OSX Finder exactly.

The source code is on github.

About the only interesting thing about it from code point of view is that I used ava testing package this time. Javascript has over 9000 different testing packages, and it's not uncommon for a single program to duct tape together 10+ of them. I'm surprised there's no clear winner yet. Some of the new ones like ava and cypress seem mostly decent.

Webpack boilerplate package for Imba and SCSS

Imba looks extremely interesting - seriously, just check out the code examples on their website.

I wanted to give it a go, there was just one tiny problem - in Javascript world you can't just gem install a few things and run them like that. Nope, painless setup is just a crazy ruby idea that never got much traction anywhere else. In Javascript universe everything always requires painfully complicated setup.

I found some boilerplate example for Imba, but it was broken on so many levels, I had to start pretty much from scratch.

So here's a working webpack boilerplate for Imba with SCSS support. Feel free to fork it into your project.

What's in it:

• Latest Webpack
• Imba
• SCSS (as plain old compile to CSS, intentionally no CSS-in-JS shenanigans)
• CSS normalize to avoid cross browser pain
• standard npm commands for development and production builds.

Everything uses sane 2 space indentation, and tries to avoid doing anything weird.

What's obviously missing is some kind of testing framework, so PRs wanted.

I haven't used it for anything more complicated than just another TODO app yet, so I don't know if there are any issues. Just report them on github.

Thanks to all the brave souls who answered webpack questions on Stack Overflow - somehow I managed to duct tape working boilerplate out of all that.

Challenges for October 2018 SecTalks London

Last month I ran another round of London SecTalks CTF.

There were 10 challenges, and the winner got 9/10 of them during the event (and last one on the following weekend), so difficulty level was about right.

Challenge files and code used to generate them available on github.

There are no answers below, but some serious hints which might make it too easy.

For previous rounds, see posts about September 2017, November 2017, May 2018, and July 2018 CTFs.

Archive (5 points)

A small variant of the 16-level 16-way nested archive, to test for basic Unix scripting skills. This time using RAR.

MonoRSA (10 points)

It's RSA-encrypted message, but it uses only one prime, not two. This is extremely insecure, and can be trivially broken, but you still need to do some math.

BCRYPT (15 points)

Each letter of the flag was encrypted with bcrypt. It's easy to break, as long as you know how bcrypt works, which isn't quite the same as plain hashes, so it was causing a bit of confusion.

RSA RNG (20 points)

It's Debian weak RSA key attack all over. We have target's public key and encrypted message. Also a lot of other people's public keys, all generated using same bad RNG. If two keys share a prime it's easy to break them, even if direct factoring isn't viable.

Python (25 points)

Small bit of reverse engineering - small Python password validator obfuscated with one of online tools for it.

MultiRSA (30 points)

A little known fact about RSA is that it works just fine with more than two primes. So this challenge uses 16 - which is fine, except key size was not adjusted appropriately, so each of those primes is small enough to break it.

Binary (35 points)

Binary password validator, provided in two versions (Linux and OSX) for convenience. It was compiled with -O3 which made its encrypting loops unroll, and what was very trivial code turned into hard to understand vectorized mess. Then again, actually running the program might reveal something interesting.

SVG XOR (40 points)

The flag is written in SVG flag, which was encrypted with a XOR cipherer. This turned out to be quite easy, as SVG files have a lot of structure which can be used to attack this.

FPGA (45 points)

Probably the most original challenge this time. A netlist of NAND gates which can validate the flag and tiny emulator were provided. Can you figure out the inputs necessary to get the validator to accept?

This was the only challenge without anyone solving it during the event, but there were some solutions afterwards.

Tweets (50 points)

A flag was hidden in collection of Donald Trump's tweets, encrypted with monoalphabetic cipher. Great exercise for frequency analysis.

Challenges for July 2018 SecTalks London

Last month I ran another round of London SecTalks CTF.

I only created 6/9 challenges this time, 3 Android challenges were created by imhotep.

Challenge files and code used to generate them available on github.

There are no answers below, but some serious hints which might make it too easy.

For previous rounds, see posts about September 2017, November 2017 CTFs, and May 2018.

Hidden treasure (5 points)

As per tradition, there was a bonus challenge with a zip bomb. This time at the end of 16-deep archive with 16 branches each there were pictures with a treasure, with EXIF tags containing either the flag or information that you failed.

Android 1 (10 points)

The simplest Android challenge wasn't even all that Android specific - as the flag was hidden directly inside the .apk.

RE1 (15 points)

Easy reverse engineering challenge was just a 32-bit Linux binary, which verified passed flag character by character, in random order.

Disassembling it, you'd get instructions like:

cmpb   $0x74,0x10(%ebx)
jne    80484b7

If you arranged them correctly, you'd see the flag. Or you could just grep for all characters, and use anagram solver - apparently that was an option too.

Android 2 (20 points)

This one required actually running the app, or fairly complex static analysis. The app saved a file with the flag to device storage. If you can find the file, the flag is yours.

RSA (25 points)

This challenge was actually quite realistic.

Bob sent a message we need to decrypt.

Probably due to bad RNG, it looks like Bob and Alice picked same N for their keys, and we managed to steal Alice's private key as well.

Perhaps there's a way to take advantage of this.

And there is a way. Having private key (n, e, d) there's algorithm to factor n into p, q. It's not completely obvious, but it's fairly short to implement. With p, q (shared between both), and Bob's e, you can trivially get Bob's d. Then you can decrypt the message.

RE2 (30 points)

It was a slightly harder version of reverse engineering challenge. The only difference was that flag was encoded into Base64 before checking, which could throw people off, but binary wasn't stripped so b64_encode method was a massive hint.

Android 3 (35 points)

The challenge app is making HTTP request, which needs to be intercepted and modified to get the flag from the server.

Once you get the request it's obvious what to do, and there were hints provided how to setup proxy on Android to get them.

CTR (40 points)

Another somewhat realistic challenge. A collection of Elon Musk's tweets and a flag was encrypted using AES-128 in CTR mode. Reusing same IV for all of them.

This turns CTR mode into XOR with random keystream, and breaking XOR cipher is fairly basic. It turns out nobody remembered mode names, so I had to give people some hints before they tried CBC or CFB attacks on it.

Croatian Monoalphabetic (50 points)

This challenge was worth the most points, but in principle it was very easy, just a bit time-consuming - and I ended up giving people hints how to approach that.

It's just statistical analysis of another Latin script language. So you can start by assuming space is the most common character, then use character frequency tables for Croatian, or lists of most common words, or take advantage of the fact that flag (very long word) is embedded inside.

Challenges for May 2018 SecTalks London

Last month I ran another round of London SecTalks CTF.

There were 8 challenges as before. And again, the winner only did 7 of 8 on time, the last one (Monoalphabetic) only after the event.

Challenge files and code used to generate them available on github.

There are no answers below, but some serious hints which might make it too easy.

For previous rounds, see posts about September 2017 and November 2017 CTFs.

Invisible Flag (5 points)

It's a third iteration of the zip challenge.

There's a zip file containing 16 zip files inside. And each contains 16 more zip files. And so on for 16 levels, until you get to the flag.

For a small additional complication, the flag is completely invisible, and made out of different types of Unicode spaces. Replacing special spaces with different characters will reveal it.

Perl (10 points)

It's a very simple password validator written in very straightforward Perl.

The validator is mostly a sequence of commands like

s/^(.{7})i(.*)$/\2\1/;

Which removes character "i" from 8th position, and swaps everything before and after it. If the answer is empty, the validation is successful.

The easy way to solve it is to work backwards from an empty answer.

Almost Invisible Flag (15 points)

The challenge is an image with 1s and 0s written onto it in a very faint color. Once you figure that out, and transcribe the numbers into a binary file, it's a zip. Unpacking it reveals the flag.

React.js (20 points)

It's a react.js based validator. Validator is build of component layers, first layer checks length of the password, then each layer checks one character and passes the data to next layer with some transformations applied.

React Developer Tools for Chrome can be quite helpful for this.

This can be solved in many ways. You could brute force each layer with a simple script, or turn React code into a recursive function, and solve in an old fashioned way.

RSA (25 points)

For this challenge p and q are very close to each other. In fact they differ by just 2. There's a very simple attack for such case.

Monoalphabetic Cipher (30 points)

This challenge is a bit of text with flag embedded in it, encrypted by monoalphabetic cipher - all lower case letters, upper case letters, digits, spaces, punctuation each getting separate character.

The trick is that symbols used by the cipher are all emoji, which doesn't make the challenge any harder, just more fun.

The slight difficulty is that the text is not actually in English, so statistical methods can suffer a small detour. It's actually in German.

This was the only challenge that wasn't solved in time, but I don't think it's really that hard or time consuming.

I really like challenges about classical ciphers, but I don't want the solution to be simply copypasting it on quipquip.

PDF (35 points)

The challenge is a PDF file encrypted by a XOR key. The content is just a result of Chrome's print to PDF function. For extra challenge the flag is censored with a black bar over it, but that's very easy to workaround.

PDFs have a lot of plaintext inside, so there's a ton to work with, however it's somewhat unpredictable where that plaintext is located.

The key is three lower case English words without spaces.

A surprise extra challenge is that I accidentally left newline character at end of the key, which I only noticed during the challenge and told the contestants.

Javascript (40 points)

It's a very simple Javascript validator, obfuscated with jsfuck to only use 6 charecters.

I expected that people would need to write a simple JSFuck decoder, and that's why it's worth so many points, but most people just used Chrome debugger instead to solve this challenge really quickly.