11 Small Improvements For Ruby - followup

Last week I posted a list of 11 things I'd like to see changed about Ruby.

Time for some followup, as I have workarounds for at least some of the issues.

Pathname#glob - now in a gem

I wrote a gem pathname-glob which provides the missing method so you can code:

Pathname("foo").glob("*.txt")

This doesn't just save you a few characters, it's the only way to get reliable globbing when the path contains special characters like [, ], {, }, ?, *, or \ - which is pretty likely if you're dealing with users' files.

Support for special characters can't be fixed in Dir.glob("foo/*.txt") style API, as it doesn't know where name of the directory ends and globbing pattern starts.

Another workaround would be to use Dir.chdir("foo"){ Dir.glob("*.txt") } - that'd deal with special characters in folder names, but would cause issues with threads.

Of all the issues, this one is probably most useful to get into ruby standard library. It's like jQuery for Pathnames.

Hash#zip - now in a gem

I wrote a gem hash-zip.

You can use it for zipping any number of Hashes together, and it will pad any missing values with nils. Usually the next step is to merge them in some meaningful way (for simple case when one overrides the other Hash#merge already exists).

default_options.zip(user_options).map{|key, (default_value, user_value)| ... }

Technically it overwrites existing #zip method Hash inherits from Enumerable, but Enumerable#zip really doesn't make any sense when applied to Hashes, so it's better this way than introducing a new name.

Missing Hash methods - now in a gem

I wrote a gem hash-polyfill, which contains a bunch of simple methods Hash is bound to get eventually.

These are:

• Hash#compact
• Hash#compact!
• Hash#select_values
• Hash#select_values!
• Hash#reject_values
• Hash#reject_values!
• Hash#transform_values
• Hash#transform_values!

As I assume these will be added eventually, gem only adds methods which don't exist yet. Gem skips already defined methods.

In particular the last two already exist in 2.4, but you can get access to them from older Ruby versions with this polyfill.

Naming convention for #compact, #compact! follows Array, and for #select_values etc. follows 2.4's #transform_values and existing Enumerable#select/reject.

Names Ruby will end up using might be different, but these are as good guesses as any.

Enumerable#count_by - easier with 2.4

The example I had last time:

posts.count_by(&:author)

is actually not too bad with 2.4:

posts.group_by(&:author).transform_values(&:size)

For very large collections (like all bytes in some huge file), group_by / transform_values is going to take a lot more memory than counting things directly, but I ran benchmarks and it seems it's usually a lot faster than Ruby-based loop implementation.

If you're not on 2.4, check out hash-polyfill gem.

Simple Terrain Mapmode Mod for Hearts of Iron 4

Paradox grand strategy games don't let mods create new map modes without some serious hacks - even something as important as truce map modes for CK2 and EU4 is still missing.

Fortunately at least here a hack worked.

If you just want to skip the explanations and get the mod:

• Vanilla version
• Kaiserreich version

How Hearts of Iron 4 map works

In most strategy games maps are generated dynamically, so there's always simple mapping from map data to what's displayed.

Even a lot of games with static map use similar technique - gameplay map drives what's displayed.

That's not how Hearts of Iron 4 and other Paradox games work. Instead, they contain a bunch of bitmaps driving what's displayed, and almost completely separate gameplay map data.

So it's possible for a tile to look like a forest visually while being a gameplay hill. In some parts of the map like Persia or Tibet majority of tiles are not what they look like.

This separation is supposed to make things look nicer, but I'd say it's not worth it. When players have a choice - like in Europa Universalis 4 - vast majority instantly disable default map mode (terrain) and nevel look back again; instead using political map mode 99% of time, and if they want to see terrain information they use simple terrain map mode which is driven by gameplay data.

Unfortunately Hearts of Iron 4 doesn't let us disable visual terrain view. And to make matters worse it contains no way to see gameplay terrain. Instead we're forced to see clutter which might or might not match gameplay terrain in every map mode.

Since visual and gameplay terrain match poorly, the result is that figuring out which way you can send your tanks is a hellish exercise of mouse-overing every tile to see its real terrain, and then trying to remember it all.

What the mod does

The mod changes visual map to make it match game map more accurately, by running some image editing scripts.

It changes terrain and tree layers. It make everything far more accurate, and while it loses some detail, I'd say map looks cleaner and better this way.

I experimented with changing heightmap as well, but that made the map look rather ugly, and other changes are usually sufficient to make terrain clear. Taiwan is an example of region where terrain and heightmap mismatch.

It doesn't completely wipe out detail on source map, just tweaking it in places where there's significant mismatch.

There are a few mods that try to make terrain look better by replacing textures used for various terrain - but they don't deal with core issue that data they're trying to display doesn't match in game map.

Compatibility

There are two versions:

• one for vanilla map
• one for Kaiserreich map

You can use them with other mods as long as they don't change the map - in particular Road to 56 mod works just fine, since it uses vanilla map.

As it's all a script if there's demand I can run it for other mods. Or you could use it anyway, it might be better than vanilla map anyway.

You can use it with mods that change map graphics.

Before

Good luck guessing where the mountains are. Or hills. Or deserts.

After

And these are the answers.

Issues

Some minor rivers flow through a tile, and it's still some guesswork if you get river penalty or not attacking it. Mod doesn't affect river placement in any way.

A number of tiles in vanilla are declared as "urban", even though game files declare them as something else. The mod believes game files. If anybody has a clue what's going on, I'd like to know. The only tiles affected that I noticed are a few VPs in Iraq region. Kaiserreich doesn't seem to be affected by this problem.

11 Small Improvements For Ruby

For followup post with some solutions, check this.

A while ago I wrote a list of big ideas for Ruby. But there's also a lot of small things it could do.

Kill 0 octal prefix

I'm limiting this list to backwards compatible changes, but this is one exception. It technically breaks backwards compatibility, but in reality it's far more likely to quietly fix bugs than to introduce them.

Here's a quick quiz. What does this code print:

p 0123
p "0123".to_i
p Integer("0123")

Now check in actual ruby and you'll see my point.

If your answer wasn't what you expected, then it proves my point. The whole damn thing is far more likely to be an accident than intentional behaviour - especially in user input.

If you actually want octal - which nobody ever uses other than Unix file permissions, use 0o123.

Add missing Hash methods

Ruby has a bad habit of treating parts of standard library as second class, and nowhere it's more mind boggling than with Hash, which might be the most commonly used object after String.

It only just got transform_values in 2.4, which was probably the most necessary one.

Some other methods which I remember needing a lot are:

• Hash#compact
• Hash#compact!
• Hash#select_values
• Hash#select_values!
• Hash#reject_values
• Hash#reject_values!

You can probably guess what they ought to do.

Hash#zip

Technically Hash#zip will call Enumerable#zip so it returns something, but that something is completely meaningless.

I needed it crazy often. With a = {x: 1, y: 2} and b = {y: 3, z: 4} to run a.zip(b) and get {x: [1, nil], y: [2,3], z: [nil, 4]}, which I can then map or transform_values to merge them in meaningful way.

Current workaround of (a.keys|b.keys).map{|k| [k, [a[k], b[k]]]}.to_h works but good luck understanding this code if you run into it, so most people would probably just loop.

Enumerable#count_by

Here's a simple SQL:

SELECT author, COUNT(*) count FROM posts GROUP BY author;

Now let's try doing this in ruby:

posts.count_by(&:author)

Well, there's nothing like it, so let's try to do it with existing API:

posts.group_by(&:author).map{|author, posts| [author, posts.size]}.to_h

For such a common operation having to do group_by / map / to_h feels real bad - and most people would just loop and += like we're coding in some javascript and not in a civilized language.

I'm not insisting on count_by - there could be a different solution (maybe some kind of posts.map(&:author).to_counts_hash).

URI query parameters access

Ruby is an old language, and it added a bunch of networking related APIs back then internet was young. I don't blame anyone for these APIs not being very good, but by now they really ought to be fixed or replaced.

One mindbogglingly missing feature is access to query parameters in URI objects to extract or modify them. The library treats the whole query as opaque string with no structure, and I guess expects people to use regular expressions and manual URI.encode / URI.decode.

There are gems like Addressable::URI that provide necessary functionality, and URI needs to either adapt or get replaced.

Replace net/http

It's similar story of API added back when internet was young and we didn't know any better. By today's needs the API feels so bad quite a few people literally use `curl ...`, and a lot more use one of hundred replacement gems.

Just pick one of those gems, and make it the new official net/http. I doubt you can do worse than what's there now.

Again, I'm not blaming anyone, but it's time to move on. Python had urllib, urllib2, urllib3, and by now it's probably up to urllib42 or so.

Make bundler chill out about binding.pry

For better or worse bundler became the standard dependencies manager for ruby, and pry its standard debugger.

But if you try to use require "pry"; binding.pry somewhere in your bundle exec enabled app, it will LoadError: cannot load such file -- pry, so you either need to add pry to every single Gemfile, or edit that, bundle install every time you need to debug anything, then undo that afterwards.

I don't really care how that's done - by moving pry to standard library, by some unbundled_require "pry", or special casing pry, the current situation is just too silly.

Actually, just make binding.pry work without any require

I have this ~/.rubyrc.rb:

begin
  require "pry"
rescue LoadError
end

which I load with RUBYOPT=-r/home/taw/.rubyrc.rb shell option.

It's such a nice quality of life improvement to type binding.pry instead of require "pry"; binding.pry, it really ought to be the default, whichever way that's implemented.

Pathname#glob

Pathname suffers from being treated as second class part of the stdlib.

Check out this code for finding all big text files in path = Pathname("some/directory"):

path.glob("*/*.txt").select{|file| file.size > 1000}

Sadly this API is missing.

In this case can use:

glob("#{path}/*/*.txt").map{|subpath| Pathname(subpath)}.select{|file| file.size > 1000}

which not only looks ugly, it would also fail if path contains any funny characters.

system should to_s its argument

If wait_time = 5 and uri = URI.parse("https://en.wikipedia.org/wiki/Fidget_Spinner"), then this code really ought to work:

system "wget", "-w", wait_time, uri

Instead we need to do this:

system "wget", "-w", wait_time.to_s, uri.to_s

There's seriously no need for this silliness.

This is especially annoying with Pathname objects, which naturally are used as command line arguments all the time. Oh and at least for Pathnames it used to work in Ruby 1.8 before they removed Pathname#to_str, so it's not like I'm asking for anything crazy.

Ruby Object Notation

Serializing some data structures to send over to another program or same in a text file is a really useful feature, and it's surprising ruby doesn't have such functionality yet.

So people use crazy things like:

• Marshal - binary code, no guarantees of compatibility, no security, can't use outside Ruby
• YAML - there's no compatibility between every library's idea of what counts as "YAML", really horrible idea
• JSON - probably best solution now, but not human readable, no comments, dumb ban on line final commas, and data loss on conversion
• JSON5 - fixes some of problems with JSON, but still data loss on conversion

What we really need is Ruby Object Notation. It would basically:

• have strict standard
• have implementations in different languages
• with comments allowed, mandatory trailing commas before newline when generated, and other such sanity features
• Would use same to_rbon / RBON.parse interface.
• And have some pretty printer.
• Support all standard Ruby objects which can be supported safely - so it could include Set.new(...), Time.new(...), URI.parse(...) etc., even though it'd actually treat them as grammar and not eval them directly.
• Optionally allow apps to explicitly support own classes, and handle missing ones with excepions.

This is unproved concept and it should be gem somewhere, not part of standard library, but I'm surprised it's not done yet.

Celebrating Cloud's 10th Birthday With Cat Pictures

Cloud is now a serious mature lady cat. Let's celebrate it the best possible way - by posting a lot of cat pictures!

Young Cloud

In her youth "smartphones" took potato-quality pictures which took a minute per picture to transfer over custom USB cables, so unfortunately I don't have many photos from that time.

The few I've got shows she already had affinity towards computers, using laptop power supply as a chair:

Cloud And Computers

Cloud is indoors cat surrounded by computers, so she took a lot of interest in them.

What a nice keyboard pillow:

I wonder what keyboard pillow tastes like?

Fixing Cabling:

3D Printing:

Using laptops as chairs:

Hooman got me new laptop chair? Did it come in a box too?

This laptop is weird, but if I fits, I sits:

Doing Hooman Things

Cloud was curious about things hoomans do, so she sometimes trying to act like one, but eventually figured out it's much better to just be a cat.

Drinking coffee:

Standing on back paws:

Packing for travel:

Exploring Her World

As indoors cat she doesn't have far to explore but she's still doing her best.

Going up:

What is my hooman doing down there?

Visiting vet:

Keeping her fur snow white:

Cloud and Chair

Cloud loves her hooman's chair. Especially when it's warm after use. There's just one problem - the hooman needs that chair too.

Using chair as bed:

Using chair for photoshoots:

OK hooman, you can keep the big one, I like the cat-sized sidechair better anyway:

Sleepy Cloud

As a proper cat, Cloud loves sleeping on everything.

Sleeping next to a computer:

Sleeping in boxes:

Sleeping on her cat tree:

Sleeping on what her hooman is trying to read:

Or write:

Playful Cloud

Unfortunately it's hard to get a good pic of Cloud in action, as she's not a very active cat, and I don't have a Go Pro.

Here's a photo of her catching a bird on a stick:

Just being cute

Like any cat she loves just being cute:

Happy Birthday Cloud!