The best kittens, technology, and video games blog in the world.

Thursday, April 18, 2013

jrpg is on github now

Abby - cat loaf 1 by DirtBikeDBA (Mike) from flickr (CC-NC-ND)

jrpg is on github now!

The scripts to make OSX and Windows packages not included, since half of them lives on the other side of VMs.

git doesn't love big binary blobs, and jrpg includes quite a few, but it should hopefully be all right. Maybe I'll rearrange them somehow to reduce their size or move them outside git repository.

There's also ton of things I could improve in jrpg, but I'm making no promises.

Sunday, April 14, 2013

magic/xml gem published

"Cat Scratch Fever!" - Ottawa 2002 by Mikey G Ottawa from flickr (CC-NC-ND)

Once upon a time I built gems for my libraries, but then rubygems site migrated like three times, and I really didn't feel like keeping track with all that, so I stopped doing anything.

Now I pushed magic-xml gem to relevant gem repositories (it has a dash like github repository name).

Apparently bkkbrad made magic_xml (with underscore) gem based on earlier version as well. Which brings me to:

Public service announcement

Everyone, it's time to talk serious business. Ruby community must decide if it wants dashes or underscores in gem name, and it must decide it now.

  • 15578 gems have underscores
  • 17127 gems have dashes
  • 1465 mix both in their name!!!
This is insanity. It's also a pretty safe bet github has something to do with it - I love you guys, but it's really time to get your act together.

I'll be using dashes, since that's what github seems to be promoting, and I tend to put my software on github these days.

Other goodies

A lot of my small utilities depend on magic/xml, so this will allow me to publish them without having to bundle magic/xml library (even it's just one file, very old school).

For now I just pushed lastfm_status program - which does precisely what its name implies - to unix-utilities repository, but I'm sure there will be more, especially once I figure out which of my programs break half of Internet's Terms of Service enough to get banned, and which only a little ;-p

Thursday, April 11, 2013

More small Unix utilities written in Ruby

eating grass 2 by lotusgreen from flickr (CC-NC-SA)

Here's the sequel to my "collection of small Unix utilities written in Ruby" post and github repository.

Useful technique - Pathname

One thing I forgot to mention the last time - Pathname library.

Pathname is an objects-oriented way to look at paths in a file system. A Pathname object is not the same as a File or Directory object since it's not opened - and might not even exist yet. It's also not like String since it has all the filesystem awareness.

For very simple scripts it's fine to use just plain Strings to represent filesystem paths, but once it gets a bit more complicated your script will get a lot more readable with Pathname - and it costs you nothing.

Let's just look at fix_permissions utility. Here's the core part:

class Pathname
  def script?
    read(2) == "#!"

  def file_type
    `file -b #{self.to_s.shellescape}`.chomp

  def should_be_executable?
    script? or file_type =~ /\b(Mach-O|executable)\b/

def fix_permissions(path)
  Pathname(path).find do |fn|
    next if
    next if fn.symlink?
    next unless fn.executable?
    fn.chmod(0644) unless fn.should_be_executable?

Since Pathname overloads #to_str method it can be transparently used in most contexts where String is expected - including printing it, file operations, system/exec commands and so on. You'll rarely need to use #to_s - mostly when you want to regexp it.

I feel Pathname#shellescape should exist, but since it doesn't that's one place where you need to use .to_s.shellescape for now.

So what does this script do? First we add a few methods to Pathname class. It already knows if something is a directory?, symlink?, and executable? (that is - has +x flag).

We want to know if it is a script. And that's easy - just read(2) as if it was a File to read first two bytes. It looks much more elegant than, 2) != "#!" we'd need if we used Strings - not to mention how String class is really no place for #script? method so we'd probably use a standalone procedure.

Next let's make file_type method - and use #shellescape to do it safely. Unfortunately that one is only defined on Strings.

After that it's just one regexp away from should_be_executable?.

Once we defined that notice how easy it is to dig into directory trees with Pathname#find, and then just use a few #query? methods to ask the path what it is about, then #chmod to setup proper flags.

Other very useful methods not present in the script are + for adding relative paths, #basename/#dirname for splitting it into components, and #relative_path_from for creating relative paths.

While I'm at it, use URI objects for URIs you want to do something complicated with rather than regexping them - usually your code will look better too.

Individual commands


Cuts long lines to specific number of characters for easy previewing.

    colcut 80 < file.xml


Removes executable flag from files which shouldn't have it. Useful for archives that went through a Windows system, zip archive, or other system not aware of Unix executable flag.

It doesn't turn +x flag, only removes it if a file neither starts with #!, nor is an executable according to file utility.

Usage example:
    fix_permissions ~/Downloads
If no parameters are passed, it fixes permissions in current directory.


Display progress for piped file.

Usage examples:

       cat /dev/urandom | progress | gzip  >/dev/null
       progress -l <file.txt | upload

By default it's in bytes mode. Use -l to specify line mode.

If progress is piped a file and it's in byte mode, it checks its size and uses that to display relative progress (like 18628608/104857600 [17%]). Otherwise it will only display number of bytes/lines piped through.

You can also specify what counts as 100% explicitly:

     progesss 123456
     progress 128m
     progress -l 42042

It will happily go over 100% on display.


Link to soup posts starting from the post before one specified.

Usage example:


Sort input through arbitrary Ruby expression. A lot more flexible than Unix sort utility.

Usage example:
    sortby '$_.length' <file.txt

RPGToolkit TST to PNM converter

Zorro at D&D by ex.libris from flickr (CC-NC-ND)

Here's a funny story - I found this converter in my ~/everything local git repository. I have no recollections of ever writing this converter, but it sure looks like my coding style from ancient days, and I think I wrote it while looking for tiles and other graphic resources for jrpg.

This tool converts TST format used by RPG Toolkit (tiles free for noncommercial use) to convenient PNM format you can then convert to whatever you wish using standard image processing tools. I ended up not using these tiles, so I forgot about the converter completely.

It's a small thing, but file format converters are one thing that's most annoying when you can't find it, so on an off chance that someone needs it someday, enjoy it on github.

Saturday, April 06, 2013

Various old projects migrated to githtub

Fluffy Buff Tom and Bike Frame by Chriss Pagani from flickr (CC-NC-ND)

Once upon a time I did "software triage"  to decide which of my software are viable, and which are dead.

Today I took another look at it, and decided to move most of my old projects - even ones that are pretty much dead - from variety of places like Sourceforge, GNU Savannah, Google Code, and tarball dumps on ftp server to github.

I don't really expect any of them to see much use, but there's always an off chance, and if I didn't move them to github, I might as well simply delete them from the Internet completely.

Here's the list of migrated projects:
  • RPU - my MSc thesis. If you want to see how to write compilers in OCaml, it might be somewhat useful.
  • bridge - I'm a Sansa Clip user now, and there's no way in hell I'm going back to iPods, but if you need this script updated (I have no idea if it still works or not), ask me, and I could probably figure out how to update it
  • tawbot - Wikipedia admin bot. I know once upon a time it had quite a few users, but I haven't heard from them in a while. If you need help with it, ask away.
  • XSS Shield for Rails 1.2.x - very similar system is included in recent Rails, so I doubt anybody needs this today.
  • freetable - HTML table generator. I know it had users once upon a time, no idea if they're still active.
  • jsme - Driver to use joystick as mouse on Linux. I made it ages ago because I accidentally my whole mouse port. I really doubt anybody would need that today, or that it would even work.
  • gtkidp - Interface for Internet Dictionary Project files. I like command line dictionaries, and I contributed to dictd stuff because Wikipedia made this kind of stuff cool, but these days it's probably not going to see much use. I don't even know if it works with recent varieties of Gtk.

Still TODO

I'm still not sure what to do with the ftp server I used to put my stuff on. Using less eye-violating styling would be a good start. I'm not entirely sure why the hell I picked that color scheme in the first place, and if it was meant as some kind of a joke or not.

And there's still my local ~/everything git repository. I put a few of its utilities on github, but there's orders of magnitude more code there - some even doesn't violate any website's ToS. There are 218 top level directories there, I'm sure at least 10% of them could be made public without any major problems.

And a lot of the software migrated to github still needs some serious work, like turning them into proper gems, compatibility with modern versions of everything and so on. If you have any special requests, just contact me.