Small tips for making Unix programming nicer

Every Unix programmer - and I tell it authoritatively based on my statistically significant sample of 1 - creates millions of tiny scripts that he never bothers publishing. Because they're small, so overhead of cleaning up, documenting, and publishing such script would be immense. And not obviously googleable/bingable/baiduable or whatever people call it these days - it might surprise you but most things in the world are not described by a few well-defined key phrases.


I decided to get a few of such scripts, and throw a bunch of tips on top of them - hopefully you'll find a useful trick or two here.

How to avoid Unix destroying your files

First, we need to fix some of the famous Unix brain damage - cp, mv, and bash overwriting your files.
Now a case could be made for rm removing files without asking for confirmation - that's what rm stands for.
But how in the world is this:

mv most_awesome_song_ever.mp3 ~/Music/

supposed to quietly overwrite existing ~/Music/most_awesome_song_ever.mp3?
In 99% of cases this is behaviour you do not want - and if you actually do,
you know to type this instead:

mv -f most_awesome_song_ever.mp3 ~/Music/

So open your ~/.bashrc and put these commands there (at least ones for mv and cp):

alias rm='rm -i'
alias mv='mv -i'
alias cp='cp -i'

There's one more way Unix can destroy your files - command >file redirection will overwrite file without asking, so it's pretty easy for accidents this way. You really don't want

cat >~/notes/girlfriend

to lose information about all your exes, do you? (example purely speculative) Get yourself into habit of always saying command >>file - appending to file. 99% of time the file in question won't exist - so the result will be the same - and in cases where file exists, you remove it first. If you make a mistake, you just ^C and you're back when you started. I haven't used overwriting redirection is years - it's completely purged from my Unix dictionary. Do the same thing.

By the way some Unix distributions do these by default. And some shells have options for asking for confirmation if > would override files. Do these anyway, just to be sure.

Make shell history useful

By default shell history stores entire 500 entries - a sensible decision back when you had 4MB of RAM, most of which taken by Emacs. It's just ridiculous these days so as first line of .bashrc put export HISTSIZE=1000000. It must be first, because if bash ever decides to exit without HISTSIZE set to a sane value, it might decide to trim your history file and lose all your history. Of course if you want to tempt the fate...

Get rid of .pyc file everywhere

Python's habit of creating .pyc files all over the place gets on my nerves - and it's dubious that they really improve performance that much. If they want to fix performance, how about dealing with GIL first instead of resorting to such hacks? Anyway export PYTHONDONTWRITEBYTECODE=1 totally fixes the problem without any side effects.

Saner output from Unix commands

Compared to fatal loss of data, these are just minor annoyances, but they're really easy to fix, so let's do it.
For stupid reasons du and df commands give sizes in units of 512 bytes or something like that.
Probably some ancient BSD file system allocated files in multiplies of that. That they care more about file system
implementation details than about human usability tells you something about the Unix mindset (if these were at least kBs
that would make sense but no...) - and in any case these assumptions are no longer true on modern operating systems,
which don't rely so mindlessly on blocks. So add these two lines to get human-readable output:

alias df='df -h'
alias du='du -h'

By the way alias command only applies to what you type in the terminal, not what scripts do. So if you run mv or df from scripts, it will not have this default -i/-h. Be cautious.

Final touches for .bashrc

If some commands require root access, and you're tired of typing sudo this and sudo that, just add a few of:

alias port='sudo port'
alias gem='sudo gem'
alias apt-get='sudo apt-get'
alias reboot='sudo reboot'

to your .bashrc. They only save you a little typing, and don't chance anything about security (you still need your admin password etc.), but why type more if you can type less.

Colorful shell

Depending on your distribution you might already have colors in your shell or not.
export CLICOLOR=1 in .bashrc will convince many commands that you want nicely colored output.
To tell that to git, you need to add to your .gitconfig:

[color]
 diff = auto
 status = auto
 branch = auto

And now your Unix comes with more rainbows.

GNU grep must die

Never use GNU grep. Use pcregrep for everything - or rak/ack if you want coloring, automatic .git directory skipping etc. Unfortunately pcregrep is ridiculously slow to type so do yourself a favour and add

alias gr='pcregrep'

to make your life even easier.

wget HTTPS nonsense

I don't know if it's just MacPorts' version of wget, or is it universal, but it seems to miss all HTTPS root CAs (take that VeriSign!). Another alias solves the problem. By "solves" I mean it opens a massive security vulnerability, but we already know that CAs will create fake certs for NSA, Mossad, RIAA, and Hackney Borough Council if asked, so the vulnerability is much less than it seems at first.

alias wget='wget --no-check-certificate'

Use ruby or perl for nontrivial actions

In early 1980s, before Perl got invented to solve exactly this problem I talk about,
people would write insanely complicated shell scripts to automate their Unix actions.

Unfortunately you cannot really serve two aims at once - being highly accepting for casual real time input,
and being highly robust for programmable interfaces - so shell sucks at both. Fortunately the problem
is solved since December 18, 1987 when Perl got invented just for this reason.

And yet - many people act as if 12/18 never happened. Wake up sheeple!

Shell is ridiculously stupid. It has arcane and fragile escaping rules.
It cannot even reliably expand a file list:

$ ls ~/porn/*.jpg
-bash: /bin/ls: Argument list too long

How useless is that? And don't even get me started with xargs, find, awk and the rest horrible mess.

How do you find top largest MP3 files - in any subdirectory?

ruby -e 'Dir["**/*.mp3"].sort_by{|fn| -File.size(fn)}[0,10].each{|fn| system "ls", "-l", fn}'

And yes, I'm calling ls 10 times here, as otherwise it would see fit to rearrange them alphabetically just for lulz.
This Ruby code is really easy and really obvious.

What would be shell solution? Something like this:

find . -name '*.mp3' -print0 | xargs -0  ls -l | sort -k25 -rn | head -n 10

You need find and xargs to avoid "argument list too long" error, then you need -print0 and -0 because file names can contain single quotes or - longcat forbid - spaces! Then you need to manually count at which position of ls -l's output is file name (conveniently ls -l uses something close enough to fixed column width to make sort work - otherwise you'd have to do some heavy awking around it), then you finally head. Personally I prefer waterboarding to having my brain suffer any of this.

Learn GUI integration basics

The chasm between nice programmable world of Unix terminals and hostile world of closed GUI programs can be to some extend lessened.

If you use KDE, dcop command gives you decent level of control over GUI programs and you can explore the interface from command line.

On OSX there's a convenient open command for opening URLs and files with the most sensible program. And osascript command, which is about as powerful as KDE's dcop except far more painful to use by trying to be "friendly" too hard, resulting in unsurprising failure. Google will help.

Editing your scripts made easy

You probably have five billion scripts in your path (you have ~/bin or ~/local/bin or ~/gitrepo/bin or such in your $PATH right?) - and you're tweaking them all the time.
Typing mate `which some_script.rb` takes forever and is not easily tabbable (Ubuntu has really good
bash autocompletion package which might alleviate this problem a lot - but most distros don't).

Wouldn't it be easier to just say some_script.rb --edit? It would also be far easier to type - somTAB --edit. It's really easy. Just put this below the shebang line of all your scripts:

# Ruby
exec 'mate', __FILE__ if ARGV[0] == '--edit'

# Perl
exec "mate", __FILE__ if $ARGV[0] eq '--edit';

# Python
import os, sys
if sys.argv[1:2] == ['--edit']: os.execlp("mate", "mate", __file__)

If someone passed --edit as first argument it will start the editor instead of running the script - otherwise it will not affect it in any way.

Python code is fairly painful because Python decided to keep low level C interface to exec* instead of providing sane Perl-style interface. And you know something is wrong if Perl is described as "sane" compared with you.

Feel free to figure out how to get this effect with C++.

Find kittens for your blog

I only want CC kittens, so nobody sues my blog. Except for defamation, that I don't mind. Here's the script

#!/usr/bin/env ruby
uri = "http://www.flickr.com/search/?q=#{ARGV.join '+'}&l=cc&ss=2&ct=0&mt=all&adv=1&s=int"
# On OSX
system "open", uri
# On Linux
#system "firefox", "-new-tab", uri

By the way could someone get Linux distros to copy open command? It's really simple and really useful.

That's it for today. Enjoy your Unix.