Sensible git diff for json files

JSON is rarely great for anything, but it's very often good enough, so one thing you'll often run into is JSON in git repositories.

Human-edited JSON (for which you should probably use something like JSON5 instead) is reasonable to work with.

Unfortunately typical machine-generated JSON is completely undiffable - it's all one big line, so trying to run git diff or git log -p will produce full content on before and after side, and you'll have no idea what actually changed without copying both before and after to external files, formatting them, and diffing the result - fairly slow and messy process.

Fortunately this is solvable. First, get latest version of my unix-utilities repositories for json_pp script for pretty-printing JSON (or any other json pretty-printer) and put it somewhere in your $PATH.

Then tell git to treat json as special file for purpose of diffing.

To do it globally:

echo "*.json diff=json" >> ~/.gitattributes

git config --global core.attributesfile ~/.gitattributes

git config --global diff.json.textconv json_pp

Or instead you can do it for just one project:

echo "*.json diff=json" >> .gitattributes

git config diff.json.textconv json_pp

You can use similar technique for other sort-of-text-but-not-really files like machine-generated XML.

textconv is only applied for human readable command output, so it doesn't affect any internal workings of git, and if you want to see raw diffs for any reasons you can always use --no-textconv argument to git diff.

Distributed dotfile management with Dropbox

Managing configuration files across multiple computers is hard - especially if you can't rely on any one of them being constantly online (the Linux home server setup from the 90s).

As far as I know nobody ever published a fully satisfactory solution to this problem. In this post I'll explain my setup - it's not perfect, but it's flexible, solves most of the problem, and is very easy to setup.

The system I'll describe will work with any combination of Linux and OSX computers, and also has enough reasonable enough support for Windows machines and smartphones.

~/everything

First, you need to designate one Linux or OSX box as master. It doesn't have to be always online, or even always on, or even your most frequently used machine - any box sitting safely at home you use (directly or via ssh) at least once a week will do perfectly fine.

On this machine, create a git repository for all the things - I call mine ~/everything.

Within this repository create ~/everything/bin for all your scripts. I'd strongly recommend making a public repository for some of them like this one, but usually at least some of the scripts will be too specific to your setup or too private to Open Source.

Now create ~/everything/dotfiles and ~/everything/dotfiles/boxname for each one of your Unix boxes.

You could also create a lot of small git repositories, but I find single repository setup more convenient here. And for that matter I find that people create far too many git repositories for things which belong together in a single one as a general rule, but that's a different story.

Install Dropbox and first round of symlinkery

The next step is installing Dropbox on all machines - making it sync ~/Dropbox directory. For now we do not want any of the dotfiles or scripts or other stuff to live there directly.

Now create symlinks on your master box from ~/Dropbox/bin to ~/everything/bin and from ~/Dropbox/dotfiles to ~/everything/dotfiles.

This is the key, since on all other machines they'll look like regular directories containing regular files, but on your master box the files will reside within your git repository, but will still be managed by Dropbox.

You do not want to put the whole git repository with .git stuff on dropbox, or symlink directly to a repository (which would make Dropbox manage your .git). That's not supported scenario for git, and it might bite you hard at some point.

Symlink all the ~/bin

Now on each machine create ~/bin directory and add it ot your $PATH. Do not put any actual files there, just create symlinks from ~/bin/somescript to ~/Dropbox/bin/somescript for each script you want to include.

Or simply add ~/Dropbox/bin to your $PATH if you don't need per-script setup.

The most common reason for wanting different scripts on different boxes is Linux vs OSX issues, and this can also be solved with ~/Dropbox/bin/linux and ~/Dropbox/bin/osx directories, selectively added to $PATH

Dotfiles

Now the fun part. On each machine for each dotfile move it to ~/Dropbox/dotfiles/boxname/.whatever and put a symlink in place of the original.

Notice how you can edit any machine's file on any other machine - and it will not only automatically update, it will also automatically update in the git repository.

Now every week or so you should login to your master server, wait for Dropbox to finish syncing, and then add proper commits to keep history of your changes.

Windows support

There's good news and bad news.

The good news is that most Windows applications let you select path for their stuff anywhere - so you can point it to c:\Dropbox\someapp or wherever you want. In the Unix world it's much more common to simply enforce some path, and if you don't like it - use symlinks.

The bad news is that while Windows has some kind of symlinks (and I'm talking real symlinks, not .lnk files it uses for "shortcuts"), but they only work one way with Dropbox.

Here are instructions how to setup Windows symlinks. Dropbox will push data to such symlink, but it won't track data changes on the other side of Windows symlink - so if you want to push data from Unix machines to Windows, such symlinks are good enough, if you want to modify them two-ways unfortunately not so much.

Fortunately you can still get two-way updates if you leave the files in c:\Dropbox\windows\stuff directory, and then symlink from c:\someplace\else to c:\Dropbox\windows\stuff.

You could even have a very fancy system of symlinks to get it all. On master: ~/Dropbox/windows/stuff symlinking to ~/everything/windows/stuff, then on Windows c:\some\place symlinking to c:\Dropbox\windows\stuff.

And everything else

Phones don't use dotfiles directly, but you can use Dropbox to store any .apk files you want to install, photos, and other such stuff.

With this kind of symlinkery you have a lot of flexibility - you can designate different masters for different parts of your dropbox, use different path names, any number of repositories and so on.

It helps with backups a bit too - since all your dotfiles are synchronized in a single git repository, you just need to snapshot your repository on one box for monthly backup (or however often you do it) instead of gathering files from a lot of boxes.

Dropbox is also sufficiently aware of file permissions that you can use it for somewhat sensitive information like low value private keys and so on. Highly sensitive information needs much better protection than that of course.

I mentioned some other uses of Dropbox in my previous post about it, so I'm not going to repeat it now.

Master Rakefile

One small technique that's somewhat related to all that is having master ~/Rakefile with all the common tasks you need. It would of course be a symlink to ~/Dropbox/dotfiles/boxname/Rakefile, which would be properly source controlled and so on.

rake searches for Rakefile up, so you'll have full access to all rake commands even when you're in ~/Downloads/my_little_pony (unless of course you have ponies-specific Rakefile).

Now the task I recommend setting up is searching your entire home directory for everything that's not supposed to be there. For every file, unless it's one of:

• symlink - OK (into Dropbox or elsewhere managed)
• empty directory - OK
• git repository - make sure to git fetch -p, then OK if clean and on master branch
• on whitelist as known garbage (Unix programs make a lot of them)
• on whitelist as Dropbox directory, or something else non-garbage that you want to be there
• for some nonempty directories you want to apply this rules recursively to their content, not to the directory directly. For example ~/Downloads, ~/Desktop, and ~/Documents with probably have some empty subdirectories, ~/bin will have symlinks etc.

Then it's a problem that needs to be dealt with.

This kind of zero tolerance for crap policy is natural extension of GTD inbox zero concept. It's really difficult to get started, but once you get used to it, you'll probably thank me. If there's enough interest, I'll write a post on how to make such Rakefiles and what to put there in the future.

Could Mercurial please die already? kthxbye

I don't understand why programmers - a crowd on whom you can usually rely to have healthy amount of hatred for competing technologies - became suddenly so lovey-dovey when it came to DVCSs.
Here's an example.

This is all wrong. Mercurial should die. So should bazaar, darcs, and countless other systems I won't even bother listing here. Why?

The best tool for the job fallacy

Very often people act as if choices made by one person in one context didn't affect other people, or even the same person in other contexts. What's the name of this problem? Externality denialism - that's what it is!

It makes very little difference what font you use in your editor - if every single person used different fonts and different colours, nobody would suffer from it, except for maybe occasional odd person forced to work on someone else's machine and have their eyes burned by Comics Sans. Best font for the job, whatever.





And it likewise makes very little difference which editor you're using (well, maybe there would be more nice plug-ins if everyone used TextMate, but the effect is not that big), or which DVCS GUI frontend, and so on.

It makes some difference what programming language you use. If everyone used one of the small number of programming languages, they would have all the libraries, all the tools, and all the support needed. But if everyone uses something else - some use RLisp, some use colorForth, some use Megan-Fox-picture - the resulting chaos means nobody has tools, libraries, or support needed to do any work done. Yes, Megan-Fox-picture might be a better language than Ruby, but the net effect of everybody using a different language is a disaster.

Does it mean everyone should use Java? Not necessarily. It only means that the threshold for choosing an unusual language should be considerably higher than "slightly better tool for the job". Is Ruby better enough than Java? Obviously. If we lived in alternative world in which everyone used Python, would it make sense to write your program in Ruby? Well, it's a borderline case, but maybe, maybe not. Should we rewrite Linux kernel in Scheme? Maybe not. Or we could have different languages for genuinely different situations - with each dominating a well specified niche.

And even with programming languages it's not such big of a deal if you use an unusual language every now and then. Programs just do stuff. They can survive on their own. And most programming languages, diverse as they are, rely on the same standards for interoperability - POSIX, Unicode, TCP/IP and so on. Even if your program is written in LOLCODE, and mine in Clojure they can probably talk with each other reasonably well.

It can get painful if interaction gets too close - for example if a C library with fake OO is used from Ruby, and memory management of both is entirely incompatible and results in random memory leaks - yes I'm talking to you Gtk+ (not that anybody cares, Linux on desktop is dead). But basic interactions between different programs sort should work even across language gap.

Version Control Systems

It's not so with DVCSs. DVCSs are interoperability technology. If I like git and you like Mercurial we cannot just both "use best tool for the job", as neither of us will be able to talk to another. Imagine that in addition to HTTP, HTML, CSS and so on, Microsoft released its Microsoft Transfer Protocol, Microsoft Markup Language, and so on - do you have any idea how much mess it would get us into? Oh wait, we're already sort of there.

Here's the game theoretic matrix for DVCS choice. Let's say half of developers like git more and half like Mercurial more:

• everyone uses git - Mercurial-lovers a bit grumpy, but otherwise we're all happy and interoperable
• everyone uses Mercurial - git-lovers a bit grumpy, but otherwise we're all happy and interoperable
• git-lovers use git, Mercurial-lovers use Mercurial - there is no interoperability and everyone suffers
• git-lovers use Mercurial, Mercurial-lovers use git - now that's just being silly

This situation is called Battle of the sexes is game theory. So imagine the real men like git, and total pussies like Mercurial. Or something.




The solution? We have to agree to one DVCS or another. Yes, some day someone will make even better one, and we'll all switch again, but neither git nor Mercurial are sufficiently better from the other to win on merits alone, and there's really no point in having them both.

Right now git is significantly more popular. So what should happen? Mercurial should die. Yes, you put a lot of effort into it, and we're very sorry etc. etc. but seriously - we need a second DVCS as much as we need a second character set, or a second .