Distributed dotfile management with Dropbox

Managing configuration files across multiple computers is hard - especially if you can't rely on any one of them being constantly online (the Linux home server setup from the 90s).

As far as I know nobody ever published a fully satisfactory solution to this problem. In this post I'll explain my setup - it's not perfect, but it's flexible, solves most of the problem, and is very easy to setup.

The system I'll describe will work with any combination of Linux and OSX computers, and also has enough reasonable enough support for Windows machines and smartphones.

~/everything

First, you need to designate one Linux or OSX box as master. It doesn't have to be always online, or even always on, or even your most frequently used machine - any box sitting safely at home you use (directly or via ssh) at least once a week will do perfectly fine.

On this machine, create a git repository for all the things - I call mine ~/everything.

Within this repository create ~/everything/bin for all your scripts. I'd strongly recommend making a public repository for some of them like this one, but usually at least some of the scripts will be too specific to your setup or too private to Open Source.

Now create ~/everything/dotfiles and ~/everything/dotfiles/boxname for each one of your Unix boxes.

You could also create a lot of small git repositories, but I find single repository setup more convenient here. And for that matter I find that people create far too many git repositories for things which belong together in a single one as a general rule, but that's a different story.

Install Dropbox and first round of symlinkery

The next step is installing Dropbox on all machines - making it sync ~/Dropbox directory. For now we do not want any of the dotfiles or scripts or other stuff to live there directly.

Now create symlinks on your master box from ~/Dropbox/bin to ~/everything/bin and from ~/Dropbox/dotfiles to ~/everything/dotfiles.

This is the key, since on all other machines they'll look like regular directories containing regular files, but on your master box the files will reside within your git repository, but will still be managed by Dropbox.

You do not want to put the whole git repository with .git stuff on dropbox, or symlink directly to a repository (which would make Dropbox manage your .git). That's not supported scenario for git, and it might bite you hard at some point.

Symlink all the ~/bin

Now on each machine create ~/bin directory and add it ot your $PATH. Do not put any actual files there, just create symlinks from ~/bin/somescript to ~/Dropbox/bin/somescript for each script you want to include.

Or simply add ~/Dropbox/bin to your $PATH if you don't need per-script setup.

The most common reason for wanting different scripts on different boxes is Linux vs OSX issues, and this can also be solved with ~/Dropbox/bin/linux and ~/Dropbox/bin/osx directories, selectively added to $PATH

Dotfiles

Now the fun part. On each machine for each dotfile move it to ~/Dropbox/dotfiles/boxname/.whatever and put a symlink in place of the original.

Notice how you can edit any machine's file on any other machine - and it will not only automatically update, it will also automatically update in the git repository.

Now every week or so you should login to your master server, wait for Dropbox to finish syncing, and then add proper commits to keep history of your changes.

Windows support

There's good news and bad news.

The good news is that most Windows applications let you select path for their stuff anywhere - so you can point it to c:\Dropbox\someapp or wherever you want. In the Unix world it's much more common to simply enforce some path, and if you don't like it - use symlinks.

The bad news is that while Windows has some kind of symlinks (and I'm talking real symlinks, not .lnk files it uses for "shortcuts"), but they only work one way with Dropbox.

Here are instructions how to setup Windows symlinks. Dropbox will push data to such symlink, but it won't track data changes on the other side of Windows symlink - so if you want to push data from Unix machines to Windows, such symlinks are good enough, if you want to modify them two-ways unfortunately not so much.

Fortunately you can still get two-way updates if you leave the files in c:\Dropbox\windows\stuff directory, and then symlink from c:\someplace\else to c:\Dropbox\windows\stuff.

You could even have a very fancy system of symlinks to get it all. On master: ~/Dropbox/windows/stuff symlinking to ~/everything/windows/stuff, then on Windows c:\some\place symlinking to c:\Dropbox\windows\stuff.

And everything else

Phones don't use dotfiles directly, but you can use Dropbox to store any .apk files you want to install, photos, and other such stuff.

With this kind of symlinkery you have a lot of flexibility - you can designate different masters for different parts of your dropbox, use different path names, any number of repositories and so on.

It helps with backups a bit too - since all your dotfiles are synchronized in a single git repository, you just need to snapshot your repository on one box for monthly backup (or however often you do it) instead of gathering files from a lot of boxes.

Dropbox is also sufficiently aware of file permissions that you can use it for somewhat sensitive information like low value private keys and so on. Highly sensitive information needs much better protection than that of course.

I mentioned some other uses of Dropbox in my previous post about it, so I'm not going to repeat it now.

Master Rakefile

One small technique that's somewhat related to all that is having master ~/Rakefile with all the common tasks you need. It would of course be a symlink to ~/Dropbox/dotfiles/boxname/Rakefile, which would be properly source controlled and so on.

rake searches for Rakefile up, so you'll have full access to all rake commands even when you're in ~/Downloads/my_little_pony (unless of course you have ponies-specific Rakefile).

Now the task I recommend setting up is searching your entire home directory for everything that's not supposed to be there. For every file, unless it's one of:

• symlink - OK (into Dropbox or elsewhere managed)
• empty directory - OK
• git repository - make sure to git fetch -p, then OK if clean and on master branch
• on whitelist as known garbage (Unix programs make a lot of them)
• on whitelist as Dropbox directory, or something else non-garbage that you want to be there
• for some nonempty directories you want to apply this rules recursively to their content, not to the directory directly. For example ~/Downloads, ~/Desktop, and ~/Documents with probably have some empty subdirectories, ~/bin will have symlinks etc.

Then it's a problem that needs to be dealt with.

This kind of zero tolerance for crap policy is natural extension of GTD inbox zero concept. It's really difficult to get started, but once you get used to it, you'll probably thank me. If there's enough interest, I'll write a post on how to make such Rakefiles and what to put there in the future.