Distributed dotfile management with Dropbox

Managing configuration files across multiple computers is hard - especially if you can't rely on any one of them being constantly online (the Linux home server setup from the 90s).

As far as I know nobody ever published a fully satisfactory solution to this problem. In this post I'll explain my setup - it's not perfect, but it's flexible, solves most of the problem, and is very easy to setup.

The system I'll describe will work with any combination of Linux and OSX computers, and also has enough reasonable enough support for Windows machines and smartphones.

~/everything

First, you need to designate one Linux or OSX box as master. It doesn't have to be always online, or even always on, or even your most frequently used machine - any box sitting safely at home you use (directly or via ssh) at least once a week will do perfectly fine.

On this machine, create a git repository for all the things - I call mine ~/everything.

Within this repository create ~/everything/bin for all your scripts. I'd strongly recommend making a public repository for some of them like this one, but usually at least some of the scripts will be too specific to your setup or too private to Open Source.

Now create ~/everything/dotfiles and ~/everything/dotfiles/boxname for each one of your Unix boxes.

You could also create a lot of small git repositories, but I find single repository setup more convenient here. And for that matter I find that people create far too many git repositories for things which belong together in a single one as a general rule, but that's a different story.

Install Dropbox and first round of symlinkery

The next step is installing Dropbox on all machines - making it sync ~/Dropbox directory. For now we do not want any of the dotfiles or scripts or other stuff to live there directly.

Now create symlinks on your master box from ~/Dropbox/bin to ~/everything/bin and from ~/Dropbox/dotfiles to ~/everything/dotfiles.

This is the key, since on all other machines they'll look like regular directories containing regular files, but on your master box the files will reside within your git repository, but will still be managed by Dropbox.

You do not want to put the whole git repository with .git stuff on dropbox, or symlink directly to a repository (which would make Dropbox manage your .git). That's not supported scenario for git, and it might bite you hard at some point.

Symlink all the ~/bin

Now on each machine create ~/bin directory and add it ot your $PATH. Do not put any actual files there, just create symlinks from ~/bin/somescript to ~/Dropbox/bin/somescript for each script you want to include.

Or simply add ~/Dropbox/bin to your $PATH if you don't need per-script setup.

The most common reason for wanting different scripts on different boxes is Linux vs OSX issues, and this can also be solved with ~/Dropbox/bin/linux and ~/Dropbox/bin/osx directories, selectively added to $PATH

Dotfiles

Now the fun part. On each machine for each dotfile move it to ~/Dropbox/dotfiles/boxname/.whatever and put a symlink in place of the original.

Notice how you can edit any machine's file on any other machine - and it will not only automatically update, it will also automatically update in the git repository.

Now every week or so you should login to your master server, wait for Dropbox to finish syncing, and then add proper commits to keep history of your changes.

Windows support

There's good news and bad news.

The good news is that most Windows applications let you select path for their stuff anywhere - so you can point it to c:\Dropbox\someapp or wherever you want. In the Unix world it's much more common to simply enforce some path, and if you don't like it - use symlinks.

The bad news is that while Windows has some kind of symlinks (and I'm talking real symlinks, not .lnk files it uses for "shortcuts"), but they only work one way with Dropbox.

Here are instructions how to setup Windows symlinks. Dropbox will push data to such symlink, but it won't track data changes on the other side of Windows symlink - so if you want to push data from Unix machines to Windows, such symlinks are good enough, if you want to modify them two-ways unfortunately not so much.

Fortunately you can still get two-way updates if you leave the files in c:\Dropbox\windows\stuff directory, and then symlink from c:\someplace\else to c:\Dropbox\windows\stuff.

You could even have a very fancy system of symlinks to get it all. On master: ~/Dropbox/windows/stuff symlinking to ~/everything/windows/stuff, then on Windows c:\some\place symlinking to c:\Dropbox\windows\stuff.

And everything else

Phones don't use dotfiles directly, but you can use Dropbox to store any .apk files you want to install, photos, and other such stuff.

With this kind of symlinkery you have a lot of flexibility - you can designate different masters for different parts of your dropbox, use different path names, any number of repositories and so on.

It helps with backups a bit too - since all your dotfiles are synchronized in a single git repository, you just need to snapshot your repository on one box for monthly backup (or however often you do it) instead of gathering files from a lot of boxes.

Dropbox is also sufficiently aware of file permissions that you can use it for somewhat sensitive information like low value private keys and so on. Highly sensitive information needs much better protection than that of course.

I mentioned some other uses of Dropbox in my previous post about it, so I'm not going to repeat it now.

Master Rakefile

One small technique that's somewhat related to all that is having master ~/Rakefile with all the common tasks you need. It would of course be a symlink to ~/Dropbox/dotfiles/boxname/Rakefile, which would be properly source controlled and so on.

rake searches for Rakefile up, so you'll have full access to all rake commands even when you're in ~/Downloads/my_little_pony (unless of course you have ponies-specific Rakefile).

Now the task I recommend setting up is searching your entire home directory for everything that's not supposed to be there. For every file, unless it's one of:

• symlink - OK (into Dropbox or elsewhere managed)
• empty directory - OK
• git repository - make sure to git fetch -p, then OK if clean and on master branch
• on whitelist as known garbage (Unix programs make a lot of them)
• on whitelist as Dropbox directory, or something else non-garbage that you want to be there
• for some nonempty directories you want to apply this rules recursively to their content, not to the directory directly. For example ~/Downloads, ~/Desktop, and ~/Documents with probably have some empty subdirectories, ~/bin will have symlinks etc.

Then it's a problem that needs to be dealt with.

This kind of zero tolerance for crap policy is natural extension of GTD inbox zero concept. It's really difficult to get started, but once you get used to it, you'll probably thank me. If there's enough interest, I'll write a post on how to make such Rakefiles and what to put there in the future.

Dropbox as solution to minor inconveniences

I'm sure most people reading this blog have multiple computers and computer-like devices (like phones, Raspberry Pis and whatnot). This comes with a huge deal of minor inconveniences - files are often on one of devices but you want to access them somewhere else.

Traditional solution was imperative file transfer - there's always some way to order a computer to transfer a file somewhere else. There's a ton of issues with this since computers are not always turned on, they're not always on networks, file transfer fail, in many cases you want to edit the same resource from different computers at different times and still want just one canonical representation of it.

Here comes Dropbox (and similar Cloud-based solutions) - its biggest innovation is replacing the imperative paradigm with declarative file transfer. You link a directory like ~/Dropbox with your account, you add, edit, and delete files and directories within it any way you like, and without your effort to babysit all that Dropbox takes care of it to make sure your ~/Dropbox directory everywhere else also reaches such state.

This solves a lot of issues like:

• You want to keep .dotfiles across multiple machines in some single versioned repository - you can solve that with Dropbox and some symlinkery (or a private github account and some symlinkery).
• You want photos you make on your camera available on your computer - at the phone should do the uploading whenever you're on some wifi, without your intervention. Dropbox completely solves that problem.
• You play a game like Cockatrice on multiple machines and you edit your decklists on multiple machines. Manually sorting out which decklist is the most recent one is pretty annoying - but Dropbox does that for you.
• A simple todo.txt file is pretty awesome if you're moving between computers a lot, even if it's obviously not a full GTD solution. In the past I did silly things like making gmail drafts for that, which I then accessed from multiple places.
• You want to write a quick note for action later, but you're somewhere outside with no paper? Simply open Dropbox directory on your phone, create somefile.txt, write whatever you wish there, and you'll get it on your computer for later access automatically. There's no way to forget about it.
• For that matter simply transferring files between computers not on shared network can be a big pain in the ass. For small files people just send emails to themselves with such files as attachment - and it's another problem Dropbox solves completely.

Ironically the only one problem that Dropbox is really horrible at solving is backups.

By the way, Dropbox gives away a lot more space than it seems at first - my free account was 2 GB initially, but they just increased that to 51.25 GB for simply connecting a few boxes and a phone to it.

Basically, the only two reasons you could have for not using Dropbox is:

• you only ever use 1 computer and 0 smartphones (pretty unlikely if you're reading this blog) 
• you use another Cloud storage service like Google Drive (with drivers installed on all your boxes)