Crosscompiling Python packages with VirtualBox and py2exe

A few weeks ago I complained that jrpg development pretty much ceased because Windows packaging became such a pain in the ass. Fortunately I managed to setup a VirtualBox environment in which I can cross-compile jrpg and other Python packages and create convenient zip files containing Windows executables. This means Windows users don't need to install Python and Pygame, and I don't need a separate Windows box.

The setup process was far from trivial, as it was my first contact with VirtualBox, and I knew very little about Cygwin and Windows in general. The first thing was to install VirtualBox. There are many freely downloadable virtual machines, this one just happened to have easily googlable Ubuntu package and made it perfectly clear what is it supposed to do, so I didn't look any further. VMware in contrast has many different packages (Server, Player, Workstation, whatever), and to a person who never used it before it isn't clear which one is meant for what.

VirtualBox packages for all major Linux distributions are available from here. Setting up VirtualBox for the first time is not simple, but the documentation (also /opt/VirtualBox-1.4.0/UserManual.pdf) described it very well. Documentation usable for non-experts is one of the things which corporation-backed software often gets better than Open Source.

After installing VirtualBox you will also need to install uml-utilities and
bridge-utils, add yourself to groups vboxusers and uml-net, log out, and in again. It will only be needed later, but it's easier to do it right now.

The next step is creating the VirtualBox. Run command VirtualBox for a pretty GUI. There's also VBoxManage CLI interface, but it's probably better to use GUI if it's your first time. Click on "New", and follow the creation wizard. Don't worry much about allocating diskspace - empty virtual diskspace doesn't take any space on the host hard drive. I gave it virtual 5GB, and it actually uses 1.5GB.

So the next step is installation of the operating system. Grab Windows XP install CD, or any other Windows install CD lying around, or torrent one in the unlikely case you cannot find any. You can tell VirtualBox to either use actual CD or an image file. Start the machine, and enjoy Windows installation. It's much less painful than real Windows installation, as you don't need any drivers or service packs. Oh, and for your sshing convenience select the same username as you have on your normal box.

After you're done start Internet Explorer on the VirtualBox. Go directly to http://www.opera.com/download/ and get a real browser (Firefox will work too). With a real browser install Python, Pygame, py2exe, WinSCP, and Cygwin. Neither Opera/Firefox nor WinSCP are strictly necessary, but I prefer Windows toasters that at least somewhat resemble real computers.

You will need to install a few more Cygwin packages - at least openssh and zip. So run the Cygwin shell. Does it work ? Great, now you need to configure ssh. The process is somewhat painful, but it's documented step by step. To verify that everything works, try sshing from Cygwin shell to localhost, and from Cygwin shell to your regular box. If it works, add ssh key from your regular box to authorized keys on the virutal box. scp real_box:~/.ssh/id_rsa.pub real_box_key; cat real_box_key >>.ssh/authorized_keys. If not, refer to Cygwin documentation. I'd describe it in detail, but .bash_history on the VirtualBox somehow evaporated, and I don't remember every single step, sorry ;-).

We are still not ready to ssh from the real machine to the VirtualBox. VirtualBox by default sets up networking in NAT mode, and connections are only possible from it, not to it. Run VirtualBox command, go to Settings, Network, switch from "NAT" to "host interface", and set interface to tap0. Now we actually need to setup some interface on the host. Make /dev/net/tun readable and writable to you, for example 0660 root.uml-net (you should have already added yourself to uml-net - remember that you need to logout and login again after you do that).

Now add to /etc/network/interfaces:

auto tap0
iface tap0 inet manual
  up ifconfig $IFACE 0.0.0.0 up
  down ifconfig $IFACE down
  tunctl_user taw

auto br0
iface br0 inet dhcp
  bridge_ports all tap0

And start both interfaces ifup tap0 br0. They are set to auto so they will start automatically the next time you start your computer. This part is very Ubuntu-specific, if you use another distribution refer to VirtualBox documentation.

Start the virtual machine again from GUI or from shell command VBoxManage startvm 'Machine Name'. Check its IP using command ipconfig. In my case host machine has IP 10.0.0.12 and the virtual machine has IP 10.0.0.6. Try logging in from the host - ssh 10.0.0.6. If everything went well we're almost done.

I use the following Rakefile task to build jrpg:

task :jrpg_windows_package do
  magicbox = "10.0.0.6"
  jrpg_package_path = FileList["/home/taw/everything/website/packages/jrpg-*.tar.gz"].sort[-1]
  jrpg_package_fn = File.basename(jrpg_package_path)
  jrpg_package_fn =~ /(\d{4}-\d{2}-\d{2})/
  windows_package_fn = "jrpg-windows-#{$1}.zip"
  windows_package_path = "/home/taw/everything/website/packages/#{windows_package_fn}"

  unless File.exists?(windows_package_path)
    # Please turn on VBox:
    # $ VBoxManage startvm 'Magic XP box
    sh "scp", jrpg_package_path, "#{magicbox}:"
    sh "ssh", magicbox, "rm -rfv jrpg/"
    sh "ssh", magicbox, "tar -xvzf '#{jrpg_package_fn}'"
    sh "scp", "/home/taw/everything/jrpg/setup.py", "#{magicbox}:jrpg/"
    sh "ssh", magicbox, "cd jrpg; c:/Python25/python.exe setup.py py2exe -b 2 -O2"
    sh "ssh", magicbox, "cd jrpg; mv dist jrpg"
    sh "ssh", magicbox, "cd jrpg; zip -r '../#{windows_package_fn}' jrpg/"
    sh "scp", "#{magicbox}:#{windows_package_fn}", windows_package_path
    # You can safely turn off the VBox
  end
end

And the following setup.py:

#!/usr/bin/python

# Run under cmd: setup.py py2exe -b 2 -O2

from distutils.core import setup
import py2exe
import glob

setup(
  windows=["jrpg.py"],
  data_files=[
    ("maps", glob.glob("maps/*")),
    ("data", glob.glob("data/*")),
    ("images", glob.glob("images/*")),
    ("font", glob.glob("font/*")),
    "README",
    "DESIGN",
  ]
)

It should be easy to customize them both to your needs. I haven't yet found how to conveniently turn the VirtualBox on and off - it can supposedly use snapshots, but they don't seem to work well with the openssh server. In any case the pain of packaging became much less unbearable. If you have any questions, go ahead and ask.