Python Virtual Environments

This document is meant to be a quick introduction to virtualenv and Python virtual environments. For more information, see the official virtualenv user guide..

What is virtualenv?

virtualenv simplifies the process of creating your own tree of Python modules, so you can install modules that are not installed system-wide, or that conflict with system-wide versions. It's not a virtual machine, just a convenient way of setting up your environment to use a different Python tree. While this can be done by hand, virtualenv automates the process and makes it significantly easier.

Walkthrough

For this example, we're going to set up a python 3.4 virtual environment on patas.ling.washington.edu, and install the Python port of Hunt the Wumpus*.

First, create the virtual environment. This is as simple as virtualenv3.4 NAME, where NAME is the directory you want the virtual environment to be in. There are several options; for this example we'll use --system-site-packages, which tells it to make the system-wide module directory accessible in our virtual environment. This will reduce the number of dependencies we have to install. You may want to avoid this if, for example, you're installing a conflicting version of a system module.

brodbd@patas:~[0]$ virtualenv3.4 --system-site-packages wumpus
Using base prefix '/opt/python-3.4.1'
New python executable in /home2/brodbd/wumpus/bin/python3.4
Also creating executable in /home2/brodbd/wumpus/bin/python
Installing setuptools, pip, wheel...done.

Next, we can source the newly-created activation script into our shell, to enter our new environment.

brodbd@patas:~[0]$ source wumpus/bin/activate
(wumpus) brodbd@patas:~[0]$

Notice the shell prompt has changed to remind us we're now in a different python environment. This will persist until we either log out or use the deactivate command. Now that we have a new Python environment exclusive to us, we can use pip to install the module we want and any dependencies.

(wumpus) brodbd@patas:~[0]$ pip install pywumpus
Collecting pywumpus
Installing collected packages: pywumpus
Successfully installed pywumpus-1.2

Pywumpus installs an executable script in the bin directory, which the activate script conveniently added to our path.

(wumpus) brodbd@patas:~[0]$ pywumpus.py
Instructions (Y-N)? n

You are in room 9.
    I smell a Wumpus.
    Bats nearby.
Tunnels lead to 8, 10, 18.
Shoot or Move (S-M)?

When you're done enjoying classic computer gaming circa 1972, you can exit back to the system python environment with deactivate:

(wumpus) brodbd@patas:~[0]$ deactivate
brodbd@patas:~[0]$

Condor usage

Since the activation script just changes environment variables, the easiest way to use a virtualenv environment from Condor is to wrap your job in a shell script. Something like this should work:

#!/bin/bash
source $HOME/my_environment/bin/activate
python3.4 my_job.py

Put this in a script file (say, my_job.sh) and use chown +x my_job.sh to make it executable. You would then use Executable=my_job.sh in your condor submit script. In this case it's unnecessary to call deactivate, since the environment changes will disappear at the end of the script.

* Hunt the Wumpus has a long history, having been written in 1972 at University of Massachusetts - Dartmouth. It was originally written for a mainframe system, in BASIC (a language that was itself invented at Dartmouth.) It was endlessly ported and re-published and many older geeks (myself included) have fond memories of typing it into our microcomputers from a listing in a book or magazine. - brodbd - 2016-05-06