HTCondorDockerUsage < Main

---+ HTCondor Docker usage

Condor has the capability to pull docker images from docker hub: https://hub.docker.com/<br /><br />To do this we must make a few small modifications to the submitfile as well as our general workflow. To start, here is an example of a submit file using the docker capabilities of condor:

---------------------------------------------------<br />Universe       = docker<br />Executable     = ./dockertest.sh<br />docker_image   = uwclms/python3.6<br />Log            = test.log<br />Output         = test.out<br />Error          = test.error<br />request_memory = 512 <br />should_transfer_files = YES <br />when_to_transfer_output = ON_EXIT<br />Queue

-----------------------------------------<br /><br />There are four important lines here.<br /><br />The first is that the "Universe" should be set to "docker"<br /><br />Next we must point condor at the proper image, this should be the namespace of the docker image on docker hub. The "uwclms/python3.6" Image is a standard image built for our environment that has a list of default python packages installed that are commonly used.

The next two pieces "should_transfer_files = YES" and "when_to_transfer_output = ON_EXIT" are features that will help move our executable into the docker container, as well as pulling output out. Anything written to the "/scratch" directory during the job will be transfered back out into the submit directory upon job completion. IE if you submit a job from "/home/userA/job/" Anything you write to "/scratch" will end up in that directory once the job as completed.<br /><br />In addition to the scratch directory however, we also mount the "/corpora" directory for input, as well as the "/projects" Directory for both input and output. This directories are mounted by HTCondor and NOT by docker. Any image you run via condor will have these directories available.
This topic: Main > HTCondorDockerUsage
Topic revision: r2 - 2019-03-07 - linghelp