It is rarely the case that a user submits just one job to the batch queue. Often a user will submit hundreds of jobs using custom scripts and tools to automate the process. HTCondor provides tools to make multiple job submission easier. There are two advantages to using these builtin features: first, job submission will be several times faster; second, HTCondor can provide more robust scheduling if sets of jobs are submitted using these tools.
To demonstrate how to submit multiple jobs using HTCondor we start with a vanilla job description file:
Universe = vanilla Executable = /bin/echo Arguments = "test job" Requirements = CPU_Speed >= 1 Rank = CPU_Speed Image_Size = 428M Priority = +20 GetEnv = True Initialdir = /experiment/u/user Input = /dev/null Output = /experiment/u/user/myjob.out Error = /experiment/u/user/myjob.err Log = /experiment/u/user/myjob.log Notify_user = user@rcf.rhic.bnl.gov +Experiment = "experiment" +Job_Type = "cas" Queue
This job description file will submit one job which executes the echo binary with the argument "test job". If we want to submit 100 such jobs we would need one simple change:
Universe = vanilla # # Common elements in file removed for brevity. # +Job_Type = "cas" Queue 100
All that was needed was to add 100 at the end of the "Queue" command. With this command, HTCondor will submit 100 identical jobs each getting the same parameters as listed in the job description file. This might add some complications such as having one log file for all 100 jobs or having each job overwrite the output file. We can easily fix that using a built-in HTCondor macro:
Universe = vanilla # # Common elements in file removed for brevity. # Output = /experiment/u/user/myjob.out.$(Process) Error = /experiment/u/user/myjob.err Log = /experiment/u/user/myjob.log.$(Process) +Experiment = "experiment" +Job_Type = "cas" Queue 100
Now each job will have its own log file and output file where "$(Process)" is numbered from 0 to 99 (note that $(Process) will not work in the 'Executable' parameter). This solves one potential problem but there still might be a need to have different requirements for a subset of the 100 jobs. This can be solved by overwriting a previous macro definition:
Universe = vanilla Executable = /bin/echo Arguments = "test job" Requirements = CPU_Speed >= 1 Rank = CPU_Speed Image_Size = 428M Priority = +20 GetEnv = True Initialdir = /experiment/u/user Input = /dev/null Output = /experiment/u/user/myjob.out.$(Process) Error = /experiment/u/user/myjob.err Log = /experiment/u/user/myjob.log.$(Process) Notify_user = user@rcf.rhic.bnl.gov +Experiment = "experiment" +Job_Type = "cas" Queue 90 Arguments = "$(Process)" Requirements = CPU_Speed == 2 Queue 9 Input = /my/file Arguments = 99 Requirements = CPU_Speed >= 3 Queue
In this example we queue up 90 jobs all with the same parameters. Next, we queue up 9 jobs with the same parameters as the previous 90 except for the "Arguments" and "Requirements" macros which will be different for those 9 jobs. Finally, we queue up the last job with a different input file, different "Arguments", and different "Requirements". Rather than call "condor_submit" 100 times we need only call it once and HTCondor will do the work to queue up the 100 jobs.