Example scripts for Casper users

MPI | OpenMP | Hybrid | Job arrays | Command file | Dependent jobs

These Slurm batch script examples work for executables generated by any compiler and MPI library installed on Casper. (The defaults are Intel and OpenMPI, respectively.) Remember to substitute your own job name and project code, and customize the other directives and commands as necessary. That includes the commands shown for setting your TMPDIR environment variable as recommended here: Storing temporary files with TMPDIR.

The examples use the long form of #SBATCH directives, in which each option begins with two dashes – as in #SBATCH --partition=dav. This provides greater clarity than the single-letter identifiers. For information on all options, run man sbatch.

Contact cislhelp@ucar.edu for assistance if you need help with any of these examples.

When your script is ready, submit your batch job for scheduling as shown here.


Batch script to run an MPI job

MPI script examples can be found here, with additional information about submitting jobs with the Slurm Workload Manager.


Batch script to run a pure OpenMP job

For tcsh users

#!/bin/tcsh
#SBATCH --job-name=OpenMP_job
#SBATCH --account=project_code
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --time=00:10:00
#SBATCH --partition=dav
#SBATCH --output=OpenMP_job.out.%j

setenv TMPDIR /glade/scratch/$USER/temp
mkdir -p $TMPDIR

### Run program
setenv OMP_NUM_THREADS 8
./executable_name

For bash users

#!/bin/bash -l
#SBATCH --job-name=OpenMP_job
#SBATCH --account=project_code
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --time=00:10:00
#SBATCH --partition=dav
#SBATCH --output=OpenMP_job.out.%j

export TMPDIR=/glade/scratch/$USER/temp
mkdir -p $TMPDIR

### Run program
export OMP_NUM_THREADS=8
./executable_name

Batch script to run a hybrid MPI/OpenMP job

For tcsh users

#!/bin/tcsh
#SBATCH --job-name=hybrid_job
#SBATCH --account=project_code
#SBATCH --ntasks=2
#SBATCH --cpus-per-task=4
#SBATCH --time=00:10:00
#SBATCH --partition=dav
#SBATCH --output=hybrid_job.out.%j

setenv TMPDIR /glade/scratch/$USER/temp
mkdir -p $TMPDIR

### Run program
setenv OMP_NUM_THREADS 4
srun ./executable_name

For bash users

#!/bin/bash -l
#SBATCH --job-name=hybrid_job
#SBATCH --account=project_code
#SBATCH --ntasks=2
#SBATCH --cpus-per-task=4
#SBATCH --time=00:10:00
#SBATCH --partition=dav
#SBATCH --output=hybrid_job.out.%j

export TMPDIR=/glade/scratch/$USER/temp
mkdir -p $TMPDIR

### Run program
export OMP_NUM_THREADS=4
srun ./executable_name

Batch script to run a job array

Job arrays are useful for submitting and managing collections of similar jobs – for example, running the same program repeatedly on different input files

This example uses environment variable $SLURM_ARRAY_TASK_ID as an argument in running the jobs. See Slurm Job Array Support for additional job array environment variables that might be useful.

For tcsh users

#!/bin/tcsh
#SBATCH --job-name=job_array
#SBATCH --account=project_code
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --time=00:10:00
#SBATCH --partition=dav
#SBATCH --array=1-5
#SBATCH --output=job_array-%A_%a.out

setenv TMPDIR /glade/scratch/$USER/temp
mkdir -p $TMPDIR

### Run program
./executable_name $SLURM_ARRAY_TASK_ID

For bash users

#!/bin/bash -l
#SBATCH --job-name=job_array
#SBATCH --account=project_code
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --time=00:10:00
#SBATCH --partition=dav
#SBATCH --array=1-5
#SBATCH --output=job_array-%A_%a.out

export TMPDIR=/glade/scratch/$USER/temp
mkdir -p $TMPDIR

### Run program
./executable_name $SLURM_ARRAY_TASK_ID

Batch script to run a command file (MPMD) job

The example scripts below show how to submit a Slurm job using a multiple program configuration file

Command/configuration file

Each line of the configuration file specifies commands to run on one or more cores.  

Follow these steps:

  • First assign task numbers, either in a list (e.g., 0,1,2) or in a range (e.g., 0-2), followed by white space.
  • Next, indicate the command to be executed and any desired arguments 
  • Commands must not specify input/output redirection, but you can specify the desired input/output redirection in subcommands as explained below the text for the tcsh and bash script examples.

Here is an example of the basic syntax to use in your configuration file when each of four commands is to run once on a single core:

0 ./cmd1.exe
1 ./cmd2.exe
2 ./cmd3.exe
3 ./cmd4.exe

Submission scripts

Customize these examples by substituting your own job name, project code, and so on. When your script is ready, run sbatch to submit the job.

sbatch script_name

For tcsh users

#!/bin/tcsh
#SBATCH --job-name=MPMD_job
#SBATCH --account=project_code
#SBATCH --ntasks=4
#SBATCH --ntasks-per-node=4
#SBATCH --time=00:10:00
#SBATCH --partition=dav
#SBATCH --output=MPMD_job.out.%j

setenv TMPDIR /glade/scratch/$USER/temp
mkdir -p $TMPDIR

### Use the multiple program flag to specify command file job
srun --multi-prog ./tasks.conf

For bash users

#!/bin/bash -l
#SBATCH --job-name=MPMD_job
#SBATCH --account=project_code
#SBATCH --ntasks=4
#SBATCH --ntasks-per-node=4
#SBATCH --time=00:10:00
#SBATCH --partition=dav
#SBATCH --output=MPMD_job.out.%j

export TMPDIR=/glade/scratch/$USER/tmp
mkdir -p $TMPDIR

### Use the multiple program flag to specify command file job
srun --multi-prog ./tasks.conf

Command file input/output redirection

If input/output redirection is required, you can achieve this by writing commands cmd1.exe, cmd2.exe, and so on as scripts that call subcommands with the desired redirection. Here is one example:

#!/bin/tcsh
### bash users replace /tcsh with /bash -l
./subcmd1.exe < input1 > output1

Follow the same pattern for the additional subcommands.


Batch script for running dependent jobs

It is possible to schedule jobs to run based on the status of other jobs. One common way to schedule such a series or chain of jobs is to use command this to submit the job with the required dependencies:

sbatch --dependency=[job-dependency-expression] 

Here’s how to submit one job, then submit a dependent job while the first job is running: 

sbatch job.first
sbatch --dependency=afterok: job.next

The <jobID> in the second command is the one associated with the first job. In this case, the dependent job will start only if the first job completes successfully (with a zero exit code).

The following shows another way to run dependent jobs. The batch script in this example submits a dependent job if the first executable runs successfully.

For tcsh users

#!/bin/tcsh
#SBATCH --job-name=dep_job1
#SBATCH --account=project_code
#SBATCH --ntasks=2
#SBATCH --cpus-per-task=4
#SBATCH --time=00:10:00
#SBATCH --partition=dav
#SBATCH --output=dep_job1.out.%j

setenv TMPDIR /glade/scratch/$USER/temp
mkdir -p $TMPDIR

### Run first program
setenv OMP_NUM_THREADS 4
srun ./executable_name

### Run job 2 if program runs successfully
if ($status .eq. 0) then
   sbatch dep_job2
endif

For bash users

#!/bin/bash -l
#SBATCH --job-name=dep_job1
#SBATCH --account=project_code
#SBATCH --ntasks=2
#SBATCH --cpus-per-task=4
#SBATCH --time=00:10:00
#SBATCH --partition=dav
#SBATCH --output=dep_job1.out.%j

export TMPDIR=/glade/scratch/$USER/temp
mkdir -p $TMPDIR

### Run first program
export OMP_NUM_THREADS=4
srun ./executable_name

### Run job 2 if program runs successfully
if [[ $? == 0 ]]; then
   sbatch dep_job2
fi