Platform LSF job script examples

Batch script examples | Dependent jobs | Waiting for batch jobs
Using array syntax | Using command files in batch jobs | Interactive jobs

When your script is ready, submit your batch job to Platform LSF for scheduling as shown in the Running jobs documentation.

Batch script examples

Script for pure MPI job

To submit a simple MPI batch job, see Submitting jobs.

Script for hybrid job

Use one of the following examples to create your batch script for submitting a hybrid MPI/OpenMP job to LSF.

For applications with large thread size, you may need to set environment variable KMP_STACKSIZE to avoid segmentation fault errors. Insert that before you invoke your executable. The default KMP_STACKSIZE on Yellowstone is 4 MB.

For tcsh users

#!/bin/tcsh
#
#BSUB -a poe                  # set parallel operating environment
#BSUB -P project_code         # project code
#BSUB -J hybrid_job_name      # job name
#BSUB -W 00:10                # wall-clock time (hrs:mins)
#BSUB -n 32                   # number of tasks in job
#BSUB -R "span[ptile=4]"      # run four MPI tasks per node
#BSUB -q regular              # queue
#BSUB -e errors.%J.hybrid     # error file name in which %J is replaced by the job ID
#BSUB -o output.%J.hybrid     # output file name in which %J is replaced by the job ID
 
setenv OMP_NUM_THREADS 4
setenv MP_TASK_AFFINITY core:$OMP_NUM_THREADS
 
mpirun.lsf ./program_name.exe

For bash users

#!/bin/bash
#
#BSUB -a poe                  # set parallel operating environment
#BSUB -P project_code         # project code
#BSUB -J hybrid_job_name      # job name
#BSUB -W 00:10                # wall-clock time (hrs:mins)
#BSUB -n 32                   # number of tasks in job
#BSUB -R "span[ptile=4]"      # run four MPI tasks per node
#BSUB -q regular              # queue
#BSUB -e errors.%J.hybrid     # error file name in which %J is replaced by the job ID
#BSUB -o output.%J.hybrid     # output file name in which %J is replaced by the job ID
 
export OMP_NUM_THREADS=4
export MP_TASK_AFFINITY=core:$OMP_NUM_THREADS   mpirun.lsf ./program_name.exe

Script for pure OpenMP job

Use one of the following examples to create your batch script for submitting a pure OpenMP job to LSF.

For applications with large thread size, you may need to set environment variable KMP_STACKSIZE to avoid segmentation fault errors. Insert that before you invoke your executable. The default KMP_STACKSIZE on Yellowstone is 4 MB.

For tcsh users

#!/bin/tcsh
#
#BSUB -a poe                              # set parallel operating environment
#BSUB -P project_code                     # project code
#BSUB -J omp_job_name                     # job name
#BSUB -W 00:10                            # wall-clock time (hrs:mins)
#BSUB -n 1                                # request one slot for this job
#BSUB -q regular                          # queue
#BSUB -e errors.%J.openmp                 # error file name in which %J is replaced by the job ID
#BSUB -o output.%J.openmp                 # output file name in which %J is replaced by the job ID
 
# request number of job threads up to and including 16
setenv OMP_NUM_THREADS 4
setenv MP_TASK_AFFINITY verbose,granularity=core
 
./program_name.exe

For bash users

#!/bin/bash
#
#BSUB -a poe                               # set parallel operating environment
#BSUB -P project_code                      # project code
#BSUB -J omp_job_name                      # job name
#BSUB -W 00:10                             # wall-clock time (hrs:mins)
#BSUB -n 1                                 # request one slot for this job
#BSUB -q regular                           # queue
#BSUB -e errors.%J.openmp                  # error file name in which %J is replaced by the job ID
#BSUB -o output.%J.openmp                  # output file name in which %J is replaced by the job ID
 
# request number of job threads up to and including 16
export OMP_NUM_THREADS=4
export MP_TASK_AFFINITY=verbose,granularity=core
 
./program_name.exe

Script for task geometry job

Consider submitting a task geometry job:

  • When a job’s workload or need for memory is not uniform, or
  • When you want to place frequently communicating tasks on the same node for efficiency. 

By default, the job scheduling system allocates an equal number of tasks to each node if the number of tasks in the job can be divided equally. If the number of tasks cannot be divided equally, fewer tasks are assigned to the last node.

Specifying task geometry lets you distribute tasks:

  1. Non-uniformly (for example, three tasks on the first node and 16 tasks on each of the other nodes), or
  2. Non-sequentially (for example, tasks 0, 16, and 19 on the
    first node; 1, 17, and 20 on a second node, and so on).

Use this sample script to create your own batch script for submitting a task geometry job. The executable could be either MPI or MPI/OpenMP.

The sample script requires three nodes. Tasks 0-7 run on the first node, 8-15 on the second, and 16-31 on the third.

If you revise the task geometry environment variable, you may also need to change the ‑n setting to 32×(desired number of nodes), to ensure the correct number of nodes. Note: the ptile=32 setting will not change.

#!/bin/bash
#
#BSUB -P project_code         # project code
#BSUB -J myjob     # job name
#BSUB -W 00:05                # wall-clock time (hrs:mins)
#BSUB -n 96                  # 32×(number of desired job nodes)
#BSUB -R "span[ptile=32]"     # run N MPI tasks per node as per task geometry
#BSUB -q small              # queue #BSUB -o myjob.%J.out # output file name in which %J is replaced by the job ID
#BSUB -e myjob.%J.err # error file name in which %J is replaced by the job ID
export MP_LABELIO="YES" export LSB_TASK_GEOMETRY="{(0,1,2,3,4,5,6,7)\
(8,9,10,11,12,13,14,15)\
(16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31)}"
mpirun.lsf ./hello_world.exe

Controlling the number of concurrent batch jobs running on a shared node

Users who submit tens or hundreds of jobs in rapid succession to a shared-node queue—those are the geyser and caldera queues on our Geyser and Caldera cluster—often notice that the jobs tend to pile up on one node instead of being distributing in round-robin manner across multiple nodes. The overloaded node may run slowly or even cease running altogether.

You can avoid overloading the nodes by stating your jobs' memory requirements when you submit them for scheduling. This lets you control the number of jobs that will run concurrently on a shared node. Jobs that don't run immediately on that node will remain pending until sufficient memory is available.

For example, you may decide that no more than two jobs should run concurrently, as in the sample scripts. The scripts reserve half of a Geyser node's usable memory. Caldera nodes' usable memory is different, so adjust accordingly if you're submitting to the caldera queue.

The scripts demonstrate how to do this for sequential jobs because individual users attempting to run a multitude of sequential batch jobs have been a major cause of shared-node job overload. Parallel jobs may require compound resource requirements as described in man bsub.

For tcsh users

#!/bin/tcsh
#
# This job reserves 0.5 TB of memory for job duration
# See "man bsub" for more information on rusage
#
#BSUB -P project_code             # project code
#BSUB -W 00:05 # wall-clock time (hrs:mins)
#BSUB -n 1 # number of tasks in job
#BSUB -R "span[ptile=1]" # run one MPI task per node
#BSUB -R "rusage[mem=512000]" # memory to reserve, in MB
#BSUB -J myjob # job name
#BSUB -o myjob.%J.out # output file name in which %J is replaced by the job ID
#BSUB -e myjob.%J.err # error file name in which %J is replaced by the job ID
#BSUB -q geyser # queue - must be either geyser or caldera ./myjob.exe

For bash users

#!/bin/bash
#
# This job reserves 0.5 TB of memory for job duration
# See "man bsub" for more information on rusage
#
#BSUB -P project_code             # project code
#BSUB -W 00:05 # wall-clock time (hrs:mins)
#BSUB -n 1 # number of tasks in job
#BSUB -R "span[ptile=1]" # run one MPI task per node
#BSUB -R "rusage[mem=512000]" # memory to reserve, in MB
#BSUB -J myjob # job name
#BSUB -o myjob.%J.out # output file name in which %J is replaced by the job ID
#BSUB -e myjob.%J.err # error file name in which %J is replaced by the job ID
#BSUB -q geyser # queue - must be either geyser or caldera

./myjob.exe

Batch script alternatives

You are not restricted to using regular login shells such as tcsh and bash to write batch scripts for submitting jobs through Platform LSF. Using more powerful languages can make some things—such as regular expressions and file manipulations—easier to handle inside a script.

Here are examples using Perl and Python.

LSF script in Perl

#!/usr/bin/perl -w
use strict;
#
#BSUB -W 00:10
#BSUB -q regular
#BSUB -n 12
#BSUB -J myjob.pl
#BSUB -P project_code
#BSUB -oo output

system("mpirun.lsf ./hello");

LSF script in Python

#!/usr/bin/python
import os
#
#BSUB -W 00:10
#BSUB -q regular
#BSUB -n 12
#BSUB -J myjob.py
#BSUB -P project_code
#BSUB -oo output

os.system("mpirun.lsf ./hello")

Dependent jobs

It is possible to schedule jobs to run only after other jobs have either started running or completed their runs. For example, you might schedule a preprocessing job to run; start a computation job when the preprocessing job is complete; then start a post-processing job when the computation is done.

To schedule such a series, use bsub -w [job-dependency-expression] to specify the job dependencies you need.

To illustrate, let's say you have you have three jobs to run:

  1. pre.lsf: a preprocessing job
  2. main.lsf: a computation job
  3. post.lsf: a post-processing job

The main job can be run only when the preprocessing job finishes, and the post-processing job can be run only when the computation job finishes.

Dependent jobs example

To schedule the jobs described above to run in sequence, follow this example.

  • Submit the preprocessing job script first:
    bsub < pre.lsf

    You will see a message like this when your job is submitted successfully:

    Job 2685 is submitted to default queue [regular].

    Note the job ID, which you will need for the next step.

  • Submit the main job, using the -w done job dependency option to indicate that it should start when the preprocessing job is done.
    bsub -w "done(2685)" < main.lsf

    You will see a message like this when the main job is submitted successfully:

    Job 2686 is submitted to default queue [regular].

    Again, note the job ID for the next step.

  • Submit the post-processing job, setting the final job dependency with the -w done option to indicate that it should start when the main job is done.
    bsub -w "done(2686)" < post.lsf

    You will see a message like this when your job is submitted successfully:

    Job 2687 is submitted to default queue [regular].
  • To monitor the jobs you submitted, use the bjobs command.
    bjobs

    Your output will be similar to this:

    JOBID  USER   STAT  QUEUE    FROM_HOST    EXEC_HOST     JOB_NAME SUBMIT_TIME
    2685   siliu  RUN   regular   yslogin1-ib 32*ys0101-ib  pre       Aug 11 14:06
    2686   siliu  PEND  regular   yslogin1-ib 32*ys0112-ib  main-job  Aug 11 14:37
    2687   siliu  PEND  regular   yslogin1-ib 32*ys0145-ib  post      Aug 11 15:12
    
  • To find out why an individual job is pending, enter bjobs -l followed by the job ID, as in this example:
    bjobs -l 2686

    Such a job typically would be pending because the job on which it depends has not met the required conditions.

Conditions and operators

Frequently used dependency conditions include the following:

  • done(job_ID | "job_name") indicates the job state is DONE (completed successfully).
  • started(job_ID | "job_name") indicates the job state is either RUN, DONE, or EXIT; or PEND or PSUSP, and it has a pre-execution command (bsub -E) that is running.
  • ended(job_ID | "job_name") indicates the job state is EXIT or DONE (completed, either successfully or unsuccessfully).
  • exit("job_name"[,[op] exit_code]) indicates the job state is EXIT (completed unsuccessfully), and its exit code satisfies the comparison test.

LSF also supports the following logic operators for complicated job-dependency control.

  • && (AND)
  • || (OR)
  • ! (NOT)

Waiting for batch jobs

Use the -K option in a batch script as shown here to submit jobs, wait for them to run to completion, and then take another action.

In this example, the output files from three jobs that run in the background will be archived using HTAR.

#!/bin/tcsh
#
bsub -K < run1.pl &
bsub -K < run2.pl &
bsub -K < run3.pl &

wait

bsub -q hpss -n 1 -W 00:30 -P project_code htar cvf output.tar output*

Using array syntax

Array syntax is an efficient way of to submit multiple jobs simultaneously. It is particularly useful when submitting large numbers of jobs—for ensemble forecasting and forecast evaluation, for example.

Here, the array [1-10] indicates you are submitting the myjob.csh job script 10 times.

bsub -J “myjob[1-10]” < myjob.csh

The output from a subsequent bjobs command demonstrates that all of the array elements have the same job ID, but element numbers are appended to the job name.

JOBID    USER      STAT  QUEUE     FROM_HOST     EXEC_HOST     JOB_NAME   SUBMIT_TIME

123456  username   RUN   caldera   caldera02-i   caldera05-i   myjob[1]   Oct  7 12:31
123456  username   RUN   caldera   caldera02-i   caldera05-i   myjob[2]   Oct  7 12:31
123456  username   RUN   caldera   caldera02-i   caldera05-i   myjob[3]   Oct  7 12:31
123456  username   RUN   caldera   caldera02-i   caldera05-i   myjob[4]   Oct  7 12:31
123456  username   RUN   caldera   caldera02-i   caldera05-i   myjob[5]   Oct  7 12:31
123456  username   RUN   caldera   caldera02-i   caldera05-i   myjob[6]   Oct  7 12:31
123456  username   RUN   caldera   caldera02-i   caldera05-i   myjob[7]   Oct  7 12:31
123456  username   PEND  caldera   caldera02-i                 myjob[8]   Oct  7 12:31
123456  username   PEND  caldera   caldera02-i                 myjob[9]   Oct  7 12:31
123456  username   PEND  caldera   caldera02-i                 myjob[10]  Oct  7 12:31

Script example

When you submit a job array, each array value is stored as variable $LSB_JOBINDEX, as shown in this sample script. This enables you to run an ensemble of forecasts by submitting one job and having the ensemble member (mem) equal $LSB_JOBINDEX.

#!/bin/csh
#
#BSUB -W 1
#BSUB -q regular
#BSUB -n 1
#BSUB -P project_code
#BSUB -R "span[ptile=32]"
#BSUB -J fcst
#BSUB -o wrf.%I
#BSUB -e wrf.%I

set mem = $LSB_JOBINDEX

touch -f mem_${mem}

exit 0

Array elements

Array elements do not have to be numbered consecutively as they appear in the sample output above [1-10].

Here are two alternatives that can be used as examples.

  • bsub -J “myjob[1,3-6,42,90]” < myjob.csh passes values of 1,3,4,5,6,42, and 90 to $LSB_JOBINDEX
  • bsub -J “myjob[1,10:2]” < myjob.csh passes values of 1,3,5,7, and 9 to $LSB_JOBINDEX

Killing jobs

To kill an entire array, just specify the single job number:

bkill 123456

To kill a single element of an array, include the element number. In this example, you're killing job myjob[5].

bkill “123456[5]”

Limiting the number of concurrent jobs

When it's desirable to limit how many array members can run concurrently, specify the limit by adding %val in the submission. In this example, no more than 15 jobs will run simultaneously.

bsub -J “myjob[1-50]%15” < myjob.sh

Dependency conditions

Arrays can be used in dependency conditions. In this example, myjob2 won't execute until all elements of myjob1 are done.

bsub -J “myjob[1-10]” < myjob1.sh
bsub -w “ended(myjob[1-10])" -J “myjob2[1-10]” < myjob2.sh

Using command files in batch jobs

Using a command file when submitting an LSF batch job can help you run numerous independent tasks in parallel and make the most efficient use of Yellowstone compute nodes.

If you have a serial data processing script, for example, and need to run many copies of it to process multiple data files, you can create a command file and job script that let you pack each node with the appropriate number of tasks and run them all in parallel.

A command file used in Multiple Program, Multiple Data (MPMD) jobs can include any set of independent commands, scripts, or other executables. They can be intermixed and in any order but are most cost-effective if they have similar wall-clock requirements.

Here are three examples of jobs that use command files.

Example 1

This example shows how to run a simple LSF batch job using a command file in which each command is a set of shell commands or scripts. This is an example of a so-called "embarrassingly parallel" workload—the commands are independent; none of them write or modify data that the other commands require as input.

Create a command file

Create a text file that lists the commands that you want to run, one command per line. In this example we assume you have a set of tcsh commands cmd1, ..., cmd30 and a set of input files input1, ..., input30 residing in your job directory. The file should not include blank lines or comment lines, which will cause the job to fail with the message "ERROR: 0031-634 Wrong command line format."

We saved the following text as "cmd_file.txt" for this example.

tcsh -c "./cmd1 < input1 > output1; ls -l output1"
tcsh -c "./cmd2 < input2 > output2; ls -l output2"
tcsh -c "./cmd3 < input3 > output3; ls -l output3"

...intervening lines deleted for brevity...

tcsh -c "./cmd28 < input28 > output28; ls -l output28"
tcsh -c "./cmd29 < input29 > output29; ls -l output29"
tcsh -c "./cmd30 < input30 > output30; ls -l output30"

Create an LSF batch file

Job control is achieved primarily by LSF directives #BSUB -n and #BSUB -R "span[ptile=" and the command file itself. In this example, we specify ptile=16 so the job will run most efficiently on Yellowstone by using all of the cores on the node. If your tasks require more memory than is available with that setting, the job will be killed with a TERM_MEMLIMIT error message and you will need to specify a smaller ptile value.

If you specify more tasks with #BSUB -n than are included in your command file, the job will fail and you will not see any expected output from the tasks. It may return Exited with exit code 2, a similar error message, or no error message at all.

Customize this sample job script as needed with your own project code, job name, command file name, and so on. We saved this as "job.lsf" for this example.

#!/bin/tcsh
#BSUB -P project_code            # your project code, such as UUOM4321 or P87654321
#BSUB -n 30                      # number of tasks in job
#BSUB -R "span[ptile=16]"        # run tasks per node
#BSUB -J myjob                   # your name for the job
#BSUB -W 0:04                    # wall-clock time (hrs:mins)
#BSUB -o myjob.%J.out            # your output file name in which %J is replaced by the job ID
#BSUB -e myjob.%J.err            # your error file name in which %J is replaced by the job ID
#BSUB -q regular                 # queue

# set environment variable
setenv MP_PGMMODEL mpmd

#run the executables identified in the designated command file
mpirun.lsf -cmdfile /glade/u/home/username/cmd_file.txt

Submit the LSF batch job

After creating your own command file and LSF batch file job.lsf , submit your job with a bsub command as shown here:

bsub < job.lsf

Retrieve output from the batch job

Your output from the example should be similar to this, except for the order of the "task" lines.

Execute poe command line: poe -cmdfile poe.cmdfile
 task 15
 task 20
 task 7
 task 4
 task 29
...remaining lines omitted for brevity...

Example 2

This example shows how to run a simple LSF batch job using a command file. In this case—another example of the "embarrassingly parallel" workload—the commands are independent; none of them write or modify data that the other commands require as input.

Create the executable

Substitute your own executable or use this Fortran 90 source code as an example for testing purposes. See Compiling code for the command to use.

The "hello world" MPI executable, which we named "hw.exe" in this example, will output your job's task numbers and node IDs.

program hw
use mpi
integer :: nt, it, len, err, buff
character(128) :: pname
call MPI_Init( err )
call MPI_Comm_size( MPI_COMM_WORLD, nt, err )
call MPI_Comm_rank( MPI_COMM_WORLD, it, err )
if(it.eq.0) buff = 1234
call MPI_Get_processor_name( pname, len, err )
call MPI_Bcast( buff, 1, MPI_INTEGER, 0, MPI_COMM_WORLD, err )
print *, 'hello world from task ',it,' of ',nt,' running on ',trim(pname)
call MPI_Finalize(err)
end program hw

Create a command file

Create a text file that lists the commands that you want to run, one command per line. The command file should not include blank lines or comment lines, which will cause the job to fail with the message "ERROR: 0031-634 Wrong command line format."

In the example, hw.exe is run four times, but other executables could be included in the file. We saved the following text as "cmd_file.txt" for this example.

./hw.exe
./hw.exe
./hw.exe
./hw.exe

Create an LSF batch file

Job control is achieved primarily by LSF directives #BSUB -n and #BSUB -R "span[ptile=" and the command file itself. In this example, the job runs four tasks, with two tasks per node, but it is easily modified for different numbers of tasks and nodes.  To run 11 "hello world" tasks on one node, you would modify the LSF job so that n is 11 and ptile is 11, and you would include 11 instances of hw.exe in the command file.

If you specify more tasks  with #BSUB -n than are included in your command file, the job will fail and you will not see any expected output from the tasks. It may return Exited with exit code 2, a similar error message, or no error message at all.

Customize this sample job script as needed with your own project code, job name, command file name, and so on. We saved this as "job.lsf" for this example.

#!/bin/tcsh
#BSUB -P project_code            # your project code, such as UUOM4321 or P87654321
#BSUB -n 4                       # number of tasks in job
#BSUB -R "span[ptile=2]"         # run 2 MPI tasks per node
#BSUB -J myjob                   # your name for the job
#BSUB -W 0:04                    # wall-clock time (hrs:mins)
#BSUB -o myjob.%J.out            # your output file name in which %J is replaced by the job ID
#BSUB -e myjob.%J.err            # your error file name in which %J is replaced by the job ID
#BSUB -q regular                 # queue

# set environment variable
setenv MP_PGMMODEL mpmd

#run the executables identified in the designated command file
mpirun.lsf -cmdfile /glade/u/home/username/cmd_file.txt

Submit the LSF batch job

After creating your own executable, command, and LSF batch files, submit your job with a bsub command as shown here:

bsub < job.lsf

Retrieve output from the batch job

Your output from the example should be similar to this, but identifying the nodes that your program used.

Execute poe command line: poe -cmdfile poe.cmdfile
 hello world from task            0  of            4  running on ys0318
 hello world from task            1  of            4  running on ys0318
 hello world from task            2  of            4  running on ys3117
 hello world from task            3  of            4  running on ys3117

Example 3

This example shows how to construct and run an LSF batch job using a command file that contains multiple independent MPI executables which may require different processor counts. This may be useful to users who have MPMD job needs greater than the "embarassingly parallel" examples above. In this example, "multiple" executables are illustrated with two executables.

Create the executables hw.exe and wh.exe

As in Example 2 above, substitute your own MPI executables or use the following Fortran 90 MPI source code as an example for testing purposes. See Compiling code for the command to use.

Use the "hello world" MPI executable from Example 1 for your first executable, named "hw.exe." The second executable—"wh.exe"—is similar to the first one except for the file name and the print line, which will say "world hello" rather than "hello world." Here is the source code for the second executable.

program wh
use mpi
integer :: nt, it, len, err, buff
character(128) :: pname
call MPI_Init( err )
call MPI_Comm_size( MPI_COMM_WORLD, nt, err )
call MPI_Comm_rank( MPI_COMM_WORLD, it, err )
if(it.eq.0) buff = 1234
call MPI_Get_processor_name( pname, len, err )
call MPI_Bcast( buff, 1, MPI_INTEGER, 0, MPI_COMM_WORLD, err )
print *, 'world hello from task ',it,' of ',nt,' running on ',trim(pname)
call MPI_Finalize(err)
end program wh

Create a command file

Create a text file that lists the commands that you want to run, one command per line. The command file should not include blank lines or comment lines, which will cause the job to fail with the message "ERROR: 0031-634 Wrong command line format."

In the example, hw.exe is run four times and wh.exe is run twice. We saved the following text as "cmd_file.txt" for this example. Each line in cmd_file.txt contains the name of the executable and the number of tasks.

./hw.exe:4
./wh.exe:2

Create an LSF batch file

Job control is achieved primarily by LSF directives #BSUB -n and #BSUB -R "span[ptile=" and the command file itself. In this example, the job runs six tasks on one node, but it is easy to modify for different numbers of tasks and nodes by changing the -n and ptile values and cmd_file.txt.

Customize this sample job script as needed with your own project code, job name, command file name, and so on. We saved this as "job.lsf" for this example.

#!/bin/tcsh
#
#BSUB -a poe
#BSUB -P project_code
#BSUB -J myjob
#BSUB -W 00:05
#BSUB -n 6
#BSUB -R "span[ptile=6]"
#BSUB -q regular
#BSUB -o myjob.%J.out
#BSUB -e myjob.%J.err

setenv MP_CMDFILE cmd_file.txt
setenv MP_PGMMODEL mpmd

mpirun.lsf

Submit the LSF batch job

After creating your own executables, command, and LSF batch files, submit your job with a bsub command as shown here:

bsub < job.lsf

Retrieve output from the batch job

Your output from the example should be similar to this, but identifying the nodes that your own programs used.

Execute poe command line: poe
 world hello from task            4  of            6  running on ys0101
 world hello from task            5  of            6  running on ys0101
 hello world from task            3  of            6  running on ys0101
 hello world from task            2  of            6  running on ys0101
 hello world from task            0  of            6  running on ys0101
 hello world from task            1  of            6  running on ys0101

Interactive jobs

Geyser and Caldera

CISL recommends using the Geyser and Caldera clusters for most interactive jobs. See Using Geyser and Caldera for documentation.

Yellowstone

Jobs that you choose to run on Yellowstone compute nodes are charged for all 16 cores on each node used, regardless of how many cores per node actually are used. 

If you do need to start an interactive session on Yellowstone, submit an LSF request specifying the "small" queue as shown just below or see our debugging documentation for information. Interactive use of the "small" queue is intended for parallel debugging and other tasks that will be making full use of the nodes.

bsub -Is -q small -W 1:00 -n number_of_tasks -P project_code $SHELL

When the job starts, load the appropriate module or modules for applications that you want to run in your shell—IDL or MATLAB, for example.

module load matlab

Execute the appropriate command to start your application.