Removing large numbers of files

The recommended way to remove thousands or hundreds of thousands of files from a GLADE directory is by running a batch job on the Geyser cluster.

Removing large numbers of files can take several hours, so you will need to provide enough wall-clock time in the job to accommodate this. The wall-clock limit for the geyser queue is 24 hours.

You can use one of the sample scripts below with your own user name, project code, job name, and other customizations.

Caution: Before removing large numbers of files, create a "play" directory in /glade/scratch/username and try the batch job with some fictional files and subdirectories to make sure that it does what you want. Carefully specify the files that you want removed before submitting a job like this.

Cheyenne users

 

The Cheyenne script will be available soon.

Yellowstone users

 

#!/bin/tcsh
#BSUB -P project_code        # project code
#BSUB -N                     # send email when job is complete
#BSUB -W 12:00               # wall-clock time (hrs:mins)
#BSUB -n 1                   # number of tasks in job
#BSUB -R "span[ptile=1]"     # run one task per node
#BSUB -J job_name            # job name
#BSUB -o job_name.%J.out     # output file name in which %J is replaced by the job ID
#BSUB -e job_name.%J.err     # error file name in which %J is replaced by the job ID
#BSUB -q geyser              # queue

# See "man rm" for explanation of options f and v
rm -fv /glade/scratch/username/directory_name/files_to_remove*

Related training courses