Removing large numbers of files

The recommended way to remove thousands or hundreds of thousands of files from a GLADE directory is by running a batch job on the Geyser cluster.

Removing large numbers of files can take several hours, so you will need to provide enough wall-clock time in the job to accommodate this. The wall-clock limit for the geyser queue is 24 hours.

You can use this sample script with your own project code, job name, and other customizations. Carefully specify the files that you want removed before submitting the job.

#BSUB -P project_code        # project code
#BSUB -N                     # send email when job is complete
#BSUB -W 12:00               # wall-clock time (hrs:mins)
#BSUB -n 1                   # number of tasks in job
#BSUB -R "span[ptile=1]"     # run one task per node
#BSUB -J job_name            # job name
#BSUB -o job_name.%J.out     # output file name in which %J is replaced by the job ID
#BSUB -e job_name.%J.err     # error file name in which %J is replaced by the job ID
#BSUB -q geyser              # queue

rm -f /glade/scratch/username/file*

Caution: Before removing large numbers of files, create a "play" directory in /glade/scratch/username and try the batch job with some fictional files and subdirectories to make sure that it does what you want.

Related training courses