Campaign Storage file system

Globus transfers | Data-access nodes | Casper use
Data retention policy | Allocations | Reports

Updated 11/7/2019: Reflects recent Casper and HPSS changes.

NCAR Campaign Storage is a resource for storing data on publication timescales by NCAR labs and by universities that have project allocations. This disk-based file system is CISL's primary long-term data storage system now that the tape-based HPSS is nearing its end of life.

Campaign Storage is accessible a number of ways that are described below:

  • through the Globus web and command-line interfaces
  • from the data-access nodes, for Globus transfers and managing data holdings
  • from the Casper cluster to facilitate data analysis and visualization workflows as well as data transfers from HPSS

Globus transfers

Globus transfers screen

The Globus endpoint established for the file system is NCAR Campaign Storage. How to make transfers to and from that endpoint is documented here:

Globus file transfers 

How to make transfers using the command line interface also is covered in detail in this tutorial:

Globus CLI tutorial

NOTE: The Globus interface for transferring data does not handle symbolic links and does not create symbolic links on a destination endpoint.


Data-access nodes

The Campaign Storage file system is mounted on the data-access nodes as /glade/campaign to:

  • enable users to manage file and directory permissions using POSIX commands.
  • facilitate transfers of small files to and from GLADE spaces such as /glade/scratch and /glade/work.
  • enable users to transfer data from HPSS.

Casper use

The Campaign Storage file system can be accessed from the Casper cluster as /glade/campaign so users can:

  • read and write data directly from their data analysis and visualization workflows.
  • submit batch scripts to migrate data from HPSS to the Campaign Storage resource.

Data retention policy

Campaign Storage has a five-year retention policy, meaning files will be purged five years after they are copied to the file system.

NCAR lab users are expected to collaborate with CISL’s Digital Asset Services Hub (DASH) to develop data migration plans for storage needs that exceed five years.

University users are expected to transfer their project data to their home institutions or other alternative storage spaces when their projects end or within the five-year time window, whichever is earlier.


Allocations

Each NCAR lab has an allocation of Campaign Storage space and the labs manage how those allocations are used.

University users can request Campaign Storage space through the NCAR Resource Allocation System as supplements to their project allocations. Requests must include detailed justification for the amount of space requested.


Reports

The Systems Accounting Manager (SAM) provides overall summary information about the use of Campaign Storage allocations and other allocations.

CISL is developing additional tools for use in allocation management.

 

Related training courses