Campaign Storage file system

Globus transfers | Data-access nodes | Casper use
Data retention policy | Allocations for labs and universities | Reports

NCAR Campaign Storage is a resource for storing data on publication timescales by NCAR labs and by universities that have project allocations. This disk-based file system is CISL's primary long-term data storage system now that the tape-based HPSS is nearing its end of life.

Campaign Storage is accessible a number of ways that are described below:

  • through the Globus web and command-line interfaces
  • from the data-access nodes, for Globus transfers and managing data holdings
  • from the Casper cluster to facilitate data analysis and visualization workflows in addition to data transfers from HPSS

Globus transfers

Globus transfers screen

The Globus endpoint established for the file system is NCAR Campaign Storage. How to make transfers to and from that endpoint is documented here:

Globus file transfers 

How to make transfers using the command line interface also is covered in detail in this tutorial:

Globus CLI tutorial

CAVEAT: The Globus interface for transferring data does not handle symbolic links and does not create symbolic links on a destination endpoint.


Data-access nodes

The Campaign Storage file system is mounted on the data-access nodes as /glade/campaign to:

  • enable users to manage file and directory permissions using POSIX commands.
  • facilitate transfers of small files to and from GLADE spaces such as /glade/scratch and /glade/work.
  • enable users to transfer data from HPSS.

Casper use

The Campaign Storage file system can be accessed from the Casper cluster as /glade/campaign so users are able to:

  • read and write data directly from their data analysis and visualization workflows.
  • submit batch scripts to migrate data from HPSS to the Campaign Storage resource.

Data retention policy

Campaign Storage has a five-year retention policy, meaning files will be purged five years after they are copied to the file system.

NCAR lab users are expected to collaborate with CISL’s Digital Asset Services Hub (DASH) to develop data migration plans for storage needs that exceed five years.

University users are expected to transfer their project data to their home institutions or other alternative storage spaces when their projects end or within the five-year time window, whichever is earlier.


Allocations

NCAR labs

Each NCAR lab has an allocation of Campaign Storage space and the labs manage how those allocations are used.

Universities

University users can request Campaign Storage space through the NCAR Resource Allocation System as supplements to their project allocations. Requests must include detailed justification for the amount of space requested.

Because NCAR is not currently funded to provide long-term data storage services to the university community, university users' requests for these allocations are prioritized based on the following factors.

Higher priority is given to requests if:

  • You have an active project, supported by an active NSF award, for using Cheyenne and you were previously awarded some HPSS storage to support this project as part of your original request.
  • Your HPSS data is associated with a project and NSF award that were completed within the last three (3) months.
  • Your request is for a period of no more than three (3) months and to support migrating of your data to your home institution.

Lower priority is given to requests if:

  • Your HPSS holdings are associated with projects or NSF awards that were completed more than three (3) months ago. 
  • Your need relates to satisfying external requirements or promises – to a publisher or agency, for example – to retain data for extended periods.

Any data requiring longer storage should be migrated to your home institution or to another appropriate repository.


Reports

The Systems Accounting Manager (SAM) provides overall summary information about the use of Campaign Storage allocations and other allocations.

CISL is developing additional tools for use in allocation management.