Campaign Storage file system

Updated 9/17/2021: The data retention policy has been revised. CISL no longer intends to purge files five years after they are copied to the file system.

Globus transfers | Data-access nodes | Casper use
Data retention policy | Allocations for labs and universities | Reports

NCAR Campaign Storage is a resource for medium-term storage of project data, typically for three to five years, by NCAR labs and universities that have project allocations.

Campaign Storage is accessible a number of ways that are described below:

  • through the Globus web and command-line interfaces
  • from the data-access nodes, for Globus transfers and managing data holdings
  • from the Casper cluster to facilitate data analysis and visualization workflows

Globus transfers

Globus transfers screen

The Globus endpoint established for the file system is NCAR Campaign Storage. How to make transfers to and from that endpoint is documented here:

Globus file transfers 

How to make transfers using the command line interface also is covered in detail in this tutorial:

Globus CLI tutorial

CAVEAT: The Globus interface for transferring data does not handle symbolic links and does not create symbolic links on a destination endpoint.


Data-access nodes

The Campaign Storage file system is mounted on the data-access nodes as /glade/campaign to:

  • enable users to manage file and directory permissions using POSIX commands.
  • facilitate transfers of small files to and from GLADE spaces such as /glade/scratch and /glade/work.

Casper use

The Campaign Storage file system can be accessed from the Casper cluster as /glade/campaign so users are able to:

  • read and write data directly from their data analysis and visualization workflows.
  • submit batch scripts to migrate data to the Campaign Storage resource.

Data retention policy

Campaign Storage is designed to provide medium-term storage for project data, typically for three to five years. While data will not be purged automatically after five years, retaining data longer will reduce the capacity for storing additional, new data. Users are expected to monitor their holdings, remove files that are no longer needed, and move necessary data to other storage options for longer-term preservation.

NCAR researchers are expected to collaborate with CISL’s Digital Asset Services Hub (DASH) to develop data migration plans for storage needs that exceed five years.

University researchers are expected to transfer their project data to their home institutions or other alternative storage repositories within one year of their NSF grant expiring. CISL will not award storage space for researchers to carry data forward from one grant to another.


Allocations

NCAR labs

Each NCAR lab has an allocation of Campaign Storage space and the labs manage how those allocations are used.

Universities

University users can request Campaign Storage space through the NCAR Resource Allocation System as supplements to their project allocations. Requests must include detailed justification for the amount of space requested.

Because NCAR is not currently funded to provide long-term data storage services to the university community, university users' requests for these allocations are prioritized based on the following factors.

Higher priority is given to requests if:

  • You have an active project, supported by an active NSF award, for using Cheyenne.
  • Your request is for a period of no more than three (3) months and to support migrating of your data to your home institution.

Lower priority is given to requests if:

  • Your need relates to satisfying external requirements or promises – to a publisher or agency, for example – to retain data for extended periods.

Any data requiring longer storage should be migrated to your home institution or to another appropriate repository.


Reports

The Systems Accounting Manager (SAM) provides overall summary information about the use of Campaign Storage allocations and other allocations.

CISL is developing additional tools for use in allocation management.