Use and storage policies

Recommended use | Data storage policies

This page was revised 9/8/2020.

1/20/2020 - HPSS is now read-only.
11/14/2019 - HPSS will be put into read-only mode on January 20, 2020.
10/15/2019 - Users advised to move or delete all HPSS data now
 
NCAR’s High-Performance Storage System (HPSS) will reach its end of life on October 1, 2021. Users are advised to begin moving their data to an alternative storage system and deleting it from HPSS immediately.
 

 

Recommended use

HPSS is best used for long-term storage of data sets larger than 100 MB. It is not for short-term storage of temporary data or for use as a disk-based file server. See File size guidelines for HPSS for additional information that will help you use this system efficiently.

Users are encouraged to work within the GLADE disk-based storage environment for short-term storage of data that are being created or analyzed, and for supporting data workflows across CISL resources. The GLADE file spaces are accessible from our HPC, data analysis, and visualization systems.

The NCAR Campaign Storage file system is the recommended alternative for storing data on publication timescales.

Do not use HPSS to back up your laptops or desktops.
Contact your desktop support team about backup options.

Concurrent transfer limits

The number of concurrent transfers an individual user can execute is limited in order to help ensure that all users have reasonable access to HPSS. There also is a global limit on concurrent file actions that sometimes results in users' receiving "EIO" error notices – even if they have not exceeded their individual limit – if the system is especially busy.

To reduce the incidence of such errors, follow these recommendations:

  • Do not submit more than one hsi cget -RA command to run concurrently. Each such command requests multiple file actions.
  • Request no more than five (5) individual file actions to be executed concurrently, whether using hsi or htar and regardless of where you start the transfers (Cheyenne, Casper, data-access nodes, for example).
  • Do not submit more than one cget command that requests retrieval of multiple files by using wildcards (such as hsi cget *file) in either a single session or in multiple sessions.
  • Be aware that HPSS may still be busy executing an action after it appears to you to have been completed. By running commands in quick succession, you may inadvertently reach your limit because some actions remain in progress. Also, once HPSS begins to execute an action, it may continue to completion even if you cancel the command.
  • If you encounter EIO errors, submit fewer htar commands simultaneously or in rapid succession. A read opens both an index file and a tar file, so you might reach your limit sooner than you expect.

HPSS and outside systems

Transferring data between HPSS and computers outside of the UCAR security perimeter is a two-stage process: downloading/uploading to disk, then using one of our data transfer methods (for example, Globus, SCP, or SFTP) to make the transfer.


Data storage policies

HPSS creates one copy of each file by default. These files are not backed up within HPSS or in any other repository. You are responsible for replicating any data that you feel should be stored at an additional location.

warning iconFiles deleted from or overwritten on HPSS cannot be recovered.

Retention

URGENT: If you have files on HPSS that you need to preserve, be aware that HPSS is nearing end of life. See Migrating files from HPSS for what to do.

Requesting file deletions

In a few situations, you may need to request CISL assistance in modifying or deleting files from HPSS. Here are two examples:

  • If you have files in HPSS but no longer have access to CISL resources (and have no need for access), simply ask that we delete or change ownership of the files.
  • A project lead may ask us to delete or reassign ownership of files that a former user created. If the former user still has an active login and another active project on CISL resources, work directly with the user to make the necessary changes. Otherwise, contact us with a compelling reason to make an exception to the standard procedure.

In both cases, CISL staff will verify your identity before deleting any data.

Curation

If data sets that you no longer need are of special community interest or have historical importance, contact the CISL Digital Asset Services Hub (DASH) to discuss the potential for curation, preservation, and storage.

Data integrity

While HPSS is a highly reliable storage system, NCAR does not guarantee that your data are immune to loss or damage. In the highly unlikely event that your files are affected by breakage of a storage tape, we may not be able to restore them entirely.

Any potential loss is more likely to be the result of mistakenly removing or overwriting stored files. To avoid such losses, please review our documentation on permissions.