File size guidelines for HPSS

Archiving files in the High Performance Storage System's robotic tape libraries is much different from writing files to disk storage such as the GLADE file spaces. File size can have a major impact on how efficiently or inefficiently files are written to HPSS, so keep these guidelines in mind when you need to archive data.

Preferred file size range

File sizes in the gigabyte range are preferred for storing in HPSS. A few files of hundreds of gigabytes each make the most efficient use of the system when you are transferring files between HPSS and GLADE.

Managing small files

Avoid transferring many small files – those in the megabyte range or smaller. Moving numerous individual files to and from tape is inefficient and time consuming and can slow the system for all users.

When you need to store many small files, use one of these two approaches:

  • Use HTAR to transfer them together as a single archive file. HTAR can bundle individual “member” files as large as 68 GB into one archive file and store it on HPSS.
  • If you need to create an archive with any member files that are larger than 68 GB, use a tar command to bundle the member files and then transfer the resulting tar file to HPSS with the HSI cput command.

File size limits

Transferring files that are 1 TB or larger increases the risk of poor system performance as well as the risk, while still low, of losing a file that contains a large amount of data. While the actual file size limit in HPSS is 5 TB, CISL recommends storing files that are no larger than 1 TB.

Also keep in mind that the peak transfer rate for HPSS tape drives is approximately 160 MBps, so retrieving a 5 TB file from tape may take nine hours or more.