Using HTAR to transfer files

How to use HTAR | HTAR help | Advanced use | Options

The HPSS Tape Archiver (HTAR) packages large numbers of files into a single archive file for efficient transfer to our High Performance Storage System (HPSS).

CISL recommends using HTAR rather than HSI if you need to archive large numbers of files that are smaller than 1 MB. Failure to follow this guideline can result in suspension of your access to HPSS. Transferring hundreds or thousands of small files with HSI commands significantly slows the entire system for all users. HTAR is more efficient for archiving and reduces the time it takes to retrieve files later.

How HTAR works:

  1. A single command (examples below) packages multiple individual "member" files on your file system into a single archive file and sends it to HPSS.
  2. The archive file ensures that all the member files are stored on a single tape, which is efficient and convenient.
  3. HTAR creates and saves an index file on HPSS (extension .idx) along with the archive file.
  4. You can retrieve individual specified member files later if you need to without copying the whole archive file back to your file system.

See File size guidelines for HPSS for additional information that will help you use this system efficiently.

Concurrent transfer limits

HPSS is shared by a large number of users, and there are individual and global limits on file actions that can be executed concurrently. See Use and storage policies for details regarding these limits.


How to use HTAR

Creating an archive file

warning icon

Caution: When you create an archive file using a file name that already exists in HPSS, HTAR will overwrite the older file without warning.

To create an archive file named myncfile.tar with all .nc files in the current directory and store the file in your HPSS home directory, follow this example.

htar -cv -f myncfile.tar *.nc 

To create an archive file named mydata.tar with the directory MyData and store the file in your HPSS home directory, follow this example.

htar -cv -f mydata.tar MyData 

Submitting a job to create an archive file

You can submit a job to create an archive file, as in this example. Be sure to use your own project code, file and directory names, and so on.

exechpss -a project_code "htar -cv -f mydata.tar MyData"

Also see Confirming HPSS transfers.

Listing archive file contents

Use the HTAR options shown here to list the contents of an archive file.

htar -tv -f myncfile.tar 

Retrieving files

You can retrieve individual files or an entire archive file using the htar command.

Caution: HTAR will overwrite local files without warning if they have the same names as the files that you are retrieving.

When fetching files from HPSS, make sure you have sufficient room in your GLADE file space. If you exceed your GLADE quota, the transfer will fail.

In this example, you are retrieving my_output0.nc from the myncfile.tar archive file.

htar -xv -f myncfile.tar my_output0.nc 

You would retrieve all member files from myncfile.tar like this:

htar -xv -f myncfile.tar 

Submitting a job to retrieve files

You can retrieve an archive file, or retrieve individual files from the archive, by submitting a job. This example shows how to retrieve an archive file. Be sure to use your own project code, file and directory names, and so on.

exechpss -a project_code "htar -xv -f mydata.tar"

Also see Optimizing HPSS file retrieval.

    Overspent storage allocations

    HPSS users who attempt to write files to the tape archive when a project's storage allocation is overspent will receive one of two error messages indicating that the command has failed:

    • Output from an HSI command such as cput or put will include the string HPSS_EDQUOT.
    • Output from an htar command will include ERROR: Error -88.

    If transfer commands are executed in a batch script, which makes such failures hard to detect, confirm that the transfers are completed successfully as described here: Confirming HPSS transfers.

    Once a project's storage allocation is overspent, users must reduce their holdings before they can write additional files. To check the status of the HPSS storage allocation for a project, log in to the Systems Accounting Manager (sam.ucar.edu) and select Reports, then My Account Statements. The statements are updated each Monday following Sunday HPSS accounting runs, so some deletions may not be reflected until several days after they are made.


    HTAR help

    See Getting help with HPSS.

    Also see the HTAR web page for more information, including links to all of the man pages and a user guide.


    Advanced use

    Soft delete

    To soft delete the member file my_output0.nc from the archive file myncfile.tar, follow this example. Run the htar command in the second line to see the updated list of file contents.

    htar -Dv -f myncfile.tar my_output0.nc
    htar -tv -f myncfile.tar 

    Undelete

    To undelete the file my_output0.nc from the archive file myncfile.tar, follow this example. Run the htar command in the second line to see the updated list of file contents.

    htar -U -f myncfile.tar my_output0.nc
    htar -tv -f myncfile.tar 

    Build a new index file

    Follow this example to build an HTAR index (or overwrite the index) for an archive file. You may want to do this if you created a large archive file using tar rather than htar and then ran an HSI command such as cput to put it on HPSS. Creating an index for it with HTAR will facilitate retrieval of individual files.

    htar -Xv -f myncfile.tar 

    Commonly used options

    Here are some of the most commonly used HTAR options. To review the full list of options and their descriptions, enter either htar -? (bash and ksh users) or htar -\? (csh and tcsh users).

    -c Creates a new archive file.
    -D Soft-deletes member files from the archive file (flags files as <deleted> in the index file).
    -f filename Specifies archive file name (required option).
    -K Verifies the contents of an existing archive file.
    -t

    Lists contents of archive file, using the index file.

    -U Undeletes soft-deleted member files in the archive file.
    -x Extracts files from the archive file to local file(s).
    -X Builds a new index file by reading an existing archive file.

     

     

     

    Related training courses