- Data Portals
- User Support
- About Us
The HSI interface is one of two primary tools for transferring data to and from HPSS within the Yellowstone environment. If you need to use HSI on NCAR systems that are outside of the Yellowstone environment, see Kerberos and HSI.
If you need to archive large numbers of individual files, use HTAR instead of HSI.
See File size guidelines for HPSS for additional information that will help you use this system efficiently.
Failure to follow these guidelines can result in suspension of your access to HPSS.
Also see HSI Help.
Files that are deleted from or overwritten in HPSS cannot be recovered. To avoid losing data, carefully review the Cautions section below.
HPSS is shared by a large number of users, and there are individual and global limits to the number of file actions that can be executed concurrently. See Use and storage policies for details regarding these limits.
Also see Optimizing HPSS file retrieval. Configuring your requests based on tape location can result in quicker retrievals in some cases.
If you need to perform an operation on 100,000 or more of your HPSS files (with commands such as chacct, chcos, chgrp, cp, mv, and rm, among others), contact the CISL Help Desk or Consulting Services Group for assistance so we can avoid slowing the system for all users.
See Submitting HPSS file metadata change requests for details that you will need to provide.
Review and follow CISL best practices for managing your files and data transfers. They will help you make the most efficient use of your computing and storage allocations.
HSI can be invoked in several modes, each of which is described here.
If you are working with our HPC, analysis, or visualization systems*, just enter hsi on your command line to start an HSI session.
Once your session starts, your command prompt will look like this, with your own username.
You can execute commands from there to archive or retrieve files and so on. See “Commands and examples” below.
To leave the HSI environment and return to your shell prompt, enter quit.
Run HSI commands from your shell prompt without starting an interactive HSI session first.
Precede your commands with “hsi” as in this example:
-bash-4.1$ hsi cput xxx : yyy
As shown in the following video, the cput command will be executed in an HSI session, then control will return to the shell. This is how you would put HSI commands in a script, or in a “system” call from a running program.
Another batch mode option is to create a file that contains the desired HSI commands and then execute one of the following:
-bash-4.1$ hsi in filename
-bash-4.1$ hsi < filename
Batch (LSF) and cron jobs can use HSI in the same way as the interactive and non-interactive jobs described above. On Yellowstone, use the hpss queue for these jobs.
Here is one example of how to submit a job on Yellowstone to execute an HPSS transfer:
-bash-4.1$ bsub -n 1 -q hpss -W 2:00 -P project_code hsi cget mydata
HSI commands include some familiar ones, such as cp and ls, that resemble their Linux and UNIX counterparts.
Some HSI commands, however, have additional options. For example, the HSI command ls has option -U for identifying the project code associated with a file. We recommend getting familiar with each command's options to be sure that you get the results you want when managing your HPSS holdings. Some of the most frequently used HSI commands are discussed below. A complete list is available here.
Executing this command in an interactive HSI session writes a file from your current working directory in GLADE to your home directory in the HPSS archive.
[HSI]/home/username-> cput filename
To write the file to HPSS with a different name, follow this example.
[HSI]/home/username-> cput filename : newfilename
Absolute path names for local or HPSS files also are acceptable. The local file always comes before the colon with both the cput and cget commands.
To put a set of files into a target directory, create the directory if it doesn't already exist, cd into it, and then run cput.
[HSI]/home/username-> cd /home/username/targetdir; cput file_pattern
UNIX users sometimes try to do the following, where targetdir is an existing directory (or a directory that is to be created with the -P option). HSI does not support this:
[HSI]/home/username-> cput file_pattern : /home/username/targetdir
You can specify a source with the recursive option (-R) in HSI, but you cannot specify a target with it. If you try, the command will interpret your source, your target, and the token “:” all as sources. This can produce unexpected and even damaging results.
Correct: The proper way to use this command is to change to the target directory first, then execute your cput command with the -R option. For example, to put the local directory mydir into an HPSS target directory /home/username/test, run this from your command line.
-bash-4.1$ hsi “cd /home/username/test; cput -R mydir”
The result is a directory /home/username/test/mydir that contains all the files and directories from your local mydir directory.
Incorrect: Here’s an example of how this often is done incorrectly by trying to write the source file tree rooted at mydir to /home/username/test/mydir:
-bash-4.1$ hsi cput mydir : /home/username/test/mydir
You will get an error message recognizing that mydir is a directory and telling you that you need to use the -R option. Take care to execute this correctly, as shown above, or you risk overwriting valuable data.
Use cget to retrieve an HPSS file into your current working directory on your local machine:
[HSI]/home/username-> cget filename
To read the HPSS file into your current working directory with a different name, follow this example:
[HSI]/home/username-> cget newfilename : filename
Also see Confirming HPSS transfers.
To change ownership of a file or files, submit a request to email@example.com.
Users do not have the necessary permissions to run the chown command.
Use the chmod command to set or change permissions on your files and directories to protect your data or to give others access to them. This is often done recursively with the –R option.
See these links for detailed examples:
Use the UNIX chgrp command to change the associated group for existing files and directories. This is often done recursively with the –R option.
[HSI]/home/username-> chgrp newgroup myfile
[HSI]/home/username-> chgrp -R newgroup mydir
See this link for more details: Changing user and group for files/directories
Use the newgrp command to change your current effective group within an HSI session.
[HSI]/home/username-> newgrp groupname
To see what your current effective group is, just enter the command with no arguments.
If you need to change your default primary group for HPSS, request the change in an email to firstname.lastname@example.org.
See this link for more information: Users and groups (HPSS)
Use the HSI chacct command if you need to change the project code—or “account ID,” in HPSS—with which existing files and directories are associated. A project code is used for the purpose of charging against your HPSS storage allocation.
Enter the command and desired project code, and identify the relevant files as shown in this example. This is often done recursively with the –R option.
[HSI]/home/username-> chacct UABC0001 myfile
[HSI]/home/username-> chacct -R UABC0001 mydir
See Projects and charges for more information.
Use the HSI newacct command to associate an individual HSI session with a project code that is not your default project code.
Enter the command and the project code that you want to use for your session as shown in this example.
[HSI]/home/username-> newacct UABC0001
See Projects and charges for more information.