HPSS permissions

Permissions and data safety | Directory permissions | File permissions | UMASK

1/20/2020 - HPSS is now read-only.
11/14/2019 - HPSS will be put into read-only mode on January 20, 2020.
10/15/2019 - Users advised to move or delete all HPSS data now
NCAR’s High-Performance Storage System (HPSS) will reach its end of life on October 1, 2021. Users are advised to begin moving their data to an alternative storage system and deleting it from HPSS immediately.

The HPSS archive that CISL manages employs HSI commands, which give the system a UNIX-like interface, including the traditional UNIX classes (user, group and other) for file and directory permissions.

The HSI commands include some familiar ones, such as cp and ls, that resemble their Linux and UNIX counterparts. Some HSI commands, however, have additional options. We recommend getting familiar with them to be sure that you get the results you want when managing your HPSS holdings. See Getting help with HPSS.

Permissions and data safety

Setting HPSS permissions and using HSI commands correctly are critical. Some commands can result in data loss, and the files and directories that you own or share in HPSS are not backed up.

Be particularly careful with these commands:

warning icon cp  mv  put  get  rm warning icon

The cp, mv, put, and get commands can overwrite data at their targets without warning. This causes a problem if you mistakenly remove or overwrite data, because it cannot be recovered. The cput and cget commands are safer alternatives to put and get because they never overwrite files.

The HSI rm command operates like the normal UNIX rm command to remove files. The operation cannot be undone, and there are no file backups. In some Linux distributions, rm defaults to rm -i and interactively asks for confirmation for each file being deleted. There is no such default in HSI, but you can include the -i option.

The HSI cp, mv, put, get, and rm commands execute an "unlink" operation (a file delete). Except for rm, this is not necessarily true of their counterparts in many UNIX systems.

The implications are important:

Write permissions on files do not protect you against data loss from these commands. File protection is a result of permissions on the parent directory.

Adding data to or deleting data from a directory requires updating that directory, and thus requires write permission on the directory. An unlink operation deletes data, and therefore requires write permission on the parent directory.

See Managing files with HSI for examples of how to use HSI commands.

Directory permissions

Read (r) permission allows users to list the files that are in the directory.

Write (w) permissions on directories allow users (owner, groups, and others) to create and delete files in those directories. This is true even if the users do not have write permission for the files, which would be unusual.

Execute (x) permission gives users full access to the files and directories that a directory contains, meaning users can cd to the directory and edit files, if they know the name of the files they wish to edit and if each of the files has suitable permissions allowing the user to edit.

For most common use cases, both read and execute permissions are needed on a directory if users are to read files or perform other operations in the directory or one of its children.

Sticky bit (t) permission on a directory protects your files from others who are able to write into the directory, and it prevents other users from deleting or overwriting each other’s files in that directory. The owner of the directory can delete files that others have written.

If you enable the writing permissions on a directory, set the sticky bit if you want to prevent others from editing or deleting your files or subdirectories.

Example 1

To set the sticky bit on an existing directory, use the chmod command followed by +t and the directory name.

chmod +t directory

The directory permissions then will include an upper-case T in place of the last execute bit, which is not set. For example, if you previously had given yourself and the “ncar” group read, write and execute permissions on that directory, but no permissions for others, command hsi ls -ld directory would report the following:

drwxrwx--T 3 username ncar 512 Jun 5 11:20 directory

Had you given execute permission for others, you would see a lower-case t rather than T in the last bit permission.

We recommend setting the sticky bit as a matter of course every time you create a new directory that you might later decide to share with others in HPSS. The following example shows how to do that explicitly.

hsi mkdir directory
hsi chmod +t directory

Example 2

You can set the sticky bit implicitly in other cases by using the -P option to create intermediate directories, or by using the -R option for a recursive cput command.

Using the chmod -d option when setting permissions recursively with the -R option sets the sticky bit for the directories but not for the files they contain. Here is an example:

hsi cput -P directory/intermediate/directories/do/not/exist/myfile
hsi chmod -d -R +t directory

Example 3

This is similar to Example 2 except that a directory is being placed on HPSS:

hsi cput -R sourcedir : targetdir
hsi chmod -d -R +t sourcedir

File permissions

Why do HPSS file permissions matter if file overwrite protection is determined by directory permissions? Two reasons:

  1. File permissions set the rules for what happens when a user opens a file or updates the file's metadata.

  2. When you run a cget or get command, the HPSS file’s permissions determine the permissions that are applied to the target UNIX client file (subject to your local UNIX umask settings).

You can change file ownership and permissions after the fact, but establishing them when creating a file reduces your chances of mistakes.

Example 1

We recommend that you always set the permissions on a file directly after writing it; do not depend on default behavior. For example, to put myfile into mydir and set permissions so you have read, write, and execute permissions on the file, and the group has read and execute permission, but no one else has write permission, execute the following from your command line.

hsi cput myfile : mydir/myfile
hsi chmod u=rwx,g=rx,o=r mydir/myfile

If necessary, you can also set more restrictive permissions for group or owner than you do for others (instead of more permissive, as is common). In the following example, group members other than the owner will be able to read and execute the files, whereas others will be unable to read, write, or execute them.

hsi cput myfile : mydir/myfile
hsi chmod u=rwx,g=rw,o-rwx mydir/myfile

Example 2

Your HPSS home directory /home/username was created with permissions u=rwxt,go-rwx. The u=rwxt means you own the directory and have read-write-execute permission on the directory. The go-rwx means no other user can navigate in your home directory, see files in that directory, or have any access to the files.

Those settings provide you with utmost security from deletion and searching by the group and others. This is suitable unless you want to share files with your group or others, or if you request help from the CISL Consulting Services Group and the consultant needs to see your files and directories. Following are two examples of how you might change those default settings to allow more access to your files.

To allow group members but no one else to see all your files:

hsi chmod -R u=rwx,g=rx,+t /home/username

To allow group members and others to see all of your files:

hsi chmod -R u=rwx,g=rx,o=rx /home/username

Example 3

Users of the HSI chmod command who are familiar with the POSIX chmod command may notice some differences in the way permissions are implemented.  For example, you might expect this invocation to remove group and owner write permissions for the designated file, but it does not work:

hsi chmod go+-w /home/username/myfile

This simpler equivalent command does remove the write permissions::

hsi chmod go-w /home/username/myfile

Best practice: Run some test examples to verify that you are setting file and directory permissions exactly as you desire before you take your project into production.


HPSS uses normal umask semantics, where the umask value sets bits that deny certain permissions but do not enable any. The umask value is combined with the permission bits on your HPSS file to determine:

  • how permissions are set on your client machine after reading a file from the archive,

  • and how permissions are set on HPSS when you write a file to the archive.

Your umask settings are client-specific and may differ from one client machine to another, so your default permission settings might differ, too.


This example demonstrates why you need to know your UMASK settings in both HPSS and GLADE to understand the behavior of the cget and get commands.

You belong to an HPSS group that we will call “groupname.” Your colleague Bob is a member of the same group and has created a file named “datafile” in /home/bob/subdirectory. When he runs the command shown here, he gets the output showing the file permissions.

hsi ls -l /home/bob/subdirectory/datafile
-rwxrwx--- 3 bob groupname 512 Jun 5 11:20 datafile

When you retrieve the file to your own GLADE home directory and list the file to examine the permissions, you see that your umask settings have changed them. Your disk copy of the file also has a newer timestamp.

hsi cget datafile: /home/bob/subdirectory/datafile
ls -l datafile
-rwxr-xr-x 3 username ncar 512 Jun 7 11:50 datafile