Using the NCAR Data Sharing Service

The NCAR Data Sharing Service enables users who have access to NCAR storage resources – such as GLADE, CISL's data-access nodes, and HPSS – to share files with anyone who has a Globus account.

An individual does not need to have an NCAR user account or token to retrieve shared data.

To retrieve data: Skip to Retrieving data from a shared endpoint at the bottom of this page.

To share data: Follow the links below to learn how to get access to the service, set up endpoints for sharing data, and share your data.


Sharing overview

The shared space is for temporary, short-term use. Data placed there are not backed up. Files that are not accessed for 45 days are deleted automatically.

The default user quota is 50 TB. Upon request, more space can be made available temporarily.

NCAR is open to considering ideas for additional use cases. Please share them by contacting CISL.

After you do the initial setup, you can share data by following these steps, which are described in detail below:

  1. Create a shared endpoint on Globus.
  2. Set permissions to define who can access your endpoint. This might be groups of users or any individual who has a Globus account.
  3. Copy files to the shared endpoint from your GLADE file spaces.

Initial setup

Start by taking these two steps:

  1. Request access to NCAR's "shared space."
  2. Create a Globus account with the appropriate username.

Requesting access to "shared space"

Request access to NCAR's Globus data-sharing space by contacting CISL.

The CISL Help Desk will notify you when your personal directory has been created on the NCAR host endpoint, which is named: NCAR Data Sharing Service. Your personal directory on that endpoint will be identified as a path like this: /glade/p/datashare/your_username. You will use that information on a Globus Transfer screen later.

Creating a Globus account with an appropriate username

Go to the Globus site to create an account.

Carefully consider your choice of Globus username. A division or group name might be appropriate in some cases.


Creating a group for sharing data (optional)

Individuals do not need to belong to a group in order for you to share data with them, but the group management features may be useful in some cases. For example, if you need to share data with several people, sharing with them as a group may be more efficient than sharing with each of them individually.

See Globus Groups to establish a group and manage group member access. You will need to know what names and email addresses your colleagues used when creating their own Globus accounts. You will use that information to add them as group members. You can add or remove individuals from a group as needed.


Creating shared endpoints - GUI

You can create your own shared endpoints after your personal directory has been created on the NCAR host endpoint.

Follow the steps below to share your data through the Globus web interface or refer to the CLI section.

  • Log in on the Globus site and go to the File Manager.
  • Use the Collection search field to locate the "NCAR Data Sharing Service" endpoint.
  • Authenticate with your UCAR username and token. You will see a panel similar to the one shown below, with a folder identified with your UCAR username.
  • Double-click your personal directory.
  • Create a subfolder in that directory.
  • Select the subfolder and click share to make it a shared endpoint.

File Manager screen

  • After you click share, name the endpoint on the following screen in the Share Display Name field (below). Name this new endpoint carefully; you will need to give it to users with whom you want to share the data.
  • Add a description.

Sharing an endpoint.

  • Click Create share at the bottom of the panel. (Continue below.)

Managing access and permissions

Use the Sharing tab to share your endpoint with others.

  • Use the Share With buttons (see image below) to designate who should have access –  individuals, groups of users, or everyone who has a Globus account.
  • Check read and write as needed.
  • Click Add Permissions.

  • After setting up the permissions, notify those with whom you are sharing the data by sending the link provided.

Sharing link

  • To share with additional users later, log in to Globus and select Endpoints.

Creating shared endpoints - CLI

After your personal directory has been created on the NCAR host endpoint, you can create shared endpoints from the Globus command line.

ssh username@data-access.ucar.edu
  • Create a subfolder in your personal directory.
mkdir /glade/p/datashare/$USER/subfolder
  • Run globus endpoint create --shared as shown below to make the subfolder a shared endpoint. The command includes the NCAR Data Sharing Service endpoint ID, your personal directory, and the name of the subfolder.
globus endpoint create --shared dd1ee92a-6d04-11e5-ba46-22000b92c6ec:/glade/p/datashare/$USER/share subfolder
Message: Shared endpoint created successfully
Endpoint ID: 69b8bd1c-6825-11e8-928b-0a6d4e044368
  • The following example creates a personal endpoint for use with Globus Connect Personal, where you will need to use the Setup Key that is provided in the output.
globus endpoint create --personal your_endpoint_name
Message: Endpoint created successfully
Endpoint ID: ed88e748-6818-11e8-928b-0a6d4e044368
Setup Key: 3325301c-6c0f-4986-a3f7-45d1b5d246ad

NCAR endpoint IDs and display names

Run globus endpoint search ncar to get a list of NCAR-related endpoints. The output will include an ID for each endpoint as well as its display name, which is used in the web interface. Here's an example showing only the first few lines of output:

globus endpoint search ncar
ID                                   | Owner                     | Display Name
------------------------------------ | ------------------------- | -------------------------------------
6b5ab960-7bbf-11e8-9450-0a6d4e044368 | ncar@globusid.org         | NCAR Campaign Storage
dd1ee92a-6d04-11e5-ba46-22000b92c6ec | ncar@globusid.org         | NCAR Data Sharing Service
d33b3614-6d04-11e5-ba46-22000b92c6ec | ncar@globusid.org         | NCAR GLADE

List shared endpoints

To see endpoints that have been shared with you or endpoints that you created for sharing with others, run this command with the ID of the NCAR Data Sharing Service host endpoint:

globus endpoint my-shared-endpoint-list

Example:

globus endpoint my-shared-endpoint-list dd1ee92a-6d04-11e5-ba46-22000b92c6ec

ID                                   | Owner                | Display Name 
------------------------------------ | -------------------- | -------------------
69b8bd1c-6825-11e8-928b-0a6d4e044368 | bjsmith@globusid.org | bj_shared_ep
a9f57e22-9e8a-11e5-99a1-22000b96db58 | bjsmith@globusid.org | CSG Shared Endpoint
90b28228-6a33-11e6-83a9-22000b97daec | bjsmith@globusid.org | Example

Additional CLI information

See Transferring files with the command line interface or enter help on your command line for additional information.


Copying files from GLADE to a shared endpoint

There are two ways to copy files from GLADE to a shared endpoint.

1. Copy with Globus

To copy files from a GLADE file space such as /glade/u/home/username to a shared endpoint, use Globus just as you would when transferring files to a non-NCAR system.

You can do that with the Globus GUI or CLI as documented here: Globus file transfers

2. Copy from data-access nodes

You can use standard Linux commands to copy files to your shared endpoints from CISL's data-access nodes, which mount both the GLADE file systems and the data-sharing file system.

Log in to data-access.ucar.edu the same way you would log in to an NCAR/CISL HPC system.

When you log in, you are in /glade/u/home/username by default. You can change to your other GLADE directories as needed.

Copy the files that you want to share from your GLADE directory to the target for the shared endpoint that you created earlier, as shown in this example.

cp file1 file2 file3 /glade/p/datashare/$USER/subfolder

Retrieving data from a shared endpoint

A user must have or create a Globus account in order to retrieve data from your shared endpoint. The user does not need to have an NCAR user account or token to retrieve data.

Give the user the name of the shared endpoint that you created, not "NCAR Data Sharing Service."

The user also needs to have access to a destination location. That location can be:

  • another endpoint such as an XSEDE or DOE host, or
  • a Globus Connect Personal client running on a Linux, Windows or Mac host, such as the user’s laptop. See Globus Connect Personal for how to make your local machine an endpoint.

To copy files from a shared endpoint to a destination endpoint, see the Globus file transfers documentation.

Related training courses