Using the NCAR Data Sharing Service

The NCAR Data Sharing Service enables users to share files with anyone who has a Globus account. The service is for users who have access to NCAR storage resources such as GLADE and HPSS, typically through the Yellowstone system or via CISL's data-access nodes.

An individual does not need to have an NCAR user account or token to retrieve shared data.

Follow the links below to learn about getting access to the service, setting up endpoints for sharing data, and then actually sharing the data.


Sharing overview

The shared space is for temporary, short-term use. Data placed there are not backed up.

The default user quota is 50 TB; more space can be made available upon request.

NCAR is open to considering ideas for additional use cases. Email cislhelp@ucar.edu.

After you do the initial setup, you can share data by following these steps, which are described in detail below:

  1. Create a shared endpoint on Globus.
  2. Set permissions to define who should have access your endpoint. This might be certain individuals, groups of users, or anyone who has a Globus account.
  3. Copy files to the shared endpoint from your GLADE file spaces.
  4. Tell the individuals or groups where to find the data.

Initial setup

There are two steps to take before you can share files with this service:

  1. Request access to NCAR's "shared space."
  2. Create a Globus account with the appropriate username.

Requesting access to "shared space"

Request access to NCAR's Globus data-sharing space by sending an email to cislhelp@ucar.edu.

The CISL Help Desk will notify you when your personal directory has been created on the NCAR host endpoint, which is named: ncar#datashare

Your personal directory on that endpoint will be identified as a path like this: /data/share/bjsmith

You will need to use that information on a Globus Transfer screen later, as shown here:

Globus Transfer panel

Creating a Globus account with an appropriate username

Go to the Globus site to create an account.

Carefully consider your choice of Globus username. Any endpoint that you create will include that username.

For example, if John Smith chooses jsmith as his Globus username, any endpoint he creates with start with "jsmith" – as in jsmith#mydata.

A division or group name might be more appropriate in some cases. The CISL Consulting Services Group, for example, might set up a Globus account with the username cislcsg to have an endpoint named cislcsg#data.


Creating a group for sharing data (optional)

Individuals do not need to belong to a group in order for you to share data with them, but the group management features may be useful in some cases. For example, if you need to share data with several people, it can be more efficient to share with them as a group rather than individually.

See Globus Groups for how to establish a group and manage group member access. First, you will need to know what names and email addresses your colleagues used when creating their own Globus accounts. You will use that information to add them as group members. You can add or remove individuals from a group as needed.

 

Creating a shared endpoint - GUI

After your personal directory has been created on the NCAR host endpoint, you can create your own shared endpoints.

Follow these steps to use the Globus web interface to share your data (or refer to the CLI section below):

  • Log in on the Globus site and click Transfer Files on the Manage Data menu.
  • Select or type in the name of the NCAR host endpoint: ncar#datashare
  • Authenticate with your UCAR username and token. You will see a panel similar to the one shown below, with a folder identified with your UCAR username.

Globus Transfer panel

  • Double-click your personal directory.
  • Create a subfolder in that directory. (See the image below to locate the drop-down menu and create a new folder.)  
  • Select the subfolder and click share from the dropdown menu to make it a shared endpoint.

Globus Transfer panel

  • After you click share, name the endpoint on the following screen. (See image below.) Add a description.

Globus shared endpoint

  • Click Create and Manage Access at the bottom of the screen. (Continue below.)

Managing access and permissions

After you name your new endpoint and click Create and Manage Access (shown in the image just above), click Add Permission.

  • Enter the folder you are sharing. (Image below.)
  • Use the Share With buttons to designate who should have access—individuals, groups of users, or everyone who has a Globus account.
  • Check read and write as needed.

To change access and permissions later, go to Manage Endpoints.

  • Select the endpoint you created.
  • Click the Sharing tab.
  • Add or remove users and groups or make other changes as needed.

GO managing permissions

  • To share with additional users later, log in to Globus and select Manage Data, then Manage Endpoints.

 


Creating a shared endpoint - CLI

After your personal directory has been created on the NCAR host endpoint, you can create shared endpoints from the Globus command line.

The host endpoint also must be activated.

  • First, log in using SSH and your Globus username.
ssh username@cli.globusonline.org
  • Activate the host endpoint as shown here. You will be asked to authenticate with your username and password or token.
endpoint-activate ncar#datashare
  • Use the endpoint-add command as shown to create your own shared endpoint.
endpoint-add <name> --sharing <host-endpoint>/<directory>
  • In the following example, the endpoint-add command creates a shared endpoint named foldername in your personal shared directory.
endpoint-add foldername --sharing=ncar#datashare/data/share/username/
  • To see a list of endpoints that you have created or that have been shared with you, run the endpoint-list command.
endpoint-list

NCAR users' output will be similar to this:

username#mydir       -
go#ep1               -
go#ep2               -
ncar#datashare       -
ncar#gridftp         -

Additional CLI information

See Transferring files with the command line interface or enter help on your command line for additional information.


Copying data from GLADE to a shared endpoint

There are two ways to copy files from GLADE to a shared endpoint.

1. Copy with Globus

To copy files from a GLADE file space such as /glade/u/home/username to a shared endpoint, use Globus just as you would when transferring files to a non-NCAR system.

You can do that with the Globus GUI or CLI as documented here: Globus file transfers

2. Copy from data-access nodes

You can use standard Linux commands to copy files to your shared endpoints from CISL's data-access nodes, which mount both the GLADE file systems and the data-sharing file system.

Log in to data-access.ucar.edu the same way you would log in to yellowstone.ucar.edu.

When you log in, you are in /glade/u/home/username by default. You can change to your other GLADE directories as needed.

Copy the files that you want to share from your GLADE directory to the target for the shared endpoint that you created earlier, as shown in this example.

cp file1 file2 file3 /data/share/username/foldername

Telling users where to find the data

Notify the users or groups that files are available.

Give them:

  1. your endpoint name—example: bjsmith#mydata
  2. the path where your data can be found—example: bjsmith#mydata, /foldername
  3. this link to instructions for retrieving the data: http://www2.cisl.ucar.edu/docs/transfer/share#retrieve

Retrieving data from a shared endpoint

A user must have or create a Globus account in order to retrieve data from a shared endpoint. The user does not need to have an NCAR user account or token to retrieve data.

The user also needs to have access to a destination location, which can be:

  • Another endpoint such as an XSEDE or DOE host,
  • A Globus Connect Personal client running on a Linux, Windows or Mac host, such as the user’s laptop. See Globus Connect Personal for how to make your local machine an endpoint.

To copy files from the shared endpoint to the destination endpoint, see the Globus file transfers documentation.

Related training courses

NCAR courses