Globus file transfers

Web interface transfers | CLI transfers | Globus Connect Personal

A Globus account enables users to transfer files between NCAR resources such as GLADE and XSEDE facilities or other sites. It has both a web interface and a command line interface (CLI). CISL recommends using Globus as an alternative to GridFTP.

To use either interface, first create a personal Globus ID account. That is the preferred account type, as some Globus features such as SSH Key management are not available if you use an organizational login.

Globus also offers a feature called Globus Connect Personal, which enables users to move files to and from a laptop or desktop computer and other "endpoints."

Several public endpoints are set up on the Globus system for transferring files to and from the GLADE disk storage system. They include:

  • NCAR GLADE (legacy name: ncar#gridftp)
  • NCAR Data Sharing Service (legacy name: ncar#datashare)
  • XSEDE NCAR GLADE (legacy name: xsede#ncar) for users who have XSEDE credentials.

Legacy names for endpoints are used in the Globus CLI.


Transferring files with the web interface

When transferring files between systems, keep in mind that your username might not be the same on each system.

Follow these steps to transfer files as shown in the video below.

  • Go to the main Globus page (globus.org) and use your Globus ID to log in. (UCAR and NCAR staff should not use the NCAR RDA organizational login.)
  • Go to File Transfer.
  • Enter your source endpoint on one side of the panel (see screen image below).
  • Specify the path where your source files are located and click Go.
  • Enter your username and token response or password when you are asked to authenticate.
  • Identify your target endpoint in the panel on the right.
  • Specify a destination path.
  • Select the files you want to copy.
  • Click the arrow button to initiate the transfer.

You can check the status of your transfers any time through the web interface and will be notified when they are complete.

Globus

Globus provides access to users' GLADE file spaces and the system's data-access nodes. Include your username in the path as shown. Changing to the parent directories /glade/scratch or /glade/u/home can result in long delays and timeouts because of the numerous child directories in each of those spaces.

Video demonstration

Start the video, then use the button at lower right to watch on YouTube.com or full-screen.


Transferring files with the command line interface

The Globus CLI can be used to make both manual and unattended file transfers. The latter feature is particularly useful if you want your job scripts to transfer files to another system without your manual intervention—at the end of a run, for example.

To prepare to use the CLI, generate an SSH key on the system that you will use to initiate transfers, then add that key to Globus. That process is described here: Making unattended file transfers.

Getting help

To get help with Globus CLI commands:

Log in to the system through which you have access to GLADE, then ssh to cli.globusonline.org with your Globus username. Your Globus username may be different from your system username.

ssh username@cli.globusonline.org

Enter help and consult the help or man pages.

Transfer command example

Use the transfer command to transfer files from one endpoint to another as in the following example. The example assumes that you have activated the endpoints.

transfer -- ncar#gridftp/glade/scratch/username/filename colorado#gridftp/home/username/filename

Your output will look like the following.

Task ID: 91a6f759-da1c-11e3-b49e-22000a971261
Created transfer task with 1 file(s)
$

The transfer command can be made recursive by adding the -r option. When using -r, the source and destination paths must end with "/" for the request to work.

Use the -s sync-level option with the transfer command to synchronize the source and destination by copying only files that are new or have been updated, with functionality similar to rsync.

Here is a basic shell script example for transferring files recursively.

#!/bin/bash
ssh username@cli.globusonline.org transfer -- ncar#gridftp/glade/u/home/username/sourcedir/ colorado#gridftp/home/username/destdir/ -r

See the transfer man page (ssh to cli.globusonline.org) for additional details.

Also see Command Line Interface.


Globus Connect Personal

To set up your laptop or desktop computer to use Globus Connect Personal for transferring files:

  • Go to Globus Connect Personal and follow the instructions to download and install it on your local system.
  • Add your local system as an endpoint by following the instructions from the Globus Connect website.
  • Start Globus Connect, and then sign in to globus.org.

Your local system should now appear as an endpoint that can be used for transferring files.

Related training courses