Python – NCAR Package Library

Python environment modules | Using the NCAR Package Library | Cloning the NPL

Updated 6/11/2020: Some version numbers were updated and the pip example was revised.

Multiple versions of Python are available on the Cheyenne and Casper systems. Additional packages for each Python version are available as part of virtual environments that we call the NCAR Package Library (NPL). The package libraries can be accessed via Python environment modules, which make NPL versions accessible.

You can see all NPL versions currently available in your computing environment by loading the ncarenv module and running the ncar_pylib command. You can also create personal clones of NPL versions and access them via ncar_pylib.


Python environment modules

Begin by loading a Python environment module before invoking the python command. Invoking python without loading a module first will start an older system version (Python 2.7.13) that is used mostly for system scripting and other administrative tasks.

To load an environment module that will provide a Python 3 environment for you to use on Cheyenne, follow this example:

module load python/3.7.5

If you load a Python 3 module but are not using the NCAR Package Library virtual environment as described below, start Python by running python3 rather than python.

python3

Python 3 is the community standard – and the default version on Cheyenne – but you can load Python 2 as an alternative by running:

module load python/2.7.16

Using the NCAR Package Library

After you have loaded the desired Python module as described above, load the NCAR-provided Python packages by activating the NCAR Package Library virtual environment.

To activate the virtual environment:

  • Load the ncarenv module.
module load ncarenv
  • Run the ncar_pylib script to activate the default version of the package library.
ncar_pylib

The script can also be used to load older versions of the library and personal clones. How to create a personal clone is described below.

To list all versions of the package library:

ncar_pylib -l

To list the packages (and version numbers) that are included in a library version:

ncar_pylib -p 20200417

Note that you can specify a library version either by its name or its full path. If two personal clones share the same name, use the full path to indicate which library you want to load.

The listings indicate whether your personal clones were created on Cheyenne or Casper. Activate a clone only on the system where it was created, and use the ncar_pylib utility. It will prevent you from mistakenly activating a virtual environment on the wrong system, which can result in runtime errors.

If a package that you want is not available in the current library, you can clone the library as described below and install the package for yourself using pip, or submit a request to have the package added by contacting CISL.

When you are done and ready to exit the NPL virtual environment, run the command deactivate.

deactivate
Using ncar_pylib in batch jobs

The ncarenv module uses a shell alias to make sourcing libraries easier via ncar_pylib, but the alias does not carry over into batch jobs or other subshells. You can address this one of two ways:

  1. Load the ncarenv module and run ncar_pylib --setup. This will configure your shell startup files to create the alias every time you login, and you will have access to ncar_pylib anywhere on Cheyenne regardless of your module configuration.
  2. Restore the alias by reloading the ncarenv module in your script.

 

Creating your own clone of the NCAR Package Library

Creating a personal clone of the package library in your GLADE space is useful if you want to add or update packages, or develop Python code while using NCAR-provided packages. Simply specify the clone option (-c) to ncar_pylib, select a version of the package library, and choose a destination path for the clone.

ncar_pylib -c 20200417 /glade/work/$USER/my_npl_clone

Then, you can use the ncar_pylib command to activate the personal clone:

ncar_pylib my_npl_clone
OR
ncar_pylib /glade/work/$USER/my_npl_clone

Once the clone is activated, you can install or upgrade packages from the Python Package Index using the pip utility. For example, to install your own copy of the seaborn plotting library, run:

pip --no-cache-dir install seaborn

Including --no-cache-dir ensures that pip doesn't install cached, out-of-date package builds that may not reflect the software environment on the system.

Removing an unwanted NPL clone

Locate the directory of the clone you want to remove by running ncar_pylib -l. Example output:

LIBRARY NAME    PYTHON   ORIGIN    SYSTEM    PATH
 ------------------------------------------------------------------
my_npl_clone1   3.7.5    20200417  Cheyenne  /glade/work/bfeller/my_npl_clone1
my_npl_clone2   3.7.5    20200417  Cheyenne  /glade/work/bfeller/my_npl_clone2

If you no longer need the second clone listed, execute command rm -fr on that path as follows. This will take a while to finish, so you may want to execute it in the background.

rm -fr /glade/work/$USER/my_npl_clone2 &

The next time you run ncar_pylib, the script will detect that the clone is gone and ask you if you want to remove it from your listings. Answer yes. When you run ncar_pylib -l again, you will see the clone has been deleted.

Creating Jupyter kernels from NPL versions

If you wish to use the NCAR Package Library inside of other Jupyter instances on Cheyenne (for example, inside a personal NPL clone, conda environment, or a JupyterHub instance), you can run the following command to produce a user-space kernel from an available NPL version:

ncar_pylib --kernel 20200417

This kernel will be stored in your home directory and available in any instance of Jupyter that you load.


Machine learning / deep learning

The information about machine learning and deep learning has been moved to this web page.