Python environment modules | Using the NCAR Package Library

Updated 11/6/2019

Multiple versions of Python are available on the Cheyenne and Casper systems. Additional packages for each Python version are available as part of virtual environments that we call the NCAR Package Library (NPL). The package libraries can be accessed via Python environment modules, which make NPL versions accessible.

You can see all NPL versions currently available in your computing environment by loading the ncarenv module and running the ncar_pylib command. You can also create personal clones of NPL versions and access them via ncar_pylib.


Python environment modules

Begin by loading a Python environment module before invoking the python command. Invoking python without loading a module first will start an older system version (Python 2.7.13) that is used mostly for system scripting and other administrative tasks.

To load an environment module that will provide a Python 3 environment for you to use on Cheyenne, follow this example:

module load python/3.6.8

If you load a Python 3 module but are not using the NCAR Package Library virtual environment as described below, start Python by running python3 rather than python.

python3

Python 3 is the community standard – and the default version on Cheyenne – but you can load Python 2 as an alternative by running:

module load python/2.7.16

Using the NCAR Package Library

After loading the desired Python module, load all NCAR-provided Python packages at once by activating the NCAR Package Library virtual environment. Then load the ncarenv module and run the ncar_pylib script to activate the default version of the package library. If you have a Python module loaded, the default version will be determined by that module. Otherwise, ncar_pylib will try to load your most recent clone. If no NPL versions are available, the command will make no changes to your compute environment and exit.

ncar_pylib

This utility can also be used to load older versions of the library as well as personal clones, which are listed by the date of creation. The listings also indicate whether your personal clones were created on Cheyenne or Casper. Activate a clone only on the system where it was created, and use the ncar_pylib utility. It will prevent you from mistakenly activating a virtual environment on the wrong system, which can result in runtime errors.

To list all versions of the package library:

ncar_pylib -l

To list the packages (and version numbers) that are included in a library version:

ncar_pylib -p 20190627

Note that you can specify a library version either by its name or its full path. If two personal clones share the same name, use the full path to indicate which library you want to load.

If a package that you want is not available in the current library, you can clone the library as described below and install the package for yourself using pip, or request a custom build by emailing cislhelp@ucar.edu.

When you are done and ready to exit the NPL virtual environment, run the command deactivate.

deactivate
11/16/2019 - Using ncar_pylib in batch jobs

The ncarenv module uses a shell alias to make sourcing libraries easier via ncar_pylib, but the alias does not carry over into batch jobs or other subshells. You can address this one of two ways:

  1. Load the ncarenv module and run ncar_pylib --setup. This will configure your shell startup files to create the alias every time you login, and you will have access to ncar_pylib anywhere on Cheyenne regardless of your module configuration.
  2. Restore the alias by reloading the ncarenv module in your script.

 

Creating your own clone of the NCAR Package Library

Creating a personal clone of the package library in your GLADE space is useful if you want to add or update packages, or develop Python code while using NCAR-provided packages. Simply specify the clone option (-c) to ncar_pylib, select a version of the package library, and choose a destination path for the clone.

ncar_pylib -c 20190627 /glade/work/$USER/my_npl_clone

Then, you can use the ncar_pylib command to activate the personal clone:

ncar_pylib my_npl_clone
OR
ncar_pylib /glade/work/$USER/my_npl_clone

Once the clone is activated, you can install or upgrade packages from the Python Package Index using the pip utility. For example, to install your own copy of the seaborn plotting library, you could run:

pip install seaborn

Creating Jupyter kernels from NPL versions

If you wish to use the NCAR Package Library inside of other Jupyter instances on Cheyenne (for example, inside a personal NPL clone, conda environment, or a JupyterHub instance), you can run the following command to produce a user-space kernel from an available NPL version:

ncar_pylib --kernel 20190627

This kernel will be stored in your home directory and available in any instance of Jupyter that you load.