Compiling code

Compilers | GPU compilers | Changing compilers | Where to compile | Native commands

Revised 10/16/2018

Cheyenne users have access to Intel, PGI, and GNU compilers. The Intel compiler module is loaded by default, which makes some Intel tools available by default as well.

After loading the compiler module that you want to use, identify and run the appropriate compilation wrapper command from the table below. (If your script already includes one of the following generic MPI commands, there is no need to change it: mpif90, mpif77, ftn; mpicc, cc; mpiCC and CC.)

Also consider using the compiler's diagnostic flags to identify potential problems.

Compiler man pages

To get the man page for a compiler, log in to the system where you intend to use it, load the module, then execute man for the compiler. 

module load pgi
man pgfortran

 

Compiler Language Commands for serial programs Commands for programs
using MPI
Flags to enable OpenMP
(for serial and MPI)
Intel (default) Fortran ifort foo.f90 mpif90 foo.f90 -qopenmp
  C icc foo.c mpicc foo.c -qopenmp
  C++ icpc foo.C mpicxx foo.C -qopenmp

Include these flags for best performance when you use the Intel compiler:
-march=corei7 -axCORE-AVX2

PGI Fortran pgfortran (or pgf90 or pgf95) foo.f90 mpif90 foo.f90 -mp
  C pgcc foo.c mpicc foo.c -mp
  C++ pgcpp (or pgCC) foo.C mpicxx foo.C -mp
GNU Fortran gfortran foo.f90 mpif90 foo.f90 -fopenmp
  C gcc foo.c mpicc foo.c -fopenmp
  C++ g++ foo.C mpicxx foo.C -fopenmp

GPU compilers

To compile CUDA code to run on the data analysis and visualization clusters, use either NVIDIA's nvcc compiler command for CUDA C and C++ code, or use PGI for CUDA Fortran code.

For detailed compilation examples, see:


Changing compilers

Should you prefer to use a compiler other than the Intel default compiler, use module swap to make the change.

In this example, you are switching from Intel to PGI:

module swap intel pgi

When you load a compiler module, the system makes other compatible modules available. This helps you establish a working environment and avoid conflicts. If you need to link your program with a library*, use module load to load the library as in this example:

module load netcdf

Then, you can just invoke the desired compilation command without adding link options such as -l netcdf. Here's an example:

mpif90 foo.f90

You can learn more about how using modules helps you manage your environment on our Environment modules page.


Where to compile

The use of multiple types of processors and operating systems in the Cheyenne environment makes your choice of where to compile your code especially important. 

System Processor
Cheyenne Intel Broadwell
Geyser Intel Westmere
Caldera Intel Sandy Bridge
Casper Intel Skylake

Cheyenne uses SUSE Enterprise Linux, while the analysis and visualization clusters run CentOS. The different operating systems provide different versions of some standard libraries, which may be incompatible with each other.

Even if you take care to build your programs for portability, you will achieve superior performance if you compile your code on the cluster where you intend for it to run.

Compile on Cheyenne if…

  • You want to aggressively optimize CPU performance.
  • You will run your code only on Cheyenne.

You can compile your code in a batch job in the "economy" or "regular" queue using your login environment (not a login node) if needed.

Compile on Geyser if…

  • You want to ensure that your code will run on Geyser.

Executables that are compiled on Geyser will run on Caldera or Casper, but likely slower than if they were compiled on those clusters.

Compile on Caldera if...

  • You will run your code on Caldera.

It will also run on Casper, but slower. Code compiled on Caldera might not run on Geyser.

Compile on Casper if...

  • You want to ensure that your code will run on Casper nodes.
  • You want to use the latest CPU optimizations in Intel's Skylake architecture.
  • Your programs use GPU tools like OpenGL, CUDA, and OpenACC.

Native commands

We recommend using the module wrapper commands described above. However, if you prefer to invoke the compilers directly, unload the NCAR default compiler wrapper environment by entering this on your command line:

module unload ncarcompilers

You can still use the environment variables that are set by the modules that remain loaded, as shown in the following examples of invoking compilers directly to compile a Fortran program.

Intel compiler

ifort -o a.out $NCAR_INC_<PROGRAM> program_name.f $NCAR_LDFLAGS_<PROGRAM> $NCAR_LIBS_<PROGRAM>

PGI compiler

pgfortran -o a.out $NCAR_INC_<PROGRAM> program_name.f $NCAR_LDFLAGS_<PROGRAM> $NCAR_LIBS_<PROGRAM>

GNU/GCC family of compilers

gfortran -o a.out $NCAR_INC_<PROGRAM> program_name.f $NCAR_LDFLAGS_<PROGRAM> $NCAR_LIBS_<PROGRAM>

To determine the correct name to substitute for <PROGRAM> in these commands, you can grep each of them as shown here.

env | grep NCAR_INC
env | grep NCAR_LDFLAGS
env | grep NCAR_LIBS

For reference, the table below shows the native commands for each of the compilers for Fortran, C, and C++ programs.

Compiler Fortran C C++
Intel ifort foo.f90 icc foo.c icpc foo.C
PGI pgfortran (or pgf90, pgf95) foo.f90 pgcc foo.c pgcpp foo.C
GNU gfortran foo.f90 gcc foo.c g++ foo.C

* In addition to multiple compilers, CISL keeps available multiple versions of libraries to accommodate a wide range of users' needs. Rather than rely on the environment variable LD_LIBRARY_PATH to find the correct libraries dynamically, we encode library paths within the binaries when you build Executable and Linkable Format (ELF) executables. To do this, we use RPATH rather than LD_LIBRARY_PATH to set the necessary paths to shared libraries.

This enables your executable to work regardless of updates to new default versions of the various libraries; it doesn't have to search dynamically at run time to load them. It also means you don't need to worry about setting the variable or loading another module, greatly reducing the likelihood of runtime errors.