Opportunity for university researchers on Derecho supercomputer

Updated 9/14/2021 – The National Center for Atmospheric Research (NCAR) will soon be accepting proposals from university-led teams to participate in the Accelerated Scientific Discovery (ASD) program on the new Derecho supercomputer. NCAR will deploy Derecho, its next-generation HPC system, early in 2022.

NCAR's Derecho supercomputer logoThe ASD program provides a unique opportunity for a small group of large-scale computational projects to have nearly exclusive use of the Derecho system for a few months. During this time, ASD teams will work closely with NCAR’s high-performance computing staff to optimize their workflows, ensure that the environment is properly configured for maximum efficiency, and demonstrate the capabilities of the 19.87-petaflops system.

For university participants in the ASD program, NCAR will make about 225 million core-hours available to five or six CPU-oriented projects and 225,000 GPU-hours to three or four GPU-oriented projects. The bulk of the ASD computational runs should be completed during the first two months after Derecho is accepted – that is, in May and June 2022 as currently planned. 

The minimum scale of CPU-oriented projects is 30 million core-hours and the minimum for GPU-oriented projects is 50,000 GPU-hours. Larger-scale proposals and hybrid CPU/GPU proposals are encouraged. NCAR will provide storage resources to support ASD projects during the computational and analysis phases; long-term storage plans should be described in the proposal.

Estimating Derecho allocation needs

Derecho users can expect to see a 1.3x improvement over the Cheyenne system's performance on a core-for-core basis. Therefore, to estimate how many CPU core-hours will be needed for a project on Derecho, multiply the total for a Cheyenne project by 0.77.

When requesting an allocation for Derecho GPU nodes, please make your request in terms of GPU-hours (number of GPUs used x wallclock hours). Derecho GPU-hour estimates can be based on any reasonable GPU performance estimate from another system, including Casper.

Consulting and visualization support

Accelerated Scientific Discovery (ASD) projects will be provided with assistance on Derecho via dedicated staff members from the Consulting Services Group. In addition, ASD projects will be granted allocations on the Casper cluster as well as staff assistance in constructing visualizations to improve understanding and presentation of results. Please describe plans or estimated needs for data analysis and visualization resources for your project.

ASD eligibility, review considerations and support

To be eligible for an ASD allocation, the research team must have an active NSF award that is generally consistent with the proposed ASD activity and is within the atmospheric, climate, oceanographic, or related sciences.

NCAR recognizes that conducting an ASD project places significant demands on a research team. To encourage ASD proposals from the broadest possible range of researchers and institutions, NCAR may be able to provide dedicated, collaborative staffing time and effort to complement a university team’s skill set. Interested researchers should contact the ASD program coordinator, David Hart (dhart@ucar.edu) to discuss possibilities for NCAR collaborative support before preparing an ASD proposal. Research teams led by or in collaboration with minority-serving institutions (MSIs) or non-R1 institutions are strongly encouraged.

ASD proposals will be reviewed by the CISL HPC Allocations Panel (CHAP), and proposals should follow the instructions for preparing large-scale allocation requests, which can be viewed on the CISL web site. In addition to the usual review criteria, the CHAP will closely assess several other factors that must be addressed in the proposal:

  • Project readiness: Successful proposals will have established, science-ready code and a team with sufficient experience to begin conducting full-scale runs on Day One.
  • Team commitment: Successful proposals will demonstrate that the research team has staffing resources available to be committed to the ASD preparation and computing periods and to complete the primary science runs.
  • Analysis and data management plan: Research teams need to show that they have staffing available to analyze the ASD model output and the resources available to handle long-term data management needs. NCAR will provide access to our analysis and storage resources for up to one year after the ASD computation period has ended.

ASD proposal and project schedule

The ASD proposal and project schedule is outlined below. Research teams must commit to have necessary staff members available for the various ASD phases.

Be sure to review these instructions before preparing a proposal.

July 1 – Sep. 10 University ASD proposals may be submitted.
Sep. 13 – Oct. 21 Review of proposals by CHAP.
Nov. 1 University ASD projects selected and notified.
Early 2022 ASD teams on Derecho test systems. Test systems should be used to familiarize teams with the Derecho environment and verify that code configurations and workflows compile, run, and produce expected results on the Derecho hardware.
May 2022 ASD teams begin primary computational runs on Derecho following system acceptance.
June 1, 2022 ASD projects not able to make sufficient progress may be ended.
July 1, 2022 ASD science runs should be largely completed; some continued access will be available to complete projects that are making good progress. ASD analysis phase begins.
May 2023 ASD projects expected to complete analysis phase and migrate data for long-term preservation.