ASD on Derecho: Soliciting NCAR projects

Updated 9/14/2021 – NCAR is seeking lab proposals for large-scale computational projects to run on its next-generation “Derecho” HPC system in the Accelerated Scientific Discovery (ASD) program. To be considered, ASD proposals must align with objectives of the NCAR Strategic Plan. NCAR is seeking to include both CPU-oriented and GPU-oriented projects.

Derecho supercomputer logoFollowing Derecho acceptance testing later this year, computational runs for the selected projects are expected to be completed in May and June 2022. Project teams will have access to test systems in early 2022 in order to prepare. 

The ASD program provides a unique opportunity for large-scale computational projects to have nearly exclusive use of the new 19.87-petaflops system for a few months, as the program did with Derecho’s predecessors in 2017 (Cheyenne) and 2012 (Yellowstone).

Projects will be evaluated and selected based on the following considerations and in the four-phase process described below. Key dates to note:

  • Lab-determined deadline: One-page proposals submitted for lab-specified review.
  • July 26: Lab-prioritized one-page proposals due to NCAR Science Advisor
  • October 1: Top projects selected for full proposal
  • January 10: Full proposals due
  • Early February: Proposal reviews complete
  • March 1: NCAR EC approves final slate or projects

Estimating Derecho allocation needs

Derecho users can expect to see a 1.3x improvement over the Cheyenne system's performance on a core-for-core basis. Therefore, to estimate how many CPU core-hours will be needed for a project on Derecho, multiply the total for a Cheyenne project by 0.77.

When requesting an allocation for Derecho GPU nodes, please make your request in terms of GPU-hours (number of GPUs used x wallclock hours). Derecho GPU-hour estimates can be based on any reasonable GPU performance estimate from another system, including Casper.

Consulting and visualization support

Accelerated Scientific Discovery (ASD) projects will be provided with assistance on Derecho via dedicated staff members from the Consulting Services Group. In addition, ASD projects will be granted allocations on the Casper cluster as well as staff assistance in constructing visualizations to improve understanding and presentation of results. Please describe plans or estimated needs for data analysis and visualization resources for your project.


Selection considerations

The final portfolio of ASD projects will span a range of domains and strategic priorities. Roughly 225M core-hours and 225k GPU-hours will be allocated to NCAR-led ASD projects. Minimum allocation size should be roughly 30M core-hours or 50k GPU-hours.

The following considerations will be evaluated during the strategic assessment phases. Projects are not expected to address all of these considerations.

  • Actionable science, including but not limited to those explicitly stated in section 3.3 of the NCAR Strategic Plan.
  • Connections to NCAR’s cross-lab activities, including the Boundary Layer Initiative, Short-Term Explicit Prediction (STEP), System for Integrated Modeling of the Atmosphere (SIMA), the NCAR Water System Program, convergent science, and Earth System predictability.
  • External collaborators committing time and effort during modeling or analysis phases. Lab collaborations with faculty and students at MSIs and non-R1 universities is encouraged.
  • Workforce development or training components (for example, inclusion of postdocs, grad students).
  • Long-term, community value of resulting data sets, such as the generation of landmark simulations.

Phase 1: Ideas and one-page proposals

NCAR lab staff are asked to submit ideas to their lab directors for possible ASD projects of appropriate scale and strategic NCAR value. Researchers/teams should produce a short project description (1 page or less) that outlines:

  1. How the project meets ASD strategic criteria and considerations.
  2. Estimated compute scale and data management needs.
  3. Code and team readiness.
  4. Staffing/funding availability or challenges.

Labs will submit up to four one-page proposals by July 26 to NCAR Science Advisor Glen Romine and ASD Coordinator Dave Hart.


Phase 2: Strategic review and selection

Labs’ project ideas will be reviewed by a panel chaired by the NCAR Science Advisor with members from the NCAR Scientist Assembly and Early Career Scientist Assembly. The panel will select a portfolio of projects based on:

  1. ASD strategic criteria and considerations.
  2. Collective breadth of strategic coverage. 
  3. Preliminary readiness assessment.
  4. Approximate fit within the compute time and storage space available to ASD.

The final slate of projects will include a small set of projects to be “on deck” in the event other projects are unable to proceed. 


Phase 3: Proposal preparation and final review

The review panel will consider how well the proposed experiments address the stated science objectives, the efficiency of the proposed runs, correctness of expected cost and data needs calculations, and the projects’ readiness for science runs in the ASD window.

All projects that proceed to this phase are likely to be approved to run on Derecho at some point, barring the discovery of significant issues. If not approved as ASD projects, some could be considered as NCAR Strategic Capability (NSC) projects.

Final selection of ASD projects will be announced at the end of August.


Phase 4: ASD period

ASD teams are expected to have access to Derecho test systems early in 2022 to familiarize them with the environment and enable them to verify that their code and workflows will produce expected results on the new hardware.

Contingent on timely Derecho hardware deliveries and successful acceptance testing, ASD teams should plan on starting their work in early May 2022, and completing most computing runs by the end of June.