CHAP: Allocation Review Criteria

The CHAP assesses the merits of large computing requests from U.S. university researchers for use of CISL supercomputing resources. Computing requests are accepted in the atmospheric and closely related sciences for projects that are supported by the National Science Foundation. The panel recommends action with respect to a prospective user's request on the basis of the computational experimental design, computational effectiveness, and availability of computing resources as described here.

Overall context

As part of reviewing the merits of the requests for CISL resource allocations, the CHAP and CISL allocations staff will ensure that:

  • All requests for resources that exceed a threshold level determined by CISL and the CHAP shall be peer-reviewed.
  • Written reviews of the resource requests shall be completed in a timely way and made available to the requestors.
  • Recommendations to CISL management for the allocation of resources based on the requests, reviews and the available resources shall be documented.
  • The allocations process shall be consistent with the conflict-of-interest policy and shall maintain confidentiality of requestors and their reviews.

Purpose and scope of reviews

In its review of allocation proposals, the CHAP verifies the suitability of the work for CISL resources, considers the ability of the research team to complete the work, and most significantly, reviews the merit of the proposed computational plan.

Scientific eligibility

A request for resources will succinctly state the scientific impact of the research to be conducted and the existing merit-reviewed support for the research, as demonstrated by current financial support from NSF. The scientific merit and approach will not be subject to further review by the CHAP. Since NCAR computing resources are provided specifically for the atmospheric and closely related sciences, the request must fall within these areas.

Merit review criteria

The justification of the resource request will be reviewed against three criteria, which apply to both computational and storage resources, with the level of detail of the review rising with the size of the requested resources:

Effectiveness of methodology. For computational resource requests, the choice of applications, methods, algorithms and techniques to be employed to accomplish the stated scientific objectives should be reasonably described and motivated. For data storage resource requests, the data to be stored should be reasonably described and motivated with respect to the stated research objectives.

Appropriateness of research plan. The steps in the research plan should explain how the scientific objectives will be achieved. For computational experiments, the proposed computations should encompass simulation parameters (grid size, time scale, ensemble parameters, etc.) that are needed to obtain accurate and meaningful results, as well as the human resources that can be devoted to the task. For storage resources, the justification should describe the rationale for determining which data will be stored. The amount of resources (of all types) requested should be derived from the methodology and research plan. If there are serious concerns about the research plan, reviewers should share their concerns with the proposer, and may decide to proceed only after the concerns have been addressed.

Efficiency of resource use. The resources requested should be used as efficiently as is reasonably possible and in accordance with the recommended use guidelines of those resources. In exceptional cases, where the reviewers conclude that the proposed methods are so inefficient that they amount to a waste of public resources, they should not approve the request until the proposer has addressed their concerns. For computational resources, performance and parallel scaling data should be provided along with a discussion of optimization or other work done to improve the applications. For storage resources, the CHAP will consider the choices made in managing the project's data, the value of that data both within the proposing research team and among the wider community, approaches for data access and dissemination, and long-term retention plans.

Prior accomplishments

For ongoing computational activities, the CHAP will consider the progress made using prior allocations, including the publication of peer-reviewed manuscripts, other communications within the community, and the effective estimation and use of CISL resources from prior requests.