IBM iDataPlex/FDR-IB

Yellowstone
Manufacturer: 
IBM
Clock Speed: 
2.60GHz
Dates Used: 
Saturday, September 15, 2012 to Saturday, December 30, 2017
Microprocessor Peak Teraflops: 
1 540.00
Memory (terabytes): 
144.54TB
Number of Processors: 
72 288.00
Electrical Power Consumption: 
1 900.00 kW
Experimental/Production: 
Production

The Yellowstone supercomputer began full-scale production at the NCAR-Wyoming Supercomputing Center (NWSC) on December 20, 2012, and performed its last calculation on December 30, 2017. CISL Director Anke Kamrath noted that Yellowstone “gave us a big boost in computing capacity over the Bluefire system. And it was the inaugural computer at the NWSC – a historic moment in NCAR’s long history of supercomputing. In addition to the sheer volume of computing it offered, Yellowstone provided NCAR with the capability to simulate Earth System physics in greater detail for longer time frames than ever before.”

Placing Yellowstone into production service in 2012 culminated a decade of CISL efforts to plan, design, and construct the NCAR-Wyoming Supercomputing Center (NWSC) while selecting the first world-class computer to be housed there. When it was installed, Yellowstone was the 13th-most powerful computer in the world, and it provided the largest-ever increase in NCAR’s computational capacity: 30 times more than Bluefire, its predecessor from 2008, and 15 million times more powerful than NCAR’s first supercomputer, the CRAY 1-A installed at NCAR in 1977. For its entire service life, Yellowstone had a total downtime of 1.96%, and its user communities utilized 92.57% of its capacity throughout those five-plus years.

While Yellowstone was still in its acceptance testing period, and during its early months of production, CISL selected a few well-prepared science proposals that tested the limits of the supercomputer’s capabilities. Those Accelerated Scientific Discovery (ASD) projects included diverse, large-scale codes that allowed researchers to answer scientific questions in a short time by using large, dedicated portions of the new system. Such dedicated use of the system is rarely feasible in a fully allocated production environment. The ASD projects made global climate predictions in the highest detail that was possible at that time; projected future summertime ozone over the United States; simulated clouds and atmospheric eddies on a global scale; analyzed earthquake hazards; and investigated magnetism dynamics inside the sun. Yellowstone supported numerous special computing campaigns throughout its lifetime.

One of the world’s first petascale computers

Yellowstone’s peak performance was rated at 1.54 petaflops. A petascale computer is a system that can perform complex calculations at a rate exceeding 1 petaflops (1 million billion floating-point operations per second). This performance is rated by running standard benchmark tests – such as the LINPACK software library that performs numerical linear algebra – on the entire system. The world’s first petascale computer went into service in 2008, and Yellowstone went into production just four years later. Yellowstone was so large and demanded so much power and cooling that it could not be operated at the NCAR Mesa Lab Computing Facility in Boulder, Colorado. CISL foresaw this need a decade in advance and built the NWSC facility to house NCAR’s first petascale computer.

During its service life, Yellowstone provided 2.49 billion CPU hours to complete 18.64 million jobs for 2,958 different users. Yellowstone contained 72,576 processor cores for user jobs, and NCAR’s computational science had advanced to the level where the average job was routinely using almost 3,000 cores in parallel. For reference, the Bluefire supercomputer offered a total of only 3,744 processor cores for user jobs, and very few of the largest jobs ever used as many as 1,000 processors.

Yellowstone also improved on Bluefire’s power efficiency. Yellowstone required only four times the power to do 30 times the work of Bluefire. These are main reasons why supercomputers become obsolete in 4-5 years: new systems can perform significantly more work in less time while consuming less power. Yellowstone became obsolete for the same reasons.

CISL’s computational scientists provided enormous support for users transitioning onto Yellowstone. Efficiently utilizing ever-larger numbers of CPUs in parallel was (and is) a complex challenge that required a great deal of advance planning, ongoing refinement, and user training. Computational scientists, software engineers, and programmers throughout CISL collaborated to modify user codes to run on Yellowstone, then they trained users to run their codes with increasing efficiency. This ongoing effort also helped NCAR’s research communities transition from Yellowstone to its successor, Cheyenne.