Yellowstone: High-performance computing resource

The Yellowstone HPC system is based on IBM's iDataPlex architecture with Intel Sandy Bridge processors. Yellowstone is now delivering computing equivalent to 28.9 Bluefire systems (see graphic below) based on the estimated performance for a typical NCAR system workload.

Core-for-core, Yellowstone is 1.5 times more powerful than Bluefire. Even though the processor clock speed is slower, Yellowstone benefits from 1600-MHz DDR3 memory (compared to Bluefire's 667-MHz DDR2), Sandy Bridge’s additional memory channels, and twice the flops/clock. Click the image at right for a video overview.

• 72,288 processor cores
  • 2.6-GHz Intel Sandy Bridge EP with Advanced Vector Extensions (AVX)
  • 8-flops clock
• 4,518 nodes
  • IBM dx360 M4, dual socket, 8 cores per socket
• 144.58 TB total system memory
  • 2 GB/core, 32 GB/node (27 GB usable), DDR3-1600
• FDR Mellanox InfiniBand interconnect 
  • Full fat tree, single plane
  • Bandwidth 13.6 GBps bidirectional per node; latency 2.5 µs
  • Peak bidirectional bisection bandwidth: 31.7 TBps
• 1.504 petaflops peak
  • 1.26 petaflops HPL
  • 28.9 Bluefire-equivalents

GLADE: Centralized file systems and data storage

This central file and data storage resource consists of file system servers and storage devices shared by the Yellowstone system and the Geyser and Caldera analysis and visualization resources. The centralized file systems allow scientists to generate model output on the supercomputer, then analyze or visualize it on the other clusters without needing to move data between separate systems.

• 4,560 3-TB disk drives
• 76 IBM DCS3700 subsystems
  • 60 drives per subsystem
• 10.7 PB usable capacity
  • 13.7 PB raw capacity
  • 90 GBps aggregate I/O bandwidth
• GPFS file system
• Phase 2 increment, Q1 2014
  • Adding 30 3-TB drives per DCS3700 subsystem
  • 20.5 PB total raw capacity (+6.84 PB)
  • 16.4 PB total usable capacity (+5.47 PB)

Geyser and Caldera: Data analysis and visualization systems

The analysis and visualization resource comprises two systems. Geyser offers large-memory nodes (1 TB in each node) and is used for large-scale analysis and post-processing tasks, including 3D visualization. Caldera is designed to run distributed-memory parallel applications and for development and testing of general-purpose GPU (GPGPU) code.

• Geyser: 16 large-memory nodes
  • 1 TB DDR3-1600 memory per node (1000 GB usable memory per node)
  • IBM x3850, quad-socket nodes
  • 10-core 2.4-GHz Intel Westmere-EX processors
  • FDR Mellanox InfiniBand, full fat tree
  • 1 NVIDIA GPU per node
• Caldera: 16 visualization nodes
  • 64 GB DDR3-1600 memory per node (63 GB usable memory per node)
  • IBM x360 M4, dual-socket nodes
  • 8-core 2.6-GHz Intel Sandy Bridge EP processors with AVX
  • FDR Mellanox InfiniBand, full fat tree
  • 2 NVIDIA GPUs per node

Pronghorn

This system of 16 IBM nodes featuring Intel's Xeon Phi coprocessors and Sandy Bridge EP/AVX cores is being installed in the spring of 2013. It will be used initially to evaluate the capabilities of Intel's Many Integrated Core (MIC) architecture.

• 16 dual-socket IBM x360 M4 nodes
• One 8-core, 2.6-GHz Intel Sandy Bridge (Xeon E5-2670) processor per socket
• 2 Intel Xeon Phi 5110P coprocessors per node
• 64 GB DDR3-1600 memory per node (63 GB usable memory per node)
• Mellanox FDR InfiniBand, full fat tree

HPSS: NCAR data archive

The High Performance Storage System (HPSS) encompasses tape libraries at the Mesa Lab and new libraries at the NCAR-Wyoming Supercomputing Center (NWSC) that feature greatly increased capacity to accommodate ever-larger data sets generated by the Yellowstone user community. The 14.5 PB in HPSS at the Mesa Lab were migrated transparently to new media at NWSC. The Mesa Lab libraries remain in service to provide offsite replication for key scientific data sets and for disaster recovery data.

• NWSC components
  • 2 StorageTek SL8500 Tape libraries – 20,000 cartridge slots
  • 26 T10000C tape drives – 240-MBps I/O rate each
  • T10000C media – 5-TB/cartridge, uncompressed
  • 20 additional T10000C drives – ca. January 2013
  • >100-PB capacity (uncompressed)
• Mesa Lab components
  • 2 StorageTek SL8500 Tape libraries – 15,000 cartridge slots
  • Current 14.5-PB archive