Data-Intensive Cluster Development

07/28/2011 - 2:05pm to 2:25pm
Main Seminar Room - ML
Dmitry Duplyakin

Dmitry Duplyakin, University of Colorado, Boulder

Abstract: Development and deployment of a computational resource specifically designed to satisfy data-intensive workflows requires a heterogeneous and highly flexible architecture and the development of new capabilities for data management, data movement, and high throughput file systems. Understanding the underlying architectural issues and design options is critical to deploy an environment that reaches the required level of performance and to satisfy this diverse set of applications and science workflows. The goal of this project was to build a prototype hardware platform for exploring these issues and using this platform to develop a number of new capabilities.

This work initially focused on the deployment of a cluster-wide Lustre parallel file system, then extended work to using RAM disks as fast local storage to explore and develop a second Lustre file system using stripes of striped RAM disks to provide a shared, incredibly high-performance, ramdisk-based parallel file system. Work was also done to allow for growing and shrinking the size of this ram-based parallel file system over time to allow for administrator-controlled but user-driven adjustments to the architecture of the resource. This file system work, paired with dedicated large-memory nodes, will become an integral part of the architecture of a larger experimental data analysis system.

Video replay of the presentation

Please install the Flash Plug-in or click here for non-flash supported devices.

Download the entire presentation (right-click/save link as)

Presented on July 28, 2011 at NCAR