NWSC receives upgrades, prepares for next supercomputer

By Marijke Unger
05/07/2015 - 12:00am

It may seem hard to believe that the NCAR-Wyoming Supercomputing Center (NWSC) has already been in operation for nearly four years, but for facility manager Gary New and the on-site staff that ensure the facility is in tip top shape and who keep the daily operations humming along smoothly, the passage of time has already required maintenance and upgrades to aging systems. And as the procurement for the NWSC’s second supercomputer is underway, NWSC staff have also sprung into action to ready the facility for a new system.

Some of the most immediate needs involved upgrades to the NWSC’s security system, adding new door security that will control traffic to different areas of the center, for example as contractors access the second supercomputing space, known as Module A, to prep for and install the new supercomputer. At the same time, cameras were added in areas that did not originally have coverage, improving safety for staff in the sparsely populated facility.

The NCAR-Wyoming Supercomputing Center, just west of Cheyenne, Wyoming, has been in operation since early 2012.

The build-out of Module A, which will reflect the features already installed in Module B, where Yellowstone is housed, are slated to start as soon as a contract is finalized, a process currently near completion.  The goal is to have all the improvements complete by mid-2016, so that a new system can be installed in the third quarter of next year, and go into production by the beginning of 2017. The new system, referred to as NWSC-2 during the procurement process, is scheduled to run concurrently with Yellowstone through the end of 2017, at which time Yellowstone is slated to be decommissioned.

Module A was originally designed as shell space, both to save on initial construction cost of the facility, and to allow for greater flexibility in accommodating new system requirements as technology advances. Another planned factor was to phase some of the construction so that the infrastructure components would not all require maintenance or warranty replacement at the same time. Module A’s floor was installed as part of the original construction, but in the coming months contractors will be adding a drop ceiling, cable trays, and air and water circulation systems. All the support systems and infrastructure have been planned for; the next steps are about provisioning them.

As Gary explains, “it’s critical that we get the major purchase items underway, because it can take six to eight months to get a 3MW transformer, even longer for the switchgear. Construction isn’t the critical timing factor, because the facility was designed to increase component growth without major construction or alterations. The critical timing factor is making sure we have those orders placed and equipment delivered in time to power, cool, and operate the new system.”

While it is easy to focus on the supercomputers and how sophisticated they are, it’s harder to see the infrastructure backbone required to support such complex machines. Yellowstone can crank out 1.5 quadrillion operations per second, but that number would drop to zero without what Gary describes as “the engine that drives the engine.”

“If you’re going to invest in a supercomputer, you also have to invest in the infrastructure to support it,” Gary says. “The complexity of that infrastructure is on the order of 30,000 data points on the mechanical side, and another 30,000 on the electrical side —things like sensors, controls, health of systems, what is turned on and off, override capabilities—that require specialized, expert staff to maintain and troubleshoot, just like the supercomputers require systems engineers and support teams to keep them in good working order.”

There’s more to that expertise, too. Operating a facility as complex as the NWSC with a small crew means that they not only have to be experts in their respective trades, but also willing to learn and overlap that skill set with other disciplines. So while it is important to have licenses, it’s also important to have a crew that understands flexibility and that can deal with emergencies or devise solutions, especially in an area that is somewhat remote and has limited resources available. 

“We value our staff’s expertise, and work to increase and diversify that expertise,” Gary says. “The more we know about our systems in-house, the better equipped we are to handle repairs and upgrades ourselves, or deal more knowledgeably with contractors and service providers, ultimately saving the organization time and money.”

Two other NWSC upgrades are already proving useful and economical: a new drainage system along the fire access road, and a metering system for the facility’s wastewater. 

New drainage system at NWSC

The poor drainage along the fire access road that skirts the northeast corner of the property became particularly noticeable in winter, when melting snow and runoff created an area of near-permanent black ice, hazardous to traffic and damaging to the pavement. In warmer months, the standing water degraded the road surface and would have caused costly repairs down the road (so to speak). A new drainage system was installed, and is working flawlessly to keep the road surface clear of water and ice.

The second of these upgrades involved the installation of a metering system to track the exact amount of wastewater returning to the public utility for processing.  The local utility, BOPU, charges users based on the amount of water consumed, and that is measured simply by how much water is metered going into a facility, based on the principle that “what goes in must come out.” However, at the NWSC, much of what goes in is evaporated by the cooling towers, rather than returned to the utility for processing.

Realizing that this charge-rate program meant paying for a service that wasn’t being used, the NWSC team reached a one-year agreement with BOPU for a discounted rate, conditional on devising a solution that could measure exactly what the difference was by the end of the one-year term. At the end of April, NWSC staff installed an infrared metering system that tracks exactly how much wastewater is returned to the utility for processing. Based on preliminary numbers, Jeremy Vaughan, who was the project lead, estimates that the system will pay for itself in about a year.

Over the next ten years, that represents savings of nearly $200,000. Not a bad return on some ingenuity and proactive thinking!