National LambdaRail lights the future for research in science and networking

By Brian Bevirt
09/05/2006 - 12:00am

A network of light winds for more than 11,000 miles back and forth around the United States, etching data trails from New York to Jacksonville, Chicago to Denver, Seattle to San Antonio. The National LambdaRail, as it’s called—lambda being the symbol for a wavelength of light—is a high-speed, nationwide fiber-optic infrastructure dedicated to research in science and networking.

Called by some the most ambitious networking initiative since the Advanced Research Projects Agency Network (ARPANET) in the late 1960s, the National LambdaRail, or NLR, became fully operational in February 2006.

NLR is the result of vision, effort, and $100 million in funding from a non-profit consortium comprised of 20 research institutions and private-sector companies. NCAR/UCAR has played a key role in the project from the start, working closely with the Front Range GigaPop—a group of universities, nonprofit corporations, and government agencies that share network services in Colorado, Wyoming, and Utah—to make NLR a reality.

The result is an ultra-high-performance infrastructure that Marla Meehl, manager of the Network Engineering and Telecommunications Section (NETS) of NCAR’s Scientific Computing Division, calls the network of the future.

Multiple lightwaves for production and experimentation

What makes NLR so remarkable is that it is capable of transmitting 40 simultaneous wavelengths of light, each of which can move data at 10 gigabits per second (Gbps). In comparison, the high-performance Abilene network developed by the Internet2 consortium currently has only a single 10-Gbps wave.

“This is the next generation of networking,” says Meehl. “You could literally have 40 Abilenes running across NLR.”

Because each NLR lightwave can support an operationally independent network, production and experimental networks can co-exist side by side without interfering with each other. Scientists can link models and move data faster over dedicated lines, enjoying guaranteed levels of reliability, availability, and performance. At the same time, network engineers can explore radically new Internet technologies without disrupting traffic.

In fact, one thing that differentiates NLR is its increased emphasis on experimentation.

“Normally, once a network goes into production, you can no longer experiment on it,” says Meehl. “But NLR bylaws state that 50 percent of the infrastructure is dedicated to research. That’s exciting for us, because NLR provides opportunities for our scientists to use these networks in new and different ways.”

Indeed, says NETS network engineer Scot Colburn, “NLR will let people in the academic community experiment with network protocols and the basic network infrastructure in a way they haven’t since ARPANET.”

The benefits of ownership

When Meehl joined NCAR in 1986, the National Science Foundation (NSF) had just launched NSFnet, a dedicated 56-kilobit-per-second network. In 1988, NSFnet was upgraded to T1 capability (1.54 megabits per second, or Mbps), expanding in 1991 to T3 (45 Mbps). The Very-high-speed Backbone Network Service, or vBNS, followed with OC3 capability (155 Mbps). After that came Abilene (2.5 Gbps), which still exists and has continuously been upgraded. (Its current capability is 10 Gbps.)

What was common to all those networks was that they used leased infrastructures. NLR, on the other hand, owns all its own fiber-optic cables and dense wavelength division optical multiplexers (the equipment used to light the fiber), as well as its own Ethernet switches and Internet Protocol routers.

“NLR members have come up with a significant financial commitment to build this infrastructure,” notes Meehl. “It’s a unique paradigm in the U.S. for a nonprofit entity to own its own fiber and network hardware devices at a national scale. Normally that’s done by commercial entities like Level 3, Qwest, or AT&T.”

Traditional network service providers aim at commodity markets. Unable to justify an investment in high-end networking because the niche is too small, they generally do not provide the advanced capabilities needed by research and education.

Since NLR belongs to its own users, however, it can be tailored directly to high-end research applications. NLR offers application scientists and network engineers unprecedented control and flexibility, promoting scientific discovery on many fronts while fostering technological advancements that are expected to make it to the commodity Internet one day.

    NETS staff
  The Network Engineering and Telecommunications Section of NCAR's Scientific Computing Division has been active in ongoing technical and management efforts to implement NLR.
   

Hard work that paid off

Getting to this point wasn’t easy.

“NLR has been four-and-a-half years in the making,” says NETS network engineer Peter O’Neil. “There have been a lot of meetings, a lot of discussions, a lot of head-scratching—and a lot of tin-cupping for money to figure out how to pay for it all.”

NETS staff have worked at local, regional, and national levels to implement NLR. Meehl is secretary of NLR's executive committee and board of directors, while O’Neil, Colburn, and NETS engineer David Mitchell are on the engineering committee. In addition to contributing technical expertise and assistance (for example, installing new routers and bringing up network connections), NETS staff have participated in ongoing consortium meetings, board meetings, personal meetings, weekly teleconferences, and private calls.

“I’d guess that most of the U.S. doesn’t realize how hard it was to get a consortium of universities and research organizations together to build something like this,” Meehl adds. “In most cases, the government sponsors this kind of effort—Canada, for instance, has had huge government support. A grassroots effort is an odd way to build a national backbone that’s for the good of the country. There’s been virtually no outside funding. The big player has been Cisco Systems, which has been a partner in NLR from the outset—they’ve slashed prices so we could afford to buy the hardware to do this, and they’re very committed to research.”

But, she notes, “the exciting thing at this point is that NLR is actually completely constructed. The fiber is all in place, the hardware is in place, and actual services are being utlized. NLR now offers a wide range of applications and network support for the atmospheric and Earth-systems science community.”

Three service levels, many projects

The NLR infrastructure provides the basis for three levels of services to members:

  • WaveNet (layer 1) uses optronics hardware to send up to 40 different-colored lightwaves over fiber-optic cables. Each color of light supports an independent, dedicated network.
  • FrameNet (layer 2) uses Ethernet technology. While Ethernet has been the basis for local area networks (e.g., across college campuses) since the 1990s, FrameNet is the nation’s first transcontinental Ethernet service.
  • PacketNet (layer 3) uses Internet Protocol routing. NCAR is now using PacketNet for its normal Internet traffic, which includes email, Web page views, FTP file transfer, and the downloading of satellite data.

These three service levels are already in use by more than a dozen cutting-edge research projects.

National TransitRail graph     
This graph shows the total network traffic on National TransitRail (in bits per second) for the 24-hour period from September 4–5, 2006. National Transit Rail is one of many projects utilizing NLR services.  
   

One such project is the National TransitRail, which will utilize layers 2 and 3 to improve nationwide peering of commodity traffic, decrease reliance on commercial providers, and increase routing efficiency and flexibility.

Launched in April 2006, National TransitRail is a pilot project aimed at reducing overall cost of Internet services to NLR members and other users. Initial participants include NCAR/UCAR and the Front Range GigaPop, the Corporation for Education Network Initiatives in California (CENIC), the Mid-Atlantic Terascale Partnership (MATP), the Pacific Northwest GigaPop, and the Pittsburgh Supercomputing Center.

“TransitRail is a testbed right now,” says Meehl. “It uses an arrangement called peering, which basically means that participants save money by exchanging network traffic less expensively. There are many different kinds of applications like this on NLR. We’re just now starting to explore the possibilities of what we can do.”

Other projects using NLR services include:

  • The NSF-sponsored Extensible Terascale Facility, or TeraGrid
  • The NSF-sponsored OptiPuter project
  • The U.S. Department of Energy’s UltraScience project
  • The Pacific Northwest GigaPop’s Pacific Wave project
  • The Community Cyberinfrastructure for Advanced Marine Microbial Ecology Research and Analysis (CAMERA) project, led by the California Institute for Telecommunications and Information Technology
  • The Circuit-switched High-speed End-to-End Transport Architecture (CHEETAH) project, let by the University of Virginia
  • The Internet2 Hybrid Optical Packet Infrastructure (HOPI)

In an era where scientists and network engineers need ever more advanced networks on which to do their research, NLR offers cost-effective, high-capacity networking among distributed sites. It facilitates the development of new applications and encourages collaboration across institutions and disciplines.

NLR capabilities may also play key role in realizing a high-performance Grid computing infrastructure on a national scale.

“It’s an impressive accomplishment,” Meehl concludes.