Physicists Set Record for Network Data Transfer

By Nicole Hemsoth

December 15, 2006

An international team of physicists, computer scientists, and network engineers led by the California Institute of Technology, CERN, and the University of Michigan and partners at the University of Florida and Vanderbilt, as well as participants from Brazil (Rio de Janeiro State University, UERJ, and the State Universities of São Paulo, USP and UNESP) and Korea (Kyungpook National University, KISTI) joined forces to set new records for sustained data transfer between storage systems during the Supercomputing 2006 (SC06) Bandwidth Challenge (BWC).

The high-energy physics team's demonstration of “High Speed Data Gathering, Distribution and Analysis for Physics Discoveries at the Large Hadron Collider” achieved a peak throughput of 17.77 gigabits per second (Gbps) between clusters of servers at the show floor and at Caltech. Following the rules set for the SC06 Bandwidth Challenge, the team used a single 10-Gbps link provided by National Lambda Rail that carried data in both directions. Sustained throughput throughout the night prior to the bandwidth challenge exceeded 16 Gbps (or two gigabytes per second) using just 10 pairs of small servers sending data at nine Gbps to Caltech from Tampa, and eight pairs of servers sending seven Gbps of data in the reverse direction.

One of the key advances in this demonstration was Fast Data Transport (FDT; http://monalisa.cern.ch/FDT), a Java application developed by Iosif Legrand of Caltech that runs on all major platforms and uses the NIO libraries to achieve stable disk reads and writes coordinated with smooth data flow across the long-range network. FDT streams a large set of files across an open TCP socket, so that a large data set composed of thousands of files, as is typical in high-energy physics applications, can be sent or received at full speed, without the network transfer restarting between files. By combining FDT with FAST TCP, developed by Steven Low of Caltech's computer science department, together with an optimized Linux kernel provided by Shawn McKee of Michigan known as the “UltraLight kernel,” the team reached unprecedented throughput levels, limited only by the speeds of the disks, that correspond to nine GB/sec reading from, or five GB/sec writing to, a single rack of 40 low-cost servers.

Overall, this year's demonstration, following the team's record memory-to-memory transfer rate of 151 Gbps using 22 10-Gbps links last year at Supercomputing 2005, represents a major milestone in providing practical, widely deployable applications. These applications exploit advances in state-of-the-art TCP-based data transport, servers (Intel Woodcrest-based systems) and the Linux kernel over the last 12 months. FDT also represents a clear advance in basic data transport capability over wide-area networks compared to last year, in that 20 Gbps could be sustained in a few streams memory-to-memory over long distances very stably for many hours, using a single 10-Gigabit Ethernet link very close to full capacity in both directions.

The two largest physics collaborations at the LHC, CMS and ATLAS, each encompass more than 2,000 physicists and engineers from 170 universities and laboratories. In order to fully exploit the potential for scientific discoveries, the many petabytes of data produced by the experiments will be processed, distributed, and analyzed using a global Grid. The key to discovery is the analysis phase, where individual physicists and small groups repeatedly access, and sometimes extract and transport, terabyte-scale data samples on demand, in order to optimally select the rare “signals” of new physics from potentially overwhelming “backgrounds” from already-understood particle interactions. This data will amount to many tens of petabytes in the early years of LHC operation, rising to the exabyte range within the coming decade.

The high-energy physics team also carried out several other demonstrations, making good use of the ten wide-area network links connected to the Caltech/CERN booth:

  • Vanderbilt demonstrated the capabilities of LStore, an integrated system that provides a single file-system image across many storage “depots” consisting of compact data servers distributed across wide-area networks. Reading and writing between sets of servers at the Vanderbilt booth at SC06 and Caltech, the team achieved a throughput of more than one GB/sec.
  • By using five of the ten 10-Gbps links coming into SC06, the team reached an aggregate throughput of more than 75 Gbps, combining disk-to-disk and memory-to-memory transfers. During this part of the demonstrations, the links between Tampa and Jacksonville, the National Lambda Rail Framenet links, and the newly commissioned Atlantic Wave link, were often loaded to full capacity at 10 Gbps in both directions, as shown on the SCInet network “weathermap.”
  • Of particular note was the use of FDT between Tampa and Daegu in South Korea, allowing the group from Kyungpook National University and KISTI to achieve 8.6 Gbps disk-to-disk over a single network path, using NLR's shared Packetnet via Atlanta, and the GLORIAD link between Seattle and Daejeon that was inaugurated in September 2005, shortly before SC05.

Professor Harvey Newman of Caltech, head of the HEP team and US CMS Collaboration Board Chair, who originated the LHC Data Grid Hierarchy concept, said, “These demonstrations allowed us to thoroughly field-test a new class of data-transport applications, together with the real-time analysis of some of the data using 'ROOTlets,' a distributed form of the ROOT system (root.cern.ch) that is an essential element of high-energy physicists' arsenal of tools for large-scale data analysis.

“These demonstrations provided a new, more agile and flexible view of the globally distributed Grid system of more than 100 laboratory- and university-based computing facilities that is now being commissioned in the U.S., Europe, Asia, and Latin America in preparation for the next generation of high-energy physics experiments at CERN's Large Hadron Collider (LHC) that will begin operation in November 2007, along with several hundred computing clusters serving individual groups of physicists. By substantially reducing the difficulty of transporting terabyte- and larger scale data sets among the sites, we are enabling physicists throughout the world to have a much greater role in the next round of physics discoveries expected soon after the LHC starts.”

David Foster, head of Communications and Networking at CERN said, “The efficient use of high-speed networks to transfer large data sets is an essential component of CERN's LCG plans to deploy computing infrastructure that will enable the LHC experiments to carry out their scientific missions. This demonstration of the high-speed transfer of physics event samples and their analysis made use of equipment at Tampa, Caltech, CERN, and elsewhere, interconnected by the same network infrastructure CERN plans to use in production, and was an important milestone on the road to ensuring full capability when the LHC starts operations in 2007.”

Iosif Legrand, senior software and distributed system engineer at Caltech and the technical coordinator for the MonALISA and FDT projects, said, “We demonstrated a realistic, worldwide deployment for distributed, data-intensive applications capable to effectively use and coordinate the network resources. A distributed agent-based system was used for dynamic discovery of resources and to monitor, configure, control, and orchestrate efficient data transfer between several hundreds of computers using hybrid networks.”

Richard Cavanaugh of the University of Florida, technical coordinator of the UltraLight project that is developing the next generation of network-integrated grids aimed at LHC data analysis, said, “Future optical networks incorporating multiple 10-Gbps links are the foundation of the Grid system that will drive scientific discoveries at the LHC. A 'hybrid' network integrating both traditional switching and routing of packets and dynamically constructed optical paths to support the largest data flows is a central part of the near-term future vision that the scientific community has adopted to meet the challenges of data-intensive science in many fields. “By demonstrating that many 10-Gbps wavelengths can be used efficiently over continental and transoceanic distances (often in both directions simultaneously), the high-energy physics team showed that this vision of a worldwide dynamic Grid supporting many terabyte and larger data transactions is practical.”

Shawn McKee, associate research scientist in the University of Michigan department of physics and leader of the UltraLight network technical group, said, “This achievement is an impressive example of what a focused network effort can accomplish. It is an important step towards the goal of delivering a highly capable end-to-end network-aware system and architecture that meet the needs of next-generation e-Science.”

Paul Sheldon of Vanderbilt University, who leads the NSF-funded Research and Education Data Depot Network (REDDnet) project that will deploy a distributed storage infrastructure of about 700TB over the next two years, commented on the innovative network storage technology that helped the group achieve such high performance in wide-area, disk-to-disk transfers.

“With IBP and the logistical network technology that Micah Beck and his group at Tennessee have developed, we were able to build middleware, L-Store, that can exploit a tremendous amount of parallelism, both in data transfers across the network and in reading and writing to disk,” said Sheldon. “And since L-Store can also do efficient erasure coding in software with minimal data movement, we can build high-quality storage clusters out of commodity parts and push depot costs down to a thousand dollars a TB.

“When you combine this network-storage technology, including its cost profile, with the remarkable tools that Harvey Newman's networking team has produced, I think we are well positioned to address the incredible infrastructure demands that the LHC experiments are going to make on our community worldwide.”

The team hopes this new demonstration will encourage scientists and engineers in many sectors of society to develop and plan to deploy a new generation of revolutionary Internet applications. Multigigabit/s end-to-end network performance will empower scientists to form “virtual organizations” on a planetary scale, sharing their collective computing and data resources in a flexible way. In particular, this is vital for projects on the frontiers of science and engineering, in “data-intensive” fields such as particle physics, astronomy, bioinformatics, global climate modeling, geosciences, fusion, and neutron science.

The new bandwidth record was achieved through extensive use of the SCInet network infrastructure at SC06. The team used all 10 of the 10-Gbps links coming into the showfloor, connected to two Cisco Systems Catalyst 6500 Series Switches at the Caltech/CERN booth, together with computing clusters provided by Hewlett Packard and a large number of 10-gigabit Ethernet server interfaces provided by Neterion and Myricom.

The 10 10-Gbps network connections included two National Lambda Rail FrameNet links, one to Los Angeles (the official BWC wavelength) and one to StarLight; two NLR PacketNet links used from Korea over GLORIAD and from the University of Michigan over MiLR; two links provided by Internet2's Abilene network used to carry traffic from Caltech and UMICH; one link from ESNET used from Brookhaven National Laboratory; and one link from FLRNET used to carry traffic from Brazil over the CHEPREO/WREN-LILA link. Also, one link provided by AtlanticWave from NYC to DC and Miami was used to carry traffic from CERN coming over the USLHCNet NYC-Geneva circuit and one link provided by UltraLight/FLR from Jacksonville.

The UltraLight/FLR circuit was the only direct WAN circuit available at SC06 and terminated directly on the Caltech equipment on the showfloor. All other circuits were connected through the SCInet infrastructure. During the test, several of the network links were shown to operate at full capacity for sustained periods. The network has been deployed through exceptional support by Cisco Systems and Nortel, as well as the network engineering staffs of National LambdaRail, Florida Lambda Rail, Internet2, ESnet, TeraGrid, CENIC, MiLR, Atlantic Wave, AMPATH, RNP and ANSP/FAPESP in Brazil, KISTI in Korea, the Starlight international peering point in Chicago, and MANLAN in New York.

As part of the SC06 demonstration, a distributed analysis of simulated LHC physics data was carried using the Grid-enabled Analysis Environment (GAE) developed at Caltech for the LHC. This demonstration involved the use of the Clarens Web Services portal developed at Caltech, the use of Root-based analysis software, and numerous architectural components developed in the framework of Caltech's “Grid Analysis Environment.” The analysis made use of a new component in the Grid system: “Rootlets” hosted by Clarens servers. Each Rootlet is a full instantiation of CERN's Root tool, created on demand by the distributed clients in the Grid.

The design and deployment of the Rootlets/Clarens system was carried out under the auspices of an STTR grant for collaboration between Deep Web Technologies (www.deepwebtech.com) of New Mexico, Caltech, and Indiana University. In addition to the Rootlets/Clarens demonstration, an innovative literature and database aggregation search tool designed specifically for scientists working in the field of particle physics and developed by Deep Web Technologies was shown. This aggregation tool allowed simultaneous queries to be made on several of the most popular document databases, the results being aggregated and presented to the user in a homogeneous fashion. Deep Web's aggregation system also powers the science.gov website.

The team used Caltech's MonALISA (MONitoring Agents using a Large Integrated Services Architecture — http://monalisa.caltech.edu) system to monitor and display the real-time data for all the network links used in the demonstration. MonALISA is a Dynamic, Distributed Service System that is capable of collecting any type of information from different systems, to analyze it in near-real time, and to provide support for automated control decisions and global optimization of workflows in complex Grid systems. It is currently used to monitor more than 300 sites, more than 50,000 computing nodes, and tens of thousands of concurrent jobs running on different Grid systems and scientific communities.

MonALISA is a highly scalable set of autonomous, self-describing, agent-based subsystems which are able to collaborate and cooperate in performing a wide range of monitoring tasks for networks and Grid systems, as well as the scientific applications themselves. Vanderbilt demonstrated the capabilities of their L-Store middleware, a scalable, open source, and wide-area-capable form of storage virtualization that builds on the Internet Backplane Protocol (IBP) and logistical networking technology developed at the University of Tennessee. Offering both scalable metadata management and software-based fault tolerance, L-Store creates an integrated system that provides a single file-system image across many IBP storage “depots” distributed across wide-area and/or local-area networks. Reading and writing between sets of depots at the Vanderbilt booth at SC06 and Caltech in California, the team achieved a network throughput, disk to disk, of more than one GB/sec. On the floor, the team was able to sustain throughputs of 3.5 GB/sec between a rack of client computers and a rack of storage depots. These two racks communicated across SCinet via four 10-GigE connections.

Further information about the demonstration may be found at
http://supercomputing.caltech.edu/, http://pcbunn.cacr.caltech.edu/sc2006/ and https://scinet.supercomp.org/2006/bwc/.

—–

Source: California Institute of Technology

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industry updates delivered to you every week!

MLPerf Inference 4.0 Results Showcase GenAI; Nvidia Still Dominates

March 28, 2024

There were no startling surprises in the latest MLPerf Inference benchmark (4.0) results released yesterday. Two new workloads — Llama 2 and Stable Diffusion XL — were added to the benchmark suite as MLPerf continues Read more…

Q&A with Nvidia’s Chief of DGX Systems on the DGX-GB200 Rack-scale System

March 27, 2024

Pictures of Nvidia's new flagship mega-server, the DGX GB200, on the GTC show floor got favorable reactions on social media for the sheer amount of computing power it brings to artificial intelligence.  Nvidia's DGX Read more…

Call for Participation in Workshop on Potential NSF CISE Quantum Initiative

March 26, 2024

Editor’s Note: Next month there will be a workshop to discuss what a quantum initiative led by NSF’s Computer, Information Science and Engineering (CISE) directorate could entail. The details are posted below in a Ca Read more…

Waseda U. Researchers Reports New Quantum Algorithm for Speeding Optimization

March 25, 2024

Optimization problems cover a wide range of applications and are often cited as good candidates for quantum computing. However, the execution time for constrained combinatorial optimization applications on quantum device Read more…

NVLink: Faster Interconnects and Switches to Help Relieve Data Bottlenecks

March 25, 2024

Nvidia’s new Blackwell architecture may have stolen the show this week at the GPU Technology Conference in San Jose, California. But an emerging bottleneck at the network layer threatens to make bigger and brawnier pro Read more…

Who is David Blackwell?

March 22, 2024

During GTC24, co-founder and president of NVIDIA Jensen Huang unveiled the Blackwell GPU. This GPU itself is heavily optimized for AI work, boasting 192GB of HBM3E memory as well as the the ability to train 1 trillion pa Read more…

MLPerf Inference 4.0 Results Showcase GenAI; Nvidia Still Dominates

March 28, 2024

There were no startling surprises in the latest MLPerf Inference benchmark (4.0) results released yesterday. Two new workloads — Llama 2 and Stable Diffusion Read more…

Q&A with Nvidia’s Chief of DGX Systems on the DGX-GB200 Rack-scale System

March 27, 2024

Pictures of Nvidia's new flagship mega-server, the DGX GB200, on the GTC show floor got favorable reactions on social media for the sheer amount of computing po Read more…

NVLink: Faster Interconnects and Switches to Help Relieve Data Bottlenecks

March 25, 2024

Nvidia’s new Blackwell architecture may have stolen the show this week at the GPU Technology Conference in San Jose, California. But an emerging bottleneck at Read more…

Who is David Blackwell?

March 22, 2024

During GTC24, co-founder and president of NVIDIA Jensen Huang unveiled the Blackwell GPU. This GPU itself is heavily optimized for AI work, boasting 192GB of HB Read more…

Nvidia Looks to Accelerate GenAI Adoption with NIM

March 19, 2024

Today at the GPU Technology Conference, Nvidia launched a new offering aimed at helping customers quickly deploy their generative AI applications in a secure, s Read more…

The Generative AI Future Is Now, Nvidia’s Huang Says

March 19, 2024

We are in the early days of a transformative shift in how business gets done thanks to the advent of generative AI, according to Nvidia CEO and cofounder Jensen Read more…

Nvidia’s New Blackwell GPU Can Train AI Models with Trillions of Parameters

March 18, 2024

Nvidia's latest and fastest GPU, codenamed Blackwell, is here and will underpin the company's AI plans this year. The chip offers performance improvements from Read more…

Nvidia Showcases Quantum Cloud, Expanding Quantum Portfolio at GTC24

March 18, 2024

Nvidia’s barrage of quantum news at GTC24 this week includes new products, signature collaborations, and a new Nvidia Quantum Cloud for quantum developers. Wh Read more…

Alibaba Shuts Down its Quantum Computing Effort

November 30, 2023

In case you missed it, China’s e-commerce giant Alibaba has shut down its quantum computing research effort. It’s not entirely clear what drove the change. Read more…

Nvidia H100: Are 550,000 GPUs Enough for This Year?

August 17, 2023

The GPU Squeeze continues to place a premium on Nvidia H100 GPUs. In a recent Financial Times article, Nvidia reports that it expects to ship 550,000 of its lat Read more…

Shutterstock 1285747942

AMD’s Horsepower-packed MI300X GPU Beats Nvidia’s Upcoming H200

December 7, 2023

AMD and Nvidia are locked in an AI performance battle – much like the gaming GPU performance clash the companies have waged for decades. AMD has claimed it Read more…

DoD Takes a Long View of Quantum Computing

December 19, 2023

Given the large sums tied to expensive weapon systems – think $100-million-plus per F-35 fighter – it’s easy to forget the U.S. Department of Defense is a Read more…

Synopsys Eats Ansys: Does HPC Get Indigestion?

February 8, 2024

Recently, it was announced that Synopsys is buying HPC tool developer Ansys. Started in Pittsburgh, Pa., in 1970 as Swanson Analysis Systems, Inc. (SASI) by John Swanson (and eventually renamed), Ansys serves the CAE (Computer Aided Engineering)/multiphysics engineering simulation market. Read more…

Choosing the Right GPU for LLM Inference and Training

December 11, 2023

Accelerating the training and inference processes of deep learning models is crucial for unleashing their true potential and NVIDIA GPUs have emerged as a game- Read more…

Intel’s Server and PC Chip Development Will Blur After 2025

January 15, 2024

Intel's dealing with much more than chip rivals breathing down its neck; it is simultaneously integrating a bevy of new technologies such as chiplets, artificia Read more…

Baidu Exits Quantum, Closely Following Alibaba’s Earlier Move

January 5, 2024

Reuters reported this week that Baidu, China’s giant e-commerce and services provider, is exiting the quantum computing development arena. Reuters reported � Read more…

Leading Solution Providers

Contributors

Comparing NVIDIA A100 and NVIDIA L40S: Which GPU is Ideal for AI and Graphics-Intensive Workloads?

October 30, 2023

With long lead times for the NVIDIA H100 and A100 GPUs, many organizations are looking at the new NVIDIA L40S GPU, which it’s a new GPU optimized for AI and g Read more…

Shutterstock 1179408610

Google Addresses the Mysteries of Its Hypercomputer 

December 28, 2023

When Google launched its Hypercomputer earlier this month (December 2023), the first reaction was, "Say what?" It turns out that the Hypercomputer is Google's t Read more…

AMD MI3000A

How AMD May Get Across the CUDA Moat

October 5, 2023

When discussing GenAI, the term "GPU" almost always enters the conversation and the topic often moves toward performance and access. Interestingly, the word "GPU" is assumed to mean "Nvidia" products. (As an aside, the popular Nvidia hardware used in GenAI are not technically... Read more…

Shutterstock 1606064203

Meta’s Zuckerberg Puts Its AI Future in the Hands of 600,000 GPUs

January 25, 2024

In under two minutes, Meta's CEO, Mark Zuckerberg, laid out the company's AI plans, which included a plan to build an artificial intelligence system with the eq Read more…

Google Introduces ‘Hypercomputer’ to Its AI Infrastructure

December 11, 2023

Google ran out of monikers to describe its new AI system released on December 7. Supercomputer perhaps wasn't an apt description, so it settled on Hypercomputer Read more…

China Is All In on a RISC-V Future

January 8, 2024

The state of RISC-V in China was discussed in a recent report released by the Jamestown Foundation, a Washington, D.C.-based think tank. The report, entitled "E Read more…

Intel Won’t Have a Xeon Max Chip with New Emerald Rapids CPU

December 14, 2023

As expected, Intel officially announced its 5th generation Xeon server chips codenamed Emerald Rapids at an event in New York City, where the focus was really o Read more…

IBM Quantum Summit: Two New QPUs, Upgraded Qiskit, 10-year Roadmap and More

December 4, 2023

IBM kicks off its annual Quantum Summit today and will announce a broad range of advances including its much-anticipated 1121-qubit Condor QPU, a smaller 133-qu Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire