Seven Days of Speed

By Linda Barney

October 18, 2010

100 people, 168 miles of fiber, and the world’s fastest network

For a week in November, New Orleans will be home to one of the most advanced networks in the world. SCinet forms the data backbone of the annual SC conference: it takes months to build and connects to the world’s most advanced data networks, but it is only active for seven days each year before it is torn down and planning starts for next year.

Each year the SCinet design team pushes the state-of-the-art in cutting edge networking technology, and 2010 is no different. This year SCinet includes a 100 Gbps circuit alongside other infrastructure capable of delivering 260 gigabits per second of aggregate data bandwidth for conference attendees and exhibitors — that’s enough data to allow the entire collection of books at the Library of Congress to be transferred in well under a minute.

Introducing SCinet

“SCinet is the fastest network in the world for the few days that it exists each year. The aggregate bandwidth in SCinet exceeds the bandwidth in all but a few countries of the world,” according to Jamie Van Randwyk, manager, Informatics and Systems Assessments Department at Sandia National Laboratories and chair of SCinet.

But what is it?

“SCinet provides a high performance, production-quality network that enables attendees and exhibitors to connect to the Internet and research networks around the world,” explains Van Randwyk. “During the conference SCinet powers everything from the mundane – email and Google searches, for example — to the extraordinary one-of-a-kind application demonstrations that can only happen in this kind of environment.”

SCinet serves as the platform for exhibitors to demonstrate the advanced computing resources of their home institutions (and elsewhere) by supporting a wide variety of bandwidth-driven applications. At its core SCinet’s capabilities are driven by multiple 10 Gbps wide area circuits and a 100 Gbps circuit that connect the exhibit floor to high performance production and research networks around the world. The infrastructure also provides the ability for exhibitors to use dynamically provisioned circuits to allow dedicated capacity between their booths and other end points.

The SCinet architecture also includes an InfiniBand (IB) network to support distributed HPC application demonstrations. This year the InfiniBand fabric will consist of Quad Data Rate (QDR) 40, 80, and 120-gigabit per second (Gbps) circuits linking together various organizations and vendors with high-speed 120 Gbps circuits providing backbone connectivity through the SCinet InfiniBand switching infrastructure.

Collaborating to Build SCinet

But SCinet is not just about what happens on the show floor. What makes this effort even more remarkable is the partnerships that connect SC to the rest of the world. During SC10, SCinet will connect exhibitors and attendees to leading research and commercial networks around the world such as the Department of Energy’s ESnet, Internet2, National LambdaRail, LONI (Louisiana Optical Network Initiative), and others.

“The story of SCinet is amazing — building SCinet is truly a collaborative effort,” states Jeff Boote, Internet2’s assistant director of research and development and chair of SCinet for 2011. SCinet is built by a group of over 100 volunteers including scientists, engineers, and students. Participants are from the United States, Canada, and Europe and they work for universities, industry, government, and US national laboratories. Vendors have donated approximately $23 million in equipment to build SCinet this year. Planning begins more than a year in advance of each SC Conference and culminates with a high-intensity installation just seven days before the conference begins.

Pushing the Boundaries with the SCinet Sandbox

Beginning this year, SCinet is introducing the SCinet Research Sandbox (SRS). The SRS provides a unique opportunity for researchers to showcase emerging technologies in network monitoring, performance optimization, network security, and other areas on the forefront of communication systems research. SRS participants will demonstrate 100G networks for petascale computing, next-generation approaches to wide area file transfer, security analysis tools, and data-intensive computing.

“Whatever network research calipers you propose, SCinet offers an unprecedented sandbox for your exploration,” says Van Randwyk.

SCinet Behind the Scenes: power, measurement, and more

The physical aspects of a network as vast and capable as SCinet are the first things about this effort that most people think of, but they only tell part of the story. Measurement, power planning, wireless communications, and more all play a pivotal role in ensuring that SCinet can successfully support the conference each year. And each area presents its own opportunities for researchers and planners to get a unique window into their area of study.

The Measurement team provides planning, performance, and network utilization tools and metrics and enables attendees to view real time network traffic stats across the infrastructure. The SCinet security teams is there to help protect SCinet assets from external hackers and malicious access, and in the process of protecting collects extensive statistics on SC network usage patterns. On-site access to these high-speed taps offers researchers a unique sample of extremely diverse security data.

The SCinet Wide Area Network (WAN) team provides national and international connectivity for exhibitor communications to external storage, compute or remote national research and education networks. SCinet’s WAN is one of the fastest networks on the planet, exceeding a peak bandwidth of 260 Gigabytes per second. This capability is used by Scientists to support e-science initiatives, but is also available to both regular exhibitors and Sandbox participants interested in technologies to advance computer communications.

The LAN Routing team provides the essential connectivity services typical of a large, diverse production network against the backdrop of a high demand and essential reliability environment. The routing team offers all services above the Optical Layer and provides consulting, configuration, optimization and trouble-shooting services for all connected exhibitors and demonstrations. Researchers interested in exploring network optimization, novel protocols or tools can propose to collaborate with this team and leverage the power and diversity of the SCinet infrastructure.

About the Author

Linda Barney owns Barney and Associates, a technical, marketing writing and Web firm in Beaverton, Oregon, that provides writing and Web content for the high tech, government, medical and scientific communities. Readers can reach her at linda@barneyassoc.com.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Activist Investor Starboard Buys 10.7% Stake in Mellanox; Sale Possible?

November 20, 2017

Starboard Value has reportedly taken a 10.7 percent stake in interconnect specialist Mellanox Technologies, and according to the Wall Street Journal, has urged the company “to improve its margins and stock and explore Read more…

By John Russell

Installation of Sierra Supercomputer Steams Along at LLNL

November 20, 2017

Sierra, the 125 petaflops (peak) machine based on IBM’s Power9 chip being built at Lawrence Livermore National Laboratory, sometimes takes a back seat to Summit, the ~200 petaflops system being built at Oak Ridge Natio Read more…

By John Russell

SC Bids Farewell to Denver, Heads to Dallas for 30th

November 17, 2017

After a jam-packed four-day expo and intensive six-day technical program, SC17 has wrapped up another successful event that brought together nearly 13,000 visitors to the Colorado Convention Center in Denver for the larg Read more…

By Tiffany Trader

HPE Extreme Performance Solutions

Harness Scalable Petabyte Storage with HPE Apollo 4510 and HPE StoreEver

As a growing number of connected devices challenges IT departments to rapidly collect, manage, and store troves of data, organizations must adopt a new generation of IT to help them operate quickly and intelligently. Read more…

SC17 Keynote – HPC Powers SKA Efforts to Peer Deep into the Cosmos

November 17, 2017

This week’s SC17 keynote – Life, the Universe and Computing: The Story of the SKA Telescope – was a powerful pitch for the potential of Big Science projects that also showcased the foundational role of high performance computing in modern science. It was also visually stunning. Read more…

By John Russell

SC Bids Farewell to Denver, Heads to Dallas for 30th

November 17, 2017

After a jam-packed four-day expo and intensive six-day technical program, SC17 has wrapped up another successful event that brought together nearly 13,000 visit Read more…

By Tiffany Trader

SC17 Keynote – HPC Powers SKA Efforts to Peer Deep into the Cosmos

November 17, 2017

This week’s SC17 keynote – Life, the Universe and Computing: The Story of the SKA Telescope – was a powerful pitch for the potential of Big Science projects that also showcased the foundational role of high performance computing in modern science. It was also visually stunning. Read more…

By John Russell

How Cities Use HPC at the Edge to Get Smarter

November 17, 2017

Cities are sensoring up, collecting vast troves of data that they’re running through predictive models and using the insights to solve problems that, in some Read more…

By Doug Black

Student Cluster LINPACK Record Shattered! More LINs Packed Than Ever before!

November 16, 2017

Nanyang Technological University, the pride of Singapore, utterly destroyed the Student Cluster Competition LINPACK record by posting a score of 51.77 TFlop/s a Read more…

By Dan Olds

Hyperion Market Update: ‘Decent’ Growth Led by HPE; AI Transparency a Risk Issue

November 15, 2017

The HPC market update from Hyperion Research (formerly IDC) at the annual SC conference is a business and social “must,” and this year’s presentation at S Read more…

By Doug Black

Nvidia Focuses Its Cloud Containers on HPC Applications

November 14, 2017

Having migrated its top-of-the-line datacenter GPU to the largest cloud vendors, Nvidia is touting its Volta architecture for a range of scientific computing ta Read more…

By George Leopold

HPE Launches ARM-based Apollo System for HPC, AI

November 14, 2017

HPE doubled down on its memory-driven computing vision while expanding its processor portfolio with the announcement yesterday of the company’s first ARM-base Read more…

By Doug Black

OpenACC Shines in Global Climate/Weather Codes

November 14, 2017

OpenACC, the directive-based parallel programming model used mostly for porting codes to GPUs for use on heterogeneous systems, came to SC17 touting impressive Read more…

By John Russell

US Coalesces Plans for First Exascale Supercomputer: Aurora in 2021

September 27, 2017

At the Advanced Scientific Computing Advisory Committee (ASCAC) meeting, in Arlington, Va., yesterday (Sept. 26), it was revealed that the "Aurora" supercompute Read more…

By Tiffany Trader

NERSC Scales Scientific Deep Learning to 15 Petaflops

August 28, 2017

A collaborative effort between Intel, NERSC and Stanford has delivered the first 15-petaflops deep learning software running on HPC platforms and is, according Read more…

By Rob Farber

Oracle Layoffs Reportedly Hit SPARC and Solaris Hard

September 7, 2017

Oracle’s latest layoffs have many wondering if this is the end of the line for the SPARC processor and Solaris OS development. As reported by multiple sources Read more…

By John Russell

AMD Showcases Growing Portfolio of EPYC and Radeon-based Systems at SC17

November 13, 2017

AMD’s charge back into HPC and the datacenter is on full display at SC17. Having launched the EPYC processor line in June along with its MI25 GPU the focus he Read more…

By John Russell

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

Google Releases Deeplearn.js to Further Democratize Machine Learning

August 17, 2017

Spreading the use of machine learning tools is one of the goals of Google’s PAIR (People + AI Research) initiative, which was introduced in early July. Last w Read more…

By John Russell

GlobalFoundries Puts Wind in AMD’s Sails with 12nm FinFET

September 24, 2017

From its annual tech conference last week (Sept. 20), where GlobalFoundries welcomed more than 600 semiconductor professionals (reaching the Santa Clara venue Read more…

By Tiffany Trader

Amazon Debuts New AMD-based GPU Instances for Graphics Acceleration

September 12, 2017

Last week Amazon Web Services (AWS) streaming service, AppStream 2.0, introduced a new GPU instance called Graphics Design intended to accelerate graphics. The Read more…

By John Russell

Leading Solution Providers

EU Funds 20 Million Euro ARM+FPGA Exascale Project

September 7, 2017

At the Barcelona Supercomputer Centre on Wednesday (Sept. 6), 16 partners gathered to launch the EuroEXA project, which invests €20 million over three-and-a-half years into exascale-focused research and development. Led by the Horizon 2020 program, EuroEXA picks up the banner of a triad of partner projects — ExaNeSt, EcoScale and ExaNoDe — building on their work... Read more…

By Tiffany Trader

Delays, Smoke, Records & Markets – A Candid Conversation with Cray CEO Peter Ungaro

October 5, 2017

Earlier this month, Tom Tabor, publisher of HPCwire and I had a very personal conversation with Cray CEO Peter Ungaro. Cray has been on something of a Cinderell Read more…

By Tiffany Trader & Tom Tabor

Reinders: “AVX-512 May Be a Hidden Gem” in Intel Xeon Scalable Processors

June 29, 2017

Imagine if we could use vector processing on something other than just floating point problems.  Today, GPUs and CPUs work tirelessly to accelerate algorithms Read more…

By James Reinders

Cray Moves to Acquire the Seagate ClusterStor Line

July 28, 2017

This week Cray announced that it is picking up Seagate's ClusterStor HPC storage array business for an undisclosed sum. "In short we're effectively transitioning the bulk of the ClusterStor product line to Cray," said CEO Peter Ungaro. Read more…

By Tiffany Trader

Intel Launches Software Tools to Ease FPGA Programming

September 5, 2017

Field Programmable Gate Arrays (FPGAs) have a reputation for being difficult to program, requiring expertise in specialty languages, like Verilog or VHDL. Easin Read more…

By Tiffany Trader

HPC Chips – A Veritable Smorgasbord?

October 10, 2017

For the first time since AMD's ill-fated launch of Bulldozer the answer to the question, 'Which CPU will be in my next HPC system?' doesn't have to be 'Whichever variety of Intel Xeon E5 they are selling when we procure'. Read more…

By Dairsie Latimer

Flipping the Flops and Reading the Top500 Tea Leaves

November 13, 2017

The 50th edition of the Top500 list, the biannual publication of the world’s fastest supercomputers based on public Linpack benchmarking results, was released Read more…

By Tiffany Trader

IBM Advances Web-based Quantum Programming

September 5, 2017

IBM Research is pairing its Jupyter-based Data Science Experience notebook environment with its cloud-based quantum computer, IBM Q, in hopes of encouraging a new class of entrepreneurial user to solve intractable problems that even exceed the capabilities of the best AI systems. Read more…

By Alex Woodie

Share This