Future Challenges of Large-Scale Computing

By Nicole Hemsoth

April 15, 2013

Now in its 28th year, the International Supercomputing Conference, ISC’13, is fast approaching. On Monday, June 17, Bill Dally, chief scientist at NVIDIA and senior vice president of NVIDIA Research, will deliver the opening keynote, titled “Future Challenges of Large-Scale Computing.”

Dally will address the multiple advances that will be necessary in order for the community to achieve the potential of HPC and data analytics going forward. The thrust of his talk will be on the challenges around power, programmability, and scalability, and most notably the role that energy-efficiency will play in determining system performance.

ISC’13 will be held from June 16-20, 2013, at the Congress Center Leipzig (CCL) in Leipzig, Germany.

In this Q&A Mr. Dally shares his views on where HPC is headed in the context of such important topics as heterogenous computing, the memory wall, government belt-tightening, and more…

HPCwire: Are different types of workloads, such as big data, HPC and Web 2.0, beginning to demand different types of processors? Will server processors diversify over the next five to ten years, or will they converge?

Bill Dally: HPC, Web servers, and big data all have similar requirement for processors. Within these applications there are program segments that are limited by single-thread performance and other segments that are limited by throughput. To meet this need, there will be a convergence on heterogeneous multicore processors where each “socket” will contain a small number of cores optimized for latency (like today’s CPU cores) and many more cores optimized for throughput (like today’s GPU cores).

HPCwire: The increase in processor performance seems to be outpacing memory technology. What can be done about the memory wall?

Dally: There are three aspects of memory relevant here: bandwidth, latency and capacity. To address the slow scaling of memory bandwidth we plan to move to memory technologies that involve placing memory dice on the same package as the processor chip and connecting them with very high-bandwidth, low-energy links. This on-package memory technology will enable us to scale memory bandwidth with processor performance holding the Byte/FLOP ratio roughly constant for the next few generations.

Memory latency is remaining roughly constant as processor performance increases. We deal with this by increasing parallelism to hide the latency. With adequate parallelism we can keep the memory pipeline full – using all of the available bandwidth.

Memory capacity is largely a matter of cost. The challenge here is that high-bandwidth memories, like on-package or stacked DRAM, cost significantly more than commodity memory. Thus, for cost-sensitive applications we are likely to see a two-tiered memory system with a moderate capacity, high-bandwidth on-package memory and a high-capacity commodity memory. A non-volatile memory technology like flash or phase-change memory could have a place in such a hierarchy as well.

HPCwire: How important will 3D stacked chip technology be to processors and memory? When do you think we’ll see the first commercial products?

Dally: Placing memory on-package will be critical to scale bandwidth. Stacking technology is important to extend the capacity of this high-bandwidth memory.

Stacked memory is shipping today. However, most of this today uses wire bonds, not through-silicon vias. At the 2013 GPU Technology Conference (GTC) this past March, we announced that we expect to introduce stacked memories with our Volta architecture-based generation of GPUs – in about 2016.

HPCwire: Government austerity restrictions look as though there could be pressure to reduce investments in exascale computing, especially in the US. How capable is industry of driving these initiatives by itself?

Dally: Industry will continue to move forward on its own on exascale projects, however progress will be much slower than with government assistance.

It is disappointing that government priorities are such that investment in computing innovation is being scaled back. At the same time, other nations like China are investing heavily in computing. Even the EU with all of its economic problems is moving forward with their exascale program. With reduced investment, the US runs a grave risk of giving up its leadership in computing.

HPCwire: With current technology, it seems as though exascale computing would require so much energy as to render it impractical. Will we see new breakthrough technologies to sufficiently reduce power consumption to make exascale practical and affordable?

Dally: Improving energy efficiency to reach the goal of a sustained exaflops on a real application in 20MW is a significant challenge. However, I am optimistic that we can meet this challenge. There are many emerging circuit, architecture and software technologies that have the potential to dramatically improve the energy efficiency of one or more parts of the system. For example, at NVIDIA we have recently developed a new signaling technology that reduces the energy required by communication by more than an order of magnitude, and we have developed an SRAM technology that permits operation at dramatically lower voltages – and hence lower power. It won’t be a single breakthrough technology that will get us to the exascale energy goal, it will be multiple breakthroughs – at least one in each of the multiple areas that require improvement – processor, communication, memory, etc. We have a number of research projects that are targeted at these different areas. If a sufficient number of these projects have successful outcomes, we will meet the goal.

These improvements, however, depend on research, which in turn will be slowed considerably without government funding.

HPCwire: What do you see as the biggest challenges to reaching exascale?

Dally: Energy efficiency and programmability are the two biggest challenges.

For energy, we will need to improve from where we are with the NVIDIA-Kepler-based Titan machine at Oak Ridge National Laboratory in Tennessee, which is about 2GFLOPS/Watt (500pJ/FLOP) to 50GFLOPS/Watt (20pJ/FLOP), a 25x improvement in efficiency while at the same time increasing scale – which tends to reduce efficiency. Of this 25x improvement we expect to get only a factor of 2x to 4x from improved semiconductor process technology.

As I described before, we are optimistic that we can meet this challenge through a number of research advances in circuits, architecture and software.

Making it easy to program a machine that requires 10 billion threads to use at full capacity is also a challenge. While a backward compatible path will be provided to allow existing MPI codes to run, MPI plus C++ or Fortran is not a productive programming environment for a machine of this scale. We need to move toward higher-level programming models where the programmer describes the algorithm with all available parallelism and locality exposed, and tools automate much of the process of efficiently mapping and tuning the program to a particular target machine.

A number of research projects are underway to develop more productive programming systems – and most importantly the tools that will permit automated mapping and tuning.

Changing a large code base, however, is a very slow process, so we need to start moving on this now. As with energy efficiency, progress will be slowed without government funding.

About Bill Dally

Bill Dally is chief scientist at NVIDIA and senior vice president of NVIDIA Research, the company’s world-class research organization, which is chartered with developing the strategic technologies that will help drive the company’s future growth and success.

Dally first joined NVIDIA in 2009 after spending 12 years at Stanford University, where he was chairman of the computer science department and the Willard R. and Inez Kerr Bell Professor of Engineering. Dally and his Stanford team developed the system architecture, network architecture, signaling, routing and synchronization technology that is found in most large parallel computers today.

Dally was previously at the Massachusetts Institute of Technology from 1986 to 1997, where he and his team built the J-Machine and M-Machine, experimental parallel computer systems that pioneered the separation of mechanism from programming models and demonstrated very low overhead synchronization and communication mechanisms. From 1983 to 1986, he was at the California Institute of Technology (Caltech), where he designed the MOSSIM Simulation Engine and the Torus Routing chip, which pioneered wormhole routing and virtual-channel flow control.

Dally is a cofounder of Velio Communications and Stream Processors. He is a member of the National Academy of Engineering, a Fellow of the American Academy of Arts & Sciences, a Fellow of the IEEE and the ACM. He received the 2010 Eckert-Mauchly Award, considered the highest prize in computer architecture, as well as the 2004 IEEE Computer Society Seymour Cray Computer Engineering Award and the 2000 ACM Maurice Wilkes Award. He has published more than 200 papers, holds more than 75 issued patents and is the author of two textbooks, “Digital Systems Engineering” and “Principles and Practices of Interconnection Networks.”

Dally received a bachelor’s degree in electrical engineering from Virginia Tech, a master’s degree in electrical engineering from Stanford University and a PhD in computer science from Caltech.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

SC Bids Farewell to Denver, Heads to Dallas for 30th

November 17, 2017

After a jam-packed four-day expo and intensive six-day technical program, SC17 has wrapped up another successful event that brought together nearly 13,000 visitors to the Colorado Convention Center in Denver for the larg Read more…

By Tiffany Trader

SC17 Keynote – HPC Powers SKA Efforts to Peer Deep into the Cosmos

November 17, 2017

This week’s SC17 keynote – Life, the Universe and Computing: The Story of the SKA Telescope – was a powerful pitch for the potential of Big Science projects that also showcased the foundational role of high performance computing in modern science. It was also visually stunning. Read more…

By John Russell

How Cities Use HPC at the Edge to Get Smarter

November 17, 2017

Cities are sensoring up, collecting vast troves of data that they’re running through predictive models and using the insights to solve problems that, in some cases, city managers didn’t even know existed. Speaking Read more…

By Doug Black

HPE Extreme Performance Solutions

Harness Scalable Petabyte Storage with HPE Apollo 4510 and HPE StoreEver

As a growing number of connected devices challenges IT departments to rapidly collect, manage, and store troves of data, organizations must adopt a new generation of IT to help them operate quickly and intelligently. Read more…

SC17 Student Cluster Competition Configurations: Fewer Nodes, Way More Accelerators

November 16, 2017

The final configurations for each of the SC17 “Donnybrook in Denver” Student Cluster Competition have been released. Fortunately, each team received their equipment shipments on time and undamaged, so the teams are r Read more…

By Dan Olds

SC Bids Farewell to Denver, Heads to Dallas for 30th

November 17, 2017

After a jam-packed four-day expo and intensive six-day technical program, SC17 has wrapped up another successful event that brought together nearly 13,000 visit Read more…

By Tiffany Trader

SC17 Keynote – HPC Powers SKA Efforts to Peer Deep into the Cosmos

November 17, 2017

This week’s SC17 keynote – Life, the Universe and Computing: The Story of the SKA Telescope – was a powerful pitch for the potential of Big Science projects that also showcased the foundational role of high performance computing in modern science. It was also visually stunning. Read more…

By John Russell

How Cities Use HPC at the Edge to Get Smarter

November 17, 2017

Cities are sensoring up, collecting vast troves of data that they’re running through predictive models and using the insights to solve problems that, in some Read more…

By Doug Black

Student Cluster LINPACK Record Shattered! More LINs Packed Than Ever before!

November 16, 2017

Nanyang Technological University, the pride of Singapore, utterly destroyed the Student Cluster Competition LINPACK record by posting a score of 51.77 TFlop/s a Read more…

By Dan Olds

Hyperion Market Update: ‘Decent’ Growth Led by HPE; AI Transparency a Risk Issue

November 15, 2017

The HPC market update from Hyperion Research (formerly IDC) at the annual SC conference is a business and social “must,” and this year’s presentation at S Read more…

By Doug Black

Nvidia Focuses Its Cloud Containers on HPC Applications

November 14, 2017

Having migrated its top-of-the-line datacenter GPU to the largest cloud vendors, Nvidia is touting its Volta architecture for a range of scientific computing ta Read more…

By George Leopold

HPE Launches ARM-based Apollo System for HPC, AI

November 14, 2017

HPE doubled down on its memory-driven computing vision while expanding its processor portfolio with the announcement yesterday of the company’s first ARM-base Read more…

By Doug Black

OpenACC Shines in Global Climate/Weather Codes

November 14, 2017

OpenACC, the directive-based parallel programming model used mostly for porting codes to GPUs for use on heterogeneous systems, came to SC17 touting impressive Read more…

By John Russell

US Coalesces Plans for First Exascale Supercomputer: Aurora in 2021

September 27, 2017

At the Advanced Scientific Computing Advisory Committee (ASCAC) meeting, in Arlington, Va., yesterday (Sept. 26), it was revealed that the "Aurora" supercompute Read more…

By Tiffany Trader

NERSC Scales Scientific Deep Learning to 15 Petaflops

August 28, 2017

A collaborative effort between Intel, NERSC and Stanford has delivered the first 15-petaflops deep learning software running on HPC platforms and is, according Read more…

By Rob Farber

Oracle Layoffs Reportedly Hit SPARC and Solaris Hard

September 7, 2017

Oracle’s latest layoffs have many wondering if this is the end of the line for the SPARC processor and Solaris OS development. As reported by multiple sources Read more…

By John Russell

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

AMD Showcases Growing Portfolio of EPYC and Radeon-based Systems at SC17

November 13, 2017

AMD’s charge back into HPC and the datacenter is on full display at SC17. Having launched the EPYC processor line in June along with its MI25 GPU the focus he Read more…

By John Russell

Google Releases Deeplearn.js to Further Democratize Machine Learning

August 17, 2017

Spreading the use of machine learning tools is one of the goals of Google’s PAIR (People + AI Research) initiative, which was introduced in early July. Last w Read more…

By John Russell

GlobalFoundries Puts Wind in AMD’s Sails with 12nm FinFET

September 24, 2017

From its annual tech conference last week (Sept. 20), where GlobalFoundries welcomed more than 600 semiconductor professionals (reaching the Santa Clara venue Read more…

By Tiffany Trader

Amazon Debuts New AMD-based GPU Instances for Graphics Acceleration

September 12, 2017

Last week Amazon Web Services (AWS) streaming service, AppStream 2.0, introduced a new GPU instance called Graphics Design intended to accelerate graphics. The Read more…

By John Russell

Leading Solution Providers

EU Funds 20 Million Euro ARM+FPGA Exascale Project

September 7, 2017

At the Barcelona Supercomputer Centre on Wednesday (Sept. 6), 16 partners gathered to launch the EuroEXA project, which invests €20 million over three-and-a-half years into exascale-focused research and development. Led by the Horizon 2020 program, EuroEXA picks up the banner of a triad of partner projects — ExaNeSt, EcoScale and ExaNoDe — building on their work... Read more…

By Tiffany Trader

Delays, Smoke, Records & Markets – A Candid Conversation with Cray CEO Peter Ungaro

October 5, 2017

Earlier this month, Tom Tabor, publisher of HPCwire and I had a very personal conversation with Cray CEO Peter Ungaro. Cray has been on something of a Cinderell Read more…

By Tiffany Trader & Tom Tabor

Reinders: “AVX-512 May Be a Hidden Gem” in Intel Xeon Scalable Processors

June 29, 2017

Imagine if we could use vector processing on something other than just floating point problems.  Today, GPUs and CPUs work tirelessly to accelerate algorithms Read more…

By James Reinders

Cray Moves to Acquire the Seagate ClusterStor Line

July 28, 2017

This week Cray announced that it is picking up Seagate's ClusterStor HPC storage array business for an undisclosed sum. "In short we're effectively transitioning the bulk of the ClusterStor product line to Cray," said CEO Peter Ungaro. Read more…

By Tiffany Trader

Intel Launches Software Tools to Ease FPGA Programming

September 5, 2017

Field Programmable Gate Arrays (FPGAs) have a reputation for being difficult to program, requiring expertise in specialty languages, like Verilog or VHDL. Easin Read more…

By Tiffany Trader

HPC Chips – A Veritable Smorgasbord?

October 10, 2017

For the first time since AMD's ill-fated launch of Bulldozer the answer to the question, 'Which CPU will be in my next HPC system?' doesn't have to be 'Whichever variety of Intel Xeon E5 they are selling when we procure'. Read more…

By Dairsie Latimer

IBM Advances Web-based Quantum Programming

September 5, 2017

IBM Research is pairing its Jupyter-based Data Science Experience notebook environment with its cloud-based quantum computer, IBM Q, in hopes of encouraging a new class of entrepreneurial user to solve intractable problems that even exceed the capabilities of the best AI systems. Read more…

By Alex Woodie

How ‘Knights Mill’ Gets Its Deep Learning Flops

June 22, 2017

Intel, the subject of much speculation regarding the delayed, rewritten or potentially canceled “Aurora” contract (the Argonne Lab part of the CORAL “ Read more…

By Tiffany Trader

  • arrow
  • Click Here for More Headlines
  • arrow
Share This