Tackling the Power and Energy Wall for Future HPC Systems

By Performance and Architecture Lab (PAL) at PNNL

December 17, 2013

A Perspective from the Paci fic Northwest National Laboratory*

As the cost of powering a supercomputer or a datacenter increases, next generation exascale systems need to be considerably more power- and energy-efficient than current supercomputers to be of practical use. Constrained power consumption (20-25MW for the entire system is the target that the DOE Office of Science gave to the HPC community) is one of the limiting factors on the road to achieve sustainable performance at exascale. In fact, the power challenge is so fundamental that other challenges can be reduced to power limitations. For example, operating at near-threshold voltage (NTV) in order to perform computation within a given power budget may considerably increase the soft-error rate (resilience challenge). Unlike petascale systems, where the primary concern was performance, exascale systems need to climb the power and energy walls in order to deliver sustainable exaflops performance. At the Pacifi c Northwest Laboratory (PNNL) we are exploring holistically energy and power efficiency aspects at all levels of granularity, from processor architecture to system integration. We are also tackling the power and energy problems from several angles, from system software and programming models to performance and power modeling of scientifi c applications and extreme scale systems.

PNNL Research Areas Fig1

PNNL’s computing facilities, such as its Institutional HPC system (PIC), and an earlier testbed, the Energy Smart Data Center (ESDC), provide research platforms to address what-if questions related to the use of suitable datacenter metrics that are meaningful to the HPC community. The measurement harness of the ESDC entailed over thousand out-of-band sensors comprising power, flow, pressure and temperature at the machine room and at the IT equipment. PIC is another example that substantiates our integrated datacenter vision to drive energy efficiency research. This system is housed in a geothermally cooled datacenter with rear-door heat exchangers. The facility is instrumented at the machine room and system level, providing insight into macro-level machine room power efficiencies and micro-level energy efficiencies at the server and mother-board component levels.

Despite its importance for future exascale systems, power is still not considered a first-class citizen, which complicates the development of power-aware software algorithms. In PNNL’s vision, power should be considered a resource, just as processing elements or memory modules, and should be managed as such by the system software. System software must be able to precisely measure (in-band) power resource utilization, i.e., how much power is consumed by each system component at any given time. More importantly, the system software should adapt the application to the contingent execution environment, e.g, by allocating sustained power to threads on the application’s critical path or promptly moving idle cores to low-power states. The design and development of such self-aware/self-adaptive system software is an active research area at PNNL. We recently analyzed the power characteristics of scientifi c applications from the DOE ASCR’s Exascale Co-Design Center and, in general, in the HPC community to identify opportunities for power savings. Given the lack of in-band, fine-grained (both in space and time) power sensors, we develop an accurate per-core proxy power sensor model that estimates the active power of each core by inspecting the cores’ activity. We use statistical regression techniques to formulate closed-form expressions for the estimated core and system power consumption. These techniques enable us to develop power-aware algorithms and characterize applications running even on non-instrumented compute nodes. Our experiments show that processes in the same application may not have the same power profi le and/or may alternate high-power with low-power phases independently from one another. These alternating behaviors raise opportunities for shifting power towards computing-demanding processes, hereby saving power without diminishing performance.

There is a strong agreement among researchers on the increasing cost of data movement with respect to computation. This ratio will further increase in future systems that will approach NTV operation levels: the energy consumption of a double precision register-to-register floating point operation is expected to decrease by 10x by 2018. The energy cost of moving data from memory to processor is not expected to follow the same trend, hence the relative energy cost of data movement with respect to performing a register-to-register operation will increase (energy wall — analogous to the memory wall). In a recent study we modeled the energy cost of moving data across the memory hierarchy of current systems and analyzed the energy cost of data movement for scientifi c applications. In this study, we answer several important questions such as what is the amount of energy spent in data movement with respect to the total energy consumption of an application or what is the dominant component of data movement energy for current and future parallel applications. Our results show that the energy cost of data movement impact di fferently on each application, ranging from 18% to 40%. This percentage might increase in the future, as the energy cost of performing computation decreases. To avoid such scenario, new technologies, such as Processing-In- Memory, Non-Volatile RAM and 3D-stacked memory, become essential for the development of sustainable exascale computing. We also noticed that the energy spent in resolving data dependency, speculation and out-of-order scheduling of instructions accounts for a considerable part of the total dynamic energy, between 22% and 35%. This cost can be reduced with simpler processor core designs that are more energy efficient.

Given the increasing complexity of future exascale applications and systems, designers need new sophisticated tools to navigate the design space. These tools must capture a range of metrics that are of interest to system and application designers, including performance and power consumption. PNNL has historically developed application-specific performance tools that model the evolution of parallel applications. While these models have shown themselves to be powerful tools for understanding the mapping of applications to complex system architectures, the metrics of interest are expanding to include power consumption as well. To this end, PNNL researchers have developed a methodology for the modeling of performance and power in concert that builds upon its experience of co-designing systems and applications. This modeling capability has been developed along three axes. The first is the deployment of a workload-specific quantitative power modeling capability. Such power models accurately capture workload phases, their impact on power consumption, and how they are impacted by system architecture and con figuration (e.g., processor clock speed). The second axis is the integration of the performance and power modeling methodologies. To this end, it is critical that both modeling methods operated at the same conceptual level. In other words, application phases or components that are captured in one model must be also reflected in the other so that trade-o ffs between power and performance may be captured and quantifi ed. The last axis of development involves integrating these models with our self-aware/self-adaptive software system that will provide mechanisms for dynamically optimizing ongoing application execution. We have developed the concept of Energy Templates, which are a mechanism for passing application-specifi c behavioral information to the underlying runtime layers. Energy Templates capture per-core idle/busy states, as well as the amount of time each core expects to remain in each state, allowing runtime software to determine appropriate opportunities to exercise power saving features provided by the hardware/software platform (e.g., Dynamic Voltage and Frequency Scaling — DVFS) without negatively impacting performance. By proactively using application-speci fic information, Energy Templates are able to exploit energy savings opportunities that are not available to mechanisms that are not application-aware.

The research at PNNL is also being applied within the new DARPA program in the Power Efficiency Revolution of Embedded Technologies (PERFECT). We see that technologies being developed both for high performance computing and embedded systems are fundamentally the same. These may well converge in the future, and thus common tools and techniques can be developed that encompass both. Within PERFECT PNNL researchers are developing a coherent framework that is able to both empirically analyze current systems and predictively assess future technologies.

Finally, PNNL’s research extends to the datacenters: this research direction is approached in an integrated fashion where IT power consumption for applications of interest to the DOE is correlated with the power consumption of the supporting infrastructure. An integrated approach allows the researcher to formulate what-if questions in an HPC setting such as the applicability and efficacy of novel cooling solutions (e.g., spray cooling) at the heat source vs. a traditional global cooling solution.

Overall, PNNL is actively participating in (and in many cases leading) several DOE and DARPA projects, as well as internal projects, that aim at understanding the impact of the power and energy walls on exascale systems and deploying power- and energy-aware solutions at all levels of the system and application design and optimization. The insights gained throughout these efforts and projects will contribute towards the design of power- and energy-efficient exascale systems.

*The following PNNL researchers contributed to this piece: Adolfy Hoisie, Kevin Barker, Roberto Gioiosa, Darren J. Kerbyson, Gokcen Kestor, Joseph Manzano, Andres Marquez, Shuaiwen Song, Nathan Tallent, Antonino Tumeo, Abhinav Vishnu

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Microsoft Wants to Speed Quantum Development

December 12, 2017

Quantum computing continues to make headlines in what remains of 2017 as tech giants jockey to establish a pole position in the race toward commercialization of quantum. This week, Microsoft took the next step in advanci Read more…

By Tiffany Trader

ESnet Now Moving More Than 1 Petabyte/wk

December 12, 2017

Optimizing ESnet (Energy Sciences Network), the world's fastest network for science, is an ongoing process. Recently a two-year collaboration by ESnet users – the Petascale DTN Project – achieved its ambitious goal t Read more…

HPC-as-a-Service Finds Toehold in Iceland

December 11, 2017

While high-demand workloads (e.g., bitcoin mining) can overheat data center cooling capabilities, at least one data center infrastructure provider has announced an HPC-as-a-service offering that features 100 percent fre Read more…

By Doug Black

HPE Extreme Performance Solutions

Explore the Origins of Space with COSMOS and Memory-Driven Computing

From the formation of black holes to the origins of space, data is the key to unlocking the secrets of the early universe. Read more…

HPC Iron, Soft, Data, People – It Takes an Ecosystem!

December 11, 2017

Cutting edge advanced computing hardware (aka big iron) does not stand by itself. These computers are the pinnacle of a myriad of technologies that must be carefully woven together by people to create the computational c Read more…

By Alex R. Larzelere

Microsoft Wants to Speed Quantum Development

December 12, 2017

Quantum computing continues to make headlines in what remains of 2017 as tech giants jockey to establish a pole position in the race toward commercialization of Read more…

By Tiffany Trader

HPC Iron, Soft, Data, People – It Takes an Ecosystem!

December 11, 2017

Cutting edge advanced computing hardware (aka big iron) does not stand by itself. These computers are the pinnacle of a myriad of technologies that must be care Read more…

By Alex R. Larzelere

IBM Begins Power9 Rollout with Backing from DOE, Google

December 6, 2017

After over a year of buildup, IBM is unveiling its first Power9 system based on the same architecture as the Department of Energy CORAL supercomputers, Summit a Read more…

By Tiffany Trader

Microsoft Spins Cycle Computing into Core Azure Product

December 5, 2017

Last August, cloud giant Microsoft acquired HPC cloud orchestration pioneer Cycle Computing. Since then the focus has been on integrating Cycle’s organization Read more…

By John Russell

GlobalFoundries, Ayar Labs Team Up to Commercialize Optical I/O

December 4, 2017

GlobalFoundries (GF) and Ayar Labs, a startup focused on using light, instead of electricity, to transfer data between chips, today announced they've entered in Read more…

By Tiffany Trader

HPE In-Memory Platform Comes to COSMOS

November 30, 2017

Hewlett Packard Enterprise is on a mission to accelerate space research. In August, it sent the first commercial-off-the-shelf HPC system into space for testing Read more…

By Tiffany Trader

SC17 Cluster Competition: Who Won and Why? Results Analyzed and Over-Analyzed

November 28, 2017

Everyone by now knows that Nanyang Technological University of Singapore (NTU) took home the highest LINPACK Award and the Overall Championship from the recently concluded SC17 Student Cluster Competition. We also already know how the teams did in the Highest LINPACK and Highest HPCG competitions, with Nanyang grabbing bragging rights for both benchmarks. Read more…

By Dan Olds

Perspective: What Really Happened at SC17?

November 22, 2017

SC is over. Now comes the myriad of follow-ups. Inboxes are filled with templated emails from vendors and other exhibitors hoping to win a place in the post-SC thinking of booth visitors. Attendees of tutorials, workshops and other technical sessions will be inundated with requests for feedback. Read more…

By Andrew Jones

US Coalesces Plans for First Exascale Supercomputer: Aurora in 2021

September 27, 2017

At the Advanced Scientific Computing Advisory Committee (ASCAC) meeting, in Arlington, Va., yesterday (Sept. 26), it was revealed that the "Aurora" supercompute Read more…

By Tiffany Trader

NERSC Scales Scientific Deep Learning to 15 Petaflops

August 28, 2017

A collaborative effort between Intel, NERSC and Stanford has delivered the first 15-petaflops deep learning software running on HPC platforms and is, according Read more…

By Rob Farber

Oracle Layoffs Reportedly Hit SPARC and Solaris Hard

September 7, 2017

Oracle’s latest layoffs have many wondering if this is the end of the line for the SPARC processor and Solaris OS development. As reported by multiple sources Read more…

By John Russell

AMD Showcases Growing Portfolio of EPYC and Radeon-based Systems at SC17

November 13, 2017

AMD’s charge back into HPC and the datacenter is on full display at SC17. Having launched the EPYC processor line in June along with its MI25 GPU the focus he Read more…

By John Russell

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

Japan Unveils Quantum Neural Network

November 22, 2017

The U.S. and China are leading the race toward productive quantum computing, but it's early enough that ultimate leadership is still something of an open questi Read more…

By Tiffany Trader

GlobalFoundries Puts Wind in AMD’s Sails with 12nm FinFET

September 24, 2017

From its annual tech conference last week (Sept. 20), where GlobalFoundries welcomed more than 600 semiconductor professionals (reaching the Santa Clara venue Read more…

By Tiffany Trader

Google Releases Deeplearn.js to Further Democratize Machine Learning

August 17, 2017

Spreading the use of machine learning tools is one of the goals of Google’s PAIR (People + AI Research) initiative, which was introduced in early July. Last w Read more…

By John Russell

Leading Solution Providers

Amazon Debuts New AMD-based GPU Instances for Graphics Acceleration

September 12, 2017

Last week Amazon Web Services (AWS) streaming service, AppStream 2.0, introduced a new GPU instance called Graphics Design intended to accelerate graphics. The Read more…

By John Russell

Perspective: What Really Happened at SC17?

November 22, 2017

SC is over. Now comes the myriad of follow-ups. Inboxes are filled with templated emails from vendors and other exhibitors hoping to win a place in the post-SC thinking of booth visitors. Attendees of tutorials, workshops and other technical sessions will be inundated with requests for feedback. Read more…

By Andrew Jones

EU Funds 20 Million Euro ARM+FPGA Exascale Project

September 7, 2017

At the Barcelona Supercomputer Centre on Wednesday (Sept. 6), 16 partners gathered to launch the EuroEXA project, which invests €20 million over three-and-a-half years into exascale-focused research and development. Led by the Horizon 2020 program, EuroEXA picks up the banner of a triad of partner projects — ExaNeSt, EcoScale and ExaNoDe — building on their work... Read more…

By Tiffany Trader

Delays, Smoke, Records & Markets – A Candid Conversation with Cray CEO Peter Ungaro

October 5, 2017

Earlier this month, Tom Tabor, publisher of HPCwire and I had a very personal conversation with Cray CEO Peter Ungaro. Cray has been on something of a Cinderell Read more…

By Tiffany Trader & Tom Tabor

Tensors Come of Age: Why the AI Revolution Will Help HPC

November 13, 2017

Thirty years ago, parallel computing was coming of age. A bitter battle began between stalwart vector computing supporters and advocates of various approaches to parallel computing. IBM skeptic Alan Karp, reacting to announcements of nCUBE’s 1024-microprocessor system and Thinking Machines’ 65,536-element array, made a public $100 wager that no one could get a parallel speedup of over 200 on real HPC workloads. Read more…

By John Gustafson & Lenore Mullin

Flipping the Flops and Reading the Top500 Tea Leaves

November 13, 2017

The 50th edition of the Top500 list, the biannual publication of the world’s fastest supercomputers based on public Linpack benchmarking results, was released Read more…

By Tiffany Trader

IBM Begins Power9 Rollout with Backing from DOE, Google

December 6, 2017

After over a year of buildup, IBM is unveiling its first Power9 system based on the same architecture as the Department of Energy CORAL supercomputers, Summit a Read more…

By Tiffany Trader

Intel Launches Software Tools to Ease FPGA Programming

September 5, 2017

Field Programmable Gate Arrays (FPGAs) have a reputation for being difficult to program, requiring expertise in specialty languages, like Verilog or VHDL. Easin Read more…

By Tiffany Trader

  • arrow
  • Click Here for More Headlines
  • arrow
Share This