Energy costs of running clusters has always been a consideration when operating an infrastructure for high-performance computing (HPC).  As clusters become larger in the drive to the next levels of computing performance, energy efficiency has emerged as one of the foremost design goals.  Relying upon contemporary technologies is simply not enough – as power demand for Exascale class systems would require hundreds of Megawatts of power.  New approaches for energy optimization are being explored which optimize throughout the whole HPC stack – from firmware and hardware through to the OS, application runtimes and workload managers.

For example, in order to achieve a sustainable power draw, future HPC systems of Exascale class will have to feature a power efficiency of approximately 50 GFLOPS/Watt and peak power demands in the range of ~20-30 Megawatts.  The current state of the art as per June 2018 TOP500 list is:

• Greenest system: 18.4 GFLOPS/Watt
• Fastest system: 889 GFLOPS/Watt 

There is a growing consensus in the HPC community that the challenge of optimizing for energy efficiency requires an orchestrated approach across hardware, firmware and all levels of software stack of HPC systems.  Power and energy management needs to happen on a node level, job/task level and cluster/cloud level using various level schedulers all working in concert.  Additionally, wide adoption of heterogeneous systems and evolving requirements for complex workflows presents additional challenges to the performance and energy efficiency optimizations. Finally, there are efforts of augmenting schedulers by insights gained from data center level telemetry data using machine learning techniques.

IBM® Research along with IBM® Spectrum Computing software have endeavored to be at the forefront of the energy efficiency research.  Our approach is illustrated in Figure 1 below.  To expand on this, energy is proportional to power. Reducing energy consumption means reducing power to solution or time to solution or both.  Managing power variations and power/performance imbalance across nodes are also essential tasks of energy management.  Our approach to energy/power management can be described as Energy Aware Scheduling (EAS), which uses performance and power consumption models and software/hardware co-design for implementing various energy/power aware scheduling policies at the node, job and cluster levels.

Figure 1 IBM approach to energy aware scheduling (EAS)

One of the notable examples is the long-term collaboration with the STFC Hartree Centre on implementing EAS techniques on several generations of their supercomputers. The outcome of the work is documented in numerous research papers and conference presentations, where energy savings of up to 12% energy savings were observed. Furthermore, this led to the research in the area of energy aware scheduling in the IBM Spectrum LSF workload manager.

More specifically we have developed machine learning based models for prediction of performance and power consumption of HPC workloads, along with scheduler plugins for cluster level energy budget management. As part of this effort, we have also ported the open source framework, Global Extensible Open Power Manager, which is a job level scheduler for energy efficiency to the OpenPOWER® architecture (POWER8®).  This work was recently presented at the International Workshop on OpenPOWER for HPC (IWOPH) 2018  and the full presentation can be viewed here.

As we have  discussed, energy efficient computing is a multi-dimensional challenge that is a key consideration in particular for Exascale computing.  We can anticipate further research efforts in this area helping to balance the need for HPC performance with the energy efficiency.

To learn more about IBM solutions for HPC and AI, join us at booth #3433 at Supercomputing 2018 this November in Dallas, Texas.  Read more about IBM at Supercomputing here.

This work was supported by the STFC Hartree Centre’s Innovation Return on Research programme, funded by the Department for Business, Energy & Industrial Strategy.