HPC Clouds — Alto Cirrus or Cumulonimbus

By Thomas Sterling and Dylan Stark

November 21, 2008

The “cloud” model of exporting user workload and services to remote, distributed and virtual environments is emerging as a powerful paradigm for improving efficiency of client and server operations, enhancing quality of service, and enabling early access to unprecedented resources for many small enterprises. From single users to major commercial organizations, cloud computing is finding numerous niche opportunities, often by simplifying rapid availability of new capabilities, with minimum time to deployment and return on requirements. Yet, one domain that challenges this model in its characteristics and needs is high performance computing (HPC).

The unique demands and decades’ long experiences of HPC on the one hand hunger for the level of service that clouds promise while on the other hand impose stringent properties, at least in some cases, that may be beyond the potential of this otherwise remarkable trend. The question is, can cloud computing reach the ethereal heights of Alto Cirrus for HPC, or will it inflict the damaging thunderclap of cumulonimbus?

While HPC immediately invokes images of TOP500 machines, the petaflops performance regime, and applications that boldly compute where no machine has calculated before, in truth this domain is multivariate with many distinct class of demand. The potential role and impact of cloud computing to HPC must be viewed across the range of disparate uses embodied by the HPC community. One possible delineation of the field (in order of most stringent first) is:

  1. Highest possible delivered capability performance (strong scaling).
  2. Weak scaling single applications.
  3. Capacity, or throughput job-stream, computing.
  4. Management of massive data sets, possibly geographically distributed.
  5. Analysis and visualization of data sets.
  6. Management and administrative workloads supporting the HPC community.

Consideration of these distinct workflows exposes opportunities for the potential exploitation of the cloud model and the benefits this might convey. Starting from the bottom of the list, the HPC community involves many everyday data processing requirements that are similar to any business or academic institution. Already some of the general infrastructure needs are quietly being outsourced to cloud-like services including databases, email, web-management, information retrieval and distribution, and other routine but critical functions. However, many of these tasks can be provided by the local set of distributed workstation and small enterprise servers. Therefore the real benefit is in reducing cost of software maintenance and per head cost of software licenses, rather than reduction of cost of hardware facilities.

Offloading tasks directly associated with doing computational science, such as data analysis and visualization, are appropriate to the use of cloud services in certain cases. This is particularly true for smaller organizations that do not have the full set of software systems that are appropriate to the local requirements. Occasionally, availability of mid-scale hardware resources, such as enterprise servers, may be useful as well if queue times do not impede fast turnaround. This domain can be expanded to include the frequent introduction of new or upgraded software packages not readily available at the local site, even if open source. Where such software is provided by ISVs, the cost of ownership or licensing may exceed the budget or even the need of occasional use.

Offerings by cloud providers may find preferable incentives for use of such software. It also removes the need for local expertise in installing, tuning, and maintaining such arcane packages. This is particularly true for small groups or individual researchers. However, a recurring theme is that HPC users tend to be in environments that incorporate high levels of expertise including motivated students and young researchers, and therefore are more likely to have access to such capabilities. The use of clouds in this case will be determined by the peculiarities of the individual and his/her situation.

Although HPC is often equated to FLOPS, it is as dependent, even sometimes more so, on bytes. Much science is data oriented, comprising data acquisition, product generation, organization, correlation, archiving, mining, and presentation. Massive data sets, especially those that are intrinsically distributed among many sites are a particularly rich target for cloud services. Maintenance of large tertiary storage facilities is particularly difficult and expensive, even for the most facilities rich environments. Data management is one area of HPC in which commercial enterprises are significantly advanced, even with respect to scientific computing expertise, with significant commercial investment being applied compared to the rarified boutique scientific computing community.

One very important factor is that confidence in data integrity of large archives may ultimately be higher among cloud resource suppliers both because of their potentially distributed nature removes issues of single point failure (like hurricanes, lightening strikes, floods), and their ability to exploit substantial investments available due to economy of scale. But one, perhaps insurmountable, challenge may impose fundamental limits in the use of clouds for data storage for some mission-critical HPC user agencies and commercial research institutions: data security. Where the potential damage for leakage or corruption of data would be strategic in nature for national security or intellectual property protection, it may be implausible that such data, no matter what the quantity or putative guarantees, will be trusted to remote and sometimes unspecified service entities.

Throughput computing is an area of strong promise for HPC in the exploitation of the emerging cloud systems. Cloud services are particularly well suited for the provisioning of resources to handle application loads of many sequential or slightly parallel (everything will have to become multicore) application tasks limited to size-constrained SMP units, such as for moderate duration parametric studies. In this case, cloud services have the potential to greatly enhance an HPC institution’s available resources and operational flexibility while improving efficiency and reducing overall cost of equipment and maintenance personnel. By offloading throughput computing workloads to cloud resources, HPC investments may be better applied to those resources unique to the needs of STEM applications not adequately served by the widely-available cloud-class processing services. However, this is tempered by the important constraint discussed above related to workloads that are security or IP sensitive.

The final two regimes of the HPC scientific and technical computing arena prove more problematic for clouds. Although weak scaling applications, where the problem size grows with the system scale such that granularity of concurrency remains approximately constant, may be suitable for a subset of the class of machines available within a cloud, the virtualization demanded by the cloud environment will preclude the hardware-specific performance tuning essential to effective HPC application execution. Virtualization is an important means of achieving user productivity, but as yet it is not a path to optimal performance, especially for high scale supercomputer grade commodity clusters (e.g., Beowulf) and MPPs (e.g., Cray XT3/4/5 and IBM BG/L/P/Q). And, while auto-tuning (as part of an autonomic framework) may one day offer a path to scalable performance, current practices at this time by users of major applications demand hands-on access to the detailed specifics of the physical machine.

Where the HPC community is already plagued with sometimes single-digit efficiencies for highly-tuned codes that may run for weeks or months to completion, the loss of substantially more performance to virtualization is untenable in many cases. An additional challenge relates to I/O bandwidth, which is sometimes a serious bottleneck if not balanced with the application needs that cannot be ensured by the abstraction of the cloud. Also, the problem of checkpoint and restart is critical to major application runs but may not be a robust service incorporated as part of most cloud systems. Therefore, a suitable system would need to make appropriate guarantees with respect to the availability of hardware and software configurations that would not be representative of the broad class of clouds.

Finally, the most challenging aspect of HPC is the constantly advancing architecture and application of capability computing systems. In their most pure form they enable strong scaling where response time is reduced for fixed sized applications with increasing system scale. Such systems imply a premium cost not just because of their mammoth size comprising upwards of a million cores and tens of terabytes of main memory, but also because of their unique design and limited market, which results in the loss of economy of scale. Even when integrating many commodity devices such as microprocessors and DRAM components, the cost of such systems may be tens of millions to over a hundred million dollars.

With the very high bandwidth, low latency internal networks with specialized functionality (e.g., combining networks) and high bandwidth storage area networks for attached secondary storage, there are few commercial user domains that can help offset the NRE costs of such major and optimized computing systems. It is unlikely that a business model can be constructed that would justify such systems being made available through cloud economics. Added to this are the same issues with virtualization versus performance optimization through hands-on performance tuning as described above. Therefore, it is unlikely that clouds will satisfy capability computing challenges for computational science in the foreseeable future.

The evolution of the cloud paradigm is an important maturing of the power of microelectronics, distributed computing, the Internet, and the rapidly expanding role of computing in all aspects of human enterprise and social context. The HPC and scientific computing community will benefit in tangential ways from the cloud environments as they evolve and where appropriate. However, challenges of virtualization and performance optimization, security and intellectual property protection, and unique requirements of scale and functionality, will result in certain critical aspects of the requirements of HPC falling outside the domain of cloud computing, relying instead on the strong foundation upon which HPC is well grounded.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

GDPR’s Impact on Scientific Research Uncertain

May 24, 2018

Amid the angst over preparations—or lack thereof—for new European Union data protections entering into force at week’s end is the equally worrisome issue of the rules’ impact on scientific research. Among the Read more…

By George Leopold

Intel Pledges First Commercial Nervana Product ‘Spring Crest’ in 2019

May 24, 2018

At its AI developer conference in San Francisco yesterday, Intel embraced a holistic approach to AI and showed off a broad AI portfolio that includes Xeon processors, Movidius technologies, FPGAs and Intel’s Nervana Neural Network Processors (NNPs), based on the technology it acquired in 2016. Read more…

By Tiffany Trader

Pattern Computer – Startup Claims Breakthrough in ‘Pattern Discovery’ Technology

May 23, 2018

If it weren’t for the heavy-hitter technology team behind start-up Pattern Computer, which emerged from stealth today in a live-streamed event from San Francisco, one would be tempted to dismiss its claims of inventing Read more…

By John Russell

HPE Extreme Performance Solutions

HPC and AI Convergence is Accelerating New Levels of Intelligence

Data analytics is the most valuable tool in the digital marketplace – so much so that organizations are employing high performance computing (HPC) capabilities to rapidly collect, share, and analyze endless streams of data. Read more…

IBM Accelerated Insights

Mastering the Big Data Challenge in Cognitive Healthcare

Patrick Chain, genomics researcher at Los Alamos National Laboratory, posed a question in a recent blog: What if a nurse could swipe a patient’s saliva and run a quick genetic test to determine if the patient’s sore throat was caused by a cold virus or a bacterial infection? Read more…

Silicon Startup Raises ‘Prodigy’ for Hyperscale/AI Workloads

May 23, 2018

There's another silicon startup coming onto the HPC/hyperscale scene with some intriguing and bold claims. Silicon Valley-based Tachyum Inc., which has been emerging from stealth over the last year and a half, is unveili Read more…

By Tiffany Trader

Intel Pledges First Commercial Nervana Product ‘Spring Crest’ in 2019

May 24, 2018

At its AI developer conference in San Francisco yesterday, Intel embraced a holistic approach to AI and showed off a broad AI portfolio that includes Xeon processors, Movidius technologies, FPGAs and Intel’s Nervana Neural Network Processors (NNPs), based on the technology it acquired in 2016. Read more…

By Tiffany Trader

Pattern Computer – Startup Claims Breakthrough in ‘Pattern Discovery’ Technology

May 23, 2018

If it weren’t for the heavy-hitter technology team behind start-up Pattern Computer, which emerged from stealth today in a live-streamed event from San Franci Read more…

By John Russell

Silicon Startup Raises ‘Prodigy’ for Hyperscale/AI Workloads

May 23, 2018

There's another silicon startup coming onto the HPC/hyperscale scene with some intriguing and bold claims. Silicon Valley-based Tachyum Inc., which has been eme Read more…

By Tiffany Trader

Japan Meteorological Agency Takes Delivery of Pair of Crays

May 21, 2018

Cray has supplied two identical Cray XC50 supercomputers to the Japan Meteorological Agency (JMA) in northwestern Tokyo. Boasting more than 18 petaflops combine Read more…

By Tiffany Trader

ASC18: Final Results Revealed & Wrapped Up

May 17, 2018

It was an exciting week at ASC18 in Nanyang, China. The student teams braved extreme heat, extremely difficult applications, and extreme competition in order to cross the cluster competition finish line. The gala awards ceremony took place on Wednesday. The auditorium was packed with student teams, various dignitaries, the media, and other interested parties. So what happened? Read more…

By Dan Olds

Spring Meetings Underscore Quantum Computing’s Rise

May 17, 2018

The month of April 2018 saw four very important and interesting meetings to discuss the state of quantum computing technologies, their potential impacts, and th Read more…

By Alex R. Larzelere

Quantum Network Hub Opens in Japan

May 17, 2018

Following on the launch of its Q Commercial quantum network last December with 12 industrial and academic partners, the official Japanese hub at Keio University is now open to facilitate the exploration of quantum applications important to science and business. The news comes a week after IBM announced that North Carolina State University was the first U.S. university to join its Q Network. Read more…

By Tiffany Trader

Democratizing HPC: OSC Releases Version 1.3 of OnDemand

May 16, 2018

Making HPC resources readily available and easier to use for scientists who may have less HPC expertise is an ongoing challenge. Open OnDemand is a project by t Read more…

By John Russell

MLPerf – Will New Machine Learning Benchmark Help Propel AI Forward?

May 2, 2018

Let the AI benchmarking wars begin. Today, a diverse group from academia and industry – Google, Baidu, Intel, AMD, Harvard, and Stanford among them – releas Read more…

By John Russell

How the Cloud Is Falling Short for HPC

March 15, 2018

The last couple of years have seen cloud computing gradually build some legitimacy within the HPC world, but still the HPC industry lies far behind enterprise I Read more…

By Chris Downing

Russian Nuclear Engineers Caught Cryptomining on Lab Supercomputer

February 12, 2018

Nuclear scientists working at the All-Russian Research Institute of Experimental Physics (RFNC-VNIIEF) have been arrested for using lab supercomputing resources to mine crypto-currency, according to a report in Russia’s Interfax News Agency. Read more…

By Tiffany Trader

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

Deep Learning at 15 PFlops Enables Training for Extreme Weather Identification at Scale

March 19, 2018

Petaflop per second deep learning training performance on the NERSC (National Energy Research Scientific Computing Center) Cori supercomputer has given climate Read more…

By Rob Farber

AI Cloud Competition Heats Up: Google’s TPUs, Amazon Building AI Chip

February 12, 2018

Competition in the white hot AI (and public cloud) market pits Google against Amazon this week, with Google offering AI hardware on its cloud platform intended Read more…

By Doug Black

US Plans $1.8 Billion Spend on DOE Exascale Supercomputing

April 11, 2018

On Monday, the United States Department of Energy announced its intention to procure up to three exascale supercomputers at a cost of up to $1.8 billion with th Read more…

By Tiffany Trader

Lenovo Unveils Warm Water Cooled ThinkSystem SD650 in Rampup to LRZ Install

February 22, 2018

This week Lenovo took the wraps off the ThinkSystem SD650 high-density server with third-generation direct water cooling technology developed in tandem with par Read more…

By Tiffany Trader

Leading Solution Providers

SC17 Booth Video Tours Playlist

Altair @ SC17


AMD @ SC17


ASRock Rack @ SC17

ASRock Rack



DDN Storage @ SC17

DDN Storage

Huawei @ SC17


IBM @ SC17


IBM Power Systems @ SC17

IBM Power Systems

Intel @ SC17


Lenovo @ SC17


Mellanox Technologies @ SC17

Mellanox Technologies

Microsoft @ SC17


Penguin Computing @ SC17

Penguin Computing

Pure Storage @ SC17

Pure Storage

Supericro @ SC17


Tyan @ SC17


Univa @ SC17


Google Chases Quantum Supremacy with 72-Qubit Processor

March 7, 2018

Google pulled ahead of the pack this week in the race toward "quantum supremacy," with the introduction of a new 72-qubit quantum processor called Bristlecone. Read more…

By Tiffany Trader

CFO Steps down in Executive Shuffle at Supermicro

January 31, 2018

Supermicro yesterday announced senior management shuffling including prominent departures, the completion of an audit linked to its delayed Nasdaq filings, and Read more…

By John Russell

HPE Wins $57 Million DoD Supercomputing Contract

February 20, 2018

Hewlett Packard Enterprise (HPE) today revealed details of its massive $57 million HPC contract with the U.S. Department of Defense (DoD). The deal calls for HP Read more…

By Tiffany Trader

Deep Learning Portends ‘Sea Change’ for Oil and Gas Sector

February 1, 2018

The billowing compute and data demands that spurred the oil and gas industry to be the largest commercial users of high-performance computing are now propelling Read more…

By Tiffany Trader

Nvidia Ups Hardware Game with 16-GPU DGX-2 Server and 18-Port NVSwitch

March 27, 2018

Nvidia unveiled a raft of new products from its annual technology conference in San Jose today, and despite not offering up a new chip architecture, there were still a few surprises in store for HPC hardware aficionados. Read more…

By Tiffany Trader

Hennessy & Patterson: A New Golden Age for Computer Architecture

April 17, 2018

On Monday June 4, 2018, 2017 A.M. Turing Award Winners John L. Hennessy and David A. Patterson will deliver the Turing Lecture at the 45th International Sympo Read more…

By Staff

Part One: Deep Dive into 2018 Trends in Life Sciences HPC

March 1, 2018

Life sciences is an interesting lens through which to see HPC. It is perhaps not an obvious choice, given life sciences’ relative newness as a heavy user of H Read more…

By John Russell

Google I/O 2018: AI Everywhere; TPU 3.0 Delivers 100+ Petaflops but Requires Liquid Cooling

May 9, 2018

All things AI dominated discussion at yesterday’s opening of Google’s I/O 2018 developers meeting covering much of Google's near-term product roadmap. The e Read more…

By John Russell

  • arrow
  • Click Here for More Headlines
  • arrow
Share This