HPC Clouds — Alto Cirrus or Cumulonimbus

By Thomas Sterling and Dylan Stark

November 21, 2008

The “cloud” model of exporting user workload and services to remote, distributed and virtual environments is emerging as a powerful paradigm for improving efficiency of client and server operations, enhancing quality of service, and enabling early access to unprecedented resources for many small enterprises. From single users to major commercial organizations, cloud computing is finding numerous niche opportunities, often by simplifying rapid availability of new capabilities, with minimum time to deployment and return on requirements. Yet, one domain that challenges this model in its characteristics and needs is high performance computing (HPC).

The unique demands and decades’ long experiences of HPC on the one hand hunger for the level of service that clouds promise while on the other hand impose stringent properties, at least in some cases, that may be beyond the potential of this otherwise remarkable trend. The question is, can cloud computing reach the ethereal heights of Alto Cirrus for HPC, or will it inflict the damaging thunderclap of cumulonimbus?

While HPC immediately invokes images of TOP500 machines, the petaflops performance regime, and applications that boldly compute where no machine has calculated before, in truth this domain is multivariate with many distinct class of demand. The potential role and impact of cloud computing to HPC must be viewed across the range of disparate uses embodied by the HPC community. One possible delineation of the field (in order of most stringent first) is:

  1. Highest possible delivered capability performance (strong scaling).
  2. Weak scaling single applications.
  3. Capacity, or throughput job-stream, computing.
  4. Management of massive data sets, possibly geographically distributed.
  5. Analysis and visualization of data sets.
  6. Management and administrative workloads supporting the HPC community.

Consideration of these distinct workflows exposes opportunities for the potential exploitation of the cloud model and the benefits this might convey. Starting from the bottom of the list, the HPC community involves many everyday data processing requirements that are similar to any business or academic institution. Already some of the general infrastructure needs are quietly being outsourced to cloud-like services including databases, email, web-management, information retrieval and distribution, and other routine but critical functions. However, many of these tasks can be provided by the local set of distributed workstation and small enterprise servers. Therefore the real benefit is in reducing cost of software maintenance and per head cost of software licenses, rather than reduction of cost of hardware facilities.

Offloading tasks directly associated with doing computational science, such as data analysis and visualization, are appropriate to the use of cloud services in certain cases. This is particularly true for smaller organizations that do not have the full set of software systems that are appropriate to the local requirements. Occasionally, availability of mid-scale hardware resources, such as enterprise servers, may be useful as well if queue times do not impede fast turnaround. This domain can be expanded to include the frequent introduction of new or upgraded software packages not readily available at the local site, even if open source. Where such software is provided by ISVs, the cost of ownership or licensing may exceed the budget or even the need of occasional use.

Offerings by cloud providers may find preferable incentives for use of such software. It also removes the need for local expertise in installing, tuning, and maintaining such arcane packages. This is particularly true for small groups or individual researchers. However, a recurring theme is that HPC users tend to be in environments that incorporate high levels of expertise including motivated students and young researchers, and therefore are more likely to have access to such capabilities. The use of clouds in this case will be determined by the peculiarities of the individual and his/her situation.

Although HPC is often equated to FLOPS, it is as dependent, even sometimes more so, on bytes. Much science is data oriented, comprising data acquisition, product generation, organization, correlation, archiving, mining, and presentation. Massive data sets, especially those that are intrinsically distributed among many sites are a particularly rich target for cloud services. Maintenance of large tertiary storage facilities is particularly difficult and expensive, even for the most facilities rich environments. Data management is one area of HPC in which commercial enterprises are significantly advanced, even with respect to scientific computing expertise, with significant commercial investment being applied compared to the rarified boutique scientific computing community.

One very important factor is that confidence in data integrity of large archives may ultimately be higher among cloud resource suppliers both because of their potentially distributed nature removes issues of single point failure (like hurricanes, lightening strikes, floods), and their ability to exploit substantial investments available due to economy of scale. But one, perhaps insurmountable, challenge may impose fundamental limits in the use of clouds for data storage for some mission-critical HPC user agencies and commercial research institutions: data security. Where the potential damage for leakage or corruption of data would be strategic in nature for national security or intellectual property protection, it may be implausible that such data, no matter what the quantity or putative guarantees, will be trusted to remote and sometimes unspecified service entities.

Throughput computing is an area of strong promise for HPC in the exploitation of the emerging cloud systems. Cloud services are particularly well suited for the provisioning of resources to handle application loads of many sequential or slightly parallel (everything will have to become multicore) application tasks limited to size-constrained SMP units, such as for moderate duration parametric studies. In this case, cloud services have the potential to greatly enhance an HPC institution’s available resources and operational flexibility while improving efficiency and reducing overall cost of equipment and maintenance personnel. By offloading throughput computing workloads to cloud resources, HPC investments may be better applied to those resources unique to the needs of STEM applications not adequately served by the widely-available cloud-class processing services. However, this is tempered by the important constraint discussed above related to workloads that are security or IP sensitive.

The final two regimes of the HPC scientific and technical computing arena prove more problematic for clouds. Although weak scaling applications, where the problem size grows with the system scale such that granularity of concurrency remains approximately constant, may be suitable for a subset of the class of machines available within a cloud, the virtualization demanded by the cloud environment will preclude the hardware-specific performance tuning essential to effective HPC application execution. Virtualization is an important means of achieving user productivity, but as yet it is not a path to optimal performance, especially for high scale supercomputer grade commodity clusters (e.g., Beowulf) and MPPs (e.g., Cray XT3/4/5 and IBM BG/L/P/Q). And, while auto-tuning (as part of an autonomic framework) may one day offer a path to scalable performance, current practices at this time by users of major applications demand hands-on access to the detailed specifics of the physical machine.

Where the HPC community is already plagued with sometimes single-digit efficiencies for highly-tuned codes that may run for weeks or months to completion, the loss of substantially more performance to virtualization is untenable in many cases. An additional challenge relates to I/O bandwidth, which is sometimes a serious bottleneck if not balanced with the application needs that cannot be ensured by the abstraction of the cloud. Also, the problem of checkpoint and restart is critical to major application runs but may not be a robust service incorporated as part of most cloud systems. Therefore, a suitable system would need to make appropriate guarantees with respect to the availability of hardware and software configurations that would not be representative of the broad class of clouds.

Finally, the most challenging aspect of HPC is the constantly advancing architecture and application of capability computing systems. In their most pure form they enable strong scaling where response time is reduced for fixed sized applications with increasing system scale. Such systems imply a premium cost not just because of their mammoth size comprising upwards of a million cores and tens of terabytes of main memory, but also because of their unique design and limited market, which results in the loss of economy of scale. Even when integrating many commodity devices such as microprocessors and DRAM components, the cost of such systems may be tens of millions to over a hundred million dollars.

With the very high bandwidth, low latency internal networks with specialized functionality (e.g., combining networks) and high bandwidth storage area networks for attached secondary storage, there are few commercial user domains that can help offset the NRE costs of such major and optimized computing systems. It is unlikely that a business model can be constructed that would justify such systems being made available through cloud economics. Added to this are the same issues with virtualization versus performance optimization through hands-on performance tuning as described above. Therefore, it is unlikely that clouds will satisfy capability computing challenges for computational science in the foreseeable future.

The evolution of the cloud paradigm is an important maturing of the power of microelectronics, distributed computing, the Internet, and the rapidly expanding role of computing in all aspects of human enterprise and social context. The HPC and scientific computing community will benefit in tangential ways from the cloud environments as they evolve and where appropriate. However, challenges of virtualization and performance optimization, security and intellectual property protection, and unique requirements of scale and functionality, will result in certain critical aspects of the requirements of HPC falling outside the domain of cloud computing, relying instead on the strong foundation upon which HPC is well grounded.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

The EU Human Brain Project Reboots but Supercomputing Still Needed

June 26, 2017

The often contentious, EU-funded Human Brain Project whose initial aim was fixed firmly on full-brain simulation is now in the midst of a reboot targeting a more modest goal – development of informatics tools and data/ Read more…

By John Russell

DOE Launches Chicago Quantum Exchange

June 26, 2017

While many of us were preoccupied with ISC 2017 last week, the launch of the Chicago Quantum Exchange went largely unnoticed. So what is such a thing? It is a Department of Energy sponsored collaboration between the Univ Read more…

By John Russell

UMass Dartmouth Reports on HPC Day 2017 Activities

June 26, 2017

UMass Dartmouth's Center for Scientific Computing & Visualization Research (CSCVR) organized and hosted the third annual "HPC Day 2017" on May 25th. This annual event showcases on-going scientific research in Massach Read more…

By Gaurav Khanna

How ‘Knights Mill’ Gets Its Deep Learning Flops

June 22, 2017

Intel, the subject of much speculation regarding the delayed, rewritten or potentially canceled “Aurora” contract (the Argonne Lab part of the CORAL “pre-exascale” award), parsed out additional information ab Read more…

By Tiffany Trader

HPE Extreme Performance Solutions

Creating a Roadmap for HPC Innovation at ISC 2017

In an era where technological advancements are driving innovation to every sector, and powering major economic and scientific breakthroughs, high performance computing (HPC) is crucial to tackle the challenges of today and tomorrow. Read more…

Tsinghua Crowned Eight-Time Student Cluster Champions at ISC

June 22, 2017

Always a hard-fought competition, the Student Cluster Competition awards were announced Wednesday, June 21, at the ISC High Performance Conference 2017. Amid whoops and hollers from the crowd, Thomas Sterling presented t Read more…

By Kim McMahon

GPUs, Power9, Figure Prominently in IBM’s Bet on Weather Forecasting

June 22, 2017

IBM jumped into the weather forecasting business roughly a year and a half ago by purchasing The Weather Company. This week at ISC 2017, Big Blue rolled out plans to push deeper into climate science and develop more gran Read more…

By John Russell

Intersect 360 at ISC: HPC Industry at $44B by 2021

June 22, 2017

The care, feeding and sustained growth of the HPC industry increasingly is in the hands of the commercial market sector – in particular, it’s the hyperscale companies and their embrace of AI and deep learning – tha Read more…

By Doug Black

At ISC – Goh on Go: Humans Can’t Scale, the Data-Centric Learning Machine Can

June 22, 2017

I've seen the future this week at ISC, it’s on display in prototype or Powerpoint form, and it’s going to dumbfound you. The future is an AI neural network designed to emulate and compete with the human brain. In thi Read more…

By Doug Black

DOE Launches Chicago Quantum Exchange

June 26, 2017

While many of us were preoccupied with ISC 2017 last week, the launch of the Chicago Quantum Exchange went largely unnoticed. So what is such a thing? It is a D Read more…

By John Russell

How ‘Knights Mill’ Gets Its Deep Learning Flops

June 22, 2017

Intel, the subject of much speculation regarding the delayed, rewritten or potentially canceled “Aurora” contract (the Argonne Lab part of the CORAL “ Read more…

By Tiffany Trader

Tsinghua Crowned Eight-Time Student Cluster Champions at ISC

June 22, 2017

Always a hard-fought competition, the Student Cluster Competition awards were announced Wednesday, June 21, at the ISC High Performance Conference 2017. Amid wh Read more…

By Kim McMahon

GPUs, Power9, Figure Prominently in IBM’s Bet on Weather Forecasting

June 22, 2017

IBM jumped into the weather forecasting business roughly a year and a half ago by purchasing The Weather Company. This week at ISC 2017, Big Blue rolled out pla Read more…

By John Russell

Intersect 360 at ISC: HPC Industry at $44B by 2021

June 22, 2017

The care, feeding and sustained growth of the HPC industry increasingly is in the hands of the commercial market sector – in particular, it’s the hyperscale Read more…

By Doug Black

At ISC – Goh on Go: Humans Can’t Scale, the Data-Centric Learning Machine Can

June 22, 2017

I've seen the future this week at ISC, it’s on display in prototype or Powerpoint form, and it’s going to dumbfound you. The future is an AI neural network Read more…

By Doug Black

Cray Brings AI and HPC Together on Flagship Supers

June 20, 2017

Cray took one more step toward the convergence of big data and high performance computing (HPC) today when it announced that it’s adding a full suite of big d Read more…

By Alex Woodie

AMD Charges Back into the Datacenter and HPC Workflows with EPYC Processor

June 20, 2017

AMD is charging back into the enterprise datacenter and select HPC workflows with its new EPYC 7000 processor line, code-named Naples, announced today at a “g Read more…

By John Russell

Quantum Bits: D-Wave and VW; Google Quantum Lab; IBM Expands Access

March 21, 2017

For a technology that’s usually characterized as far off and in a distant galaxy, quantum computing has been steadily picking up steam. Just how close real-wo Read more…

By John Russell

Trump Budget Targets NIH, DOE, and EPA; No Mention of NSF

March 16, 2017

President Trump’s proposed U.S. fiscal 2018 budget issued today sharply cuts science spending while bolstering military spending as he promised during the cam Read more…

By John Russell

HPC Compiler Company PathScale Seeks Life Raft

March 23, 2017

HPCwire has learned that HPC compiler company PathScale has fallen on difficult times and is asking the community for help or actively seeking a buyer for its a Read more…

By Tiffany Trader

Google Pulls Back the Covers on Its First Machine Learning Chip

April 6, 2017

This week Google released a report detailing the design and performance characteristics of the Tensor Processing Unit (TPU), its custom ASIC for the inference Read more…

By Tiffany Trader

CPU-based Visualization Positions for Exascale Supercomputing

March 16, 2017

In this contributed perspective piece, Intel’s Jim Jeffers makes the case that CPU-based visualization is now widely adopted and as such is no longer a contrarian view, but is rather an exascale requirement. Read more…

By Jim Jeffers, Principal Engineer and Engineering Leader, Intel

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

Nvidia’s Mammoth Volta GPU Aims High for AI, HPC

May 10, 2017

At Nvidia's GPU Technology Conference (GTC17) in San Jose, Calif., this morning, CEO Jensen Huang announced the company's much-anticipated Volta architecture a Read more…

By Tiffany Trader

Facebook Open Sources Caffe2; Nvidia, Intel Rush to Optimize

April 18, 2017

From its F8 developer conference in San Jose, Calif., today, Facebook announced Caffe2, a new open-source, cross-platform framework for deep learning. Caffe2 is the successor to Caffe, the deep learning framework developed by Berkeley AI Research and community contributors. Read more…

By Tiffany Trader

Leading Solution Providers

MIT Mathematician Spins Up 220,000-Core Google Compute Cluster

April 21, 2017

On Thursday, Google announced that MIT math professor and computational number theorist Andrew V. Sutherland had set a record for the largest Google Compute Engine (GCE) job. Sutherland ran the massive mathematics workload on 220,000 GCE cores using preemptible virtual machine instances. Read more…

By Tiffany Trader

Google Debuts TPU v2 and will Add to Google Cloud

May 25, 2017

Not long after stirring attention in the deep learning/AI community by revealing the details of its Tensor Processing Unit (TPU), Google last week announced the Read more…

By John Russell

US Supercomputing Leaders Tackle the China Question

March 15, 2017

Joint DOE-NSA report responds to the increased global pressures impacting the competitiveness of U.S. supercomputing. Read more…

By Tiffany Trader

Russian Researchers Claim First Quantum-Safe Blockchain

May 25, 2017

The Russian Quantum Center today announced it has overcome the threat of quantum cryptography by creating the first quantum-safe blockchain, securing cryptocurrencies like Bitcoin, along with classified government communications and other sensitive digital transfers. Read more…

By Doug Black

Groq This: New AI Chips to Give GPUs a Run for Deep Learning Money

April 24, 2017

CPUs and GPUs, move over. Thanks to recent revelations surrounding Google’s new Tensor Processing Unit (TPU), the computing world appears to be on the cusp of Read more…

By Alex Woodie

DOE Supercomputer Achieves Record 45-Qubit Quantum Simulation

April 13, 2017

In order to simulate larger and larger quantum systems and usher in an age of “quantum supremacy,” researchers are stretching the limits of today’s most advanced supercomputers. Read more…

By Tiffany Trader

Messina Update: The US Path to Exascale in 16 Slides

April 26, 2017

Paul Messina, director of the U.S. Exascale Computing Project, provided a wide-ranging review of ECP’s evolving plans last week at the HPC User Forum. Read more…

By John Russell

Six Exascale PathForward Vendors Selected; DoE Providing $258M

June 15, 2017

The much-anticipated PathForward awards for hardware R&D in support of the Exascale Computing Project were announced today with six vendors selected – AMD Read more…

By John Russell

  • arrow
  • Click Here for More Headlines
  • arrow
Share This