Top 10 Objections to GPU Computing Reconsidered

By Dr. Vincent Natoli

June 9, 2011

By any measure, the emergence of GPU computing in the HPC world over the last few years has been a remarkable and unprecedented development. It has all the markings of a disruptive technology. It began with a small cadre of adherents willing to brave the complications of shoehorning graphics languages like Cg and OpenGL’s GLSL into performing general-purpose calculations. It was championed by a hungry new player in HPC. It provides step out, order of magnitude performance gains over conventional solutions in more cases than people would like to admit and it’s just plain different.

As we approach the four-year release anniversary of NVIDIA CUDA, arguably the ground zero of the GPGPU movement, there are many who have flirted, piloted and adopted the technology, but many more who are sitting on the sidelines for various reasons. In our work, we have come across many of the latter, and have thus compiled a list of the most common questions, concerns and assertions that preempt efforts to evaluate the technology.

What follows below is an attempt to list and address these objections, reconsidering them in light of recent developments and our best guess at how the future will unfold. To be clear, GPGPU is not the solution for all HPC applications but many have found the technology cost effective in domains as diverse as seismic imaging, electromagnetics, molecular dynamics, financial valuation, medical imaging and others.

1. I don’t want to rewrite my code or learn a new language

It’s true that you will have to rewrite your code to make use of GPUs. However if you are developing a parallel CPU implementation from a mostly serial code rewriting your code is inevitable. The only question is which platform will you target. Targeting multicore CPUs presents a three-tier model with parallelism at the process, thread and register levels where you will use MPI, OpenMP/pthreads and SSE/AVX extensions. Programming for GPUs using CUDA is no more difficult and the advantage for both compute-bound and memory-bound codes can be significant, as we’ll see.

If you already have a parallel code, what benefit will you reap from a GPU port? Back-of-the-envelope calculations yield between 5x and 40x improvements in chip-to-chip comparisons on typical codes. This metric is also what we have seen anecdotally and in the many publications on GPU enabled applications. These comparisons have remained fairly constant over the past few years spanning two generations of Intel and NVIDIA products.

CUDA is an extension to C and relatively easily picked up by experienced coders. The model for parallel programming that will take us to the exascale is far from settled, however I believe that the eventual solution will look more like the data parallel CUDA model than the task parallel CPU alternatives. In previous HPCwire contributions I have expressed the opinion that CUDA, by forcing developers to consider the irreducible level of parallel work in their problems and mapping that to threads, is a better programming model for extensible parallelism, lending itself more naturally to scalability across multiple GPUs on a single node and across multiple nodes.

Steps in this direction are already underway with examples from the academic world with the excellent work on Global Memory for Accelerators (GMAC) being jointly developed by Universitat Politecnica de Catalunya and the IMPACT Research Group at the University of Illinois, and the commercial world with the unique scaling capabilities of the HUESPACE compute API offered by Oslo-based startup HUE and their sister company focusing in oil and gas, Headwave.

2. I don’t know what kind of performance to expect

HPC codes are either compute-bound or memory-bound. For compute-bound codes we compare the NVIDIA Fermi M2090 with the Intel Westmere. The Fermi has 512 cores running at about 1.3GHz while Westmere has 6 cores running at up to 3.4 GHz. The ratio of core-Hz yields a factor of about 32x. If your CPU code is effectively using SSE instructions, that might give you an additional boost of 4x on the CPU side reducing GPU efficacy to 8x (close to the ratio of peak GFLOPS).

For memory-bound codes we compare GPU main memory bandwidth of 177 GB/second to 32 GB/second available per processor on CPU motherboards for a ratio of about 5.5x. The bottom line is that if your code is compute bound expect a GPU boost of between 5x for highly optimized SSE implemented codes and 20x or more for typical codes. If your code is memory bound expect a boost of about 5x in chip to chip comparisons.

It’s helpful when considering a parallel solution to think about the problem in terms of marginal cost. If your code is memory bound then you should consider the least expensive option for adding bandwidth. One can add another GPU card and buy bandwidth for about $15 per GB/second. Alternatively one can add a separate node at a nominal cost of about $80 per GB/second. The latter solution also increases the compute footprint and adds another O/S instance.

For compute-bound codes a similar calculation can be performed to produce the marginal cost of gigaflops. For codes that have a mixture of compute-bound and memory-bound sections i.e., most codes, the GPU does a better job of hiding the latency of memory-bound sections by juggling thousands of threads and computing continuously on those threads that have their data registers filled and ready to go. By hiding memory latency the GPU is better able to achieve the higher compute-bound performance limits.

3. The PCIe bandwidth will kill my performance

The argument against GPU computing based on PCIe bandwidth is really one of computational intensity. Computational intensity is variously defined but for these purposes let us say it is the number of floating point operations performed on each float of data transferred. There is a threshold of work which must be applied to every byte of data transferred to the GPU board in order for a GPU compute effort to be worthwhile.

For example, PCIe v2.0 x16 bandwidth totals about 6 GB/second in practice. In perspective it can fill the 6 GB of main memory on the M2090 board in about a second. At 665 double precision gigaflops peak performance the M2090 is a floating point monster and can clearly do a lot of processing in the second that it takes to fill the board memory. If for example you’d like the PCIe transfer time to be no more than 10 percent of the compute time, the M2090 must do thousands of floating point operations on the stored data before flushing it. GPU algorithms must therefore strive to retain data on the boards for as long as possible.

In addition, CUDA allows the asynchronous overlap of PCIe data transfer with calculation. Clever use of this feature lets the developer hide some or all of the PCIe data transfer time behind computations. Algorithms that work well include time stepping algorithms with local physics such as Finite Difference Time Domain (FDTD) or classical molecular dynamics where the N2 particle-particle interactions lead to significant data reuse and high computational intensity.

Algorithms that are not effective in isolation on GPUs such as simple vector dot products have very low computational intensity. If problems are mapped across multiple GPUs they should minimize data transfer, for example sending only border data in a domain decomposition.

4. What about Amdahl’s law?

Amdahl’s law quantitatively expresses the fact that if you are going to accelerate part of a large serial code either by whatever means, parallelization or magic gnomes, you had better accelerate a significant portion or you will not realize much benefit. Amdahl’s law is often held forth as a talisman against any suggestion that code performance can be improved via parallelization.

Imagine, for example, you could reduce the runtime of a part of your serial code to zero. If that portion of the code accounted for 50 percent of the runtime then your overall speedup would only be 2x; if it accounted for 90 percent of runtime your speedup would approach 10x. For more realistic accelerations speedup results are lower.

The most effective counter to an Amdahl’s law objection to GPU computing is the observation that performance on modern architectures requires all codes to be written for massive parallelism reducing their serial portions to the minimum possible…there is no other game in town. This is true on CPU platforms as well as on GPU platforms.

The only HPC compute platforms we have going forward are massively parallel and Amdahl’s law applies equally well to CPU implementations. The real question is where are you going to run the parallel portion of your code…on the CPU or on the GPU?

5. What if NVIDIA goes away?

The history of HPC is littered with the carcasses of many a supercomputing company that tried to make the leap to parallel computing. A short list includes Thinking Machines, Maspar, KSR, Mitrion and many others. These often heroic efforts and the people behind them should be recognized for their vision and the role they played in the creative disruption that over time led to a greater collective understanding of what works and what doesn’t. We owe a debt of gratitude to them all.

NVIDIA, however, is not a supercomputing company. It’s a $5 billion corporation that makes most of its money from video cards and embedded processors that it sells to the huge avid market of PC gamers. Its relative independence from HPC is a strength, and if all HPC usage of GPU computing disappeared NVIDIA would still have a fine and profitable business. As long as there are caffeine-addicted adolescents willing to play virtual war games NVIDIA will be around. The fact is that NVIDIA’s staying power in the market is greater and more secure than that of long-time HPC icon Cray.

Further, NVIDIA has publicly posted its vision and roadmap for technology development out to roughly 6 years. Reading between the lines it is extremely aggressive with ambitions to move the GPU from its ancillary role living on an accelerator card to a more central role in the compute infrastructure. Along the way they have some powerful compute engines planned.

6. GPU boards don’t offer enough main memory for my problem.

GPU board memory is currently limited to 6 GB available on the M2090 and M2070. This can be particularly problematic for algorithms that require large chunks of memory in excess of that. This problem is somewhat alleviated by access to multiple cards available to a single node.

For example, the Dell C410x PCIe expansion chassis that can contain up to 16 NVIDIA boards for a total of 96 GB. Domain decomposing your problem on 16 separate GPUs can be a very effective way to side-step the memory limitation and it works quite well for problems effecting local physics that calculate properties on volumes and share data at surfaces.

The most problematic are algorithms that require essentially random access to large arrays, for example, huge hash tables or other problems that require random array lookups. The current generation of GPU boards are not effective solutions in this case, however, memory being relatively inexpensive and constantly improving in density, one can expect future generations of boards to come with increased amounts.

7. I will wait for more CPU cores / Knights Corner

More cores will help with compute-bound applications, however one needs to consider that as more cores are added to CPU dies, the same will be true for GPUs. Comparing roadmaps over the past two technology generations shows a persistent gap in compute and bandwidth between CPUs and GPUs that sometimes grows and sometimes wanes but is consistently non-zero. Expect this to continue going forward. For bandwidth-bound problems the situation is somewhat worse as it appears easier to add cores than to add bandwidth.

Intel’s plans for Knights Corner, announced just over a year ago, recognizes the need for an x86 data parallel competitor to GPUs. Full details on Knights Corner are still unknown however using specs from the Knights Ferry prototype as a baseline, we might expect 50 or more 1.2 GHz cores, each with its own 512-bit vector processing unit supporting up to 4 threads each, making it a formidable HPC competitor. However Intel’s plans for the development model, pricing, release dates and other critical information is, at best, poorly understood and communicated at this time.

For Knights Corner to be a success it will have to conform to the commodity market arguments that have allowed the x86 architecture to dominate HPC computing. It must find a broad-based market outside the cloistered world of HPC scientists. Commodity graphics is a logical option for this broader market, but it is already well settled by NVIDIA and AMD.

8. I don’t like proprietary languages

Proprietary languages here refers to languages supported by one organization that may take development into an unknown or unwanted direction or drop support altogether. CUDA would fall into this category. The advantage to using CUDA is quite clear: 1) it can implement NVIDIA hardware specific optimizations; 2) there is no committee to make roadmap decisions; and 3) It supports new NVIDIA hardware features more quickly.

However if proprietary languages are a show-stopper in your organization, then OpenCL is a perfectly good option for non-proprietary language development. OpenCL, supported by Apple, NVIDIA, AMD, Intel and many others provides functional portability across multiple hardware platforms. I emphasize functional portability in contrast to performance portability that is still lagging. OpenCL kernels are very similar to CUDA kernels, with more differences found in the host-based setup and launch code.

9. I’m waiting for the magic CPU-to-GPU code converter

There’s good news and bad news here. The good news is that CPU-to-GPU converters already exist. The bad news is that its unlikely to produce code that is as performant as a native port by experts. With no experience using these tools and affiliation with a company that does significant technical code ports using native CUDA, the author is a poor and not unbiased source for reliable data on such approaches, however it would seem that a trial license for PGI Workstation from The Portland Group (PGI) and/or the CAPS HMPP workbench, two providers of such compilers, is easy enough to obtain and this option can be put to the test. Up the road we may expect some standardization in these compiler directives as they are incorporated into OpenMP.

10. I have N codes but one IT budget

More colloquially this may be referred to as the “go big or go home” dilemma. To add GPU-enabled nodes to the infrastructure of most organizations on a fixed IT budget requires a choice between fewer more powerful heterogeneous GPU nodes or more, less powerful, traditional CPU nodes. For economies of scale it makes sense for some organizations to have either 100 percent or 0 percent GPU-enabled nodes. This is particularly true of margin-based businesses working in competitive markets where cluster farms crunch data 24/7/365. Splitting the IT infrastructure complicates scheduling and requires, in the worst case, two versions of everything: cluster management scripts, scheduling, compilers, testing and validation, application code, etc.

Technology adoption in large commercial organizations must be done with keen attention to return on investment (ROI). The “go big or go home” argument expresses the anxiety that smart, thoughtful organizations give to this difficult problem in trying to quantify the known costs and speculate about the unknown costs of a technology shift. This last point as well as the previous nine, relate in some manner to either the investment (code development, people skills, new hardware, retraining) or the return (performance, scalability, power).

Each company must work out its unique ROI equation with fear and trembling, a healthy respect for the hurdles they will face and conservative margins. Using traditional financial analysis, capital investments should generate returns to shareholders commensurate with the organization’s weighted cost of capital and must be compared with other investment opportunities available to the company in the area of its domain expertise.

In summary GPU computing has held tenaciously on to the HPC market making notable gains in adoption over the last four years. The ten objections presented above are the most commonly voiced by individuals and organizations and here we have tried to address them. As noted at the beginning GPGPU, is not the solution to all HPC problems, but organizations may be missing out on significant performance gains and cost savings by ignoring the technology for the wrong reasons.

Finally, organizations should not make a move into GPU compute with the idea that it is this year’s solution but rather as a result of a deliberate strategy that determines it to be both currently cost effective and the best solution going forward based on architecture, programming model and power arguments to take us to the exascale.

About the Author

Vincent Natoli is the president and founder of Stone Ridge Technology. He is a computational physicist with 20 years experience in the field of high performance computing. He worked as a technical director at High Performance Technologies (HPTi) and before that for 10 years as a senior physicist at ExxonMobil Corporation, at their Corporate Research Lab in Clinton, New Jersey, and in the Upstream Research Center in Houston, Texas. Dr. Natoli holds Bachelor’s and Master’s degrees from MIT, a PhD in Physics from the University of Illinois Urbana-Champaign, and a Masters in Technology Management from the University of Pennsylvania and the Wharton School. Stone Ridge Technology is a professional services firm focused on authoring, profiling, optimizing and porting high performance technical codes to multicore CPUs, GPUs, and FPGAs.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Why HPC Storage Matters More Now Than Ever: Analyst Q&A

September 17, 2021

With soaring data volumes and insatiable computing driving nearly every facet of economic, social and scientific progress, data storage is seizing the spotlight. Hyperion Research analyst and noted storage expert Mark No Read more…

GigaIO Gets $14.7M in Series B Funding to Expand Its Composable Fabric Technology to Customers

September 16, 2021

Just before the COVID-19 pandemic began in March 2020, GigaIO introduced its Universal Composable Fabric technology, which allows enterprises to bring together any HPC and AI resources and integrate them with networking, Read more…

What’s New in HPC Research: Solar Power, ExaWorks, Optane & More

September 16, 2021

In this regular feature, HPCwire highlights newly published research in the high-performance computing community and related domains. From parallel programming to exascale to quantum computing, the details are here. Read more…

Cerebras Brings Its Wafer-Scale Engine AI System to the Cloud

September 16, 2021

Five months ago, when Cerebras Systems debuted its second-generation wafer-scale silicon system (CS-2), co-founder and CEO Andrew Feldman hinted of the company’s coming cloud plans, and now those plans have come to fruition. Today, Cerebras and Cirrascale Cloud Services are launching... Read more…

AI Hardware Summit: Panel on Memory Looks Forward

September 15, 2021

What will system memory look like in five years? Good question. While Monday's panel, Designing AI Super-Chips at the Speed of Memory, at the AI Hardware Summit, tackled several topics, the panelists also took a brief glimpse into the future. Unlike compute, storage and networking, which... Read more…

AWS Solution Channel

Supporting Climate Model Simulations to Accelerate Climate Science

The Amazon Sustainability Data Initiative (ASDI), AWS is donating cloud resources, technical support, and access to scalable infrastructure and fast networking providing high performance computing (HPC) solutions to support simulations of near-term climate using the National Center for Atmospheric Research (NCAR) Community Earth System Model Version 2 (CESM2) and its Whole Atmosphere Community Climate Model (WACCM). Read more…

ECMWF Opens Bologna Datacenter in Preparation for Atos Supercomputer

September 14, 2021

In January 2020, the European Centre for Medium-Range Weather Forecasts (ECMWF) – a juggernaut in the weather forecasting scene – signed a four-year, $89-million contract with European tech firm Atos to quintuple its supercomputing capacity. With the deal approaching the two-year mark, ECMWF... Read more…

Why HPC Storage Matters More Now Than Ever: Analyst Q&A

September 17, 2021

With soaring data volumes and insatiable computing driving nearly every facet of economic, social and scientific progress, data storage is seizing the spotlight Read more…

Cerebras Brings Its Wafer-Scale Engine AI System to the Cloud

September 16, 2021

Five months ago, when Cerebras Systems debuted its second-generation wafer-scale silicon system (CS-2), co-founder and CEO Andrew Feldman hinted of the company’s coming cloud plans, and now those plans have come to fruition. Today, Cerebras and Cirrascale Cloud Services are launching... Read more…

AI Hardware Summit: Panel on Memory Looks Forward

September 15, 2021

What will system memory look like in five years? Good question. While Monday's panel, Designing AI Super-Chips at the Speed of Memory, at the AI Hardware Summit, tackled several topics, the panelists also took a brief glimpse into the future. Unlike compute, storage and networking, which... Read more…

ECMWF Opens Bologna Datacenter in Preparation for Atos Supercomputer

September 14, 2021

In January 2020, the European Centre for Medium-Range Weather Forecasts (ECMWF) – a juggernaut in the weather forecasting scene – signed a four-year, $89-million contract with European tech firm Atos to quintuple its supercomputing capacity. With the deal approaching the two-year mark, ECMWF... Read more…

Quantum Computer Market Headed to $830M in 2024

September 13, 2021

What is one to make of the quantum computing market? Energized (lots of funding) but still chaotic and advancing in unpredictable ways (e.g. competing qubit tec Read more…

Amazon, NCAR, SilverLining Team for Unprecedented Cloud Climate Simulations

September 10, 2021

Earth’s climate is, to put it mildly, not in a good place. In the wake of a damning report from the Intergovernmental Panel on Climate Change (IPCC), scientis Read more…

After Roadblocks and Renewals, EuroHPC Targets a Bigger, Quantum Future

September 9, 2021

The EuroHPC Joint Undertaking (JU) was formalized in 2018, beginning a new era of European supercomputing that began to bear fruit this year with the launch of several of the first EuroHPC systems. The undertaking, however, has not been without its speed bumps, and the Union faces an uphill... Read more…

How Argonne Is Preparing for Exascale in 2022

September 8, 2021

Additional details came to light on Argonne National Laboratory’s preparation for the 2022 Aurora exascale-class supercomputer, during the HPC User Forum, held virtually this week on account of pandemic. Exascale Computing Project director Doug Kothe reviewed some of the 'early exascale hardware' at Argonne, Oak Ridge and NERSC (Perlmutter), while Ti Leggett, Deputy Project Director & Deputy Director... Read more…

Ahead of ‘Dojo,’ Tesla Reveals Its Massive Precursor Supercomputer

June 22, 2021

In spring 2019, Tesla made cryptic reference to a project called Dojo, a “super-powerful training computer” for video data processing. Then, in summer 2020, Tesla CEO Elon Musk tweeted: “Tesla is developing a [neural network] training computer called Dojo to process truly vast amounts of video data. It’s a beast! … A truly useful exaflop at de facto FP32.” Read more…

Berkeley Lab Debuts Perlmutter, World’s Fastest AI Supercomputer

May 27, 2021

A ribbon-cutting ceremony held virtually at Berkeley Lab's National Energy Research Scientific Computing Center (NERSC) today marked the official launch of Perlmutter – aka NERSC-9 – the GPU-accelerated supercomputer built by HPE in partnership with Nvidia and AMD. Read more…

Esperanto, Silicon in Hand, Champions the Efficiency of Its 1,092-Core RISC-V Chip

August 27, 2021

Esperanto Technologies made waves last December when it announced ET-SoC-1, a new RISC-V-based chip aimed at machine learning that packed nearly 1,100 cores onto a package small enough to fit six times over on a single PCIe card. Now, Esperanto is back, silicon in-hand and taking aim... Read more…

Enter Dojo: Tesla Reveals Design for Modular Supercomputer & D1 Chip

August 20, 2021

Two months ago, Tesla revealed a massive GPU cluster that it said was “roughly the number five supercomputer in the world,” and which was just a precursor to Tesla’s real supercomputing moonshot: the long-rumored, little-detailed Dojo system. “We’ve been scaling our neural network training compute dramatically over the last few years,” said Milan Kovac, Tesla’s director of autopilot engineering. Read more…

CentOS Replacement Rocky Linux Is Now in GA and Under Independent Control

June 21, 2021

The Rocky Enterprise Software Foundation (RESF) is announcing the general availability of Rocky Linux, release 8.4, designed as a drop-in replacement for the soon-to-be discontinued CentOS. The GA release is launching six-and-a-half months after Red Hat deprecated its support for the widely popular, free CentOS server operating system. The Rocky Linux development effort... Read more…

Google Launches TPU v4 AI Chips

May 20, 2021

Google CEO Sundar Pichai spoke for only one minute and 42 seconds about the company’s latest TPU v4 Tensor Processing Units during his keynote at the Google I Read more…

Intel Completes LLVM Adoption; Will End Updates to Classic C/C++ Compilers in Future

August 10, 2021

Intel reported in a blog this week that its adoption of the open source LLVM architecture for Intel’s C/C++ compiler is complete. The transition is part of In Read more…

AMD-Xilinx Deal Gains UK, EU Approvals — China’s Decision Still Pending

July 1, 2021

AMD’s planned acquisition of FPGA maker Xilinx is now in the hands of Chinese regulators after needed antitrust approvals for the $35 billion deal were receiv Read more…

Leading Solution Providers


Hot Chips: Here Come the DPUs and IPUs from Arm, Nvidia and Intel

August 25, 2021

The emergence of data processing units (DPU) and infrastructure processing units (IPU) as potentially important pieces in cloud and datacenter architectures was Read more…

Julia Update: Adoption Keeps Climbing; Is It a Python Challenger?

January 13, 2021

The rapid adoption of Julia, the open source, high level programing language with roots at MIT, shows no sign of slowing according to data from I Read more…

10nm, 7nm, 5nm…. Should the Chip Nanometer Metric Be Replaced?

June 1, 2020

The biggest cool factor in server chips is the nanometer. AMD beating Intel to a CPU built on a 7nm process node* – with 5nm and 3nm on the way – has been i Read more…

HPE Wins $2B GreenLake HPC-as-a-Service Deal with NSA

September 1, 2021

In the heated, oft-contentious, government IT space, HPE has won a massive $2 billion contract to provide HPC and AI services to the United States’ National Security Agency (NSA). Following on the heels of the now-canceled $10 billion JEDI contract (reissued as JWCC) and a $10 billion... Read more…

Quantum Roundup: IBM, Rigetti, Phasecraft, Oxford QC, China, and More

July 13, 2021

IBM yesterday announced a proof for a quantum ML algorithm. A week ago, it unveiled a new topology for its quantum processors. Last Friday, the Technical Univer Read more…

Intel Launches 10nm ‘Ice Lake’ Datacenter CPU with Up to 40 Cores

April 6, 2021

The wait is over. Today Intel officially launched its 10nm datacenter CPU, the third-generation Intel Xeon Scalable processor, codenamed Ice Lake. With up to 40 Read more…

Frontier to Meet 20MW Exascale Power Target Set by DARPA in 2008

July 14, 2021

After more than a decade of planning, the United States’ first exascale computer, Frontier, is set to arrive at Oak Ridge National Laboratory (ORNL) later this year. Crossing this “1,000x” horizon required overcoming four major challenges: power demand, reliability, extreme parallelism and data movement. Read more…

Intel Unveils New Node Names; Sapphire Rapids Is Now an ‘Intel 7’ CPU

July 27, 2021

What's a preeminent chip company to do when its process node technology lags the competition by (roughly) one generation, but outmoded naming conventions make it seem like it's two nodes behind? For Intel, the response was to change how it refers to its nodes with the aim of better reflecting its positioning within the leadership semiconductor manufacturing space. Intel revealed its new node nomenclature, and... Read more…

  • arrow
  • Click Here for More Headlines
  • arrow