HPCwire: Top 10 Objections to GPU Computing Reconsidered

June 09, 2011

Top 10 Objections to GPU Computing Reconsidered

Dr. Vincent Natoli, Stone Ridge Technology

By any measure, the emergence of GPU computing in the HPC world over the last few years has been a remarkable and unprecedented development. It has all the markings of a disruptive technology. It began with a small cadre of adherents willing to brave the complications of shoehorning graphics languages like Cg and OpenGL's GLSL into performing general-purpose calculations. It was championed by a hungry new player in HPC. It provides step out, order of magnitude performance gains over conventional solutions in more cases than people would like to admit and it's just plain different.

As we approach the four-year release anniversary of NVIDIA CUDA, arguably the ground zero of the GPGPU movement, there are many who have flirted, piloted and adopted the technology, but many more who are sitting on the sidelines for various reasons. In our work, we have come across many of the latter, and have thus compiled a list of the most common questions, concerns and assertions that preempt efforts to evaluate the technology.

What follows below is an attempt to list and address these objections, reconsidering them in light of recent developments and our best guess at how the future will unfold. To be clear, GPGPU is not the solution for all HPC applications but many have found the technology cost effective in domains as diverse as seismic imaging, electromagnetics, molecular dynamics, financial valuation, medical imaging and others.

1. I don't want to rewrite my code or learn a new language

It's true that you will have to rewrite your code to make use of GPUs. However if you are developing a parallel CPU implementation from a mostly serial code rewriting your code is inevitable. The only question is which platform will you target. Targeting multicore CPUs presents a three-tier model with parallelism at the process, thread and register levels where you will use MPI, OpenMP/pthreads and SSE/AVX extensions. Programming for GPUs using CUDA is no more difficult and the advantage for both compute-bound and memory-bound codes can be significant, as we'll see.

If you already have a parallel code, what benefit will you reap from a GPU port? Back-of-the-envelope calculations yield between 5x and 40x improvements in chip-to-chip comparisons on typical codes. This metric is also what we have seen anecdotally and in the many publications on GPU enabled applications. These comparisons have remained fairly constant over the past few years spanning two generations of Intel and NVIDIA products.

CUDA is an extension to C and relatively easily picked up by experienced coders. The model for parallel programming that will take us to the exascale is far from settled, however I believe that the eventual solution will look more like the data parallel CUDA model than the task parallel CPU alternatives. In previous HPCwire contributions I have expressed the opinion that CUDA, by forcing developers to consider the irreducible level of parallel work in their problems and mapping that to threads, is a better programming model for extensible parallelism, lending itself more naturally to scalability across multiple GPUs on a single node and across multiple nodes.

Steps in this direction are already underway with examples from the academic world with the excellent work on Global Memory for Accelerators (GMAC) being jointly developed by Universitat Politecnica de Catalunya and the IMPACT Research Group at the University of Illinois, and the commercial world with the unique scaling capabilities of the HUESPACE compute API offered by Oslo-based startup HUE and their sister company focusing in oil and gas, Headwave.

2. I don't know what kind of performance to expect

HPC codes are either compute-bound or memory-bound. For compute-bound codes we compare the NVIDIA Fermi M2090 with the Intel Westmere. The Fermi has 512 cores running at about 1.3GHz while Westmere has 6 cores running at up to 3.4 GHz. The ratio of core-Hz yields a factor of about 32x. If your CPU code is effectively using SSE instructions, that might give you an additional boost of 4x on the CPU side reducing GPU efficacy to 8x (close to the ratio of peak GFLOPS).

For memory-bound codes we compare GPU main memory bandwidth of 177 GB/second to 32 GB/second available per processor on CPU motherboards for a ratio of about 5.5x. The bottom line is that if your code is compute bound expect a GPU boost of between 5x for highly optimized SSE implemented codes and 20x or more for typical codes. If your code is memory bound expect a boost of about 5x in chip to chip comparisons.

It's helpful when considering a parallel solution to think about the problem in terms of marginal cost. If your code is memory bound then you should consider the least expensive option for adding bandwidth. One can add another GPU card and buy bandwidth for about $15 per GB/second. Alternatively one can add a separate node at a nominal cost of about $80 per GB/second. The latter solution also increases the compute footprint and adds another O/S instance.

For compute-bound codes a similar calculation can be performed to produce the marginal cost of gigaflops. For codes that have a mixture of compute-bound and memory-bound sections i.e., most codes, the GPU does a better job of hiding the latency of memory-bound sections by juggling thousands of threads and computing continuously on those threads that have their data registers filled and ready to go. By hiding memory latency the GPU is better able to achieve the higher compute-bound performance limits.

3. The PCIe bandwidth will kill my performance

The argument against GPU computing based on PCIe bandwidth is really one of computational intensity. Computational intensity is variously defined but for these purposes let us say it is the number of floating point operations performed on each float of data transferred. There is a threshold of work which must be applied to every byte of data transferred to the GPU board in order for a GPU compute effort to be worthwhile.

For example, PCIe v2.0 x16 bandwidth totals about 6 GB/second in practice. In perspective it can fill the 6 GB of main memory on the M2090 board in about a second. At 665 double precision gigaflops peak performance the M2090 is a floating point monster and can clearly do a lot of processing in the second that it takes to fill the board memory. If for example you'd like the PCIe transfer time to be no more than 10 percent of the compute time, the M2090 must do thousands of floating point operations on the stored data before flushing it. GPU algorithms must therefore strive to retain data on the boards for as long as possible.

In addition, CUDA allows the asynchronous overlap of PCIe data transfer with calculation. Clever use of this feature lets the developer hide some or all of the PCIe data transfer time behind computations. Algorithms that work well include time stepping algorithms with local physics such as Finite Difference Time Domain (FDTD) or classical molecular dynamics where the N2 particle-particle interactions lead to significant data reuse and high computational intensity.

Algorithms that are not effective in isolation on GPUs such as simple vector dot products have very low computational intensity. If problems are mapped across multiple GPUs they should minimize data transfer, for example sending only border data in a domain decomposition.

4. What about Amdahl's law?

Amdahl's law quantitatively expresses the fact that if you are going to accelerate part of a large serial code either by whatever means, parallelization or magic gnomes, you had better accelerate a significant portion or you will not realize much benefit. Amdahl's law is often held forth as a talisman against any suggestion that code performance can be improved via parallelization.

Imagine, for example, you could reduce the runtime of a part of your serial code to zero. If that portion of the code accounted for 50 percent of the runtime then your overall speedup would only be 2x; if it accounted for 90 percent of runtime your speedup would approach 10x. For more realistic accelerations speedup results are lower.

The most effective counter to an Amdahl's law objection to GPU computing is the observation that performance on modern architectures requires all codes to be written for massive parallelism reducing their serial portions to the minimum possible…there is no other game in town. This is true on CPU platforms as well as on GPU platforms.

The only HPC compute platforms we have going forward are massively parallel and Amdahl's law applies equally well to CPU implementations. The real question is where are you going to run the parallel portion of your code…on the CPU or on the GPU?

5. What if NVIDIA goes away?

The history of HPC is littered with the carcasses of many a supercomputing company that tried to make the leap to parallel computing. A short list includes Thinking Machines, Maspar, KSR, Mitrion and many others. These often heroic efforts and the people behind them should be recognized for their vision and the role they played in the creative disruption that over time led to a greater collective understanding of what works and what doesn't. We owe a debt of gratitude to them all.

NVIDIA, however, is not a supercomputing company. It's a $5 billion corporation that makes most of its money from video cards and embedded processors that it sells to the huge avid market of PC gamers. Its relative independence from HPC is a strength, and if all HPC usage of GPU computing disappeared NVIDIA would still have a fine and profitable business. As long as there are caffeine-addicted adolescents willing to play virtual war games NVIDIA will be around. The fact is that NVIDIA's staying power in the market is greater and more secure than that of long-time HPC icon Cray.

Further, NVIDIA has publicly posted its vision and roadmap for technology development out to roughly 6 years. Reading between the lines it is extremely aggressive with ambitions to move the GPU from its ancillary role living on an accelerator card to a more central role in the compute infrastructure. Along the way they have some powerful compute engines planned.

6. GPU boards don't offer enough main memory for my problem.

GPU board memory is currently limited to 6 GB available on the M2090 and M2070. This can be particularly problematic for algorithms that require large chunks of memory in excess of that. This problem is somewhat alleviated by access to multiple cards available to a single node.

For example, the Dell C410x PCIe expansion chassis that can contain up to 16 NVIDIA boards for a total of 96 GB. Domain decomposing your problem on 16 separate GPUs can be a very effective way to side-step the memory limitation and it works quite well for problems effecting local physics that calculate properties on volumes and share data at surfaces.

The most problematic are algorithms that require essentially random access to large arrays, for example, huge hash tables or other problems that require random array lookups. The current generation of GPU boards are not effective solutions in this case, however, memory being relatively inexpensive and constantly improving in density, one can expect future generations of boards to come with increased amounts.

7. I will wait for more CPU cores / Knights Corner

More cores will help with compute-bound applications, however one needs to consider that as more cores are added to CPU dies, the same will be true for GPUs. Comparing roadmaps over the past two technology generations shows a persistent gap in compute and bandwidth between CPUs and GPUs that sometimes grows and sometimes wanes but is consistently non-zero. Expect this to continue going forward. For bandwidth-bound problems the situation is somewhat worse as it appears easier to add cores than to add bandwidth.

Intel's plans for Knights Corner, announced just over a year ago, recognizes the need for an x86 data parallel competitor to GPUs. Full details on Knights Corner are still unknown however using specs from the Knights Ferry prototype as a baseline, we might expect 50 or more 1.2 GHz cores, each with its own 512-bit vector processing unit supporting up to 4 threads each, making it a formidable HPC competitor. However Intel's plans for the development model, pricing, release dates and other critical information is, at best, poorly understood and communicated at this time.

For Knights Corner to be a success it will have to conform to the commodity market arguments that have allowed the x86 architecture to dominate HPC computing. It must find a broad-based market outside the cloistered world of HPC scientists. Commodity graphics is a logical option for this broader market, but it is already well settled by NVIDIA and AMD.

8. I don't like proprietary languages

Proprietary languages here refers to languages supported by one organization that may take development into an unknown or unwanted direction or drop support altogether. CUDA would fall into this category. The advantage to using CUDA is quite clear: 1) it can implement NVIDIA hardware specific optimizations; 2) there is no committee to make roadmap decisions; and 3) It supports new NVIDIA hardware features more quickly.

However if proprietary languages are a show-stopper in your organization, then OpenCL is a perfectly good option for non-proprietary language development. OpenCL, supported by Apple, NVIDIA, AMD, Intel and many others provides functional portability across multiple hardware platforms. I emphasize functional portability in contrast to performance portability that is still lagging. OpenCL kernels are very similar to CUDA kernels, with more differences found in the host-based setup and launch code.

9. I'm waiting for the magic CPU-to-GPU code converter

There's good news and bad news here. The good news is that CPU-to-GPU converters already exist. The bad news is that its unlikely to produce code that is as performant as a native port by experts. With no experience using these tools and affiliation with a company that does significant technical code ports using native CUDA, the author is a poor and not unbiased source for reliable data on such approaches, however it would seem that a trial license for PGI Workstation from The Portland Group (PGI) and/or the CAPS HMPP workbench, two providers of such compilers, is easy enough to obtain and this option can be put to the test. Up the road we may expect some standardization in these compiler directives as they are incorporated into OpenMP.

10. I have N codes but one IT budget

More colloquially this may be referred to as the "go big or go home" dilemma. To add GPU-enabled nodes to the infrastructure of most organizations on a fixed IT budget requires a choice between fewer more powerful heterogeneous GPU nodes or more, less powerful, traditional CPU nodes. For economies of scale it makes sense for some organizations to have either 100 percent or 0 percent GPU-enabled nodes. This is particularly true of margin-based businesses working in competitive markets where cluster farms crunch data 24/7/365. Splitting the IT infrastructure complicates scheduling and requires, in the worst case, two versions of everything: cluster management scripts, scheduling, compilers, testing and validation, application code, etc.

Technology adoption in large commercial organizations must be done with keen attention to return on investment (ROI). The "go big or go home" argument expresses the anxiety that smart, thoughtful organizations give to this difficult problem in trying to quantify the known costs and speculate about the unknown costs of a technology shift. This last point as well as the previous nine, relate in some manner to either the investment (code development, people skills, new hardware, retraining) or the return (performance, scalability, power).

Each company must work out its unique ROI equation with fear and trembling, a healthy respect for the hurdles they will face and conservative margins. Using traditional financial analysis, capital investments should generate returns to shareholders commensurate with the organization's weighted cost of capital and must be compared with other investment opportunities available to the company in the area of its domain expertise.

In summary GPU computing has held tenaciously on to the HPC market making notable gains in adoption over the last four years. The ten objections presented above are the most commonly voiced by individuals and organizations and here we have tried to address them. As noted at the beginning GPGPU, is not the solution to all HPC problems, but organizations may be missing out on significant performance gains and cost savings by ignoring the technology for the wrong reasons.

Finally, organizations should not make a move into GPU compute with the idea that it is this year's solution but rather as a result of a deliberate strategy that determines it to be both currently cost effective and the best solution going forward based on architecture, programming model and power arguments to take us to the exascale.

About the Author

Vincent Natoli is the president and founder of Stone Ridge Technology. He is a computational physicist with 20 years experience in the field of high performance computing. He worked as a technical director at High Performance Technologies (HPTi) and before that for 10 years as a senior physicist at ExxonMobil Corporation, at their Corporate Research Lab in Clinton, New Jersey, and in the Upstream Research Center in Houston, Texas. Dr. Natoli holds Bachelor's and Master's degrees from MIT, a PhD in Physics from the University of Illinois Urbana-Champaign, and a Masters in Technology Management from the University of Pennsylvania and the Wharton School. Stone Ridge Technology is a professional services firm focused on authoring, profiling, optimizing and porting high performance technical codes to multicore CPUs, GPUs, and FPGAs.

Share Options

Subscribe to HPCwire

Discussion

There are 10 discussion items posted.

Where are Cell BE (Broadband Engine) and FPGA now ?
Submitted by Hurricane on Jun 10, 2011 @ 2:20 AM EDT

GPU is not the first accelerator idea ever happened. Cell BE and FPGA were also touted as the next big thing...where are they now ? I am not too concerned about difficulty/applicability/performance of GPU programming or IT budget, but nVidia's future is uncertain. The author should read this: http://perilsofparallel.blogspot.com/2010/08/nvidia-based-cheap-supercomputing.html Why nVidia is doing ARM now ? Because they know Intel and AMD's integrated GPU (Sandy Bridge, Llano, APU, Bulldozer, to name a few) will eat up nVidia's discrete GPU market share. This happens before (remember nVidia also used to make x86 chipsets ? Think about why they quit that business...) Knights Ferry, if Intel is serious about it, would be the CUDA killer: For at least two reasons: it's x86 compatible, and it shares the same memory address space as the main CPU. That means you probably can run a Linux on a Knights Ferry and use all existing x86 toolchains with little or no modification. How great is that ? I am also pretty confident Cray will stay (Uncle Sam won't let it fail, as it has nationally strategic value and IBM needs a competitor.), but nVidia ? ( What happened to Sun ? Voodoo ? DEC ? )

Post #1

Re: Where are Cell BE (Broadband Engine) and FPGA now ?
Submitted by cgorac on Jun 10, 2011 @ 2:34 AM EDT

If you linked to the blog post above, you could have linked to follow-up post http://perilsofparallel.blogspot.com/2011/05/sandy-bridge-graphics-disappoints.html, where author basically had to admit that lots of Intel promises were smoke. So if you still want to trust them and wait, it's up to you - for people that have some real HPC work to do, GPUs are here to be used. Cell BE never caught up because its programming model was overly complicated, thus it was doomed to fail. FPGAs are completely different story, these are alive and kicking but they are different type of solution and not that much of competitor to GPUs.

Post #2

Submitted by Hurricane on Jun 10, 2011 @ 11:12 AM EDT

I don't see CUDA programming is any simpler than Cell BE programming. They all follow similar progrmming model. And just 2-3 years ago the first super to break the 1 PF barrier is Cell-based ! Things like accelerators (no matter in what form) can only be temporary or in an extremely niche space (e.g. GRAPE or Anton for MD calculations): At its peak there were no less than half a dozen companies selling 80x87 co-processors (notably Weitek, AMD, Cyrix), and 80x87 has the same programming model as the CPU, so only a re-compilation of the user programs is needed. (Don't tell me HMPP or PGI Accelerator or OpenMP-like efforts... their performance is abysmal.) Regarding the viability of nVidia, with the constant decline of high-end PC gaming (thanks to rampant piracy) and the ever growing popularity of browser-based leisure games, I don't see why someone would pay $100-200 to upgrade his video card every 2-3 years just to be able to play the latest, high-definition games on *PC*. Remember PS3 and XBOX360 are said to have a lifecycle of (at least) 7-8 years ! nVidia already saw this coming and tried very hard to diversify (first x86 chipsets, and now ARM/mobile devices).. The problem is, ARM is already a very crowded space, and I don't see nVidia has any edge over other giants like Apple, Qualcomm, Broadcom, Marvell, Samsung, TI, to name a few. All of these companies except Samsung are fabless and are at mercy of TSMC and GlobalFoundries.

Post #3

Re:
Submitted by cgorac on Jun 11, 2011 @ 2:57 AM EDT

CUDA programming is much simpler than Cell BE programming: with Cell, you'd have to mess with SIMD, "local storage" memory model is more complicated than CUDA shared memory model, and PPE code driving SPE execution is more complicated than CUDA host code. Furthermore, Cell development systems were not that available - your best option was PS3, and then at some point Sony just disabled its use for this purpose. So - if it was of any good, Cell BE would get picked up by people, but it died instead. Further: I don't know what you mean by stating that x87 programming model is the same as for x86 CPU - it was actually completely different (stack-based); at some point x86 and x87 programming models get unified when compilers became able to generate x87 code, but one could state this line of development is already in-place for GPUs too - I'd agree that for PGI Accelerator or HMPP performance is not there yet, but it will be at some point. Regarding your opinions on the market - I don't know. Basically, your claims are alike to what is mentioned in the blog post you linked in your first message, and you may be right. Still, game consoles are there for many years, and people still do gaming on their PCs, so they may as well keep paying for video-card upgrades and such. In any case, the bottom line is that GPUs are very good option as compute accelerators today, and as I mentioned above - for people that has some real work to do, this is the only thing that matters.

Post #4

The objections to GPU in the article are equally applicable to all HPC
Submitted by Jack Horner on Jun 11, 2011 @ 11:03 AM EDT

Stripped of inessential references to GPU-oriented technology, the perceived objections to GPU-based computing considered in the article apply equally well to any parallel HPC regime. There is no relief at the bleeding edge.

Post #5

Weak arguments
Submitted by Christof Stork on Jun 15, 2011 @ 12:04 PM EDT

Many of the arguments in this article are weak. This article is actually good at summarizing the anti-Nvidia GPU arguments but doesnât counter them well. A few things that are left out of this article are: a) Nvidia GPUs are a very specific type of GPU computeâit is not representative of all possible GPU architectures. b) It is very hard to get the peak performance from Nvidia GPU boardsâmuch harder than from CPUs. c) You have to totally re-write your algorithm for Nvidia GPU boards and the re-write wonât work on other future GPU architectures. Also, the Nvidia architecture makes it impossible to implement algorithms optimizations that you can make on CPUs.

Post #6

Forget specifics
Submitted by bb_vb on Jun 16, 2011 @ 7:20 PM EDT

A very interesting article. While some use specifics like NVIDIA hardware or software to argue against the longevity of the GPGPU movement, the truth is that the use of many-core architectures is so much more general than that. Once the parallelism in an algorithm has been identified, you're most of the way there -- the hardware and software-specific implementation is then simply a detail. Porting becomes easy once the initial shift from sequential to parallel has been made. See this paper for answers to questions 2, 3 and 4 based on the use of simple algorithm analysis: http://adsabs.harvard.edu/abs/2010MNRAS.408.1936B

Post #7

Cell and FPGA
Submitted by Vincent Natoli on Jun 22, 2011 @ 11:28 AM EDT

Thanks for your comments on the article. I have to agree with cgorac above. GPU coding is much easier than Cell. The model for Cell as a serious HPC component was broken from the beginning . First there was not enough effort put into the development environment and second the only way to get access to the hardware was to buy a game console however the game console vendor discounts the console to make money on the games and their licensing. Clearly they are not interested in a bunch of HPC folks buying dozens of consoles to build clusters. FPGAs are still around. They did not fulfill the role of general accelerator for a variety of reasons including the fragmented hardware market and the vastly different programming model. However they are fantastic accelerators for a number of key application areas like Bioinformatics and Finance. I see them playing a role in products that can be treated more like an appliance than a general purpose programmable box.

Post #8

Re: Weak arguments
Submitted by Vincent Natoli on Jun 22, 2011 @ 11:41 AM EDT

I actually went way over in the length of the article and I'm grateful to the editor for publishing it anyway. Because of the overall length I didn't have as much space to lay out expanded arguments. Fewer points in more depth would have been an alternative. To respond to some of the points raised: a) Theoretical GPU architectures may be interesting however practically speaking you can go out to Best Buy or Newegg today and buy a GPU card from NVIDIA or AMD and start programming. b) Our experience is that with equivalent effort you can get closer to peak on GPU. I'm interested in the experiences others have had. c) You do have to rewrite the code. If you write for NVIDIA or ATI hardware I'm pretty sure it will port to future hardware platforms from these vendors. Incompatibility with CPU optimizations is certainly true but there is no reason to expect it. There are CPU optimizations that are also not compatible between CPUs.

Post #9

GPU or Hybrid computing will happen
Submitted by Ben HPC Smith on Jul 5, 2011 @ 7:19 PM EDT

Most of the issues described do not hold water. If I wouldnât rewrite my code multiple times already, and if I would be concerned of what if, I would still be running my programs on my old (very old) Commodore 64â¦. Hybrid computing (separate GPUs or integrated GPUs) will happen if we want to continue to increase our HPC capabilities.
Ben "HPC" Smith (see my posts at hpcopinion dot blogspot dot com).

Post #10

Join the Discussion