A Call to Arms for Parallel Programming Standards

By Nicole Hemsoth

November 16, 2010

Although the parallel programming landscape is relatively young, it’s already easy to get lost in. Beside legacy frameworks like MPI and OpenMP, we now have NVIDIA’s CUDA, OpenCL, Cilk, Intel Threading Building Blocks, Microsoft’s parallel programming extensions for .NET, and a whole gamut of PGAS languages.

And according to Intel’s Tim Mattson, that’s not necessarily a good thing. Mattson, who is a principal engineer (and parallel programming evangelist) at the company’s Visual Applications Research Lab, says all these software frameworks are leading to what he calls “choice overload” and this concerns him greatly.

From his point of view, the road to parallel programming need to be paved with open industry standards. And today then means MPI, OpenMP, and OpenCL. Given that some of Intel’s parallel software offerings such as the Cilk Plus and Array Building Blocks are proprietary, that viewpoint sometimes puts him at odds with his own company. But Mattson’s role as an Intel researcher forces him to look beyond the one or two-year timeframe of product cycles. He’s in it for the long-term, and that means Mattson is looking at what is best for the ecosystem ten years out. “First and foremost, we have to make sure that the right standards exist and they run best on Intel products,” he says.

At SC10 this year, Mattson will be in full software evangelist mode, speaking at seven different tutorials, BoFs and panels on various parallel programming topics. Three of these are geared to fire up the troops for OpenCL, an open standard parallel programming framework for heterogeneous multicore architectures. HPCwire spoke with Mattson shortly before the conference about the importance of open standards, his unapologetic enthusiasm for OpenCL, and his open animosity for the CUDA programming language.

HPCwire: What is the significance of OpenCL and why are you devoting so much time talking about it at SC10?

Tim Mattson: I think OpenCL is perhaps the most important development in the last five, if not the last ten years. The reason I make such an over-the-top statement is that I believe the core to solving the parallel programming challenge is standards. Only an idiot software developer would write code using a propriety API. Since I don’t like to work with idiots (laughs), I want to support good software developers out there by making sure they have the full suite of standards that they need.

So we have message passing covered: MPI. It’s great. We have shared memory covered: OpenMP. It’s great. The glaring hole — because frankly I don’t think any of us saw it coming in the early 2000s — is heterogenous platforms. So we have to fill that hole, and that’s what OpenCL does. So I think it’s incredibly important because now with MPI, OpenMP and OpenCL we’ve got the space covered with these low-level basic programming standards that are required to move things forward.

HPCwire: Well as far as openness goes, NVIDIA’s CUDA programming API is available for any vendor to implement for their particular parallel hardware architecture. For example, AMD could support CUDA for their x86 and GPU platforms. So couldn’t CUDA be adopted as a standard as well?

Mattson: Well, just think about it. I can’t speak for AMD, but why would Intel put resources into an API or language that we have absolutely no say in over how it’s going to evolve? To call CUDA a standard is just insulting. It’s not a standard until the various players can all have a voice in it. It’s ridiculous. If NVIDIA was serious about it, they would create an industry working group that owns the development of CUDA’s API and languages and that has a full voice in what happens with it. Oh, by the way, that’s what OpenCL is.

HPCwire: But there is at least one example of a standard language that emerged from a vendor initiative. Java was controlled by Sun Microsystems for many years and was adopted as a commercial standard because it became so popular across the industry. Don’t you think CUDA could follow that model?

Mattson: Well I know that’s what NVIDIA would like to see happen. And yes, Java is the one instance that would call into question how absolute my statement is. Java though was coming into a very different market and was tightly associated with the Web browser — a platform that cut across the industry. And Sun showed very early on that they were willing to support it as a cross-platform language. They had Java available on x86 and Sparc and showed a willingness to work across the vendors. NVIDIA — rationally, by the way — isn’t doing this with CUDA.

When you look closely at OpenCL, it covers everything CUDA can do and more. OpenCL has all the key vendors and covers a much wider space than CUDA. We’ve got the embedded people, the cell phone vendors, and game vendors all involved. So OpenCL is the right way to go; CUDA is the wrong way to go.

HPCwire: In the high performance computing community, though, there has been criticism that OpenCL doesn’t deliver the kind of performance required for HPC codes. Do you think that’s a fair assessment?

Mattson: That’s a statement that’s both true and false. There’s nothing pathological in the definition of OpenCL that prevents it from being every bit as efficient as CUDA. The thing about OpenCL is that it’s young; it just hasn’t been out very long. So it really comes down to the vendors as far as the quality of their implementations.

I think it’s important for the programmers out there — and let’s face it, they are the end user community for these technologies — to steer things in the right direction by insisting on standards. Look at how MPI and OpenMP came into existence. In both those cases, the user community insisted that these standards be the foundation of the software ecosystem, and the vendors stood behind them. We need people to do the same thing here and not get caught up with point solutions.

If NVIDIA engineers spent as much time optimizing OpenCL, it would run as fast as CUDA. So the performance arguments don’t hold a lot of sway with me, except when someone can say this feature of the language as defined is fundamentally going to be inefficient regardless of the quality of implementation. When people find those, we in the OpenCL group take it very seriously.

We’re roughly on a two-year cadence of coming out with new releases of the OpenCL spec, and we’re very focused on finding the weakness in OpenCL and aggressively evolving the language and to stay right in line with the latest hardware trends.

HPCwire: There are plenty of other languages that address multicore parallelism, some of which have been introduced by Intel. How does OpenCL fit in?

Mattson: Let me be really clear. There are three distinct standards that address multicore. MPI, for example, works great on multicore. OpenMP, if you have a shared address space, works really well too. And OpenCL covers heterogenous architectures. It’s really the trio that I’m pushing and Intel is 100 percent behind them.

On the other hand, yes, there is a trend that I find deeply disturbing of vendors wanting to distinguish themselves by creating new languages and proprietary APIs. It’s disturbing because time spent on a new language or proprietary API is time not spent on improving and establishing these standards. So this is where I’m kind of at odds with some of my colleagues at Intel. That’s just the way it goes.

Let’s face it. Vendors, left to their devices, want people to adopt a proprietary API that lock them into their platform. That’s not bad. NVIDIA is completely rational in wanting to lock people into their platform with CUDA. If I was working at NVIDIA, I’d probably be trying to do the same think. I think it’s up to the user community to refuse to let users get away with that game. They can do that by insisting on standards or open source solutions.

The three standards I mentioned are where I think most of the resource should go. But Intel did release Task Building Blocks –TBB — as open source. That was a very responsible thing to do. I was very excited, as was the TBB team, when that happened.

HPCwire: Another language is Ct, which started as a research project at Intel and has now been commercialized. How does that fit in to this parallel language ecosystem?

Mattson: Ct, which by the way, is now called Array Building Blocks, is a higher level abstraction of parallelism. While I’m a huge supporter of what they are doing with Array Building Blocks, [as for] how useful it will be in the marketplace, I’m not sure because it is proprietary. But I think some of the optimizations it does under the covers is very important. There are a lot of really important things about that project.

But I think we should distinguish creating good technologies versus confusing the market by having too many options out there. In 2004, if you wanted to do parallel programming you had Windows threads, pthreads on Linux, OpenMP, and MPI. Five options was fine. Now there are a dozen or more parallel programming languages out there. So I think we’re losing ground. I think choice overload is real. And that concerns me deeply.

HPCwire: Do you think parallelizing established languages like Java and Python is a positive development?

Mattson: Let me tell you where I think things are going and where we’ll be in 10 years. The question is do we get there cleanly or do we get there with messy detours along the way. Ultimately I think we have to raise the level of abstraction, which is what you see with these efforts around building parallelism into Python. We need to focus on the higher level frameworks that people are increasingly using to write software.

This is really what I spend the bulk of my time doing in my personal research, and with a group at UC Berkeley — to define patterned languages from which we can derive the frameworks, which then map down to the lower-level languages. I really want to make it so that only a small number of performance-oriented, efficiency layer programmers worry about these low-level languages — OpenCL, MPI, OpenMP, or TBB. But beyond that, people need to have some higher level framework they can work with. A parallel Python project like Copperhead is such an example. I’m very excited about it because I think that’s clearly the direction things are moving.

I learned this most clearly looking at the gaming industry, because that industry has been the leader in adopting multicore, and I mean adopting multicore as a successful business venture. Researchers have been playing with it for a long time, but in terms of creating multicore software, selling it, and building a profitable businesses around it, the gaming industry has led the charge.

They have these separations of concerns very sharply defined, and it works extremely well for them. Most of their programmers work in a higher level scripting language or with collections of libraries written in C++. And then they have a small number of “technology programmers” which are on the order of 10 percent of their software developers. They’re the guys who do the low-level stuff. And I think that kind of separation of concerns is what’s absolutely critical.

HPCwire: So you think higher level frameworks will be key to enabling these low-level parallel programming APIs you’re talking about?

Mattson: When we were sitting around creating OpenCL, we explicitly talked about that as our goal. In fact, you’ll find some places in the spec where we describe OpenCL as a hardware abstraction layer. We’re perfectly aware that OpenCL is obnoxiously low-level. It exposes so many details of the underlying platform. We achieve extreme portability by exposing everything and abstracting as little as we can. The reason we think that’s the right thing to do is because we view OpenCL ultimately as being a target for higher level frameworks. It’s young, so those higher level frameworks don’t exist yet, but I think they will and I think that will be the long range trend, not just for OpenCL, but for all these parallel languages.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Scientists Conduct First Quantum Simulation of Atomic Nucleus

May 23, 2018

OAK RIDGE, Tenn., May 23, 2018—Scientists at the Department of Energy’s Oak Ridge National Laboratory are the first to successfully simulate an atomic nucleus using a quantum computer. The results, published in Ph Read more…

By Rachel Harken, ORNL

Pattern Computer – Startup Claims Breakthrough in ‘Pattern Discovery’ Technology

May 23, 2018

If it weren’t for the heavy-hitter technology team behind start-up Pattern Computer, which emerged from stealth today in a live-streamed event from San Francisco, one would be tempted to dismiss its claims of inventing Read more…

By John Russell

Intel, Micro Debut Quad-Level Cell NAND Flash

May 22, 2018

Chipmakers continue to gear designs toward AI and other demanding cloud workloads that take advantage of datacenter flash storage capacity. To that end, memory specialist Micron Technology Inc. began shipping compact sol Read more…

By George Leopold

HPE Extreme Performance Solutions

HPC and AI Convergence is Accelerating New Levels of Intelligence

Data analytics is the most valuable tool in the digital marketplace – so much so that organizations are employing high performance computing (HPC) capabilities to rapidly collect, share, and analyze endless streams of data. Read more…

IBM Accelerated Insights

Mastering the Big Data Challenge in Cognitive Healthcare

Patrick Chain, genomics researcher at Los Alamos National Laboratory, posed a question in a recent blog: What if a nurse could swipe a patient’s saliva and run a quick genetic test to determine if the patient’s sore throat was caused by a cold virus or a bacterial infection? Read more…

Japan Meteorological Agency Takes Delivery of Pair of Crays

May 21, 2018

Cray has supplied two identical Cray XC50 supercomputers to the Japan Meteorological Agency (JMA) in northwestern Tokyo. Boasting more than 18 petaflops combined peak computing capacity, the new systems will extend the a Read more…

By Tiffany Trader

Pattern Computer – Startup Claims Breakthrough in ‘Pattern Discovery’ Technology

May 23, 2018

If it weren’t for the heavy-hitter technology team behind start-up Pattern Computer, which emerged from stealth today in a live-streamed event from San Franci Read more…

By John Russell

Japan Meteorological Agency Takes Delivery of Pair of Crays

May 21, 2018

Cray has supplied two identical Cray XC50 supercomputers to the Japan Meteorological Agency (JMA) in northwestern Tokyo. Boasting more than 18 petaflops combine Read more…

By Tiffany Trader

ASC18: Final Results Revealed & Wrapped Up

May 17, 2018

It was an exciting week at ASC18 in Nanyang, China. The student teams braved extreme heat, extremely difficult applications, and extreme competition in order to cross the cluster competition finish line. The gala awards ceremony took place on Wednesday. The auditorium was packed with student teams, various dignitaries, the media, and other interested parties. So what happened? Read more…

By Dan Olds

Spring Meetings Underscore Quantum Computing’s Rise

May 17, 2018

The month of April 2018 saw four very important and interesting meetings to discuss the state of quantum computing technologies, their potential impacts, and th Read more…

By Alex R. Larzelere

Quantum Network Hub Opens in Japan

May 17, 2018

Following on the launch of its Q Commercial quantum network last December with 12 industrial and academic partners, the official Japanese hub at Keio University is now open to facilitate the exploration of quantum applications important to science and business. The news comes a week after IBM announced that North Carolina State University was the first U.S. university to join its Q Network. Read more…

By Tiffany Trader

Democratizing HPC: OSC Releases Version 1.3 of OnDemand

May 16, 2018

Making HPC resources readily available and easier to use for scientists who may have less HPC expertise is an ongoing challenge. Open OnDemand is a project by t Read more…

By John Russell

PRACE 2017 Annual Report: Exascale Aspirations; Industry Collaboration; HPC Training

May 15, 2018

The Partnership for Advanced Computing in Europe (PRACE) today released its annual report showcasing 2017 activities and providing a glimpse into thinking about Read more…

By John Russell

US Forms AI Brain Trust

May 11, 2018

Amid calls for a U.S. strategy for promoting AI development, the Trump administration is forming a senior-level panel to help coordinate government and industry research efforts. The Select Committee on Artificial Intelligence was announced Thursday (May 10) during a White House summit organized by the Office of Science and Technology Policy (OSTP). Read more…

By George Leopold

MLPerf – Will New Machine Learning Benchmark Help Propel AI Forward?

May 2, 2018

Let the AI benchmarking wars begin. Today, a diverse group from academia and industry – Google, Baidu, Intel, AMD, Harvard, and Stanford among them – releas Read more…

By John Russell

How the Cloud Is Falling Short for HPC

March 15, 2018

The last couple of years have seen cloud computing gradually build some legitimacy within the HPC world, but still the HPC industry lies far behind enterprise I Read more…

By Chris Downing

Russian Nuclear Engineers Caught Cryptomining on Lab Supercomputer

February 12, 2018

Nuclear scientists working at the All-Russian Research Institute of Experimental Physics (RFNC-VNIIEF) have been arrested for using lab supercomputing resources to mine crypto-currency, according to a report in Russia’s Interfax News Agency. Read more…

By Tiffany Trader

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

Deep Learning at 15 PFlops Enables Training for Extreme Weather Identification at Scale

March 19, 2018

Petaflop per second deep learning training performance on the NERSC (National Energy Research Scientific Computing Center) Cori supercomputer has given climate Read more…

By Rob Farber

AI Cloud Competition Heats Up: Google’s TPUs, Amazon Building AI Chip

February 12, 2018

Competition in the white hot AI (and public cloud) market pits Google against Amazon this week, with Google offering AI hardware on its cloud platform intended Read more…

By Doug Black

US Plans $1.8 Billion Spend on DOE Exascale Supercomputing

April 11, 2018

On Monday, the United States Department of Energy announced its intention to procure up to three exascale supercomputers at a cost of up to $1.8 billion with th Read more…

By Tiffany Trader

Lenovo Unveils Warm Water Cooled ThinkSystem SD650 in Rampup to LRZ Install

February 22, 2018

This week Lenovo took the wraps off the ThinkSystem SD650 high-density server with third-generation direct water cooling technology developed in tandem with par Read more…

By Tiffany Trader

Leading Solution Providers

HPC and AI – Two Communities Same Future

January 25, 2018

According to Al Gara (Intel Fellow, Data Center Group), high performance computing and artificial intelligence will increasingly intertwine as we transition to Read more…

By Rob Farber

Google Chases Quantum Supremacy with 72-Qubit Processor

March 7, 2018

Google pulled ahead of the pack this week in the race toward "quantum supremacy," with the introduction of a new 72-qubit quantum processor called Bristlecone. Read more…

By Tiffany Trader

HPE Wins $57 Million DoD Supercomputing Contract

February 20, 2018

Hewlett Packard Enterprise (HPE) today revealed details of its massive $57 million HPC contract with the U.S. Department of Defense (DoD). The deal calls for HP Read more…

By Tiffany Trader

CFO Steps down in Executive Shuffle at Supermicro

January 31, 2018

Supermicro yesterday announced senior management shuffling including prominent departures, the completion of an audit linked to its delayed Nasdaq filings, and Read more…

By John Russell

Deep Learning Portends ‘Sea Change’ for Oil and Gas Sector

February 1, 2018

The billowing compute and data demands that spurred the oil and gas industry to be the largest commercial users of high-performance computing are now propelling Read more…

By Tiffany Trader

Nvidia Ups Hardware Game with 16-GPU DGX-2 Server and 18-Port NVSwitch

March 27, 2018

Nvidia unveiled a raft of new products from its annual technology conference in San Jose today, and despite not offering up a new chip architecture, there were still a few surprises in store for HPC hardware aficionados. Read more…

By Tiffany Trader

Hennessy & Patterson: A New Golden Age for Computer Architecture

April 17, 2018

On Monday June 4, 2018, 2017 A.M. Turing Award Winners John L. Hennessy and David A. Patterson will deliver the Turing Lecture at the 45th International Sympo Read more…

By Staff

Part One: Deep Dive into 2018 Trends in Life Sciences HPC

March 1, 2018

Life sciences is an interesting lens through which to see HPC. It is perhaps not an obvious choice, given life sciences’ relative newness as a heavy user of H Read more…

By John Russell

  • arrow
  • Click Here for More Headlines
  • arrow
Share This