HPC Programming in the Age of Multicore: One Man’s View

By Michael Feldman

January 14, 2013

Application developers looking for a foundational treatment of HPC programming need look no further than the 2010 text, Introduction to High Performance Computing for Scientists and Engineers, which describes the art of extracting maximum performance from modern processors and HPC platforms. Gerhard Wellein, who co-authored the book with colleague Georg Hager, has made a career of teaching HPC techniques to aspiring students in science and engineering programs.

Currently a professor at the Department for Computer Science at the University of Erlangen-Nuremberg, Wellein also heads the HPC group at the Erlangen Regional Computing Center (RRZE). Like many in HPC today, he came from a science background, in this case physics, and specifically solid state physics. Wellein holds a PhD in the subject from Germany’s University of Bayreuth, and his interest in parallel and HPC programming stem from the intense computation demands of his original field of study.

At this June’s International Supercomputing Conference (ISC’13) in Leipzig, Germany, Wellein will be delivering a keynote titled, Fooling the Masses with Performance Results: Old Classics & Some New Ideas. The presentation was inspired by David H. Bailey 1991 classic article Twelve Ways to Fool the Masses When Giving Performance Results on Parallel Computers, and is intended to follow the humorous tone of Bailey’s original work.

HPCwire caught up with Wellein and asked him to preview some of the themes of his upcoming talk and expound on strategies for performance programming in the multicore era.

HPCwire: Multicore processors have been with us for more than a decade. In general, do you think programmers and programming has caught up to this computing paradigm?

Gerhard Wellein: Many software developers mainly focus on extendibility, flexibility and maintainability for their frameworks. Performance or better time to solution has not been an issue for them for two decades. Teaching in computer science still often ignores performance issues.

The same holds for parallelization – here the developers often hope that some other software layer – compiler or libraries – will do the job. However, people should become aware that improving time to solution, through hardware efficient code structures and parallelization, is not for free and often orthogonal to widely accepted concepts in modern programming languages and software engineering paradigms.

In a very provocative statement I would say that “abstraction is the natural enemy of performance.” To mitigate this problem, software developers need to understand the interaction of the different software layers, including compilers, with the basic hardware structures at least at a coarse level, and consider this relationship when designing and implementing the performance-critical path of the software.

HPCwire: In general, why is it so hard to extract peak performance from multicore processors?

Wellein: Modern multicore processors draw their performance from thread-level parallelism and data parallelism, that is, multiple cores and wide SIMD units per core. The programmer has to address both hardware features at the same time and needs optimize for them; otherwise he loses a substantial fraction of peak performance. If the compiler, for some reason, refuses to vectorize your single precision arithmetic code, you immediately lose almost 90 percent of the available peak performance on the latest Intel processors!

Then you need to do the job on your own. But who uses SIMD intrinsics or even programs assembly language? The same holds for multicore parallelization. Here load imbalances or dependencies can severely limit multicore efficiency from the start.

Moreover multicores have plenty of shared resources such as caches and memory interfaces which carry the risk of contention. One also should be aware that the maximum attainable performance always depends on the problem to be solved. Sparse matrix solvers working on irregular problems typically can only achieve a single digit fraction of peak performance, due to main memory bandwidth limitations. The architecture of modern multicores strongly favors problems which run from cache and have high spatial and temporal locality in their data access, which excludes large application areas from getting close to peak performance.
HPCwire: What kinds of tools and techniques for multicore programming are available that are not being commonly employed now?

Wellein: There is no lack of advanced tools. Often I have the impression that there are too many tools, frameworks and parallel programming models out there and many software developers get lost.

What is missing is a comprehensive understanding of the attainable performance for a give code or algorithm on a given hardware platform. Performance modeling is a structured process to analyze the interaction of hardware and software in order to predict the best performance levels both in serial and parallel applications. Comparing actual and predicted performance identifies inefficient code parts and opportunities for code optimization. Thus I feel that for time-critical applications, performance modeling should be integral part of the software development cycle.

HPCwire: If you had to pick one or two attributes of modern processors that hinders the ability of HPC developers to extract performance, what would it be?

Wellein: Performance improvements in multicores are driven mainly by two factors – SIMD and multicore parallelism – which need to be explicitly addressed by the programmer. The compiler alone will not do the job! In the good old days of the clock speed race, performance was boosted by the increasing processor frequency for free.

HPCwire: Is programming for performance different that programming for energy efficiency?

Wellein: No, this is just another side of the same medal. Programming for performance aims to reduce overall runtime. It may increase the power consumption mainly of the processor chip. But what counts at the end of the day is energy to solution, which is the product of both. All optimizations which substantially improve time to solution also reduce overall energy consumption.

What is even more interesting is that with a well optimized serial code you need fewer cores to saturate performance on the multicore chip. This now offers the interesting option to substantially reduce the clock speed of multicores or switch off cores completely! Some people may still complain that this makes your code less scalable on multicores. That is true but it improves energy efficiency and time to solution.

HPCwire: What do you think of the latest vector accelerators: GPUs and Intel’s latest Xeon Phi line, or even the general-purpose DSPs from Texas Instruments? Is this a reasonable approach for vector computing or do you see this as an intermediate step to something else?

Wellein: Those approaches trade in hardware complexity for additional performance, thus shifting additional work load to the programmer. Application performance on those architectures is typically much more susceptible to programmer’s capability to write highly efficient code. At the “core” level they draw their performance from the same concepts as vector computers did, namely fine-grained data parallelism with a wide SIMD approach. The product of those two factors on an Intel Xeon Phi processor makes performance approximately 15 times higher than on a state-of-the art Xeon EP processor, making the accelerator even more sensitive to inefficient codes.

From classical vector computers they differ in the sense that accelerators are mainly optimized for FLOPS and not for FLOPS-to-byte ratios. This also becomes evident if you look at the hierarchical memory subsystems which do not provide the ease of use of the simple flat designs of vector computers.

Increasing fine-grained data parallelism in hardware may further boost the FLOPS of future accelerators, but will further decrease the FLOPS-to-byte ratios. Also non-SIMD parts in the codes may soon become severe bottlenecks – with greetings from Gene Amdahl!
HPCwire: Does the mass market for processors – generic servers, PCs, and now especially mobile devices – serve the needs of HPC? Do you think there is a reasonable cost/benefit for developing more specialized processors for this domain?

Wellein: The rapid development cycles and cost reductions of the mass markets have substantially fostered progress in computational science and engineering in the past decade. These achievements opened HPC capacities for the “masses” in science and engineering.

I do not feel that most of those users are today willing to pay a high price premium to develop specialized hardware solutions, which means for them to trade in capability for capacity. What is happening to the last vendor of specialized processors for supercomputing – NEC with its vector computing processors – is a good example. In recent years NEC lost many of their customers in the weather and climate community, one of the few communities that asked for specialized HPC solutions for a long time. The Lattice QCD community is still building specialized solutions, such as the QPACE project, but also gave up designing its own processors.

The requirements to an “optimal processor” differ substantially among the application communities and making specialized designs for each of them is economically unattractive for most of the users. They take what they get from the mass market vendors. As long as those deliver processors which provide a good compromise between “time to solution,” aka capability, and “throughput,” aka capacity, for most users, there is not enough pressure for large-scale investments in specialized HPC processors.

I do not feel comfortable with that trend because there is tendency to blame the application that it cannot make efficient use of those processors and one should adapt the solver or even the physical model to the processor! A provocative summary of this trend: “Seymour Cray built machines to solve problems. Today we look for problems which we can solve on the machines available!”

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Pfizer HPC Engineer Aims to Automate Software Stack Testing

January 17, 2019

Seeking to reign in the tediousness of manual software testing, Pfizer HPC Engineer Shahzeb Siddiqui is developing an open source software tool called buildtest, aimed at automating software stack testing by providing the community with a central repository of tests for common HPC apps and the ability to automate execution of testing. Read more…

By Tiffany Trader

Senegal Prepares to Take Delivery of Atos Supercomputer

January 16, 2019

In just a few months time, Senegal will be operating the second largest HPC system in sub-Saharan Africa. The Minister of Higher Education, Research and Innovation Mary Teuw Niane made the announcement on Monday (Jan. 14 Read more…

By Tiffany Trader

Google Cloud Platform Extends GPU Instance Options

January 16, 2019

If it's Nvidia GPUs you're after to power your AI/HPC/visualization workload, Google Cloud has them, now claiming "broadest GPU availability." Each of the three big public cloud vendors has by turn touted the latest and Read more…

By Tiffany Trader

HPE Extreme Performance Solutions

HPE Systems With Intel Omni-Path: Architected for Value and Accessible High-Performance Computing

Today’s high-performance computing (HPC) and artificial intelligence (AI) users value high performing clusters. And the higher the performance that their system can deliver, the better. Read more…

IBM Accelerated Insights

Resource Management in the Age of Artificial Intelligence

New challenges demand fresh approaches

Fueled by GPUs, big data, and rapid advances in software, the AI revolution is upon us. Read more…

STAC Floats ML Benchmark for Financial Services Workloads

January 16, 2019

STAC (Securities Technology Analysis Center) recently released an ‘exploratory’ benchmark for machine learning which it hopes will evolve into a firm benchmark or suite of benchmarking tools to compare the performanc Read more…

By John Russell

Google Cloud Platform Extends GPU Instance Options

January 16, 2019

If it's Nvidia GPUs you're after to power your AI/HPC/visualization workload, Google Cloud has them, now claiming "broadest GPU availability." Each of the three Read more…

By Tiffany Trader

STAC Floats ML Benchmark for Financial Services Workloads

January 16, 2019

STAC (Securities Technology Analysis Center) recently released an ‘exploratory’ benchmark for machine learning which it hopes will evolve into a firm benchm Read more…

By John Russell

A Big Data Journey While Seeking to Catalog our Universe

January 16, 2019

It turns out, astronomers have lots of photos of the sky but seek knowledge about what the photos mean. Sound familiar? Big data problems are often characterize Read more…

By James Reinders

Intel Bets Big on 2-Track Quantum Strategy

January 15, 2019

Quantum computing has lived so long in the future it’s taken on a futuristic life of its own, with a Gartner-style hype cycle that includes triggers of innovation, inflated expectations and – though a useful quantum system is still years away – anticipatory troughs of disillusionment. Read more…

By Doug Black

IBM Quantum Update: Q System One Launch, New Collaborators, and QC Center Plans

January 10, 2019

IBM made three significant quantum computing announcements at CES this week. One was introduction of IBM Q System One; it’s really the integration of IBM’s Read more…

By John Russell

IBM’s New Global Weather Forecasting System Runs on GPUs

January 9, 2019

Anyone who has checked a forecast to decide whether or not to pack an umbrella knows that weather prediction can be a mercurial endeavor. It is a Herculean task: the constant modeling of incredibly complex systems to a high degree of accuracy at a local level within very short spans of time. Read more…

By Oliver Peckham

The Case Against ‘The Case Against Quantum Computing’

January 9, 2019

It’s not easy to be a physicist. Richard Feynman (basically the Jimi Hendrix of physicists) once said: “The first principle is that you must not fool yourse Read more…

By Ben Criger

The Deep500 – Researchers Tackle an HPC Benchmark for Deep Learning

January 7, 2019

How do you know if an HPC system, particularly a larger-scale system, is well-suited for deep learning workloads? Today, that’s not an easy question to answer Read more…

By John Russell

Quantum Computing Will Never Work

November 27, 2018

Amid the gush of money and enthusiastic predictions being thrown at quantum computing comes a proposed cold shower in the form of an essay by physicist Mikhail Read more…

By John Russell

Cray Unveils Shasta, Lands NERSC-9 Contract

October 30, 2018

Cray revealed today the details of its next-gen supercomputing architecture, Shasta, selected to be the next flagship system at NERSC. We've known of the code-name "Shasta" since the Argonne slice of the CORAL project was announced in 2015 and although the details of that plan have changed considerably, Cray didn't slow down its timeline for Shasta. Read more…

By Tiffany Trader

AMD Sets Up for Epyc Epoch

November 16, 2018

It’s been a good two weeks, AMD’s Gary Silcott and Andy Parma told me on the last day of SC18 in Dallas at the restaurant where we met to discuss their show news and recent successes. Heck, it’s been a good year. Read more…

By Tiffany Trader

The Case Against ‘The Case Against Quantum Computing’

January 9, 2019

It’s not easy to be a physicist. Richard Feynman (basically the Jimi Hendrix of physicists) once said: “The first principle is that you must not fool yourse Read more…

By Ben Criger

US Leads Supercomputing with #1, #2 Systems & Petascale Arm

November 12, 2018

The 31st Supercomputing Conference (SC) - commemorating 30 years since the first Supercomputing in 1988 - kicked off in Dallas yesterday, taking over the Kay Ba Read more…

By Tiffany Trader

Contract Signed for New Finnish Supercomputer

December 13, 2018

After the official contract signing yesterday, configuration details were made public for the new BullSequana system that the Finnish IT Center for Science (CSC Read more…

By Tiffany Trader

Summit Supercomputer is Already Making its Mark on Science

September 20, 2018

Summit, now the fastest supercomputer in the world, is quickly making its mark in science – five of the six finalists just announced for the prestigious 2018 Read more…

By John Russell

Nvidia’s Jensen Huang Delivers Vision for the New HPC

November 14, 2018

For nearly two hours on Monday at SC18, Jensen Huang, CEO of Nvidia, presented his expansive view of the future of HPC (and computing in general) as only he can do. Animated. Backstopped by a stream of data charts, product photos, and even a beautiful image of supernovae... Read more…

By John Russell

Leading Solution Providers

SC 18 Virtual Booth Video Tour

Advania @ SC18 AMD @ SC18
ASRock Rack @ SC18
DDN Storage @ SC18
HPE @ SC18
IBM @ SC18
Lenovo @ SC18 Mellanox Technologies @ SC18
NVIDIA @ SC18
One Stop Systems @ SC18
Oracle @ SC18 Panasas @ SC18
Supermicro @ SC18 SUSE @ SC18 TYAN @ SC18
Verne Global @ SC18

HPE No. 1, IBM Surges, in ‘Bucking Bronco’ High Performance Server Market

September 27, 2018

Riding healthy U.S. and global economies, strong demand for AI-capable hardware and other tailwind trends, the high performance computing server market jumped 28 percent in the second quarter 2018 to $3.7 billion, up from $2.9 billion for the same period last year, according to industry analyst firm Hyperion Research. Read more…

By Doug Black

HPC Reflections and (Mostly Hopeful) Predictions

December 19, 2018

So much ‘spaghetti’ gets tossed on walls by the technology community (vendors and researchers) to see what sticks that it is often difficult to peer through Read more…

By John Russell

Intel Confirms 48-Core Cascade Lake-AP for 2019

November 4, 2018

As part of the run-up to SC18, taking place in Dallas next week (Nov. 11-16), Intel is doling out info on its next-gen Cascade Lake family of Xeon processors, specifically the “Advanced Processor” version (Cascade Lake-AP), architected for high-performance computing, artificial intelligence and infrastructure-as-a-service workloads. Read more…

By Tiffany Trader

Germany Celebrates Launch of Two Fastest Supercomputers

September 26, 2018

The new high-performance computer SuperMUC-NG at the Leibniz Supercomputing Center (LRZ) in Garching is the fastest computer in Germany and one of the fastest i Read more…

By Tiffany Trader

Houston to Field Massive, ‘Geophysically Configured’ Cloud Supercomputer

October 11, 2018

Based on some news stories out today, one might get the impression that the next system to crack number one on the Top500 would be an industrial oil and gas mon Read more…

By Tiffany Trader

Microsoft to Buy Mellanox?

December 20, 2018

Networking equipment powerhouse Mellanox could be an acquisition target by Microsoft, according to a published report in an Israeli financial publication. Microsoft has reportedly gone so far as to engage Goldman Sachs to handle negotiations with Mellanox. Read more…

By Doug Black

House Passes $1.275B National Quantum Initiative

September 17, 2018

Last Thursday the U.S. House of Representatives passed the National Quantum Initiative Act (NQIA) intended to accelerate quantum computing research and developm Read more…

By John Russell

The Deep500 – Researchers Tackle an HPC Benchmark for Deep Learning

January 7, 2019

How do you know if an HPC system, particularly a larger-scale system, is well-suited for deep learning workloads? Today, that’s not an easy question to answer Read more…

By John Russell

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This