Compilers and More: Are Optimizing Compilers Important?

By Michael Wolfe

October 19, 2007

Recently I participated on a panel to discuss what has changed in the parallel computing research community in the past 20 years, specifically dealing with languages and compilers. Many things look the same, but one thing that has changed a great deal is the presence, acceptance and use of standard benchmarks. Twenty years ago, we compared supercomputers and minisupercomputers using any number of microkernels and small programs (remember the Livermore Loops!). We still fall into this trap, rating the top 500 supercomputers using a single benchmark program. However, even 20 years ago, there were several efforts to collect benchmarks suites to represent real applications, including the Perfect Club, RiCEPS, and Mendez suites. These were also used to compare the behavior and performance of compiler optimizations.

Processor, system and compiler performance are now frequently measured using the SPEC CPU benchmarks. The original suite was released as ten programs in 1989, each of which ran for about 5 minutes on then-current workstations, with performance normalized to that of a VAX 11/780 (that very VAX now sits in the Paul G. Allen Center, housing the Computer Science and Engineering department at the University of Washington). Good luck finding those programs today, SPEC seems to have disavowed all knowledge of them. If you can, it’s fun to run them or the 1992 or 1995 versions; today’s machines are so fast that the programs finish almost before the return key bounces back. The latest suite, SPEC CPU2006, comprises 28 programs written in C++, C and Fortran.

As originally intended, the SPEC suite was to be used by customers as a reasonably comprehensive and vendor-neutral benchmark for comparing system performance. It has since become much more important. For instance, the list price for a new computer system may be raised or lowered depending on whether the SPEC benchmark score is higher or lower than that of its competition, so SPEC performance can affect the profitability of a vendor. To be fair, SPEC is not the only benchmark that is so used, but it is one of the most visible.

The components that affect SPEC performance are the processor, cache, memory and memory bus, and the compiler. The cache, memory and bus are typically determined by the processor, so there are really only two variables. Processor designers add or optimize features to improve performance, often looking at instructions used in these benchmarks to decide what to optimize. The days of increasing speed by pumping up the clock rate seem to be gone, but there are still opportunities for improvement in implementation technology and microarchitecture.

It’s illuminating to look at some historical SPEC CPU2000 results between its initial release in late 1999 until its retirement in late 2006. Dell published a run in November 1999 on a top-of-the-line Precision Workstation 420 which delivered a SPEC CINT2000 base ratio of 336, and CFP2000 ratio of 242. Seven years later, Dell published a run on a then-current Precision Workstation 390 with a CINT2000 base ratio of 2829 and CFP2000 ratio of 2679, for a factor of 8 improvement in CINT and 11 in CFP performance. Most of this improvement was undoubtedly due to the move from the 733 MHz Intel Pentium III (with 256KB L2 cache) to the 2.66GHz Intel Core 2 Extreme (with 4MB L2 cache). We might predict a 3.5X improvement just from the clock rate, with additional speedup due to larger cache, double precision SSE2 instructions, and aggressive superscalar instruction issue with out-of-order execution. Indeed, the lowest speedup was 4.5 (for 164.gzip).

However, the speedups are quite nonuniform. For instance, 171.swim was the best CFP performer in 1999, but was in the middle of the pack in 2006, with a speedup of 6.3;, the second best CFP performer in 1999 benefited from a speedup of almost 30 to become the best (by far) in 2006. On the CINT side, 181.mcf was the worst performer in 1999, but was second best in 2006 (speedup of 18), so something obviously changed for the better there, whereas 164.gzip, the third best CINT in 1999, was the worst performer in 2006.

I’d like to explore how much of the performance improvement is due to clock and implementation differences, how much is due to the instruction set changes with the addition of SSE2 and the move to x86-64, and in particular whether the compiler had any effect. To demonstrate this, I’m going to show some results of one particular SPEC CPU2000 benchmark, 172.mgrid, for reasons that I’ll explain below. I ran mgrid on two different machines with two different compilers. Note: These are not official SPEC runs, so the results are only estimates. The two machine profiles are:

  • Intel Pentium III, 550MHz, 512KB L2 cache, 1GB memory, Linux Red Hat 8.0
  • Intel Xeon 5160, 3GHz, 4MB L2 cache, 4GB memory, Linux SLES 10

The two compilers are the PGI compiler suite from 1999 (3.2-4a, using -fast) and our most recent release, 7.0-7 (using -fast -Mipa=fast,inline). I estimate the total speedup from 1999 to 2006 by comparing runs on the Pentium III using the 3.2-4 compiler to runs on the Xeon using the 7.0-7 compiler. For mgrid, the speedup was 28.

My first experiment isolates the compiler, testing only the clock and implementation improvements from the Pentium III to the Xeon. I compiled and ran mgrid using the 3.2-4a compiler on the Pentium III, then ran the same binary on the Xeon machine; this gives a speedup of 12.5, not quite half the total. A factor of eight is easy to explain with the clock speed and microarchitectural improvements; the higher number is possibly due to more of the working set fitting entirely in the much larger L2 cache.

My second experiment repeats the first, but using the newer compiler. I compiled and ran mgrid using the 7.0-7 compiler on the Pentium III, then ran the same binary on the Xeon machine. The newer compilers have some new features, but we should expect more or less the same improvements from the older machine to the Xeon; here I saw a speedup of 15. The new binary ran faster on both machines, but the improvement was more significant on the Xeon, even though the compiler was optimizing for the Pentium III. From these two experiments, I conclude that clock, microarchitecture and other implementation improvements delivered a speedup factor of between 12-15. Very impressive, but this is only about half the total speedup for mgrid between 1999 and 2006.

My third experiment tries to isolate the improvements due to the instruction set enhancements added since the Pentium III, such as SSE2 instructions. I compiled and ran mgrid using the 7.0-7 compiler, targetting the 32-bit instruction set of the Xeon, and compared it to the run targeting the Pentium III instruction set. We should expect some improvement here, since mgrid has lots of vectorized double precision operations. However, my results show only about a 2 percent improvement, quite a surprise.

My final architectural experiment isolates the benefits of moving to the x86-64 instruction set. I compiled and ran mgrid with the 7.0-7 compiler targetting the full 64-bit instruction set, and compared this to the run using the 32-bit instruction set. For mgrid, this improvement is about 22 percent. So the total speedup due to hardware and instruction set is between 15-19.

But this is supposed to be a compiler column. Can we attribute the rest of the speedup to compiler improvements? Certainly, to take advantage of the new instructions (SSE2, 64-bit instructions), compilers had to be significantly enhanced. However, it’s not fair to chalk up all those speedups in the compiler column.

Commercial compilers have advanced greatly since 1999; for instance, as I’ve mentioned in past columns, all current commercial compilers now use some form of interprocedural or whole program analysis and optimization. Some of this controls inlining, some propagates other information across subprogram and file boundaries.

I ran two more experiments to isolate the effects of the compiler. I took the 3.2-4 compiled code and the 7.0-7 compiled code, and ran both on the Pentium III. Here I saw a speedup of 1.5, 50 percent improvement just from compiler improvements. Not nearly the factor of 15-19 we get from hardware, but these are multiplicative improvements. I then took those same binaries and ran them on the Xeon. Note, these binaries were optimized for the Pentium-III, not the Xeon. Nevertheless, the speedup on the Xeon was even better, almost 1.8.

So, what is it about mgrid that lets the compiler deliver a 50-80 percent performance improvement over seven years. I chose mgrid for a reason, and not because it benefits most from compiler improvements since 1999, but because there’s one specific optimization that applies. Let’s look at one of the key loops in mgrid:

      DO I3 = 2, N-1
DO I2 = 2, N-1
DO I1 = 2, N-1
> -A(0)*( U(I1, I2, I3 ) )
> -A(1)*( U(I1-1,I2, I3 ) + U(I1+1,I2, I3 )
> + U(I1, I2-1,I3 ) + U(I1, I2+1,I3 )
> + U(I1, I2, I3-1) + U(I1, I2, I3+1) )
> -A(2)*( U(I1-1,I2-1,I3 ) + U(I1+1,I2-1,I3 )
> + U(I1-1,I2+1,I3 ) + U(I1+1,I2+1,I3 )
> + U(I1, I2-1,I3-1) + U(I1, I2+1,I3-1)
> + U(I1, I2-1,I3+1) + U(I1, I2+1,I3+1)
> + U(I1-1,I2, I3-1) + U(I1-1,I2, I3+1)
> + U(I1+1,I2, I3-1) + U(I1+1,I2, I3+1) )
> -A(3)*( U(I1-1,I2-1,I3-1) + U(I1+1,I2-1,I3-1)
> + U(I1-1,I2+1,I3-1) + U(I1+1,I2+1,I3-1)
> + U(I1-1,I2-1,I3+1) + U(I1+1,I2-1,I3+1)
> + U(I1-1,I2+1,I3+1) + U(I1+1,I2+1,I3+1) )

Several loops in mgrid are similar to this one. This loop fetches 28 array elements and performs 27 double precision floating point additions. However, compilers now recognize that in the inner loop, the value computed as ‘U(I1+1,I2,I3-1)+U(I1+1,I2,I3+1)’ will be used again in the next iteration as ‘U(I1,I2,I3-1)+U(I1,I2,I3+1)’. Saving that value in a register eliminates two array element fetches and one addition. In fact, this pattern occurs so often in this loop that the optimized inner loop only loads 12 array elements and performs 15 floating point additions, a savings of about 1/2. Such optimizations have appeared in academic literature with research languages, such as ZPL, but had not been implemented in production commercial compilers before SPEC CPU2000 and mgrid. At PGI, we designed our implementation as an enhancement of the Scalar Replacement optimization developed at Rice University. We call it Loop-Carried Redundancy Elimination, or LRE, and it is responsible for improving our compiler’s performance on mgrid 15-20 percent, depending on the target machine.

To answer the question in the title, yes, optimizing compilers are important. Even without the LRE optimization, our compilers have improved the performance of mgrid somewhere between 35-60 percent since 1999. With LRE, the improvement is 50-80 percent, and that’s compared to optimized code. This is like progressing a processor generation or two beyond what you can buy today.

One might argue whether using LRE to optimize mgrid is fair, as the optimization seems targetted specifically at that benchmark. Happily, LRE has turned out to be quite useful in many cases in Fortran, C, and C++ numerical applications as well as in signal processing applications for embedded systems. Quite possibly, LRE would have been investigated and implemented even without mgrid. However, it’s hard not to agree that the importance of the SPEC CPU benchmark spurs the study of optimizations for those programs.

But there’s a dark side as well. There are several dangers when introducing a compiler optimization to improve a specific benchmark. It’s an additional feature in the compiler, which must be tested and maintained; this can be a nontrivial cost over the lifetime of the compiler. It may affect the stability of the compiler, both in correctness, if the initial implementation is rushed, and in performance, if it speeds up some programs and slows down others. And it may raise expectations by customers, who hope to see the same 15-20 percent improvements in their own codes, and can be quite vocally disappointed when they don’t. While it may make some benchmark numbers look good, implementing a bunch of benchmark-specific optimizations doesn’t make for a quality compiler.

To address this, the SPEC run rules disallow optimizations that specifically target the SPEC benchmark suites. But who’s to say that some optimization targets one specific benchmark. Certainly when we implemented LRE, we had mgrid in mind; it was only afterward that we realized how often this pattern appeared and how important it was. I’ll explore this topic in more detail in my next column; as a teaser, what might account for that factor of 30 performance improvement in

SPEC is a registered trademark of the Standard Performance Evaluation Corporation (


Michael Wolfe has developed compilers for over 30 years in both academia and industry, and is now a senior compiler engineer at The Portland Group, Inc. (, a wholly-owned subsidiary of STMicroelectronics, Inc. The opinions stated here are those of the author, and do not represent opinions of The Portland Group, Inc. or STMicroelectronics, Inc.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

AI-Focused ‘Genius’ Supercomputer Installed at KU Leuven

April 24, 2018

Hewlett Packard Enterprise has deployed a new approximately half-petaflops supercomputer, named Genius, at Flemish research university KU Leuven. The system is built to run artificial intelligence (AI) workloads and, as Read more…

By Tiffany Trader

New Exascale System for Earth Simulation Introduced

April 23, 2018

After four years of development, the Energy Exascale Earth System Model (E3SM) will be unveiled today and released to the broader scientific community this month. The E3SM project is supported by the Department of Energy Read more…

By Staff

RSC Reports 500Tflops, Hot Water Cooled System Deployed at JINR

April 18, 2018

RSC, developer of supercomputers and advanced HPC systems based in Russia, today reported deployment of “the world's first 100% ‘hot water’ liquid cooled supercomputer” at Joint Institute for Nuclear Research (JI Read more…

By Staff

HPE Extreme Performance Solutions

Hybrid HPC is Speeding Time to Insight and Revolutionizing Medicine

High performance computing (HPC) is a key driver of success in many verticals today, and health and life science industries are extensively leveraging these capabilities. Read more…

New Device Spots Quantum Particle ‘Fingerprint’

April 18, 2018

Majorana particles have been observed by university researchers employing a device consisting of layers of magnetic insulators on a superconducting material. The advance opens the door to controlling the elusive particle Read more…

By George Leopold

AI-Focused ‘Genius’ Supercomputer Installed at KU Leuven

April 24, 2018

Hewlett Packard Enterprise has deployed a new approximately half-petaflops supercomputer, named Genius, at Flemish research university KU Leuven. The system is Read more…

By Tiffany Trader

Cray Rolls Out AMD-Based CS500; More to Follow?

April 18, 2018

Cray was the latest OEM to bring AMD back into the fold with introduction today of a CS500 option based on AMD’s Epyc processor line. The move follows Cray’ Read more…

By John Russell

IBM: Software Ecosystem for OpenPOWER is Ready for Prime Time

April 16, 2018

With key pieces of the IBM/OpenPOWER versus Intel/x86 gambit settling into place – e.g., the arrival of Power9 chips and Power9-based systems, hyperscaler sup Read more…

By John Russell

US Plans $1.8 Billion Spend on DOE Exascale Supercomputing

April 11, 2018

On Monday, the United States Department of Energy announced its intention to procure up to three exascale supercomputers at a cost of up to $1.8 billion with th Read more…

By Tiffany Trader

Cloud-Readiness and Looking Beyond Application Scaling

April 11, 2018

There are two aspects to consider when determining if an application is suitable for running in the cloud. The first, which we will discuss here under the title Read more…

By Chris Downing

Transitioning from Big Data to Discovery: Data Management as a Keystone Analytics Strategy

April 9, 2018

The past 10-15 years has seen a stark rise in the density, size, and diversity of scientific data being generated in every scientific discipline in the world. Key among the sciences has been the explosion of laboratory technologies that generate large amounts of data in life-sciences and healthcare research. Large amounts of data are now being stored in very large storage name spaces, with little to no organization and a general unease about how to approach analyzing it. Read more…

By Ari Berman, BioTeam, Inc.

IBM Expands Quantum Computing Network

April 5, 2018

IBM is positioning itself as a first mover in establishing the era of commercial quantum computing. The company believes in order for quantum to work, taming qu Read more…

By Tiffany Trader

FY18 Budget & CORAL-2 – Exascale USA Continues to Move Ahead

April 2, 2018

It was not pretty. However, despite some twists and turns, the federal government’s Fiscal Year 2018 (FY18) budget is complete and ended with some very positi Read more…

By Alex R. Larzelere

Inventor Claims to Have Solved Floating Point Error Problem

January 17, 2018

"The decades-old floating point error problem has been solved," proclaims a press release from inventor Alan Jorgensen. The computer scientist has filed for and Read more…

By Tiffany Trader

Researchers Measure Impact of ‘Meltdown’ and ‘Spectre’ Patches on HPC Workloads

January 17, 2018

Computer scientists from the Center for Computational Research, State University of New York (SUNY), University at Buffalo have examined the effect of Meltdown Read more…

By Tiffany Trader

How the Cloud Is Falling Short for HPC

March 15, 2018

The last couple of years have seen cloud computing gradually build some legitimacy within the HPC world, but still the HPC industry lies far behind enterprise I Read more…

By Chris Downing

Russian Nuclear Engineers Caught Cryptomining on Lab Supercomputer

February 12, 2018

Nuclear scientists working at the All-Russian Research Institute of Experimental Physics (RFNC-VNIIEF) have been arrested for using lab supercomputing resources to mine crypto-currency, according to a report in Russia’s Interfax News Agency. Read more…

By Tiffany Trader

Chip Flaws ‘Meltdown’ and ‘Spectre’ Loom Large

January 4, 2018

The HPC and wider tech community have been abuzz this week over the discovery of critical design flaws that impact virtually all contemporary microprocessors. T Read more…

By Tiffany Trader

How Meltdown and Spectre Patches Will Affect HPC Workloads

January 10, 2018

There have been claims that the fixes for the Meltdown and Spectre security vulnerabilities, named the KPTI (aka KAISER) patches, are going to affect applicatio Read more…

By Rosemary Francis

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

Deep Learning at 15 PFlops Enables Training for Extreme Weather Identification at Scale

March 19, 2018

Petaflop per second deep learning training performance on the NERSC (National Energy Research Scientific Computing Center) Cori supercomputer has given climate Read more…

By Rob Farber

Leading Solution Providers

Fast Forward: Five HPC Predictions for 2018

December 21, 2017

What’s on your list of high (and low) lights for 2017? Volta 100’s arrival on the heels of the P100? Appearance, albeit late in the year, of IBM’s Power9? Read more…

By John Russell

Lenovo Unveils Warm Water Cooled ThinkSystem SD650 in Rampup to LRZ Install

February 22, 2018

This week Lenovo took the wraps off the ThinkSystem SD650 high-density server with third-generation direct water cooling technology developed in tandem with par Read more…

By Tiffany Trader

HPC and AI – Two Communities Same Future

January 25, 2018

According to Al Gara (Intel Fellow, Data Center Group), high performance computing and artificial intelligence will increasingly intertwine as we transition to Read more…

By Rob Farber

AI Cloud Competition Heats Up: Google’s TPUs, Amazon Building AI Chip

February 12, 2018

Competition in the white hot AI (and public cloud) market pits Google against Amazon this week, with Google offering AI hardware on its cloud platform intended Read more…

By Doug Black

New Blueprint for Converging HPC, Big Data

January 18, 2018

After five annual workshops on Big Data and Extreme-Scale Computing (BDEC), a group of international HPC heavyweights including Jack Dongarra (University of Te Read more…

By John Russell

US Plans $1.8 Billion Spend on DOE Exascale Supercomputing

April 11, 2018

On Monday, the United States Department of Energy announced its intention to procure up to three exascale supercomputers at a cost of up to $1.8 billion with th Read more…

By Tiffany Trader

Momentum Builds for US Exascale

January 9, 2018

2018 looks to be a great year for the U.S. exascale program. The last several months of 2017 revealed a number of important developments that help put the U.S. Read more…

By Alex R. Larzelere

Google Chases Quantum Supremacy with 72-Qubit Processor

March 7, 2018

Google pulled ahead of the pack this week in the race toward "quantum supremacy," with the introduction of a new 72-qubit quantum processor called Bristlecone. Read more…

By Tiffany Trader

  • arrow
  • Click Here for More Headlines
  • arrow
Share This