Compilers and More: Precision and Accuracy

By Michael Wolfe

May 11, 2007

According to Wikipedia, precision is defined in engineering and science as the degree of agreement among a series of individual measurements or results. For instance, if I measure my height (nonmetrically) five days in a row and come up with 73.8″, 73.5″, 73.75″, 73.7″ and 73.6″, I could state that I’m just over 6 feet tall. The precision of the measurements is about three tenths of an inch, the range of the measurements.

Accuracy is the degree of agreement of a measure value with the actual value. If you know the actual value, you hardly need to measure it, except perhaps to determine the accuracy of the measuring tool. If you don’t know the actual value, you might infer the accuracy of the measurement from the precision of your measurements, if you believe your measuring tool is reasonably accurate. When asked, I usually say I’m 6’2″ tall, which is reasonably accurate, though not exact.

In computing, we use precision in two ways. One is the number of bits in the representation of the mantissa of a floating point number; for instance, single-precision IEEE floating point gives 24 bits of precision (counting the hidden bit for normalized numbers), about 7 decimal digits; double-precision (or full precision, as numerical analysts used to call it) provides 53 bits of precision, about 15 decimal digits.

We also use precision as a measure of how many of these bits are correct. For instance, if I represent a very precise measurement of 1.000100 as a single precision number, then subtract 1.0 from that, the floating point result is 1.000000E-4; since the original measurement was only precise to seven digits, this difference is only precise to three digits, despite being represented with 7 digits. As another simple example, try summing 1/i in single precision where i ranges from 1 to 10,000,000; then sum the numbers in the other order, from 1/10,000,000 to 1/1; here, the answers differ in the second digit. The computations are mathematically identical, but computationally different, because of the limited accuracy of the computer floating point representation. The key is the order of computations can change the floating point answers, due to differences in the order and magnitude of rounding, itself due to the limited size of the representation. Numerical analysts earn a living determining how accurate and precise a computed answer is, and how to formulate the computation to improve accuracy and precision. The rest of us address the problem by going to double precision and hoping for the best.

It’s well known that a compiler can change the order of computations, and yes, this can affect the answers delivered. This dates back at least to the early days of vectorizing compilers for the Texas Instruments ASC, Control Data Cyber 205, and Cray 1. The simplest example was the way a summation is accumulated in vector mode. In our simple example, we showed two orders (forward and backward). These vector computers accumulated some number of intermediate partial sums, then add up the partial sums for a final result. The Cray compiler, for instance, would accumulate a sum in groups of 64 partial sums, the length of the vector register. The Cyber 205 would accumulate eight partial sums, the pipeline depth of the floating point adder. In all cases, the compiler had a flag to preserve the same floating point computation order, but that could be significantly slower.

This is not just historically interesting; many current processors have multimedia or SIMD instruction set extensions, such as the SSE and SSE2 instructions on the Intel and AMD x64 processors. Current compilers use the same code generation scheme to vectorize summations as compilers did 30 years ago for vector machines, though the ‘vector length’ may be only 2 or 4, the size of the multimedia registers.

In addition, a language definition may allow some freedom of evaluation order. Fortran allows a compiler to evaluate any mathematically equivalent expression, provided that parentheses are not violated. C does not even protect parenthesized subexpressions, explicitly allowing reassociation and even redistribution of multiply over addition.

Why this matters

It’s also well known that floating point operations, in spite of gigahertz clocks, functional unit pipelines and all the other architectural magic, are relatively slow. Multiplication is slower than addition, and division or square root is up to ten times slower than that. For those in the high performance world, reducing the multiply count, or replacing multiplication by addition, or, even better, replacing division by multiplication can give a big performance boost. It’s hard to underestimate the importance of benchmark performance in the world of compiler and processor vendors, where bragging rights, pricing, and sales can all depend on a particular benchmark or suite.

So let’s take the case of a particular benchmark, Gromacs, from the SPEC CPU2006 suite. This benchmark has a large number of single precision square root function calls, many of which are in the denominator, such as 1.0/sqrt(rsq11). Several current microprocessors have an instruction to approximate a single-precision inverse square root, such as the PowerPC (AltiVec), the Intel (SSE) and AMD (3D-Now! and SSE). In all cases, the instruction gives a low precision approximation, about half the number of bits. One Newton-Raphson iteration is enough to improve the result to full precision.

Such a technique is well known; the Cray 1 implemented a reciprocal approximate instead of a divide instruction, requiring the compiler to generate the Newton-Raphson iteration to generate a full precision result. This allowed divides to be vector pipelined without undue hardware requirements.

The performance of the square root and divide units are so slow that doing the reciprocal square root approximate instruction, followed by a Newton-Raphson iteration involving four multiplies and a substract, is still quite a bit faster. In some applications (graphics, for example), full precision may be overkill anyway, so perhaps the approximation by itself is sufficient.

So, all is well, right?

The problem is that one iteration of Newton-Raphson is enough to bring the approximation to full precision, except for the last ULP (unit in the last place) or two. This means that using the optimized code in this benchmark generates slightly different results than the standard code that uses a library or full-precision square root and divide instructions.

So, are the answers right or not?

This is tough question, perhaps one that doesn’t make sense. Recall that there’s a certain amount of roundoff error for any computer floating point arithmetic, so the answer is “wrong” whichever way you compute this value. Moreover, the computation is probably based on some initial data that was measured or approximated and so doesn’t have that much precision to start with.

So, are the answers okay or not?

As long as the programmer and the user realize and agree with the limitations of the generated code, yes, they are okay. There are really only two problems. In the history of floating point arithmetic, we’ve had a myriad of floating point formats, from base 16 to two, with or without rounding, with wildly varying dynamic range. In the 1980s, IEEE sponsored an effort to standardize on floating point formats, which all current mainstream processors use. This means that the answers computed on one brand processor, be it AMD, IBM, or Intel, are going to be the same as on any other brand, such as Motorola, MIPS, or SPARC. But here, we’re changing the computation, so the first problem is that answers on different hardware will be close, but may no longer be exactly the same.

The other problem has to do with the programmer and user agreeing with the limitations. Compilers that implement this performance optimization will enable or disable it under programmer control, usually with a command-line option (for those of us still using command lines). The issue is what’s the default? Would you like to specify an additional option to relax the precision of these operations, or specify an additional option to maintain their precision? This is largely a marketing question, though having an option to ‘maintain precision’ makes one wonder what the default behavior is.

As with all SPEC CPU2006 benchmarks, Gromacs checks that the computed answers are correct. However, the check is weak enough that the reciprocal square root approximation is good enough; the Newton-Raphson iteration isn’t even necessary. And yes, some of those SPEC submissions use this “optimization” in their peak numbers. Beware! Dragons lurk here!

Perhaps compilers could take another step and provide an option that would aggressively reorder and reassociate operations, and in general play heck with the floating point rounding properties of the program. If you run your program in standard and reordered mode and the answers are the same, you could have some degree of confidence that the language, the compiler, and the hardware did nothing bad to your computation (though the physics, of course, is up to you). However, if the answers differed wildly, then perhaps you need to bring in that numerical analyst to determine just what makes your program so sensitive.

As clock rates peak, we will explore ever more bizarre mechanisms to improve performance. There’s a lot of buzz about GPGPUs (General Purpose computing on Graphics Processing Units), which are cheap and fast, but which don’t (yet) implement all the IEEE rounding modes. We could quickly degrade into a world much like 20 years ago, where you must consider the hardware floating point representation and implementation before choosing your hardware.

Precision control can also affect performance. Recent numerical library work at Oak Ridge National Labs and the University of Tennessee uses the fact that single precision (32-bit) floating point operations run much faster than double precision on many current processors. The packed SSE instructions on the x64 processors, for instance, can do twice as many 32-bit operations per cycle. This encourages them to redesign their libraries to do the bulk of the work in single precision, then to use iterative refinement in double precision to provide a high precision result with low precision speed.

On the other hand, some scientists are concerned about how quickly roundoff error accumulates when performing peta-operations, perhaps reducing the number of significant digits to zero. Eighteen years ago, David Bailey and others predicted that IEEE double precision floating point would run out of bits as problem sizes increased, and suggested that hardware 128-bit floating point would be needed. Perhaps those SSE registers will soon be holding a single value.


Michael Wolfe has developed compilers for over 30 years in both academia and industry, and is now a senior compiler engineer at The Portland Group, Inc. (, a wholly-owned subsidiary of STMicroelectronics, Inc. The opinions stated here are those of the author, and do not represent opinions of The Portland Group, Inc. or STMicroelectronics, Inc.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

GlobalFoundries Puts Wind in AMD’s Sails with 12nm FinFET

September 24, 2017

From its annual tech conference last week (Sept. 13), where GlobalFoundries welcomed more than 600 semiconductor professionals (reaching the Santa Clara venue’s max capacity and doubling 2016 attendee numbers), the one Read more…

By Tiffany Trader

Machine Learning at HPC User Forum: Drilling into Specific Use Cases

September 22, 2017

The 66th HPC User Forum held September 5-7, in Milwaukee, Wisconsin, at the elegant and historic Pfister Hotel, highlighting the 1893 Victorian décor and art of “The Grand Hotel Of The West,” contrasted nicely with Read more…

By Arno Kolster

Google Cloud Makes Good on Promise to Add Nvidia P100 GPUs

September 21, 2017

Google has taken down the notice on its cloud platform website that says Nvidia Tesla P100s are “coming soon.” That's because the search giant has announced the beta launch of the high-end P100 Nvidia Tesla GPUs on t Read more…

By George Leopold

HPE Extreme Performance Solutions

HPE Prepares Customers for Success with the HPC Software Portfolio

High performance computing (HPC) software is key to harnessing the full power of HPC environments. Development and management tools enable IT departments to streamline installation and maintenance of their systems as well as create, optimize, and run their HPC applications. Read more…

Cray Wins $48M Supercomputer Contract from KISTI

September 21, 2017

It was a good day for Cray which won a $48 million contract from the Korea Institute of Science and Technology Information (KISTI) for a 128-rack CS500 cluster supercomputer. The new system, equipped with Intel Xeon Scal Read more…

By John Russell

GlobalFoundries Puts Wind in AMD’s Sails with 12nm FinFET

September 24, 2017

From its annual tech conference last week (Sept. 13), where GlobalFoundries welcomed more than 600 semiconductor professionals (reaching the Santa Clara venue Read more…

By Tiffany Trader

Machine Learning at HPC User Forum: Drilling into Specific Use Cases

September 22, 2017

The 66th HPC User Forum held September 5-7, in Milwaukee, Wisconsin, at the elegant and historic Pfister Hotel, highlighting the 1893 Victorian décor and art o Read more…

By Arno Kolster

Stanford University and UberCloud Achieve Breakthrough in Living Heart Simulations

September 21, 2017

Cardiac arrhythmia can be an undesirable and potentially lethal side effect of drugs. During this condition, the electrical activity of the heart turns chaotic, Read more…

By Wolfgang Gentzsch, UberCloud, and Francisco Sahli, Stanford University

PNNL’s Center for Advanced Tech Evaluation Seeks Wider HPC Community Ties

September 21, 2017

Two years ago the Department of Energy established the Center for Advanced Technology Evaluation (CENATE) at Pacific Northwest National Laboratory (PNNL). CENAT Read more…

By John Russell

Exascale Computing Project Names Doug Kothe as Director

September 20, 2017

The Department of Energy’s Exascale Computing Project (ECP) has named Doug Kothe as its new director effective October 1. He replaces Paul Messina, who is stepping down after two years to return to Argonne National Laboratory. Kothe is a 32-year veteran of DOE’s National Laboratory System. Read more…

Takeaways from the Milwaukee HPC User Forum

September 19, 2017

Milwaukee’s elegant Pfister Hotel hosted approximately 100 attendees for the 66th HPC User Forum (September 5-7, 2017). In the original home city of Pabst Blu Read more…

By Merle Giles

Kathy Yelick Charts the Promise and Progress of Exascale Science

September 15, 2017

On Friday, Sept. 8, Kathy Yelick of Lawrence Berkeley National Laboratory and the University of California, Berkeley, delivered the keynote address on “Breakthrough Science at the Exascale” at the ACM Europe Conference in Barcelona. In conjunction with her presentation, Yelick agreed to a short Q&A discussion with HPCwire. Read more…

By Tiffany Trader

DARPA Pledges Another $300 Million for Post-Moore’s Readiness

September 14, 2017

The Defense Advanced Research Projects Agency (DARPA) launched a giant funding effort to ensure the United States can sustain the pace of electronic innovation vital to both a flourishing economy and a secure military. Under the banner of the Electronics Resurgence Initiative (ERI), some $500-$800 million will be invested in post-Moore’s Law technologies. Read more…

By Tiffany Trader

How ‘Knights Mill’ Gets Its Deep Learning Flops

June 22, 2017

Intel, the subject of much speculation regarding the delayed, rewritten or potentially canceled “Aurora” contract (the Argonne Lab part of the CORAL “ Read more…

By Tiffany Trader

Reinders: “AVX-512 May Be a Hidden Gem” in Intel Xeon Scalable Processors

June 29, 2017

Imagine if we could use vector processing on something other than just floating point problems.  Today, GPUs and CPUs work tirelessly to accelerate algorithms Read more…

By James Reinders

NERSC Scales Scientific Deep Learning to 15 Petaflops

August 28, 2017

A collaborative effort between Intel, NERSC and Stanford has delivered the first 15-petaflops deep learning software running on HPC platforms and is, according Read more…

By Rob Farber

Oracle Layoffs Reportedly Hit SPARC and Solaris Hard

September 7, 2017

Oracle’s latest layoffs have many wondering if this is the end of the line for the SPARC processor and Solaris OS development. As reported by multiple sources Read more…

By John Russell

Six Exascale PathForward Vendors Selected; DoE Providing $258M

June 15, 2017

The much-anticipated PathForward awards for hardware R&D in support of the Exascale Computing Project were announced today with six vendors selected – AMD Read more…

By John Russell

Top500 Results: Latest List Trends and What’s in Store

June 19, 2017

Greetings from Frankfurt and the 2017 International Supercomputing Conference where the latest Top500 list has just been revealed. Although there were no major Read more…

By Tiffany Trader

IBM Clears Path to 5nm with Silicon Nanosheets

June 5, 2017

Two years since announcing the industry’s first 7nm node test chip, IBM and its research alliance partners GlobalFoundries and Samsung have developed a proces Read more…

By Tiffany Trader

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

Leading Solution Providers

Graphcore Readies Launch of 16nm Colossus-IPU Chip

July 20, 2017

A second $30 million funding round for U.K. AI chip developer Graphcore sets up the company to go to market with its “intelligent processing unit” (IPU) in Read more…

By Tiffany Trader

Google Releases Deeplearn.js to Further Democratize Machine Learning

August 17, 2017

Spreading the use of machine learning tools is one of the goals of Google’s PAIR (People + AI Research) initiative, which was introduced in early July. Last w Read more…

By John Russell

Russian Researchers Claim First Quantum-Safe Blockchain

May 25, 2017

The Russian Quantum Center today announced it has overcome the threat of quantum cryptography by creating the first quantum-safe blockchain, securing cryptocurrencies like Bitcoin, along with classified government communications and other sensitive digital transfers. Read more…

By Doug Black

Google Debuts TPU v2 and will Add to Google Cloud

May 25, 2017

Not long after stirring attention in the deep learning/AI community by revealing the details of its Tensor Processing Unit (TPU), Google last week announced the Read more…

By John Russell

EU Funds 20 Million Euro ARM+FPGA Exascale Project

September 7, 2017

At the Barcelona Supercomputer Centre on Wednesday (Sept. 6), 16 partners gathered to launch the EuroEXA project, which invests €20 million over three-and-a-half years into exascale-focused research and development. Led by the Horizon 2020 program, EuroEXA picks up the banner of a triad of partner projects — ExaNeSt, EcoScale and ExaNoDe — building on their work... Read more…

By Tiffany Trader

Amazon Debuts New AMD-based GPU Instances for Graphics Acceleration

September 12, 2017

Last week Amazon Web Services (AWS) streaming service, AppStream 2.0, introduced a new GPU instance called Graphics Design intended to accelerate graphics. The Read more…

By John Russell

Cray Moves to Acquire the Seagate ClusterStor Line

July 28, 2017

This week Cray announced that it is picking up Seagate's ClusterStor HPC storage array business for an undisclosed sum. "In short we're effectively transitioning the bulk of the ClusterStor product line to Cray," said CEO Peter Ungaro. Read more…

By Tiffany Trader

GlobalFoundries: 7nm Chips Coming in 2018, EUV in 2019

June 13, 2017

GlobalFoundries has formally announced that its 7nm technology is ready for customer engagement with product tape outs expected for the first half of 2018. The Read more…

By Tiffany Trader

  • arrow
  • Click Here for More Headlines
  • arrow
Share This