FPGA Floating Point Performance

By Dave Strenski

January 12, 2007

HPC programmers are evaluating alternative accelerators to boost the performance of their applications. When looking at FPGAs, they are confronted with an array of new terminologies and concepts that can be difficult to understand at first. This article will walk the HPC programmer through understanding double precision (64-bit) floating-point performance of Xilinx Virtex-4 LX200 and Virtex-5 LX330 FPGAs and compares them to the performance of a 2.5 GHz, dual-core Opteron processor.

The FPGA (Field Programmable Gate Array) can be thought of as a reconfigurable co-processor. The chip consists of an array of Look Up Tables (LUT), Flip-Flops (FF), and Digital Signal Processing (DSP) blocks that all can be reprogrammed on the order of milliseconds. To use FPGAs to accelerate an application, the programmer must first implement a design for the chip. The microprocessor can then call the FPGA loaded with this design to accelerate the application.

The easiest example to envision is an application that uses matrix multiply during its calculation. For the best performance, the programmer would call a highly tuned vendor supplied math library like DGEMM, and pass pointers of the matrices being multiplied. In the ideal FPGA situation, the programmer would call a vendor supplied routine called FPGA_DGEMM and pass the same pointers. In the first case, the DGEMM function would be performed on the microprocessor, reading and writing to the microprocessor’s memory. In the second case, the microprocessor would initiate a Direct Memory Access (DMA) transfer, and move the data to memory associated with the attached FPGA, or directly to the memory located within the FPGA. The results would then be calculated using the logic on the FPGA and returned to the microprocessor’s memory.

Obviously the transfer times between the microprocessor and the FPGA can greatly affect the performance, but for our microprocessor comparison consider the FPGA’s capabilities itself. When a microprocessor’s peak performance is quoted, it is usually calculated by the number of 64-bit floating-point operations it can perform per clock, multiplied by the clock frequency of the chip. In the new world of multi-core processors, this calculation has been expanded by multiplying that result by the number of cores on the chip. So a 2.5 GHz dual-core Opteron, which can perform one add and one multiply per clock, has a peak of (2.5 x 2 x 2) = 10 Gflop/s. An FPGA has neither floating-point adders nor multipliers, only generic logic that can be configured any way the user would like. So to get an equivalent type of 64-bit floating-point performance, we need to figure out how many add and multiply function units will fit on an FPGA and at what clock frequency that design might run.

Doing the Calculations

To start this pencil and paper calculation, we need three reference documents from Xilinx: “Virtex-4 Family Overview” (DS112 v1.6), “Virtex-5 Family Overview LX and LXT Platforms” (DS100 v2.1), and “Floating-Point Operator v3.0” (DS 335), all of which are available at http://www.xilinx.com/. Using the first two documents we can find out how many resources are available on the Virtex-4 LX200 and the Virtex-5 LX330 FPGAs. The last document will tell us how many resources are needed to implement 64-bit multiply, add, divide, square root and other functions, and at what clock frequency those function units will run. Dividing the resources needed per function unit into the resources available on the FPGA will tell how many function units will fit on the chip. Multiplying this by the clock frequency of the function units gives us a peak performance for the FPGA, similar to the peak for the Opteron. Here is a table summarizing the resources available on the LX200, LX330 and other Virtex FPGAs.

——– —– ——  ——  ————- ———–
Virtex-4 Speed Logic   DSP48   Block RAM     Total
         MHz   slices  slices  18-bit/36-bit Kbits (MB)
——– —– ——  ——  ————- ———–
LX160    500   67,584  96      288/0         5,185 (0.6)
LX200    500   89,088  96      336/0         6,048 (0.7)
——– —– ——  ——  ————- ———–
Virtex-5 Speed Logic   DSP48E  Block RAM     Total
         MHz   slices  slices  18-bit/36-bit Kbits (MB)
——– —– ——  ——  ————- ———–
LX220    550   34,560  128     384/192        6,912 (0.8)
LX330    550   51,840  192     576/288       10,368 (1.3)

The Virtex-4 LX200 is listed as having 89,088 logic slices and 96 DSP48 slices, and the Virtex-5 LX330 is listed as having 51,840 logic slices and 192 DSP48E slices. Reading the footnotes in those reference documents shows that a Virtex-4 logic slice contains 2 LUTs and 2 FFs whereas the Virtex-5 logic slice contains 4 LUTs and 4 FFs. Similarly, the Virtex-4 DSP48 slices have 18 x 18 bit hardware multiplier/accumulators whereas the Virtex-5 DSP48E slices have 18 x 25 bit hardware multiplier/accumulators.

Before calculating the number of function units that will fit on an FPGA, we need to subtract some portion of the logic slices for the I/O interface. Remember that an FPGA is generic logic, it does not know how to talk to the microprocessor until you implement and load an interface. For the purposes of these calculations we will assume that we need 13,500 slices on the LX200 and 6,750 slices on the LX330. This leaves the LX200 with 75,588 and the LX330 with 44,790 logic slices available for function units.

The other limiting factor for the number of function units that can be placed on an FPGA is the total amount of on-chip memory available for building 64-bit registers that the function units can read and write. The LX200 has only 18-bit dual-port block RAMs and the LX330 has a combination of 18-bit and 36-bit dual-port block RAMs. Dual-ported means the block RAM can read (or write) two values every clock cycle. Grouping these into 64-bit registers we can make  ((336*2)/4) = 168 registers on the LX200 and ((576*2)/4 + (288*2/2)) = 576 registers on the LX330. Assume we will need at most two registers for each function unit since many of them will be chained or pipelined together with one feeding the next. So the upper bound on function units is (168/2) = 84 for the LX200, and (576/2) = 288 for the LX330.

The “Floating-Point Operators” reference shows that we can build 64-bit multipliers three different ways with the full implementation yielding the highest function unit density, so that is what is used in these calculations. We will first implement as many function units as possible using the DSP slices, then fill up the rest of the FPGA with function units built out of only logic slices. This technique will yield the maximum number of function units, but they will all have to run at the slower all-logic clock frequency. The table below uses the expression (dsp+logic):reg to show how many of the function units were built with a combination of DSP and logic slices, and how many are implemented with logic slices alone. The last number in the expression compares that sum against the upper bound imposed by the number of available 64-bit registers made from on-chip memory. The peak Gops/s value is the minimum of these two numbers multiplied by the minimum of the two clock frequencies shown in the next column.

——–  —— —– —– ——  —————  ——— ——
Function  DSP48  LUTs  FFs   Logic    Virtex-4        Freq      Peak
64-bit    slices             slices   LX200           MHz       Gops/s
                                     (dsp+logic):reg  dsp:logic
——–  —— —– —– ——  —————  ——— ——
Multiply  16     550   774   387     (6+59):84        303:185   12.0
Multiply  0      2311  2457  1229        61:84          185     11.3
Adder     0      1274  1139  637        118:84          284     23.9
——–  —— —– —– ——  —————  ——— ——
Function  DSP48E LUTs  FFs   Logic    Virtex-5        Freq      Peak
64-bit    slices             slices   LX330           MHz       Gops/s
                                     (dsp+logic):reg  dsp:logic
——–  —— —– —– ——  —————  ——— ——
Multiply  12     424   669   168     (16+68):288      369:237   19.9
Multiply  0      2309  2457  615          73:288        237     17.3
Adder     0      804   1060  265         170:288        316     53.7

We need to consider one more adjustment to these results before we can compare them to the dual-core 2.5 GHz Opteron. The results above assume there are only multipliers or adders on the chip, not both. If both multipliers and adders are in the same design, we need to make sure we have enough DSP slices for both, and run the mixed design at the slower of the two clock frequencies. After several iterations, the optimal mixed mult/add implementation for the LX200 is 43 multipliers and 43 adders running at a clock frequency of 185 MHz. This design implements 6 multipliers using the DSP full design, 37 multipliers in all logic and 43 adders in logic. For the LX330 the optimal mixed design is 59 multipliers and 59 adders running at 237 MHz. Again using the DSP full implementation for 16 of the multipliers, 43 multipliers in logic slices, and 59 adders in logic. Multiplying that out, the LX200 gets (43+43)*185 = 15.9 Gflop/s and the LX330 gets (59+59)*237 = 28.0 Gflop/s.

              ———   ———   ———
              Opteron     Virtex-4    Virtex-5
              Dual-core   LX200       LX330
              2.5 GHz     185 MHz     237 MHz
              (Gflop/s)  (Gflop/s)   (Gflop/s)
              ———   ———   ———
Mult/Add         10         15.9        28.0
Mult only        5          12.0        19.9
Add only         5          23.9        55.3

Practical Considerations

In terms of market availability, the LX200 and dual-core Opteron are both readily available and can be purchased today. The LX330 is available now, but in very limited numbers, becoming more available towards the end of 2007, so the analysis should be performed again comparing the LX330 with the quad-core Opterons.

Another consideration is the percent of peak that can be obtained. With more flexibility in the FPGA architecture the programmer should be able to achieve a much high percentage of peak on typical code; whereas the more cores that are placed on a multi-core microprocessor the percent or peak continues to fall. One can actually think of an FPGA as a dense multi-core processor with a very fast crossbar connecting all the function units and registers.

Power consumption is another consideration that is getting more important these days. The dual-core Opteron is rated as requiring 95 watts. The pathological worst case power rating for the LX200 in a current system is 42 watts. A more realistic power rating for this design would be about 25 watts on the LX200, with the LX330 being somewhat higher. So the FPGA designs would use about half to one quarter of the power of the Opteron. Considering the lifetime cost of a system, this reduced power consumption would lower the machines operating costs for both electricity and cooling capacity.

A more aggressive design might also consider using a lower precision calculation. 32-bit function units take about a quarter of the real estate as 64-bit floating-point function units, and the on-chip memory would hold twice as many 32-bit registers as 64-bit registers. Since FPGAs are completely programmable, one could use any bit width or numerical representation needed.

Converting from floating-point to fixed-point or integer would greatly benefit the FPGA’s performance.

This pencil and paper exercise shows that FPGAs can be competitive compared to standard microprocessors at 64-bit floating-point operations. Naturally many details have been left out, such as the speed of the interface between the microprocessor and the FPGA, the amount of additional logic needed to implement a given design, and the larger issue pertaining to the amount of programming effort it takes to implement an efficient design on the FPGA. Nevertheless, this article should provide the motivation for programmers to start leaning how to program these accelerators. Programmers who want to experiment with a HyperTransport attached FPGA architecture may wish to look at the Cray XD1 supercomputer with a Virtex-4 LX160 attached per node (www.cray.com/products/xd1/acceleration.html), or the DRC Development System 2000 from DRC Computer Corporation with one or more attached Virtex-4 LX200 (www.drccomputer.com/pdfs/DRC_DS2000_datasheet.pdf).

—–

Dave Strenski is an applications analyst for Cray Inc., which designs and manufactures high performance computing systems. Prior to Cray, Dave held a variety of technical positions at several computer and research organizations. He holds degrees in Land Surveying, Civil and Mechanical Engineering. His publications include works in the areas of parallel computing, numerical consistency, genomic data searching algorithms, and field programmable gate arrays. He also holds a patent on a meshing algorithm for threaded fasteners. As a hobby, Dave plays with solar power.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

How the United States Invests in Supercomputing

November 14, 2018

The CORAL supercomputers Summit and Sierra are now the world's fastest computers and are already contributing to science with early applications. Ahead of SC18, Maciej Chojnowski with ICM at the University of Warsaw discussed the details of the CORAL project with Dr. Dimitri Kusnezov from the U.S. Department of Energy. Read more…

By Maciej Chojnowski

At SC18: Humanitarianism Amid Boom Times for HPC

November 14, 2018

At SC18 in Dallas, the feeling on the ground is one of forward-looking buoyancy. Like boom times that cycle through the Texas oil fields, the HPC industry is enjoying a prosperity seen only every few decades, one driven Read more…

By Doug Black

Nvidia’s Jensen Huang Delivers Vision for the New HPC

November 14, 2018

For nearly two hours on Monday at SC18, Jensen Huang, CEO of Nvidia, presented his expansive view of the future of HPC (and computing in general) as only he can do. Animated. Backstopped by a stream of data charts, produ Read more…

By John Russell

HPE Extreme Performance Solutions

AI Can Be Scary. But Choosing the Wrong Partners Can Be Mortifying!

As you continue to dive deeper into AI, you will discover it is more than just deep learning. AI is an extremely complex set of machine learning, deep learning, reinforcement, and analytics algorithms with varying compute, storage, memory, and communications needs. Read more…

IBM Accelerated Insights

From Deep Blue to Summit – 30 Years of Supercomputing Innovation

This week, in honor of the 30th anniversary of the SC conference, we are highlighting some of the most significant IBM contributions to supercomputing over the past 30 years. Read more…

New Panasas High Performance Storage Straddles Commercial-Traditional HPC

November 13, 2018

High performance storage vendor Panasas has launched a new version of its ActiveStor product line this morning featuring what the company said is the industry’s first plug-and-play, portable parallel file system that delivers up to 75 Gb/s per rack on industry standard hardware combined with “enterprise-grade reliability and manageability.” Read more…

By Doug Black

How the United States Invests in Supercomputing

November 14, 2018

The CORAL supercomputers Summit and Sierra are now the world's fastest computers and are already contributing to science with early applications. Ahead of SC18, Maciej Chojnowski with ICM at the University of Warsaw discussed the details of the CORAL project with Dr. Dimitri Kusnezov from the U.S. Department of Energy. Read more…

By Maciej Chojnowski

At SC18: Humanitarianism Amid Boom Times for HPC

November 14, 2018

At SC18 in Dallas, the feeling on the ground is one of forward-looking buoyancy. Like boom times that cycle through the Texas oil fields, the HPC industry is en Read more…

By Doug Black

Nvidia’s Jensen Huang Delivers Vision for the New HPC

November 14, 2018

For nearly two hours on Monday at SC18, Jensen Huang, CEO of Nvidia, presented his expansive view of the future of HPC (and computing in general) as only he can Read more…

By John Russell

New Panasas High Performance Storage Straddles Commercial-Traditional HPC

November 13, 2018

High performance storage vendor Panasas has launched a new version of its ActiveStor product line this morning featuring what the company said is the industry’s first plug-and-play, portable parallel file system that delivers up to 75 Gb/s per rack on industry standard hardware combined with “enterprise-grade reliability and manageability.” Read more…

By Doug Black

SC18 Student Cluster Competition – Revealing the Field

November 13, 2018

It’s November again and we’re almost ready for the kick-off of one of the greatest computer sports events in the world – the SC Student Cluster Competitio Read more…

By Dan Olds

US Leads Supercomputing with #1, #2 Systems & Petascale Arm

November 12, 2018

The 31st Supercomputing Conference (SC) - commemorating 30 years since the first Supercomputing in 1988 - kicked off in Dallas yesterday, taking over the Kay Ba Read more…

By Tiffany Trader

OpenACC Talks Up Summit and Community Momentum at SC18

November 12, 2018

OpenACC – the directives-based parallel programing model for optimizing applications on heterogeneous architectures – is showcasing user traction and HPC im Read more…

By John Russell

How ASCI Revolutionized the World of High-Performance Computing and Advanced Modeling and Simulation

November 9, 2018

The 1993 Supercomputing Conference was held in Portland, Oregon. That conference and it’s show floor provided a good snapshot of the uncertainty that U.S. supercomputing was facing in the early 1990s. Many of the companies exhibiting that year would soon be gone, either bankrupt or acquired by somebody else. Read more…

By Alex R. Larzelere

Cray Unveils Shasta, Lands NERSC-9 Contract

October 30, 2018

Cray revealed today the details of its next-gen supercomputing architecture, Shasta, selected to be the next flagship system at NERSC. We've known of the code-name "Shasta" since the Argonne slice of the CORAL project was announced in 2015 and although the details of that plan have changed considerably, Cray didn't slow down its timeline for Shasta. Read more…

By Tiffany Trader

TACC Wins Next NSF-funded Major Supercomputer

July 30, 2018

The Texas Advanced Computing Center (TACC) has won the next NSF-funded big supercomputer beating out rivals including the National Center for Supercomputing Ap Read more…

By John Russell

IBM at Hot Chips: What’s Next for Power

August 23, 2018

With processor, memory and networking technologies all racing to fill in for an ailing Moore’s law, the era of the heterogeneous datacenter is well underway, Read more…

By Tiffany Trader

Requiem for a Phi: Knights Landing Discontinued

July 25, 2018

On Monday, Intel made public its end of life strategy for the Knights Landing "KNL" Phi product set. The announcement makes official what has already been wide Read more…

By Tiffany Trader

House Passes $1.275B National Quantum Initiative

September 17, 2018

Last Thursday the U.S. House of Representatives passed the National Quantum Initiative Act (NQIA) intended to accelerate quantum computing research and developm Read more…

By John Russell

CERN Project Sees Orders-of-Magnitude Speedup with AI Approach

August 14, 2018

An award-winning effort at CERN has demonstrated potential to significantly change how the physics based modeling and simulation communities view machine learni Read more…

By Rob Farber

Summit Supercomputer is Already Making its Mark on Science

September 20, 2018

Summit, now the fastest supercomputer in the world, is quickly making its mark in science – five of the six finalists just announced for the prestigious 2018 Read more…

By John Russell

New Deep Learning Algorithm Solves Rubik’s Cube

July 25, 2018

Solving (and attempting to solve) Rubik’s Cube has delighted millions of puzzle lovers since 1974 when the cube was invented by Hungarian sculptor and archite Read more…

By John Russell

Leading Solution Providers

US Leads Supercomputing with #1, #2 Systems & Petascale Arm

November 12, 2018

The 31st Supercomputing Conference (SC) - commemorating 30 years since the first Supercomputing in 1988 - kicked off in Dallas yesterday, taking over the Kay Ba Read more…

By Tiffany Trader

TACC’s ‘Frontera’ Supercomputer Expands Horizon for Extreme-Scale Science

August 29, 2018

The National Science Foundation and the Texas Advanced Computing Center announced today that a new system, called Frontera, will overtake Stampede 2 as the fast Read more…

By Tiffany Trader

HPE No. 1, IBM Surges, in ‘Bucking Bronco’ High Performance Server Market

September 27, 2018

Riding healthy U.S. and global economies, strong demand for AI-capable hardware and other tailwind trends, the high performance computing server market jumped 28 percent in the second quarter 2018 to $3.7 billion, up from $2.9 billion for the same period last year, according to industry analyst firm Hyperion Research. Read more…

By Doug Black

Intel Announces Cooper Lake, Advances AI Strategy

August 9, 2018

Intel's chief datacenter exec Navin Shenoy kicked off the company's Data-Centric Innovation Summit Wednesday, the day-long program devoted to Intel's datacenter Read more…

By Tiffany Trader

Germany Celebrates Launch of Two Fastest Supercomputers

September 26, 2018

The new high-performance computer SuperMUC-NG at the Leibniz Supercomputing Center (LRZ) in Garching is the fastest computer in Germany and one of the fastest i Read more…

By Tiffany Trader

Houston to Field Massive, ‘Geophysically Configured’ Cloud Supercomputer

October 11, 2018

Based on some news stories out today, one might get the impression that the next system to crack number one on the Top500 would be an industrial oil and gas mon Read more…

By Tiffany Trader

MLPerf – Will New Machine Learning Benchmark Help Propel AI Forward?

May 2, 2018

Let the AI benchmarking wars begin. Today, a diverse group from academia and industry – Google, Baidu, Intel, AMD, Harvard, and Stanford among them – releas Read more…

By John Russell

Google Releases Machine Learning “What-If” Analysis Tool

September 12, 2018

Training machine learning models has long been time-consuming process. Yesterday, Google released a “What-If Tool” for probing how data point changes affect a model’s prediction. The new tool is being launched as a new feature of the open source TensorBoard web application... Read more…

By John Russell

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This