FPGA Floating Point Performance

By Dave Strenski

January 12, 2007

HPC programmers are evaluating alternative accelerators to boost the performance of their applications. When looking at FPGAs, they are confronted with an array of new terminologies and concepts that can be difficult to understand at first. This article will walk the HPC programmer through understanding double precision (64-bit) floating-point performance of Xilinx Virtex-4 LX200 and Virtex-5 LX330 FPGAs and compares them to the performance of a 2.5 GHz, dual-core Opteron processor.

The FPGA (Field Programmable Gate Array) can be thought of as a reconfigurable co-processor. The chip consists of an array of Look Up Tables (LUT), Flip-Flops (FF), and Digital Signal Processing (DSP) blocks that all can be reprogrammed on the order of milliseconds. To use FPGAs to accelerate an application, the programmer must first implement a design for the chip. The microprocessor can then call the FPGA loaded with this design to accelerate the application.

The easiest example to envision is an application that uses matrix multiply during its calculation. For the best performance, the programmer would call a highly tuned vendor supplied math library like DGEMM, and pass pointers of the matrices being multiplied. In the ideal FPGA situation, the programmer would call a vendor supplied routine called FPGA_DGEMM and pass the same pointers. In the first case, the DGEMM function would be performed on the microprocessor, reading and writing to the microprocessor’s memory. In the second case, the microprocessor would initiate a Direct Memory Access (DMA) transfer, and move the data to memory associated with the attached FPGA, or directly to the memory located within the FPGA. The results would then be calculated using the logic on the FPGA and returned to the microprocessor’s memory.

Obviously the transfer times between the microprocessor and the FPGA can greatly affect the performance, but for our microprocessor comparison consider the FPGA’s capabilities itself. When a microprocessor’s peak performance is quoted, it is usually calculated by the number of 64-bit floating-point operations it can perform per clock, multiplied by the clock frequency of the chip. In the new world of multi-core processors, this calculation has been expanded by multiplying that result by the number of cores on the chip. So a 2.5 GHz dual-core Opteron, which can perform one add and one multiply per clock, has a peak of (2.5 x 2 x 2) = 10 Gflop/s. An FPGA has neither floating-point adders nor multipliers, only generic logic that can be configured any way the user would like. So to get an equivalent type of 64-bit floating-point performance, we need to figure out how many add and multiply function units will fit on an FPGA and at what clock frequency that design might run.

Doing the Calculations

To start this pencil and paper calculation, we need three reference documents from Xilinx: “Virtex-4 Family Overview” (DS112 v1.6), “Virtex-5 Family Overview LX and LXT Platforms” (DS100 v2.1), and “Floating-Point Operator v3.0” (DS 335), all of which are available at http://www.xilinx.com/. Using the first two documents we can find out how many resources are available on the Virtex-4 LX200 and the Virtex-5 LX330 FPGAs. The last document will tell us how many resources are needed to implement 64-bit multiply, add, divide, square root and other functions, and at what clock frequency those function units will run. Dividing the resources needed per function unit into the resources available on the FPGA will tell how many function units will fit on the chip. Multiplying this by the clock frequency of the function units gives us a peak performance for the FPGA, similar to the peak for the Opteron. Here is a table summarizing the resources available on the LX200, LX330 and other Virtex FPGAs.

——– —– ——  ——  ————- ———–
Virtex-4 Speed Logic   DSP48   Block RAM     Total
         MHz   slices  slices  18-bit/36-bit Kbits (MB)
——– —– ——  ——  ————- ———–
LX160    500   67,584  96      288/0         5,185 (0.6)
LX200    500   89,088  96      336/0         6,048 (0.7)
——– —– ——  ——  ————- ———–
Virtex-5 Speed Logic   DSP48E  Block RAM     Total
         MHz   slices  slices  18-bit/36-bit Kbits (MB)
——– —– ——  ——  ————- ———–
LX220    550   34,560  128     384/192        6,912 (0.8)
LX330    550   51,840  192     576/288       10,368 (1.3)

The Virtex-4 LX200 is listed as having 89,088 logic slices and 96 DSP48 slices, and the Virtex-5 LX330 is listed as having 51,840 logic slices and 192 DSP48E slices. Reading the footnotes in those reference documents shows that a Virtex-4 logic slice contains 2 LUTs and 2 FFs whereas the Virtex-5 logic slice contains 4 LUTs and 4 FFs. Similarly, the Virtex-4 DSP48 slices have 18 x 18 bit hardware multiplier/accumulators whereas the Virtex-5 DSP48E slices have 18 x 25 bit hardware multiplier/accumulators.

Before calculating the number of function units that will fit on an FPGA, we need to subtract some portion of the logic slices for the I/O interface. Remember that an FPGA is generic logic, it does not know how to talk to the microprocessor until you implement and load an interface. For the purposes of these calculations we will assume that we need 13,500 slices on the LX200 and 6,750 slices on the LX330. This leaves the LX200 with 75,588 and the LX330 with 44,790 logic slices available for function units.

The other limiting factor for the number of function units that can be placed on an FPGA is the total amount of on-chip memory available for building 64-bit registers that the function units can read and write. The LX200 has only 18-bit dual-port block RAMs and the LX330 has a combination of 18-bit and 36-bit dual-port block RAMs. Dual-ported means the block RAM can read (or write) two values every clock cycle. Grouping these into 64-bit registers we can make  ((336*2)/4) = 168 registers on the LX200 and ((576*2)/4 + (288*2/2)) = 576 registers on the LX330. Assume we will need at most two registers for each function unit since many of them will be chained or pipelined together with one feeding the next. So the upper bound on function units is (168/2) = 84 for the LX200, and (576/2) = 288 for the LX330.

The “Floating-Point Operators” reference shows that we can build 64-bit multipliers three different ways with the full implementation yielding the highest function unit density, so that is what is used in these calculations. We will first implement as many function units as possible using the DSP slices, then fill up the rest of the FPGA with function units built out of only logic slices. This technique will yield the maximum number of function units, but they will all have to run at the slower all-logic clock frequency. The table below uses the expression (dsp+logic):reg to show how many of the function units were built with a combination of DSP and logic slices, and how many are implemented with logic slices alone. The last number in the expression compares that sum against the upper bound imposed by the number of available 64-bit registers made from on-chip memory. The peak Gops/s value is the minimum of these two numbers multiplied by the minimum of the two clock frequencies shown in the next column.

——–  —— —– —– ——  —————  ——— ——
Function  DSP48  LUTs  FFs   Logic    Virtex-4        Freq      Peak
64-bit    slices             slices   LX200           MHz       Gops/s
                                     (dsp+logic):reg  dsp:logic
——–  —— —– —– ——  —————  ——— ——
Multiply  16     550   774   387     (6+59):84        303:185   12.0
Multiply  0      2311  2457  1229        61:84          185     11.3
Adder     0      1274  1139  637        118:84          284     23.9
——–  —— —– —– ——  —————  ——— ——
Function  DSP48E LUTs  FFs   Logic    Virtex-5        Freq      Peak
64-bit    slices             slices   LX330           MHz       Gops/s
                                     (dsp+logic):reg  dsp:logic
——–  —— —– —– ——  —————  ——— ——
Multiply  12     424   669   168     (16+68):288      369:237   19.9
Multiply  0      2309  2457  615          73:288        237     17.3
Adder     0      804   1060  265         170:288        316     53.7

We need to consider one more adjustment to these results before we can compare them to the dual-core 2.5 GHz Opteron. The results above assume there are only multipliers or adders on the chip, not both. If both multipliers and adders are in the same design, we need to make sure we have enough DSP slices for both, and run the mixed design at the slower of the two clock frequencies. After several iterations, the optimal mixed mult/add implementation for the LX200 is 43 multipliers and 43 adders running at a clock frequency of 185 MHz. This design implements 6 multipliers using the DSP full design, 37 multipliers in all logic and 43 adders in logic. For the LX330 the optimal mixed design is 59 multipliers and 59 adders running at 237 MHz. Again using the DSP full implementation for 16 of the multipliers, 43 multipliers in logic slices, and 59 adders in logic. Multiplying that out, the LX200 gets (43+43)*185 = 15.9 Gflop/s and the LX330 gets (59+59)*237 = 28.0 Gflop/s.

              ———   ———   ———
              Opteron     Virtex-4    Virtex-5
              Dual-core   LX200       LX330
              2.5 GHz     185 MHz     237 MHz
              (Gflop/s)  (Gflop/s)   (Gflop/s)
              ———   ———   ———
Mult/Add         10         15.9        28.0
Mult only        5          12.0        19.9
Add only         5          23.9        55.3

Practical Considerations

In terms of market availability, the LX200 and dual-core Opteron are both readily available and can be purchased today. The LX330 is available now, but in very limited numbers, becoming more available towards the end of 2007, so the analysis should be performed again comparing the LX330 with the quad-core Opterons.

Another consideration is the percent of peak that can be obtained. With more flexibility in the FPGA architecture the programmer should be able to achieve a much high percentage of peak on typical code; whereas the more cores that are placed on a multi-core microprocessor the percent or peak continues to fall. One can actually think of an FPGA as a dense multi-core processor with a very fast crossbar connecting all the function units and registers.

Power consumption is another consideration that is getting more important these days. The dual-core Opteron is rated as requiring 95 watts. The pathological worst case power rating for the LX200 in a current system is 42 watts. A more realistic power rating for this design would be about 25 watts on the LX200, with the LX330 being somewhat higher. So the FPGA designs would use about half to one quarter of the power of the Opteron. Considering the lifetime cost of a system, this reduced power consumption would lower the machines operating costs for both electricity and cooling capacity.

A more aggressive design might also consider using a lower precision calculation. 32-bit function units take about a quarter of the real estate as 64-bit floating-point function units, and the on-chip memory would hold twice as many 32-bit registers as 64-bit registers. Since FPGAs are completely programmable, one could use any bit width or numerical representation needed.

Converting from floating-point to fixed-point or integer would greatly benefit the FPGA’s performance.

This pencil and paper exercise shows that FPGAs can be competitive compared to standard microprocessors at 64-bit floating-point operations. Naturally many details have been left out, such as the speed of the interface between the microprocessor and the FPGA, the amount of additional logic needed to implement a given design, and the larger issue pertaining to the amount of programming effort it takes to implement an efficient design on the FPGA. Nevertheless, this article should provide the motivation for programmers to start leaning how to program these accelerators. Programmers who want to experiment with a HyperTransport attached FPGA architecture may wish to look at the Cray XD1 supercomputer with a Virtex-4 LX160 attached per node (www.cray.com/products/xd1/acceleration.html), or the DRC Development System 2000 from DRC Computer Corporation with one or more attached Virtex-4 LX200 (www.drccomputer.com/pdfs/DRC_DS2000_datasheet.pdf).


Dave Strenski is an applications analyst for Cray Inc., which designs and manufactures high performance computing systems. Prior to Cray, Dave held a variety of technical positions at several computer and research organizations. He holds degrees in Land Surveying, Civil and Mechanical Engineering. His publications include works in the areas of parallel computing, numerical consistency, genomic data searching algorithms, and field programmable gate arrays. He also holds a patent on a meshing algorithm for threaded fasteners. As a hobby, Dave plays with solar power.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Is Data Science the Fourth Pillar of the Scientific Method?

April 18, 2019

Nvidia CEO Jensen Huang revived a decade-old debate last month when he said that modern data science (AI plus HPC) has become the fourth pillar of the scientific method. While some disagree with the notion that statistic Read more…

By Alex Woodie

At ASF 2019: The Virtuous Circle of Big Data, AI and HPC

April 18, 2019

We've entered a new phase in IT -- in the world, really -- where the combination of big data, artificial intelligence, and high performance computing is pushing the bounds of what's possible in business and science, in w Read more…

By Alex Woodie with Doug Black and Tiffany Trader

Google Open Sources TensorFlow Version of MorphNet DL Tool

April 18, 2019

Designing optimum deep neural networks remains a non-trivial exercise. “Given the large search space of possible architectures, designing a network from scratch for your specific application can be prohibitively expens Read more…

By John Russell

HPE Extreme Performance Solutions

HPE and Intel® Omni-Path Architecture: How to Power a Cloud

Learn how HPE and Intel® Omni-Path Architecture provide critical infrastructure for leading Nordic HPC provider’s HPCFLOW cloud service.

powercloud_blog.jpgFor decades, HPE has been at the forefront of high-performance computing, and we’ve powered some of the fastest and most robust supercomputers in the world. Read more…

IBM Accelerated Insights

Bridging HPC and Cloud Native Development with Kubernetes

The HPC community has historically developed its own specialized software stack including schedulers, filesystems, developer tools, container technologies tuned for performance and large-scale on-premises deployments. Read more…

Interview with 2019 Person to Watch Michela Taufer

April 18, 2019

Today, as part of our ongoing HPCwire People to Watch focus series, we are highlighting our interview with 2019 Person to Watch Michela Taufer. Michela -- the General Chair of SC19 -- is an ACM Distinguished Scientist. Read more…

By HPCwire Editorial Team

At ASF 2019: The Virtuous Circle of Big Data, AI and HPC

April 18, 2019

We've entered a new phase in IT -- in the world, really -- where the combination of big data, artificial intelligence, and high performance computing is pushing Read more…

By Alex Woodie with Doug Black and Tiffany Trader

Interview with 2019 Person to Watch Michela Taufer

April 18, 2019

Today, as part of our ongoing HPCwire People to Watch focus series, we are highlighting our interview with 2019 Person to Watch Michela Taufer. Michela -- the Read more…

By HPCwire Editorial Team

Intel Gold U-Series SKUs Reveal Single Socket Intentions

April 18, 2019

Intel plans to jump into the single socket market with a portion of its just announced Cascade Lake microprocessor line according to one media report. This isn Read more…

By John Russell

BSC Researchers Shrink Floating Point Formats to Accelerate Deep Neural Network Training

April 15, 2019

Sometimes calculating solutions as precisely as a computer can wastes more CPU resources than is necessary. A case in point is with deep learning. In early stag Read more…

By Ken Strandberg

Intel Extends FPGA Ecosystem with 10nm Agilex

April 11, 2019

The insatiable appetite for higher throughput and lower latency – particularly where edge analytics and AI, network functions, or for a range of datacenter ac Read more…

By Doug Black

Nvidia Doubles Down on Medical AI

April 9, 2019

Nvidia is collaborating with medical groups to push GPU-powered AI tools into clinical settings, including radiology and drug discovery. The GPU leader said Monday it will collaborate with the American College of Radiology (ACR) to provide clinicians with its Clara AI tool kit. The partnership would allow radiologists to leverage AI techniques for diagnostic imaging using their own clinical data. Read more…

By George Leopold

Digging into MLPerf Benchmark Suite to Inform AI Infrastructure Decisions

April 9, 2019

With machine learning and deep learning storming into the datacenter, the new challenge is optimizing infrastructure choices to support diverse ML and DL workfl Read more…

By John Russell

AI and Enterprise Datacenters Boost HPC Server Revenues Past Expectations – Hyperion

April 9, 2019

Building on the big year of 2017 and spurred in part by the convergence of AI and HPC, global revenue for high performance servers jumped 15.6 percent last year Read more…

By Doug Black

The Case Against ‘The Case Against Quantum Computing’

January 9, 2019

It’s not easy to be a physicist. Richard Feynman (basically the Jimi Hendrix of physicists) once said: “The first principle is that you must not fool yourse Read more…

By Ben Criger

Why Nvidia Bought Mellanox: ‘Future Datacenters Will Be…Like High Performance Computers’

March 14, 2019

“Future datacenters of all kinds will be built like high performance computers,” said Nvidia CEO Jensen Huang during a phone briefing on Monday after Nvidia revealed scooping up the high performance networking company Mellanox for $6.9 billion. Read more…

By Tiffany Trader

ClusterVision in Bankruptcy, Fate Uncertain

February 13, 2019

ClusterVision, European HPC specialists that have built and installed over 20 Top500-ranked systems in their nearly 17-year history, appear to be in the midst o Read more…

By Tiffany Trader

Intel Reportedly in $6B Bid for Mellanox

January 30, 2019

The latest rumors and reports around an acquisition of Mellanox focus on Intel, which has reportedly offered a $6 billion bid for the high performance interconn Read more…

By Doug Black

It’s Official: Aurora on Track to Be First US Exascale Computer in 2021

March 18, 2019

The U.S. Department of Energy along with Intel and Cray confirmed today that an Intel/Cray supercomputer, "Aurora," capable of sustained performance of one exaf Read more…

By Tiffany Trader

Looking for Light Reading? NSF-backed ‘Comic Books’ Tackle Quantum Computing

January 28, 2019

Still baffled by quantum computing? How about turning to comic books (graphic novels for the well-read among you) for some clarity and a little humor on QC. The Read more…

By John Russell

IBM Quantum Update: Q System One Launch, New Collaborators, and QC Center Plans

January 10, 2019

IBM made three significant quantum computing announcements at CES this week. One was introduction of IBM Q System One; it’s really the integration of IBM’s Read more…

By John Russell

Deep500: ETH Researchers Introduce New Deep Learning Benchmark for HPC

February 5, 2019

ETH researchers have developed a new deep learning benchmarking environment – Deep500 – they say is “the first distributed and reproducible benchmarking s Read more…

By John Russell

Leading Solution Providers

SC 18 Virtual Booth Video Tour

Advania @ SC18 AMD @ SC18
ASRock Rack @ SC18
DDN Storage @ SC18
HPE @ SC18
IBM @ SC18
Lenovo @ SC18 Mellanox Technologies @ SC18
One Stop Systems @ SC18
Oracle @ SC18 Panasas @ SC18
Supermicro @ SC18 SUSE @ SC18 TYAN @ SC18
Verne Global @ SC18

IBM Bets $2B Seeking 1000X AI Hardware Performance Boost

February 7, 2019

For now, AI systems are mostly machine learning-based and “narrow” – powerful as they are by today's standards, they're limited to performing a few, narro Read more…

By Doug Black

The Deep500 – Researchers Tackle an HPC Benchmark for Deep Learning

January 7, 2019

How do you know if an HPC system, particularly a larger-scale system, is well-suited for deep learning workloads? Today, that’s not an easy question to answer Read more…

By John Russell

Arm Unveils Neoverse N1 Platform with up to 128-Cores

February 20, 2019

Following on its Neoverse roadmap announcement last October, Arm today revealed its next-gen Neoverse microarchitecture with compute and throughput-optimized si Read more…

By Tiffany Trader

France to Deploy AI-Focused Supercomputer: Jean Zay

January 22, 2019

HPE announced today that it won the contract to build a supercomputer that will drive France’s AI and HPC efforts. The computer will be part of GENCI, the Fre Read more…

By Tiffany Trader

Intel Launches Cascade Lake Xeons with Up to 56 Cores

April 2, 2019

At Intel's Data-Centric Innovation Day in San Francisco (April 2), the company unveiled its second-generation Xeon Scalable (Cascade Lake) family and debuted it Read more…

By Tiffany Trader

Oil and Gas Supercloud Clears Out Remaining Knights Landing Inventory: All 38,000 Wafers

March 13, 2019

The McCloud HPC service being built by Australia’s DownUnder GeoSolutions (DUG) outside Houston is set to become the largest oil and gas cloud in the world th Read more…

By Tiffany Trader

HPC Reflections and (Mostly Hopeful) Predictions

December 19, 2018

So much ‘spaghetti’ gets tossed on walls by the technology community (vendors and researchers) to see what sticks that it is often difficult to peer through Read more…

By John Russell

Intel Extends FPGA Ecosystem with 10nm Agilex

April 11, 2019

The insatiable appetite for higher throughput and lower latency – particularly where edge analytics and AI, network functions, or for a range of datacenter ac Read more…

By Doug Black

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This