FPGA Floating Point Performance

By Dave Strenski

January 12, 2007

HPC programmers are evaluating alternative accelerators to boost the performance of their applications. When looking at FPGAs, they are confronted with an array of new terminologies and concepts that can be difficult to understand at first. This article will walk the HPC programmer through understanding double precision (64-bit) floating-point performance of Xilinx Virtex-4 LX200 and Virtex-5 LX330 FPGAs and compares them to the performance of a 2.5 GHz, dual-core Opteron processor.

The FPGA (Field Programmable Gate Array) can be thought of as a reconfigurable co-processor. The chip consists of an array of Look Up Tables (LUT), Flip-Flops (FF), and Digital Signal Processing (DSP) blocks that all can be reprogrammed on the order of milliseconds. To use FPGAs to accelerate an application, the programmer must first implement a design for the chip. The microprocessor can then call the FPGA loaded with this design to accelerate the application.

The easiest example to envision is an application that uses matrix multiply during its calculation. For the best performance, the programmer would call a highly tuned vendor supplied math library like DGEMM, and pass pointers of the matrices being multiplied. In the ideal FPGA situation, the programmer would call a vendor supplied routine called FPGA_DGEMM and pass the same pointers. In the first case, the DGEMM function would be performed on the microprocessor, reading and writing to the microprocessor’s memory. In the second case, the microprocessor would initiate a Direct Memory Access (DMA) transfer, and move the data to memory associated with the attached FPGA, or directly to the memory located within the FPGA. The results would then be calculated using the logic on the FPGA and returned to the microprocessor’s memory.

Obviously the transfer times between the microprocessor and the FPGA can greatly affect the performance, but for our microprocessor comparison consider the FPGA’s capabilities itself. When a microprocessor’s peak performance is quoted, it is usually calculated by the number of 64-bit floating-point operations it can perform per clock, multiplied by the clock frequency of the chip. In the new world of multi-core processors, this calculation has been expanded by multiplying that result by the number of cores on the chip. So a 2.5 GHz dual-core Opteron, which can perform one add and one multiply per clock, has a peak of (2.5 x 2 x 2) = 10 Gflop/s. An FPGA has neither floating-point adders nor multipliers, only generic logic that can be configured any way the user would like. So to get an equivalent type of 64-bit floating-point performance, we need to figure out how many add and multiply function units will fit on an FPGA and at what clock frequency that design might run.

Doing the Calculations

To start this pencil and paper calculation, we need three reference documents from Xilinx: “Virtex-4 Family Overview” (DS112 v1.6), “Virtex-5 Family Overview LX and LXT Platforms” (DS100 v2.1), and “Floating-Point Operator v3.0” (DS 335), all of which are available at http://www.xilinx.com/. Using the first two documents we can find out how many resources are available on the Virtex-4 LX200 and the Virtex-5 LX330 FPGAs. The last document will tell us how many resources are needed to implement 64-bit multiply, add, divide, square root and other functions, and at what clock frequency those function units will run. Dividing the resources needed per function unit into the resources available on the FPGA will tell how many function units will fit on the chip. Multiplying this by the clock frequency of the function units gives us a peak performance for the FPGA, similar to the peak for the Opteron. Here is a table summarizing the resources available on the LX200, LX330 and other Virtex FPGAs.

——– —– ——  ——  ————- ———–
Virtex-4 Speed Logic   DSP48   Block RAM     Total
         MHz   slices  slices  18-bit/36-bit Kbits (MB)
——– —– ——  ——  ————- ———–
LX160    500   67,584  96      288/0         5,185 (0.6)
LX200    500   89,088  96      336/0         6,048 (0.7)
——– —– ——  ——  ————- ———–
Virtex-5 Speed Logic   DSP48E  Block RAM     Total
         MHz   slices  slices  18-bit/36-bit Kbits (MB)
——– —– ——  ——  ————- ———–
LX220    550   34,560  128     384/192        6,912 (0.8)
LX330    550   51,840  192     576/288       10,368 (1.3)

The Virtex-4 LX200 is listed as having 89,088 logic slices and 96 DSP48 slices, and the Virtex-5 LX330 is listed as having 51,840 logic slices and 192 DSP48E slices. Reading the footnotes in those reference documents shows that a Virtex-4 logic slice contains 2 LUTs and 2 FFs whereas the Virtex-5 logic slice contains 4 LUTs and 4 FFs. Similarly, the Virtex-4 DSP48 slices have 18 x 18 bit hardware multiplier/accumulators whereas the Virtex-5 DSP48E slices have 18 x 25 bit hardware multiplier/accumulators.

Before calculating the number of function units that will fit on an FPGA, we need to subtract some portion of the logic slices for the I/O interface. Remember that an FPGA is generic logic, it does not know how to talk to the microprocessor until you implement and load an interface. For the purposes of these calculations we will assume that we need 13,500 slices on the LX200 and 6,750 slices on the LX330. This leaves the LX200 with 75,588 and the LX330 with 44,790 logic slices available for function units.

The other limiting factor for the number of function units that can be placed on an FPGA is the total amount of on-chip memory available for building 64-bit registers that the function units can read and write. The LX200 has only 18-bit dual-port block RAMs and the LX330 has a combination of 18-bit and 36-bit dual-port block RAMs. Dual-ported means the block RAM can read (or write) two values every clock cycle. Grouping these into 64-bit registers we can make  ((336*2)/4) = 168 registers on the LX200 and ((576*2)/4 + (288*2/2)) = 576 registers on the LX330. Assume we will need at most two registers for each function unit since many of them will be chained or pipelined together with one feeding the next. So the upper bound on function units is (168/2) = 84 for the LX200, and (576/2) = 288 for the LX330.

The “Floating-Point Operators” reference shows that we can build 64-bit multipliers three different ways with the full implementation yielding the highest function unit density, so that is what is used in these calculations. We will first implement as many function units as possible using the DSP slices, then fill up the rest of the FPGA with function units built out of only logic slices. This technique will yield the maximum number of function units, but they will all have to run at the slower all-logic clock frequency. The table below uses the expression (dsp+logic):reg to show how many of the function units were built with a combination of DSP and logic slices, and how many are implemented with logic slices alone. The last number in the expression compares that sum against the upper bound imposed by the number of available 64-bit registers made from on-chip memory. The peak Gops/s value is the minimum of these two numbers multiplied by the minimum of the two clock frequencies shown in the next column.

——–  —— —– —– ——  —————  ——— ——
Function  DSP48  LUTs  FFs   Logic    Virtex-4        Freq      Peak
64-bit    slices             slices   LX200           MHz       Gops/s
                                     (dsp+logic):reg  dsp:logic
——–  —— —– —– ——  —————  ——— ——
Multiply  16     550   774   387     (6+59):84        303:185   12.0
Multiply  0      2311  2457  1229        61:84          185     11.3
Adder     0      1274  1139  637        118:84          284     23.9
——–  —— —– —– ——  —————  ——— ——
Function  DSP48E LUTs  FFs   Logic    Virtex-5        Freq      Peak
64-bit    slices             slices   LX330           MHz       Gops/s
                                     (dsp+logic):reg  dsp:logic
——–  —— —– —– ——  —————  ——— ——
Multiply  12     424   669   168     (16+68):288      369:237   19.9
Multiply  0      2309  2457  615          73:288        237     17.3
Adder     0      804   1060  265         170:288        316     53.7

We need to consider one more adjustment to these results before we can compare them to the dual-core 2.5 GHz Opteron. The results above assume there are only multipliers or adders on the chip, not both. If both multipliers and adders are in the same design, we need to make sure we have enough DSP slices for both, and run the mixed design at the slower of the two clock frequencies. After several iterations, the optimal mixed mult/add implementation for the LX200 is 43 multipliers and 43 adders running at a clock frequency of 185 MHz. This design implements 6 multipliers using the DSP full design, 37 multipliers in all logic and 43 adders in logic. For the LX330 the optimal mixed design is 59 multipliers and 59 adders running at 237 MHz. Again using the DSP full implementation for 16 of the multipliers, 43 multipliers in logic slices, and 59 adders in logic. Multiplying that out, the LX200 gets (43+43)*185 = 15.9 Gflop/s and the LX330 gets (59+59)*237 = 28.0 Gflop/s.

              ———   ———   ———
              Opteron     Virtex-4    Virtex-5
              Dual-core   LX200       LX330
              2.5 GHz     185 MHz     237 MHz
              (Gflop/s)  (Gflop/s)   (Gflop/s)
              ———   ———   ———
Mult/Add         10         15.9        28.0
Mult only        5          12.0        19.9
Add only         5          23.9        55.3

Practical Considerations

In terms of market availability, the LX200 and dual-core Opteron are both readily available and can be purchased today. The LX330 is available now, but in very limited numbers, becoming more available towards the end of 2007, so the analysis should be performed again comparing the LX330 with the quad-core Opterons.

Another consideration is the percent of peak that can be obtained. With more flexibility in the FPGA architecture the programmer should be able to achieve a much high percentage of peak on typical code; whereas the more cores that are placed on a multi-core microprocessor the percent or peak continues to fall. One can actually think of an FPGA as a dense multi-core processor with a very fast crossbar connecting all the function units and registers.

Power consumption is another consideration that is getting more important these days. The dual-core Opteron is rated as requiring 95 watts. The pathological worst case power rating for the LX200 in a current system is 42 watts. A more realistic power rating for this design would be about 25 watts on the LX200, with the LX330 being somewhat higher. So the FPGA designs would use about half to one quarter of the power of the Opteron. Considering the lifetime cost of a system, this reduced power consumption would lower the machines operating costs for both electricity and cooling capacity.

A more aggressive design might also consider using a lower precision calculation. 32-bit function units take about a quarter of the real estate as 64-bit floating-point function units, and the on-chip memory would hold twice as many 32-bit registers as 64-bit registers. Since FPGAs are completely programmable, one could use any bit width or numerical representation needed.

Converting from floating-point to fixed-point or integer would greatly benefit the FPGA’s performance.

This pencil and paper exercise shows that FPGAs can be competitive compared to standard microprocessors at 64-bit floating-point operations. Naturally many details have been left out, such as the speed of the interface between the microprocessor and the FPGA, the amount of additional logic needed to implement a given design, and the larger issue pertaining to the amount of programming effort it takes to implement an efficient design on the FPGA. Nevertheless, this article should provide the motivation for programmers to start leaning how to program these accelerators. Programmers who want to experiment with a HyperTransport attached FPGA architecture may wish to look at the Cray XD1 supercomputer with a Virtex-4 LX160 attached per node (www.cray.com/products/xd1/acceleration.html), or the DRC Development System 2000 from DRC Computer Corporation with one or more attached Virtex-4 LX200 (www.drccomputer.com/pdfs/DRC_DS2000_datasheet.pdf).

—–

Dave Strenski is an applications analyst for Cray Inc., which designs and manufactures high performance computing systems. Prior to Cray, Dave held a variety of technical positions at several computer and research organizations. He holds degrees in Land Surveying, Civil and Mechanical Engineering. His publications include works in the areas of parallel computing, numerical consistency, genomic data searching algorithms, and field programmable gate arrays. He also holds a patent on a meshing algorithm for threaded fasteners. As a hobby, Dave plays with solar power.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Pfizer HPC Engineer Aims to Automate Software Stack Testing

January 17, 2019

Seeking to reign in the tediousness of manual software testing, Pfizer HPC Engineer Shahzeb Siddiqui is developing an open source software tool called buildtest, aimed at automating software stack testing by providing the community with a central repository of tests for common HPC apps and the ability to automate execution of testing. Read more…

By Tiffany Trader

Senegal Prepares to Take Delivery of Atos Supercomputer

January 16, 2019

In just a few months time, Senegal will be operating the second largest HPC system in sub-Saharan Africa. The Minister of Higher Education, Research and Innovation Mary Teuw Niane made the announcement on Monday (Jan. 14 Read more…

By Tiffany Trader

Google Cloud Platform Extends GPU Instance Options

January 16, 2019

If it's Nvidia GPUs you're after to power your AI/HPC/visualization workload, Google Cloud has them, now claiming "broadest GPU availability." Each of the three big public cloud vendors has by turn touted the latest and Read more…

By Tiffany Trader

HPE Extreme Performance Solutions

HPE Systems With Intel Omni-Path: Architected for Value and Accessible High-Performance Computing

Today’s high-performance computing (HPC) and artificial intelligence (AI) users value high performing clusters. And the higher the performance that their system can deliver, the better. Read more…

IBM Accelerated Insights

Resource Management in the Age of Artificial Intelligence

New challenges demand fresh approaches

Fueled by GPUs, big data, and rapid advances in software, the AI revolution is upon us. Read more…

STAC Floats ML Benchmark for Financial Services Workloads

January 16, 2019

STAC (Securities Technology Analysis Center) recently released an ‘exploratory’ benchmark for machine learning which it hopes will evolve into a firm benchmark or suite of benchmarking tools to compare the performanc Read more…

By John Russell

Google Cloud Platform Extends GPU Instance Options

January 16, 2019

If it's Nvidia GPUs you're after to power your AI/HPC/visualization workload, Google Cloud has them, now claiming "broadest GPU availability." Each of the three Read more…

By Tiffany Trader

STAC Floats ML Benchmark for Financial Services Workloads

January 16, 2019

STAC (Securities Technology Analysis Center) recently released an ‘exploratory’ benchmark for machine learning which it hopes will evolve into a firm benchm Read more…

By John Russell

A Big Data Journey While Seeking to Catalog our Universe

January 16, 2019

It turns out, astronomers have lots of photos of the sky but seek knowledge about what the photos mean. Sound familiar? Big data problems are often characterize Read more…

By James Reinders

Intel Bets Big on 2-Track Quantum Strategy

January 15, 2019

Quantum computing has lived so long in the future it’s taken on a futuristic life of its own, with a Gartner-style hype cycle that includes triggers of innovation, inflated expectations and – though a useful quantum system is still years away – anticipatory troughs of disillusionment. Read more…

By Doug Black

IBM Quantum Update: Q System One Launch, New Collaborators, and QC Center Plans

January 10, 2019

IBM made three significant quantum computing announcements at CES this week. One was introduction of IBM Q System One; it’s really the integration of IBM’s Read more…

By John Russell

IBM’s New Global Weather Forecasting System Runs on GPUs

January 9, 2019

Anyone who has checked a forecast to decide whether or not to pack an umbrella knows that weather prediction can be a mercurial endeavor. It is a Herculean task: the constant modeling of incredibly complex systems to a high degree of accuracy at a local level within very short spans of time. Read more…

By Oliver Peckham

The Case Against ‘The Case Against Quantum Computing’

January 9, 2019

It’s not easy to be a physicist. Richard Feynman (basically the Jimi Hendrix of physicists) once said: “The first principle is that you must not fool yourse Read more…

By Ben Criger

The Deep500 – Researchers Tackle an HPC Benchmark for Deep Learning

January 7, 2019

How do you know if an HPC system, particularly a larger-scale system, is well-suited for deep learning workloads? Today, that’s not an easy question to answer Read more…

By John Russell

Quantum Computing Will Never Work

November 27, 2018

Amid the gush of money and enthusiastic predictions being thrown at quantum computing comes a proposed cold shower in the form of an essay by physicist Mikhail Read more…

By John Russell

Cray Unveils Shasta, Lands NERSC-9 Contract

October 30, 2018

Cray revealed today the details of its next-gen supercomputing architecture, Shasta, selected to be the next flagship system at NERSC. We've known of the code-name "Shasta" since the Argonne slice of the CORAL project was announced in 2015 and although the details of that plan have changed considerably, Cray didn't slow down its timeline for Shasta. Read more…

By Tiffany Trader

AMD Sets Up for Epyc Epoch

November 16, 2018

It’s been a good two weeks, AMD’s Gary Silcott and Andy Parma told me on the last day of SC18 in Dallas at the restaurant where we met to discuss their show news and recent successes. Heck, it’s been a good year. Read more…

By Tiffany Trader

The Case Against ‘The Case Against Quantum Computing’

January 9, 2019

It’s not easy to be a physicist. Richard Feynman (basically the Jimi Hendrix of physicists) once said: “The first principle is that you must not fool yourse Read more…

By Ben Criger

US Leads Supercomputing with #1, #2 Systems & Petascale Arm

November 12, 2018

The 31st Supercomputing Conference (SC) - commemorating 30 years since the first Supercomputing in 1988 - kicked off in Dallas yesterday, taking over the Kay Ba Read more…

By Tiffany Trader

Contract Signed for New Finnish Supercomputer

December 13, 2018

After the official contract signing yesterday, configuration details were made public for the new BullSequana system that the Finnish IT Center for Science (CSC Read more…

By Tiffany Trader

Nvidia’s Jensen Huang Delivers Vision for the New HPC

November 14, 2018

For nearly two hours on Monday at SC18, Jensen Huang, CEO of Nvidia, presented his expansive view of the future of HPC (and computing in general) as only he can do. Animated. Backstopped by a stream of data charts, product photos, and even a beautiful image of supernovae... Read more…

By John Russell

HPE No. 1, IBM Surges, in ‘Bucking Bronco’ High Performance Server Market

September 27, 2018

Riding healthy U.S. and global economies, strong demand for AI-capable hardware and other tailwind trends, the high performance computing server market jumped 28 percent in the second quarter 2018 to $3.7 billion, up from $2.9 billion for the same period last year, according to industry analyst firm Hyperion Research. Read more…

By Doug Black

Leading Solution Providers

SC 18 Virtual Booth Video Tour

Advania @ SC18 AMD @ SC18
ASRock Rack @ SC18
DDN Storage @ SC18
HPE @ SC18
IBM @ SC18
Lenovo @ SC18 Mellanox Technologies @ SC18
NVIDIA @ SC18
One Stop Systems @ SC18
Oracle @ SC18 Panasas @ SC18
Supermicro @ SC18 SUSE @ SC18 TYAN @ SC18
Verne Global @ SC18

Summit Supercomputer is Already Making its Mark on Science

September 20, 2018

Summit, now the fastest supercomputer in the world, is quickly making its mark in science – five of the six finalists just announced for the prestigious 2018 Read more…

By John Russell

HPC Reflections and (Mostly Hopeful) Predictions

December 19, 2018

So much ‘spaghetti’ gets tossed on walls by the technology community (vendors and researchers) to see what sticks that it is often difficult to peer through Read more…

By John Russell

Intel Confirms 48-Core Cascade Lake-AP for 2019

November 4, 2018

As part of the run-up to SC18, taking place in Dallas next week (Nov. 11-16), Intel is doling out info on its next-gen Cascade Lake family of Xeon processors, specifically the “Advanced Processor” version (Cascade Lake-AP), architected for high-performance computing, artificial intelligence and infrastructure-as-a-service workloads. Read more…

By Tiffany Trader

Germany Celebrates Launch of Two Fastest Supercomputers

September 26, 2018

The new high-performance computer SuperMUC-NG at the Leibniz Supercomputing Center (LRZ) in Garching is the fastest computer in Germany and one of the fastest i Read more…

By Tiffany Trader

Houston to Field Massive, ‘Geophysically Configured’ Cloud Supercomputer

October 11, 2018

Based on some news stories out today, one might get the impression that the next system to crack number one on the Top500 would be an industrial oil and gas mon Read more…

By Tiffany Trader

Microsoft to Buy Mellanox?

December 20, 2018

Networking equipment powerhouse Mellanox could be an acquisition target by Microsoft, according to a published report in an Israeli financial publication. Microsoft has reportedly gone so far as to engage Goldman Sachs to handle negotiations with Mellanox. Read more…

By Doug Black

The Deep500 – Researchers Tackle an HPC Benchmark for Deep Learning

January 7, 2019

How do you know if an HPC system, particularly a larger-scale system, is well-suited for deep learning workloads? Today, that’s not an easy question to answer Read more…

By John Russell

House Passes $1.275B National Quantum Initiative

September 17, 2018

Last Thursday the U.S. House of Representatives passed the National Quantum Initiative Act (NQIA) intended to accelerate quantum computing research and developm Read more…

By John Russell

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This