HPC programmers are evaluating alternative accelerators to boost the performance of their applications. When looking at FPGAs, they are confronted with an array of new terminologies and concepts that can be difficult to understand at first. This article will walk the HPC programmer through understanding double precision (64-bit) floating-point performance of Xilinx Virtex-4 LX200 and Virtex-5 LX330 FPGAs and compares them to the performance of a 2.5 GHz, dual-core Opteron processor.

The FPGA (Field Programmable Gate Array) can be thought of as a reconfigurable co-processor. The chip consists of an array of Look Up Tables (LUT), Flip-Flops (FF), and Digital Signal Processing (DSP) blocks that all can be reprogrammed on the order of milliseconds. To use FPGAs to accelerate an application, the programmer must first implement a design for the chip. The microprocessor can then call the FPGA loaded with this design to accelerate the application.

The easiest example to envision is an application that uses matrix multiply during its calculation. For the best performance, the programmer would call a highly tuned vendor supplied math library like DGEMM, and pass pointers of the matrices being multiplied. In the ideal FPGA situation, the programmer would call a vendor supplied routine called FPGA_DGEMM and pass the same pointers. In the first case, the DGEMM function would be performed on the microprocessor, reading and writing to the microprocessor’s memory. In the second case, the microprocessor would initiate a Direct Memory Access (DMA) transfer, and move the data to memory associated with the attached FPGA, or directly to the memory located within the FPGA. The results would then be calculated using the logic on the FPGA and returned to the microprocessor’s memory.

Obviously the transfer times between the microprocessor and the FPGA can greatly affect the performance, but for our microprocessor comparison consider the FPGA’s capabilities itself. When a microprocessor’s peak performance is quoted, it is usually calculated by the number of 64-bit floating-point operations it can perform per clock, multiplied by the clock frequency of the chip. In the new world of multi-core processors, this calculation has been expanded by multiplying that result by the number of cores on the chip. So a 2.5 GHz dual-core Opteron, which can perform one add and one multiply per clock, has a peak of (2.5 x 2 x 2) = 10 Gflop/s. An FPGA has neither floating-point adders nor multipliers, only generic logic that can be configured any way the user would like. So to get an equivalent type of 64-bit floating-point performance, we need to figure out how many add and multiply function units will fit on an FPGA and at what clock frequency that design might run.

Doing the Calculations

To start this pencil and paper calculation, we need three reference documents from Xilinx: “Virtex-4 Family Overview” (DS112 v1.6), “Virtex-5 Family Overview LX and LXT Platforms” (DS100 v2.1), and “Floating-Point Operator v3.0” (DS 335), all of which are available at http://www.xilinx.com/. Using the first two documents we can find out how many resources are available on the Virtex-4 LX200 and the Virtex-5 LX330 FPGAs. The last document will tell us how many resources are needed to implement 64-bit multiply, add, divide, square root and other functions, and at what clock frequency those function units will run. Dividing the resources needed per function unit into the resources available on the FPGA will tell how many function units will fit on the chip. Multiplying this by the clock frequency of the function units gives us a peak performance for the FPGA, similar to the peak for the Opteron. Here is a table summarizing the resources available on the LX200, LX330 and other Virtex FPGAs.

——– —– ——  ——  ————- ———–
Virtex-4 Speed Logic   DSP48   Block RAM     Total
         MHz   slices  slices  18-bit/36-bit Kbits (MB)
——– —– ——  ——  ————- ———–
LX160    500   67,584  96      288/0         5,185 (0.6)
LX200    500   89,088  96      336/0         6,048 (0.7)
——– —– ——  ——  ————- ———–
Virtex-5 Speed Logic   DSP48E  Block RAM     Total
         MHz   slices  slices  18-bit/36-bit Kbits (MB)
——– —– ——  ——  ————- ———–
LX220    550   34,560  128     384/192        6,912 (0.8)
LX330    550   51,840  192     576/288       10,368 (1.3)

The Virtex-4 LX200 is listed as having 89,088 logic slices and 96 DSP48 slices, and the Virtex-5 LX330 is listed as having 51,840 logic slices and 192 DSP48E slices. Reading the footnotes in those reference documents shows that a Virtex-4 logic slice contains 2 LUTs and 2 FFs whereas the Virtex-5 logic slice contains 4 LUTs and 4 FFs. Similarly, the Virtex-4 DSP48 slices have 18 x 18 bit hardware multiplier/accumulators whereas the Virtex-5 DSP48E slices have 18 x 25 bit hardware multiplier/accumulators.

Before calculating the number of function units that will fit on an FPGA, we need to subtract some portion of the logic slices for the I/O interface. Remember that an FPGA is generic logic, it does not know how to talk to the microprocessor until you implement and load an interface. For the purposes of these calculations we will assume that we need 13,500 slices on the LX200 and 6,750 slices on the LX330. This leaves the LX200 with 75,588 and the LX330 with 44,790 logic slices available for function units.

The other limiting factor for the number of function units that can be placed on an FPGA is the total amount of on-chip memory available for building 64-bit registers that the function units can read and write. The LX200 has only 18-bit dual-port block RAMs and the LX330 has a combination of 18-bit and 36-bit dual-port block RAMs. Dual-ported means the block RAM can read (or write) two values every clock cycle. Grouping these into 64-bit registers we can make  ((336*2)/4) = 168 registers on the LX200 and ((576*2)/4 + (288*2/2)) = 576 registers on the LX330. Assume we will need at most two registers for each function unit since many of them will be chained or pipelined together with one feeding the next. So the upper bound on function units is (168/2) = 84 for the LX200, and (576/2) = 288 for the LX330.

The “Floating-Point Operators” reference shows that we can build 64-bit multipliers three different ways with the full implementation yielding the highest function unit density, so that is what is used in these calculations. We will first implement as many function units as possible using the DSP slices, then fill up the rest of the FPGA with function units built out of only logic slices. This technique will yield the maximum number of function units, but they will all have to run at the slower all-logic clock frequency. The table below uses the expression (dsp+logic):reg to show how many of the function units were built with a combination of DSP and logic slices, and how many are implemented with logic slices alone. The last number in the expression compares that sum against the upper bound imposed by the number of available 64-bit registers made from on-chip memory. The peak Gops/s value is the minimum of these two numbers multiplied by the minimum of the two clock frequencies shown in the next column.

——–  —— —– —– ——  —————  ——— ——
Function  DSP48  LUTs  FFs   Logic    Virtex-4        Freq      Peak
64-bit    slices             slices   LX200           MHz       Gops/s
                                     (dsp+logic):reg  dsp:logic
——–  —— —– —– ——  —————  ——— ——
Multiply  16     550   774   387     (6+59):84        303:185   12.0
Multiply  0      2311  2457  1229        61:84          185     11.3
Adder     0      1274  1139  637        118:84          284     23.9
——–  —— —– —– ——  —————  ——— ——
Function  DSP48E LUTs  FFs   Logic    Virtex-5        Freq      Peak
64-bit    slices             slices   LX330           MHz       Gops/s
                                     (dsp+logic):reg  dsp:logic
——–  —— —– —– ——  —————  ——— ——
Multiply  12     424   669   168     (16+68):288      369:237   19.9
Multiply  0      2309  2457  615          73:288        237     17.3
Adder     0      804   1060  265         170:288        316     53.7

We need to consider one more adjustment to these results before we can compare them to the dual-core 2.5 GHz Opteron. The results above assume there are only multipliers or adders on the chip, not both. If both multipliers and adders are in the same design, we need to make sure we have enough DSP slices for both, and run the mixed design at the slower of the two clock frequencies. After several iterations, the optimal mixed mult/add implementation for the LX200 is 43 multipliers and 43 adders running at a clock frequency of 185 MHz. This design implements 6 multipliers using the DSP full design, 37 multipliers in all logic and 43 adders in logic. For the LX330 the optimal mixed design is 59 multipliers and 59 adders running at 237 MHz. Again using the DSP full implementation for 16 of the multipliers, 43 multipliers in logic slices, and 59 adders in logic. Multiplying that out, the LX200 gets (43+43)*185 = 15.9 Gflop/s and the LX330 gets (59+59)*237 = 28.0 Gflop/s.

              ———   ———   ———
              Opteron     Virtex-4    Virtex-5
              Dual-core   LX200       LX330
              2.5 GHz     185 MHz     237 MHz
              (Gflop/s)  (Gflop/s)   (Gflop/s)
              ———   ———   ———
Mult/Add         10         15.9        28.0
Mult only        5          12.0        19.9
Add only         5          23.9        55.3

Practical Considerations

In terms of market availability, the LX200 and dual-core Opteron are both readily available and can be purchased today. The LX330 is available now, but in very limited numbers, becoming more available towards the end of 2007, so the analysis should be performed again comparing the LX330 with the quad-core Opterons.

Another consideration is the percent of peak that can be obtained. With more flexibility in the FPGA architecture the programmer should be able to achieve a much high percentage of peak on typical code; whereas the more cores that are placed on a multi-core microprocessor the percent or peak continues to fall. One can actually think of an FPGA as a dense multi-core processor with a very fast crossbar connecting all the function units and registers.

Power consumption is another consideration that is getting more important these days. The dual-core Opteron is rated as requiring 95 watts. The pathological worst case power rating for the LX200 in a current system is 42 watts. A more realistic power rating for this design would be about 25 watts on the LX200, with the LX330 being somewhat higher. So the FPGA designs would use about half to one quarter of the power of the Opteron. Considering the lifetime cost of a system, this reduced power consumption would lower the machines operating costs for both electricity and cooling capacity.

A more aggressive design might also consider using a lower precision calculation. 32-bit function units take about a quarter of the real estate as 64-bit floating-point function units, and the on-chip memory would hold twice as many 32-bit registers as 64-bit registers. Since FPGAs are completely programmable, one could use any bit width or numerical representation needed.

Converting from floating-point to fixed-point or integer would greatly benefit the FPGA’s performance.

This pencil and paper exercise shows that FPGAs can be competitive compared to standard microprocessors at 64-bit floating-point operations. Naturally many details have been left out, such as the speed of the interface between the microprocessor and the FPGA, the amount of additional logic needed to implement a given design, and the larger issue pertaining to the amount of programming effort it takes to implement an efficient design on the FPGA. Nevertheless, this article should provide the motivation for programmers to start leaning how to program these accelerators. Programmers who want to experiment with a HyperTransport attached FPGA architecture may wish to look at the Cray XD1 supercomputer with a Virtex-4 LX160 attached per node (www.cray.com/products/xd1/acceleration.html), or the DRC Development System 2000 from DRC Computer Corporation with one or more attached Virtex-4 LX200 (www.drccomputer.com/pdfs/DRC_DS2000_datasheet.pdf).

—–

Dave Strenski is an applications analyst for Cray Inc., which designs and manufactures high performance computing systems. Prior to Cray, Dave held a variety of technical positions at several computer and research organizations. He holds degrees in Land Surveying, Civil and Mechanical Engineering. His publications include works in the areas of parallel computing, numerical consistency, genomic data searching algorithms, and field programmable gate arrays. He also holds a patent on a meshing algorithm for threaded fasteners. As a hobby, Dave plays with solar power.