Visit additional Tabor Communication Publications
July 13, 2011
One of the largest financial institutions in the world is using FPGA-based supercomputing for analyzing some of its largest and most complex credit derivative portfolios. JP Morgan, along with Maxeler Technologies, has built and deployed a state-of-the art HPC system capable of number-crunching the company's collateralized debt obligation (CDO) portfolio in near real-time.
CDOs are instruments in which the credit assets are divided into different bundles or tranches, according to their relative risk of default. During the credit crisis of 2007-2008, CDO valuation tanked as the value of the underlying assets, mostly mortgages, fell off a cliff. Part of the problem was that many of the computer models didn't assess the risk parameters of the various mortgages correctly. The less obvious aspect was that these instruments were so complex that it was difficult for the models using traditional computer technology to analyze these portfolios effectively.
With the credit crisis in full swing in 2008, Stephen Weston joined JP Morgan's London office, heading up a team devoted to making the company's financial algorithms and models run more effectively. In what started out as a blue-sky technology project almost three years ago, Weston's group has implemented a production-ready solution that speeds up the company's CDO risk models by a factor of more than 130. "This, to us, is a step change," said Weston, talking about the project during a presentation at Stanford University in May.
Execution time was the critical factor. Prior to the FPGA solution, JP Morgan's main risk model for analyzing their CDO portfolio took 8 to 12 hours to complete -- an overnight run requiring a cluster of thousands of x86 cores. If the model failed to execute correctly, there was no time to resubmit the application for that day. Worse yet, the credit risks and valuation are in constant flux. That snapshot of the previous day may no longer be useful. "It was a bit like driving your car on the freeway at 90 miles per hour by looking in the rear view mirror," said Weston. "It could be fun, but there's a high probability it could be a destructive activity."
With the speedup, the same risk model took four minutes, with the FPGA processing eating up just 12 seconds of that. It's not just that they could run the models faster though. The better performance allowed them to run multiple trading/risk scenarios throughout the day. So traders can evaluate more scenarios using different combinations of default criteria. In a nutshell, the time compression allowed JP Morgan to get a better handle on the risk profile of their CDO assets.
In general, porting legacy applications like these financial risk models to FPGAs is no small task. Programming them with low-level VHDL, the traditional programming language of FPGAs, is time-consuming, tedious, and generally unsuited for application developers. Weston knew that it would be a tough sell to convince the quants and management types at the company that this could be a viable solution for a production environment.
In fact, initially JP Morgan looked at GPUs for acceleration. They ported one of their models to the graphics architecture and were able to get a 14- to 15-fold performance boost. But they thought they could do even better with FPGAs. The problem was that it was going to take about 6 months for an initial port. That's when they went to Maxeler and initiated a proof-of-concept engagement with them.
Maxeler is a London-based technology vendor specializing in FPGA acceleration for high performance computing applications. Unlike most FPGA vendors though, Maxeler offers a vertically integrated solution: hardware, high-level compilers (Java), runtime support, development tools, and FPGA porting expertise. As such, the company is able to meet application programmers on their own turf and help them navigate the eccentricities of FPGA software development. At least, that's Maxeler's pitch.
With JP Morgan, it all seemed to work. With Maxeler's help, Weston's group was able to port the time-critical, compute-intensive pieces of their C++ risk model (the Copula and Convoluter kernels, in particular) to the FPGA platform in about 3 months. The end result was something Weston felt was sustainable for their production environment.
Part of the effort to port to risk model involved redesigning the original C++ code, which was chock full of templates and objects. Those languages structures are great for application abstraction, said Weston, but they effectively kill parallelism, and thus performance. So the first phase of the code migrations was to remove all uses of classes, templates, and other C++ abstractions that got in the way of parallelization.
With the lower level code exposed, it became much simpler to tease out the parallelism that could be exploited by the FPGAs. In this case, the flattened C++ source was ported to Java, which the Maxeler compiler is able to convert to VHDL.
Hardware-wise, the final target system is a 40-node hybrid HPC cluster from Maxeler. Each node houses eight Xeon cores hooked up to two Xilinx Virtex-5 (SX240T) FPGAs via PCIe links. Memory is split between the CPU (24GB) and the two FPGAs (12 GB each). Two terabytes of hard disk storage are hung off an Ethernet connection.
The advantage of the FPGA is that it is built for parallelism and allow the application to be intimately mapped onto the hardware. The devices are especially suited to applications that can exploit fine-grained parallelism and very deep pipelines. Unlike linear computations on fast CPUs (~2.6 GHz), parallel computation on slower FPGAs (~200 MHz) can yield many more calculations per watt. As Weston put it, "We went from computing in time to computing in space."
Right now the company is in the final stages of the project to integrate it with the rest of their production infrastructure. They are also looking to move the technology into other areas of their business like FX trading and high frequency trading, and in some cases are seeing even better performance improvements. Their Monte Carlo model, for example, was able to realize a 260- to 280-fold speedup using FPGA acceleration.
Apparently JP Morgan feels bullish enough about the technology to warrant a direct investment. In March, they acquired a 20 percent stake in Maxeler for an undisclosed amount. Although the investment is probably just a rounding error for the financial giant, it signals company's interest in making sure Maxeler's intellectual assets are intact.
There is certainly plenty of room to expand the Maxeler footprint at JP Morgan. To run all aspects of their financial business, the company currently has 14 thousand applications running on 50 thousand servers spread across more than 42 datacenters worldwide. Only a fraction of those applications will be amenable to acceleration, but each one has the potential to raise the company's bottom line.
"If we can compress the space, the time and the energy required to do these calculations, then it has hard business value for us," noted Weston. "It gives us, ultimately, a competitive edge."
May 23, 2013 |
The study of climate change is one of those scientific problems where it is almost essential to model the entire Earth to attain accurate results and make worthwhile predictions. In an attempt to make climate science more accessible to smaller research facilities, NASA introduced what they call ‘Climate in a Box,’ a system they note acts as a desktop supercomputer.
May 22, 2013 |
At some point in the not-too-distant future, building powerful, miniature computing systems will be considered a hobby for high schoolers, just as robotics or even Lego-building are today. That could be made possible through recent advancements made with the Raspberry Pi computers.
May 16, 2013 |
When it comes to cloud, long distances mean unacceptably high latencies. Researchers from the University of Bonn in Germany examined those latency issues of doing CFD modeling in the cloud by utilizing a common CFD and its utilization in HPC instance types including both CPU and GPU cores of Amazon EC2.
May 15, 2013 |
Supercomputers at the Department of Energy’s National Energy Research Scientific Computing Center (NERSC) have worked on important computational problems such as collapse of the atomic state, the optimization of chemical catalysts, and now modeling popping bubbles.
05/10/2013 | Cleversafe, Cray, DDN, NetApp, & Panasas | From Wall Street to Hollywood, drug discovery to homeland security, companies and organizations of all sizes and stripes are coming face to face with the challenges – and opportunities – afforded by Big Data. Before anyone can utilize these extraordinary data repositories, however, they must first harness and manage their data stores, and do so utilizing technologies that underscore affordability, security, and scalability.
04/15/2013 | Bull | “50% of HPC users say their largest jobs scale to 120 cores or less.” How about yours? Are your codes ready to take advantage of today’s and tomorrow’s ultra-parallel HPC systems? Download this White Paper by Analysts Intersect360 Research to see what Bull and Intel’s Center for Excellence in Parallel Programming can do for your codes.
In this demonstration of SGI DMF ZeroWatt disk solution, Dr. Eng Lim Goh, SGI CTO, discusses a function of SGI DMF software to reduce costs and power consumption in an exascale (Big Data) storage datacenter.
The Cray CS300-AC cluster supercomputer offers energy efficient, air-cooled design based on modular, industry-standard platforms featuring the latest processor and network technologies and a wide range of datacenter cooling requirements.