In the wake of Watson’s dominant performance on Jeopardy last month, IBM has taken the technology on the road to showcase it to anyone who’ll listen. On Monday, Watson — or rather, its keepers — headlined the opening session of the High Performance Computing Linux Financial Markets Conference in New York City. There was even a Watson demo at the conference, attracting crowds in IBM’s exhibitor booth.

If presenting a research project to a Wall Street crowd seems unusual, keep in mind that IBM does not intend to keep Watson in the lab forever. Commercialization of the technology is clearly in the company’s plan.

Following its Jeopardy win, the supercomputer’s next task will be to apply its analytic smarts to healthcare applications. IBM and Nuance Communications in collaboration with Columbia University Medical Center and the University of Maryland School of Medicine are looking for ways to use Watson to help doctors with patient diagnoses in a real-world medical setting.

Since I was in town for the HPC Wall Street conference, I got the opportunity to chat with two of the IBM’ers that spoke on the topic — Jean Staten Healy, director of the Cross-IBM Linux group, and Edward Epstein, manager of Unstructured Information at IBM Research — and ask them about the how the technology could be applied to financial services.

First though, I wanted to find out more about the Watson design and how it evolved over the three-year project. Since Epstein was one of the primary developers of the Watson software, he was able to give me a rundown on the supercomputer’s path to Jeopardy stardom.

According to him, Watson had a rather unimpressive start. In its first incarnation, it took two hours to spit out the answer to a question (or rather the question to the answer), which obviously wouldn’t do for a prime-time game show. The IBM engineers soon realized they had to do a serious redesign of the 750,000 lines of code if they were ever to be competitive on Jeopardy.

First off, all the data (dictionaries, encyclopedias, historical texts, etc.) had to be placed into RAM. Waiting precious milliseconds for disk reads is a performance killer, so everything got stuffed into memory for lightening-fast access.

But most of the initial effort to boost execution speed involved scaling out the software such that the hundreds of analytics algorithms and natural language processing (NLP) code could be run in parallel. The algorithms were parallelized across the analytics framework — in this case Apache UIMA (Unstructured Information Management Architecture), an open source information management environment that was at the heart of Watson’s software. Also, the search algorithms that looked up data references were distributed across the available cores of the Watson cluster. When the initial scale-out effort was done, there were about 200 Java processes as well as an additional 200 C++ processes running in parallel on Watson’s hardware.

According to Epstein that effort reduced the average answer time to just over 14 seconds. Since, in Jeopardy, you need have the answer in just a few seconds — in most cases just a fraction of a second right after the clue is read — they still needed another four-fold performance boost. Most of that was achieved by precomputing the deep NLP analysis of the pre-canned text and by hammering on every computation outlier. With that accomplished, the average answer time was trimmed to 3.6 seconds — on par with a human Jeopardy champ.

The software development work and the initial sparring matches for Watson were done on an IBM x86 blade cluster, outfitted with Xeon Nehalem CPUs. That system had the ability to store intermediate results, so that during test runs, the software team could execute a partial scenario, and return to it later to run a new calculation based on those intermediates. Also during development, it was important to run thousands of questions simultaneously, rather than a single question for fast real-time execution. So the system was scaled differently than the final Power 750 cluster that was used in the Jeopardy match.
 
The x86 development cluster had much less powerful processors, less memory, and most importantly less memory bandwidth compared to the Power 750 machine. Fundamentally, Watson is a big data app that feeds large amounts of information through a complex framework of analytics software. The fact the this needs to be done interactively puts particular constraints on performance.

According to Epstein, they needed the performance of the Power 750 to be competitive in Jeopardy. Fortunately, porting the software from the x86 blades system to the Power cluster was fairly straightforward, given that the software stack is all based on portable technology (Java, C++, Linux, and UIMA).

A single 750 node has four 8-core 3.5 GHz Power7 CPUs, and the entire system consisted of ninety such nodes, encapsulating 2,880 CPUs and 16 TB of RAM. The peak performance of the Jeopardy system is estimated to be about 80 teraflops.

The Watson software team added a number of Power7 optimizations to bump up the performance a bit more. Most of that involved using NUMA control to pin software processes to specific resources in the machine. “If you’re really trying to get that last edge in performance, then you do things like that,” said Epstein.

The ninety cluster Power7 was probably a bit of overkill for the Jeopardy match. Epstein estimates that CPU utilization was in the neighborhood of 30 percent during the clue processing (So theoretically, Watson could have been playing two additional Jeopardy matches simultaneously.) In any case, it was Epstein’s task to win the match at any cost, CPU utilization be damned. “I had the luxury of having enough hardware to do this job for Jeopardy,” he explained.

So what is Watson doing on Wall Street? IBM might be looking to attract some willing partners for a Watson-style financial analytics project analogous to the aforementioned healthcare research initiative. Big Blue is obviously proud of the technology and believes the system can be applied to all sorts of deep analytics work.

Epstein himself is currently working in the group involved in the healthcare project, but there are a number of individuals who are exploring “other opportunities.” One group is specifically focused on the financial application space.

IBM’s Healy believes a major focus for the technology in the financial arena will involve risk management. The idea is to provide results that will enable investors and money managers to make very fast decisions based on market conditions. Healy said it would not just involve spitting out a single answer like in Jeopardy, but also provide metrics of confidence about that answer, as well as some sort of evidence trail of its analysis.

Healy also suggested the possibility that Watson could serve as a resource for individuals making personal investments decisions. One could envision a sort of “Ask Watson” application that could serve thousands or even millions of investors simultaneously (assuming the machine was scaled appropriately). For this type of work, Watson might have to solicit information from the user based on the specific investment question. In that sense, Watson couldn’t just be an answer machine; it would need some rudimentary conversational skills as well. While Healy concedes the technology is still in the research stage, from her perspective, it has many applications going forward.

I suspect Watson will show up at a lot of conferences this year as IBM tests the waters for the technology. Deep analytics is broadly applicable to many domains and this has all the makings of a high-margin business for IBM. They just need to gather some proof points.