Must See TV: IBM Watson Heads for Jeopardy Showdown

By Michael Feldman

February 9, 2011

Next week the IBM supercomputer known as “Watson” will take on two of the most accomplished Jeopardy players of all time, Ken Jennings and Brad Rutter, in a three-game match starting on February 14. If Watson manages to best the humans, it will represent the most important advance in machine intelligence since IBM’s “Deep Blue” beat chess grandmaster Garry Kasparov in 1997. But this time around, the company also plans to make a business case for the technology. Trivial pursuit this is not.

And impressive technology it is. On the hardware side, Watson is comprised of 90 Power 750 servers, 16 TB of memory and 4 TB of disk storage, all housed in a relatively compact ten racks. The 750 is IBM’s elite Power7-based server targeted for high-end enterprise analytics. (The Power 755 is geared toward high performance technical computing and differs only marginally in CPU speed, memory capacity, and storage options.) Although the enterprise version can be ordered with 1 to 4 sockets of 6-core or 8-core Power7 chips, Watson is maxed out with the 4-socket, 8-core configuration using the top bin 3.55 GHz processors.

The 360 Power7 chips that make up Watson’s brain represent IBM’s best and brightest processor technology. Each Power7 is capable of over 500 GB/second of aggregate bandwidth, making it particularly adept at manipulating data at high speeds. FLOPS-wise, a 3.55 GHz Power7 delivers 218 Linpack gigaflops. For comparison, the POWER2 SC processor, which was the chip that powered cyber-chessmaster Deep Blue, managed a paltry 0.48 gigaflops, with the whole machine delivering a mere 11.4 Linpack gigaflops.

But FLOPS are not the real story here. Watson’s question-answering software presumably makes little use of floating-point number crunching. To deal with the game scenario, the system had to be endowed with a rather advanced version of natural language processing. But according to David Ferrucci, principal investigator for the project, it goes far beyond language smarts. The software system, called DeepQA, also incorporates machine learning, knowledge representation, and deep analytics.

Even so, the whole application rests on first understanding the Jeopardy clues, which, because they employ colloquialisms and often obscure references, can be challenging even for humans. That’s why this is such a good test case for natural language processing. Ferrucci says the ability to understand language is destined to become a very important aspect of computers. “It has to be that way,” he says. “We just cant imagine a future without it.”

But it’s the analysis component that we associate with real “intelligence.” The approach here reflects the open domain nature of the problem. According to Ferrucci, it wouldn’t have made sense to simply construct a database corresponding to possible Jeopardy clues. Such a model would have supported only a small fraction of the possible topics available to Jeopardy. Rather their approach was to use “as is” information sources — encyclopedias, dictionaries, thesauri, plays, books, etc. — and make the correlations dynamically.

The trick of course is to do all the processing in real-time. Contestants, at least the successful ones, need to provide an answer in just a few seconds. When the software was run on a lone 2.6 GHz CPU, it took around 2 hours to process a typical Jeopardy clue — not a very practical implementation. But when they parallelized the algorithms across the 2,880-core Watson, they were able to cut the processing time from a couple of hours to between 2 and 6 seconds.

Even at that, Watson doesn’t just spit out the answers. It forms hypotheses based on the evidence it finds and scores them at various confidence levels. Watson is programmed not to buzz in until it reaches a confidence of at least 50 percent, although this parameter can be self-adjusted depending on the game situation.

To accomplish all this, DeepQA employs an ensemble of algorithms — about a million lines of code — to gather and score the evidence. These include temporal reasoning algorithms to correlate times with events, statistical paraphrasing algorithms to evaluate semantic context, and geospatial reasoning to correlate locations.

It can also dynamically form associations, both in training and at game time, to connect disparate ideas. For example it can learn that inventors can patent information or that officials can submit resignations. Watson also shifts the weight it assigns to different algorithms based on which ones are delivering the more accurate correlations. This aspect of machine learning allows Watson to get “smarter” the more it plays the game.

The DeepQA programmers have also been refining the algorithms themselves over the past several years. In 2007, Watson could only answer a small fraction of Jeopardy clues with reasonable confidence and even at that, was only correct 47 percent of the time. When forced to answer the majority of the clues, like a grand champion would, it could only answer 15 percent correctly. By IBM’s own admission, Watson was playing “terrible.” The highest performing Jeopardy grand champions, like Jennings and Rutter, typically buzz in on 70 to 80 percent of the entries and give the correct answer 85 to 95 percent of time.

By 2010 Watson started playing at that level. Ferrucci says that while the system can’t buzz in on every question, it can now answer the vast majority of them in competitive time. “We can compete with grand champions in terms of precision, in terms of confidence, and in terms of speed,” he says.

In dozens of practice rounds against former Jeopardy champs, the computer was beating the humans with a 65 percent win rate. Watson also prevailed in a 15-question round against Jennings and Rutter in early January of this year. See the performance below.

None of this is a guarantee that Watson will prevail next week. But even if the machine just makes a decent showing, IBM will have pulled off quite possibly the best product placement in television history. Open domain question answering is not only one of the Holy Grails of artificial intelligence but has enormous potential for commercial applications. In areas as disparate as healthcare, tech support, business intelligence, security and finance, this type of platform could change those businesses irrevocably. John Kelly, senior vice president and director of IBM Research, boasts, “We’re going to revolutionize industries at a level that has never been done before.”

In the case of healthcare, it’s not a huge leap to imagine “expert” question answering systems helping doctors with medical diagnosis. A differential diagnosis is not much different from what Watson does when it analyzes a Jeopardy clue. Before it replaces Dr. House, though, the machine will have to prove itself in the game show arena.

If Jennings and Rutter defeat the supercomputer this time around, IBM will almost certainly ask for a rematch, as it did when Deep Blue initially lost its first chess match with Kasparov in 1996. The engineers will keep stroking the code and retraining the computer until Watson is truly unbeatable. Eventually the machine will prevail.

—–

For a broader discussion on this topic between the author and InterSect360 Research CEO Addison Snell, download this week’s HPCwire Soundbite podcast.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

UCSD, AIST Forge Tighter Alliance with AI-Focused MOU

January 18, 2018

The rich history of collaboration between UC San Diego and AIST in Japan is getting richer. The organizations entered into a five-year memorandum of understanding on January 10. The MOU represents the continuation of a 1 Read more…

By Tiffany Trader

New Blueprint for Converging HPC, Big Data

January 18, 2018

After five annual workshops on Big Data and Extreme-Scale Computing (BDEC), a group of international HPC heavyweights including Jack Dongarra (University of Tennessee), Satoshi Matsuoka (Tokyo Institute of Technology), Read more…

By John Russell

Researchers Measure Impact of ‘Meltdown’ and ‘Spectre’ Patches on HPC Workloads

January 17, 2018

Computer scientists from the Center for Computational Research, State University of New York (SUNY), University at Buffalo have examined the effect of Meltdown and Spectre security updates on the performance of popular H Read more…

By Tiffany Trader

HPE Extreme Performance Solutions

HPE and NREL Take Steps to Create a Sustainable, Energy-Efficient Data Center with an H2 Fuel Cell

As enterprises attempt to manage rising volumes of data, unplanned data center outages are becoming more common and more expensive. As the cost of downtime rises, enterprises lose out on productivity and valuable competitive advantage without access to their critical data. Read more…

Fostering Lustre Advancement Through Development and Contributions

January 17, 2018

Six months after organizational changes at Intel's High Performance Data (HPDD) division, most in the Lustre community have shed any initial apprehension around the potential changes that could affect or disrupt Lustre Read more…

By Carlos Aoki Thomaz

UCSD, AIST Forge Tighter Alliance with AI-Focused MOU

January 18, 2018

The rich history of collaboration between UC San Diego and AIST in Japan is getting richer. The organizations entered into a five-year memorandum of understandi Read more…

By Tiffany Trader

New Blueprint for Converging HPC, Big Data

January 18, 2018

After five annual workshops on Big Data and Extreme-Scale Computing (BDEC), a group of international HPC heavyweights including Jack Dongarra (University of Te Read more…

By John Russell

Researchers Measure Impact of ‘Meltdown’ and ‘Spectre’ Patches on HPC Workloads

January 17, 2018

Computer scientists from the Center for Computational Research, State University of New York (SUNY), University at Buffalo have examined the effect of Meltdown Read more…

By Tiffany Trader

Fostering Lustre Advancement Through Development and Contributions

January 17, 2018

Six months after organizational changes at Intel's High Performance Data (HPDD) division, most in the Lustre community have shed any initial apprehension aroun Read more…

By Carlos Aoki Thomaz

When the Chips Are Down

January 11, 2018

In the last article, "The High Stakes Semiconductor Game that Drives HPC Diversity," I alluded to the challenges facing the semiconductor industry and how that may impact the evolution of HPC systems over the next few years. I thought I’d lift the covers a little and look at some of the commercial challenges that impact the component technology we use in HPC. Read more…

By Dairsie Latimer

How Meltdown and Spectre Patches Will Affect HPC Workloads

January 10, 2018

There have been claims that the fixes for the Meltdown and Spectre security vulnerabilities, named the KPTI (aka KAISER) patches, are going to affect applicatio Read more…

By Rosemary Francis

Momentum Builds for US Exascale

January 9, 2018

2018 looks to be a great year for the U.S. exascale program. The last several months of 2017 revealed a number of important developments that help put the U.S. Read more…

By Alex R. Larzelere

ANL’s Rick Stevens on CANDLE, ARM, Quantum, and More

January 8, 2018

Late last year HPCwire caught up with Rick Stevens, associate laboratory director for computing, environment and life Sciences at Argonne National Laboratory, f Read more…

By John Russell

Inventor Claims to Have Solved Floating Point Error Problem

January 17, 2018

"The decades-old floating point error problem has been solved," proclaims a press release from inventor Alan Jorgensen. The computer scientist has filed for and Read more…

By Tiffany Trader

US Coalesces Plans for First Exascale Supercomputer: Aurora in 2021

September 27, 2017

At the Advanced Scientific Computing Advisory Committee (ASCAC) meeting, in Arlington, Va., yesterday (Sept. 26), it was revealed that the "Aurora" supercompute Read more…

By Tiffany Trader

Japan Unveils Quantum Neural Network

November 22, 2017

The U.S. and China are leading the race toward productive quantum computing, but it's early enough that ultimate leadership is still something of an open questi Read more…

By Tiffany Trader

AMD Showcases Growing Portfolio of EPYC and Radeon-based Systems at SC17

November 13, 2017

AMD’s charge back into HPC and the datacenter is on full display at SC17. Having launched the EPYC processor line in June along with its MI25 GPU the focus he Read more…

By John Russell

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

IBM Begins Power9 Rollout with Backing from DOE, Google

December 6, 2017

After over a year of buildup, IBM is unveiling its first Power9 system based on the same architecture as the Department of Energy CORAL supercomputers, Summit a Read more…

By Tiffany Trader

Fast Forward: Five HPC Predictions for 2018

December 21, 2017

What’s on your list of high (and low) lights for 2017? Volta 100’s arrival on the heels of the P100? Appearance, albeit late in the year, of IBM’s Power9? Read more…

By John Russell

Chip Flaws ‘Meltdown’ and ‘Spectre’ Loom Large

January 4, 2018

The HPC and wider tech community have been abuzz this week over the discovery of critical design flaws that impact virtually all contemporary microprocessors. T Read more…

By Tiffany Trader

Leading Solution Providers

Perspective: What Really Happened at SC17?

November 22, 2017

SC is over. Now comes the myriad of follow-ups. Inboxes are filled with templated emails from vendors and other exhibitors hoping to win a place in the post-SC thinking of booth visitors. Attendees of tutorials, workshops and other technical sessions will be inundated with requests for feedback. Read more…

By Andrew Jones

Tensors Come of Age: Why the AI Revolution Will Help HPC

November 13, 2017

Thirty years ago, parallel computing was coming of age. A bitter battle began between stalwart vector computing supporters and advocates of various approaches to parallel computing. IBM skeptic Alan Karp, reacting to announcements of nCUBE’s 1024-microprocessor system and Thinking Machines’ 65,536-element array, made a public $100 wager that no one could get a parallel speedup of over 200 on real HPC workloads. Read more…

By John Gustafson & Lenore Mullin

Delays, Smoke, Records & Markets – A Candid Conversation with Cray CEO Peter Ungaro

October 5, 2017

Earlier this month, Tom Tabor, publisher of HPCwire and I had a very personal conversation with Cray CEO Peter Ungaro. Cray has been on something of a Cinderell Read more…

By Tiffany Trader & Tom Tabor

Flipping the Flops and Reading the Top500 Tea Leaves

November 13, 2017

The 50th edition of the Top500 list, the biannual publication of the world’s fastest supercomputers based on public Linpack benchmarking results, was released Read more…

By Tiffany Trader

How Meltdown and Spectre Patches Will Affect HPC Workloads

January 10, 2018

There have been claims that the fixes for the Meltdown and Spectre security vulnerabilities, named the KPTI (aka KAISER) patches, are going to affect applicatio Read more…

By Rosemary Francis

GlobalFoundries, Ayar Labs Team Up to Commercialize Optical I/O

December 4, 2017

GlobalFoundries (GF) and Ayar Labs, a startup focused on using light, instead of electricity, to transfer data between chips, today announced they've entered in Read more…

By Tiffany Trader

Researchers Measure Impact of ‘Meltdown’ and ‘Spectre’ Patches on HPC Workloads

January 17, 2018

Computer scientists from the Center for Computational Research, State University of New York (SUNY), University at Buffalo have examined the effect of Meltdown Read more…

By Tiffany Trader

HPC Chips – A Veritable Smorgasbord?

October 10, 2017

For the first time since AMD's ill-fated launch of Bulldozer the answer to the question, 'Which CPU will be in my next HPC system?' doesn't have to be 'Whichever variety of Intel Xeon E5 they are selling when we procure'. Read more…

By Dairsie Latimer

  • arrow
  • Click Here for More Headlines
  • arrow
Share This