At ISC – Goh on Go: Humans Can’t Scale, the Data-Centric Learning Machine Can

By Doug Black

June 22, 2017

I’ve seen the future this week at ISC, it’s on display in prototype or Powerpoint form, and it’s going to dumbfound you. The future is an AI neural network designed to emulate and compete with the human brain. In this game, the brain doesn’t stand a chance.

Scoff at such talk as farfetched or far off in a hazy utopic/dystopic future. Roll your eyes and say we’ve heard the hype before (some of us remember a supercomputer company 25 years ago with the inflated name of “Thinking Machines,” long defunct). But it’s neither futuristic nor hype, it’s happening now, the technology pieces are taking shape, and the implications for business, for the work world and for our everyday lives – for good or ill – are as staggering as they are real.

Aside: It’s somewhat unsettling that conference attendees here in Frankfurt don’t seem particularly interested in those implications. For the moment, ISC is at the gathering point of computer scientists bringing about massive technological change, but nearly all the talk here is about the “how” of AI systems, not the “what then?” But there’s one anecdote making the rounds that has raised eyebrows: when Google engineers were asked to how its AlphaGo machine the winning move against the world champion of Go (the world’s most complex board game), the answer was: “We don’t know” (more on this below).

Quite consciously, engineers are architecting HPC systems along the lines of our brain. The new architecture is an emerging style of computing called “data intensive” or “data centric.” It replaces processing with memory (i.e., data) at the center of the computing universe. Combined with advanced algorithms, new memory and processor technologies are coming on line to make the new architecture a practical reality. Once the pieces are in place, the next step will be to scale these systems beyond all measure of human brain capacity.

What does data centric computing mean? How does it work? Why does it represent a major shift in advanced scale computing?

Let’s start answering those questions by first looking at how data centric systems are measured. The benchmark for new AI systems isn’t how fast they solve linear algebra problems (i.e., Linpack). That’s how processor-centric systems have been measured for decades, and considering the capabilities of data-centric systems under development, that benchmark seems wholly inadequate.

Rather than throughput, AI-based systems are measured in relation to people: their ability to compete with humans at our most intellectually challenging games of reason – checkers, chess, Go, poker. The standard of success isn’t training the system to become perfect at it, or to “solve” the game (i.e., work out every possible combination of moves). The benchmark is playing the game better than any human.

That’s the objective. Once the system is better than any of us, it’s ready to move into an advisory role, providing guidance and suggestions, augmenting our capabilities. For now. In a decade or so, these systems will take over tasks for us altogether.

Driving is a prime example. If driving were a game, humans would still beat machines – even though statistics show we’re getting worse at it (according to Dr. Pradeep Dubey, Intel Fellow, Intel Labs & Director, Parallel Computing Lab, who presented at ISC on autonomous vehicle technology). Around the world, two people are killed in car accidents each minute. In the U.S., 40,000 people are killed annually and 2 million suffer permanent injuries.

Meanwhile, AI is enabling machines to get better at driving. A convergence point is coming. For now, the car’s intelligence is limited to navigating, warning us about traffic conditions and setting off beepers when we get close to curbs and other cars.

The next step: our roads will have special lanes where we’ll temporarily hand over operation of the car to itself. A few years after that, we won’t drive at all. Driving is a game in which machines will soon be much better than we are.

Dr. Eng Lim Goh, Vice President of HPE and an industry visionary for decades, is a prime driver of new AI system development. At ISC this week, he discussed why AI in all its forms – machine learning, deep learning, strategic reasoning, etc. – is the driving force bringing about “data intensive” computing architectures.

Here’s his schema for the data intensive computer:

The left side of the diagram is old-style, LINPACK-benchmarked, processor-centric computing. That’s where HPC happens. The processor is at the center. Data is sent to the CPU, simulations are run, and new, and more, data comes out. These systems have hit a wall of their own making. The problem occurs when HPC systems run their simulations, generating exponentially more machine-generated data than they started with. They’re producing data beyond the capability of data scientists to analyze. Big data isn’t big enough.

“For 30 years we’ve lived in this world where small amounts of data go in, and we apply supercomputing power onto our partial differential equations, or our models, to generate lots of data,” he said.

Already, Goh pointed out, there aren’t enough data scientists to meet demand for today’s data analytics requirements. For the torrents of machine-generated data to come, there’s an overwhelming need to automate how data is analyzed.

Take for example seismic exploration.

For exploration of energy reserves at sea, ships drag cables with hydrophones, fire shots into the ocean floor and collect the echo on sensors. Goh said for every 10TB of data collected by the sensors, 1PB of simulation data is produced – 100X the original data.

That’s where the right side of the diagram comes in: high performance analytics (HPA), self-learning AI systems that can take voluminous amounts of data produced by HPC, put it in memory, and work up answers to questions.

Dr. Eng Lim Goh

The key to the data-centric system of the future is the border area in the middle of the diagram. That’s where memory (i.e., data) resides, like a queen bee. It will be surrounded by a variety of processors (CPUs, GPUs, FPGAs, ASICs, assigned jobs appropriate for their capabilities) operating around the data, like drones.

Looked at this way, in a world where most companies have analyzed only about 3 percent of their data on average, traditional HPC systems seem glaringly incomplete. Combining the left side of the diagram and the right, integrating HPC with HPA – that takes supercomputing somewhere new. That’s a machine with a new soul.

But Goh conceded there are barriers to HPC and HPA joining forces.

“The two worlds are very different,” Goh said. “The HPC world where I lived, I’m guilty of this. All these years we assumed data movement was free. Guess what? When Linpack started 20 years ago we didn’t consider data movement. Yet we’re still ranking our Top500 systems that way. We’re still guilty that way.

“But the data scientists of the world also have something to say about us,” he added. “They assume compute is free. Take Hadoop. Hadoop is a technique where you map your data out onto compute nodes, then do your computation, then you reduce the data you bring back. The data world called this MapReduce. So we have to bring the two worlds together. More and more now, people should be investing in one system of left and right, not just the left.”

Goh pointed to the middle of his diagram and said that’s where the big architectural challenge lies. “If you have to move an exabyte of data between system A and B, if they are two different systems, it will be impractical. The world will come to this (integration of HPC and HPA).”

That’s why the U.S. effort to develop a “capable” exascale computer by the early 2020’s puts as much emphasis on compute power as memory capacity. A mission document issued by the Exascale Computing Project said its intent to build a system not just with an exaflop of processing power but one that also can handle an Exabyte of data in memory.

Goh described HPE’s “Bridges” system at the Pittsburgh Supercomputer Center as a data-centric supercomputer that incorporates HPC and HPA, designed specifically for “scalable deep learning.”

“Essentially, it’s a bandwidth machine,” Goh said. “It’s a supercomputer, but really it’s a data mover. Not only are NVlinks all connected, they’re also GPU-connected, so clumps of four GPUs can talk to other clumps of four GPUs directly. Then we have four OPA’s coming out of each node, giving one OPA per GPU. So this is really a data machine.”

The Bridges supercomputer pulled off one of the most impressive game wins of the emerging AI era when it defeated four of the world’s top poker players earlier this year. Actually, the competition stretched across two years, Goh said, with the AI system losing $700,000 to the players the first year they played. The second year, with 10X more compute from the Bridges computer, the AI system (“Libratus”) took the four humans for $1.7 million, a classic hustle.

While IBM Deep Blue (chess) and Google’s AlphaGo have grabbed most of the machine-defeated-human headlines of late, it’s less well known that machines have beaten humans at checkers, which has 1020 “naïve” (or possible) combinations, since the early 1990s, several years before IBM beat the world’s top chess player. Chess has 1047 naïve combinations. How big is 1047? An exascale machine running for 100 years would complete only 1028 combinations. The point being that without integrated AI techniques, processing only gets you so far.

Go, meanwhile, has 10171 combinations. Poker, with “only” 10160 combinations, offers up the added complexity of “incomplete information.” By contrast with the three board games, in which you can see the pieces held by your opponent, in poker, you don’t know what your opponents have in their hands.

“So we didn’t solve chess, machines didn’t solve chess,” Goh said, “all they did was be good enough to be superhuman – to beat any human. That’s a term were going to hear more and more now.”

After Goh’s presentation, he was asked to response to Google not understanding how AlpaGo won the Go tournament. The issue, he said, is overcoming opacity.

“We’re working very hard to increasing transparency,” he said. “Some people have discussed the idea that there are many stages in a neural network, to intercept it in between those stages, and take its output and see if you can make sense of it.”

Leaving a strong role for human supervision also is important. He pointed out that since the Industrial Revolution, workers get promoted from first operating a machine to supervising machines.

He also discussed the distinction between the “correct” and the “right” answer. An AI-based system may deliver a correct answer, but whether it’s “right” – acceptable within human social mores, the bounds of business ethics, or even an aesthetic judgment – is something only humans can decide.

“Societal values need to be applied, human values need to be applied,” he said.

 

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Quantum Rolls – DOE Dishes $218M; NSF Awards $31M; US Releases ‘Strategic Overview’

September 24, 2018

It was quite a day for U.S. quantum computing. In conjunction with the White House Summit on Advancing American Leadership in Quantum Information Science (QIS) held today, the Department of Energy announced $218 million Read more…

By John Russell

Russian and American Scientists Achieve 50% Increase in Data Transmission Speed

September 20, 2018

As high-performance computing becomes increasingly data-intensive and the demand for shorter turnaround times grows, data transfer speed becomes an ever more important bottleneck. Now, in an article published in IEEE Tra Read more…

By Oliver Peckham

IBM to Brand Rescale’s HPC-in-Cloud Platform

September 20, 2018

HPC (or big compute)-in-the-cloud platform provider Rescale has formalized the work it’s been doing in partnership with public cloud vendors by announcing its Powered by Rescale program – with IBM as its first named Read more…

By Doug Black

HPE Extreme Performance Solutions

Introducing the First Integrated System Management Software for HPC Clusters from HPE

How do you manage your complex, growing cluster environments? Answer that big challenge with the new HPC cluster management solution: HPE Performance Cluster Manager. Read more…

IBM Accelerated Insights

Clouds Over the Ocean – a Healthcare Perspective

Advances in precision medicine, genomics, and imaging; the widespread adoption of electronic health records; and the proliferation of medical Internet of Things (IoT) and mobile devices are resulting in an explosion of structured and unstructured healthcare-related data. Read more…

Democratization of HPC Part 1: Simulation Sheds Light on Building Dispute

September 20, 2018

This is the first of three articles demonstrating the growing acceptance of High Performance Computing especially in new user communities and application areas. Major reasons for this trend are the ongoing improvements i Read more…

By Wolfgang Gentzsch

Quantum Rolls – DOE Dishes $218M; NSF Awards $31M; US Releases ‘Strategic Overview’

September 24, 2018

It was quite a day for U.S. quantum computing. In conjunction with the White House Summit on Advancing American Leadership in Quantum Information Science (QIS) Read more…

By John Russell

Summit Supercomputer is Already Making its Mark on Science

September 20, 2018

Summit, now the fastest supercomputer in the world, is quickly making its mark in science – five of the six finalists just announced for the prestigious 2018 Read more…

By John Russell

House Passes $1.275B National Quantum Initiative

September 17, 2018

Last Thursday the U.S. House of Representatives passed the National Quantum Initiative Act (NQIA) intended to accelerate quantum computing research and developm Read more…

By John Russell

Nvidia Accelerates AI Inference in the Datacenter with T4 GPU

September 14, 2018

Nvidia is upping its game for AI inference in the datacenter with a new platform consisting of an inference accelerator chip--the new Turing-based Tesla T4 GPU- Read more…

By George Leopold

DeepSense Combines HPC and AI to Bolster Canada’s Ocean Economy

September 13, 2018

We often hear scientists say that we know less than 10 percent of the life of the oceans. This week, IBM and a group of Canadian industry and government partner Read more…

By Tiffany Trader

Rigetti (and Others) Pursuit of Quantum Advantage

September 11, 2018

Remember ‘quantum supremacy’, the much-touted but little-loved idea that the age of quantum computing would be signaled when quantum computers could tackle Read more…

By John Russell

How FPGAs Accelerate Financial Services Workloads

September 11, 2018

While FSI companies are unlikely, for competitive reasons, to disclose their FPGA strategies, James Reinders offers insights into the case for FPGAs as accelerators for FSI by discussing performance, power, size, latency, jitter and inline processing. Read more…

By James Reinders

Update from Gregory Kurtzer on Singularity’s Push into FS and the Enterprise

September 11, 2018

Container technology is hardly new but it has undergone rapid evolution in the HPC space in recent years to accommodate traditional science workloads and HPC systems requirements. While Docker containers continue to dominate in the enterprise, other variants are becoming important and one alternative with distinctly HPC roots – Singularity – is making an enterprise push targeting advanced scale workload inclusive of HPC. Read more…

By John Russell

TACC Wins Next NSF-funded Major Supercomputer

July 30, 2018

The Texas Advanced Computing Center (TACC) has won the next NSF-funded big supercomputer beating out rivals including the National Center for Supercomputing Ap Read more…

By John Russell

IBM at Hot Chips: What’s Next for Power

August 23, 2018

With processor, memory and networking technologies all racing to fill in for an ailing Moore’s law, the era of the heterogeneous datacenter is well underway, Read more…

By Tiffany Trader

Requiem for a Phi: Knights Landing Discontinued

July 25, 2018

On Monday, Intel made public its end of life strategy for the Knights Landing "KNL" Phi product set. The announcement makes official what has already been wide Read more…

By Tiffany Trader

CERN Project Sees Orders-of-Magnitude Speedup with AI Approach

August 14, 2018

An award-winning effort at CERN has demonstrated potential to significantly change how the physics based modeling and simulation communities view machine learni Read more…

By Rob Farber

ORNL Summit Supercomputer Is Officially Here

June 8, 2018

Oak Ridge National Laboratory (ORNL) together with IBM and Nvidia celebrated the official unveiling of the Department of Energy (DOE) Summit supercomputer toda Read more…

By Tiffany Trader

New Deep Learning Algorithm Solves Rubik’s Cube

July 25, 2018

Solving (and attempting to solve) Rubik’s Cube has delighted millions of puzzle lovers since 1974 when the cube was invented by Hungarian sculptor and archite Read more…

By John Russell

House Passes $1.275B National Quantum Initiative

September 17, 2018

Last Thursday the U.S. House of Representatives passed the National Quantum Initiative Act (NQIA) intended to accelerate quantum computing research and developm Read more…

By John Russell

AMD’s EPYC Road to Redemption in Six Slides

June 21, 2018

A year ago AMD returned to the server market with its EPYC processor line. The earth didn’t tremble but folks took notice. People remember the Opteron fondly Read more…

By John Russell

Leading Solution Providers

SC17 Booth Video Tours Playlist

Altair @ SC17

Altair

AMD @ SC17

AMD

ASRock Rack @ SC17

ASRock Rack

CEJN @ SC17

CEJN

DDN Storage @ SC17

DDN Storage

Huawei @ SC17

Huawei

IBM @ SC17

IBM

IBM Power Systems @ SC17

IBM Power Systems

Intel @ SC17

Intel

Lenovo @ SC17

Lenovo

Mellanox Technologies @ SC17

Mellanox Technologies

Microsoft @ SC17

Microsoft

Penguin Computing @ SC17

Penguin Computing

Pure Storage @ SC17

Pure Storage

Supericro @ SC17

Supericro

Tyan @ SC17

Tyan

Univa @ SC17

Univa

Summit Supercomputer is Already Making its Mark on Science

September 20, 2018

Summit, now the fastest supercomputer in the world, is quickly making its mark in science – five of the six finalists just announced for the prestigious 2018 Read more…

By John Russell

Sandia to Take Delivery of World’s Largest Arm System

June 18, 2018

While the enterprise remains circumspect on prospects for Arm servers in the datacenter, the leadership HPC community is taking a bolder, brighter view of the x86 server CPU alternative. Amongst current and planned Arm HPC installations – i.e., the innovative Mont-Blanc project, led by Bull/Atos, the 'Isambard’ Cray XC50 going into the University of Bristol, and commitments from both Japan and France among others -- HPE is announcing that it will be supply the United States National Nuclear Security Administration (NNSA) with a 2.3 petaflops peak Arm-based system, named Astra. Read more…

By Tiffany Trader

D-Wave Breaks New Ground in Quantum Simulation

July 16, 2018

Last Friday D-Wave scientists and colleagues published work in Science which they say represents the first fulfillment of Richard Feynman’s 1982 notion that Read more…

By John Russell

MLPerf – Will New Machine Learning Benchmark Help Propel AI Forward?

May 2, 2018

Let the AI benchmarking wars begin. Today, a diverse group from academia and industry – Google, Baidu, Intel, AMD, Harvard, and Stanford among them – releas Read more…

By John Russell

TACC’s ‘Frontera’ Supercomputer Expands Horizon for Extreme-Scale Science

August 29, 2018

The National Science Foundation and the Texas Advanced Computing Center announced today that a new system, called Frontera, will overtake Stampede 2 as the fast Read more…

By Tiffany Trader

Intel Announces Cooper Lake, Advances AI Strategy

August 9, 2018

Intel's chief datacenter exec Navin Shenoy kicked off the company's Data-Centric Innovation Summit Wednesday, the day-long program devoted to Intel's datacenter Read more…

By Tiffany Trader

GPUs Power Five of World’s Top Seven Supercomputers

June 25, 2018

The top 10 echelon of the newly minted Top500 list boasts three powerful new systems with one common engine: the Nvidia Volta V100 general-purpose graphics proc Read more…

By Tiffany Trader

The Machine Learning Hype Cycle and HPC

June 14, 2018

Like many other HPC professionals I’m following the hype cycle around machine learning/deep learning with interest. I subscribe to the view that we’re probably approaching the ‘peak of inflated expectation’ but not quite yet starting the descent into the ‘trough of disillusionment. This still raises the probability that... Read more…

By Dairsie Latimer

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This