The Masters of Uncertainty

By Nicole Hemsoth

September 13, 2013

According to Dr. Houman Owhadi and Dr. Clint Scovel, both from the California Institute of Technology, Bayesian methods are becoming more prevalent as high performance computing advances continue. In this special audio-based feature interview, we talk with both about what these methods will contribute to a number of research and enterprise endeavors, what computational requirements exist as we move toward more advanced questions, and how the field is evolving—and will continue to evolve with exascale (or even quantum) class systems.

HPCwire: In the context of high performance computing the two of you have argued recently that Bayesian methods are becoming even more popular than they ever were before to quantify uncertainty both in science and industry. What is it about these methods that are necessary and even more necessary as we move towards ever more advanced systems?

Owhadi: Bayesian inference goes all the way back to a formula discovered by Reverend Thomas Bayes, and when he found that formula, Pierre-Simon Laplace took his research and developed further into a field called Baysian Inference. This started a 250 year old controversy. What is the controversy about? In Bayesian inference, you have some prior about what reality could be, then you condition your prior with some data that you observe. This is basically the Bayes’ rule.

Now Pierre-Simon Laplace took that one step further where he said, ok, I don’t really need to have an exact prior we’re representing – an exact measure corresponding to what reality could be. I could just make up a prior, or a belief about what it could be, then I could use Bayes’ formula to update my belief. And this started the field that we know today as Bayesian inference.

In the 50’s, Bayesian inference was mainly looked at as a curiosity because we couldn’t really compute those Bayesian posteriors for complex systems. But now with the advent of high performance computing, we can actually compute those posterior probabilities. Since Bayesian inference is also an elegant and simple way of combining information with beliefs, it’s becoming increasingly popular.

HPCwire: Dr. Scovel to build on what he just said – HPC is so often considered to be about increasing fidelity and resolution, moving as close to reality as possible, so where does uncertaintly fit in in the next generation of systems and applications.

Scovel: Well certainly no one believes that those systems compute exact, anything. There’s always an error, and so having some confidence in what the results are is always going to be of interest. There’s another way that these things all fit together, and that is that not only is uncertainty quantification useful for high performance computing, but high performance computing is useful for uncertainty quantification; because that computation that he was saying that has just come about in the 50’s is basically about our ability to do these numerical computations – in particular Markov chain Monte Carlo simulations to compute these posteriors.

HPCwire: Let’s talk about what areas of industry and scientific computing most important. Where is this most valuable?

Owhadi: Risk analysis. Climate modeling. Take for instance Boeing. So when Boeing is developing a new plane, most of the budget goes into the safety assessment of their new model. What you have to understand with respect to that industry is that they have to certify that their new model of airplane has a probability of catastrophic event that is smaller than 10 to the power of minus nine per hour of flight. Now this is really small.

And of course, they cannot fly one billion airplanes and just see how many crash, so they have to assess the safety of their system and they have limited information. Take another example, you are JPL and you wanted to design your new satellite and you want your satellite to go around the planet in the solar system, and you are spending a lot of money for it. How do you certify that your system is not going to crash? One way is to build 1,000 of these satellites and just count how many crash, but that would be too costly. So you have to do it with a very limited amount of data. This has created a new field called uncertainty quantification, which is an emerging field. It is basically a field that is at the interface between probability and statistics and computer science.

It mainly has to do with engineering systems characterized by low number of samples, and complex information. The way we are seeing that it should be pushed forward is basically to be able to process information in an optimal way, to assess the risk in an optimal way, without making assumptions that may not be true, and without ignoring relevant information.

So let me explain – at the end of the day our point of view is that you cannot really say if a piece of information or piece of data is accurate or not unless you test it. But once this information is given to you, the best thing that you can do is to process it in an optimal way. Basically what we are striving to do is to develop an algorithmic framework to allow us to do just that: process information in an optimal way. Now you can imagine that there are plenty of places where you can apply these things.

HPCwire: Dr. Scovel, I believe this leads into your pet project right now which is the scientific computation of optimal estimators. Where does that fit into this conversation we’re having. Can you describe it more thoroughly.

Scovel: Yes, when he was talking doing this in the optimal way, the first question is what does that mean. Instead of providing solutions, the first thing that we do is actually formulate a problem that incorporates everybody’s – the customers objectives that they’re interested in, the available information, what we know about the information, what the domain experts know about the information et cetera – and then you formulate this optimization problem which essentially defines what it means to be an optimal solution to this question that you’ve asked – like how reliable is that satellite going to be.

Where this is new is that historically what has been done, you provide some model for this process and you see what happens with the model. Ours is different. We’re saying we want to formulate this problem that we’re trying to solve and we’re going to use our computing capability – in particular, high performance computing – to solve these problems. I think historically is very similar to Bayes methods in the past. Historically, the reason why people didn’t go down this path is we didn’t have the computing power to do it, but I think that we now do have the computing power to actually solve these problems defining optimal estimation problems, or optimal prediction problems given optimal being optimal over some set of assumptions that we’re all willing to agree on.

HPCwire: I noticed that both of you have, just in your body of research, talked about solving exascale class problems. Those are the types of systems you’re looking at to be able to do this at a much higher level, is that correct?

Owhadi: Yes. I could take another analogy here – 200 years ago, if I were to ask you to solve a partial differential equation, you wouldn’t use a computer, you would probably use your brain. You would probably not come with a quantitiative estimate of the solution, but only qualitative estimates.

Now, if ask you the same question today, you will not use your brain to solve the partial differential equation – you will use a computer. But you will still use your brain to program the computer that will crunch numbers for you to solve the partial differential equation. This paradigm shift can be traced back to seminal work by John Von Neumann and (Herman) Goldstein in the 50’s, and humans organized as computers in the beginning of the previous century.

Now today if I asked you to find a statistical estimator, or to find the best possible climate model, or to find a test that will tell me if some data that I’m observing is corrupted or not – you’re not going to use a computer to do that. You’re going to use your brain and guess work. What we want to do here is basically turn this guesswork into an algorithm that we’ll be able to implement on a high performance cluster.

If you look at the mathematics behind this problem of turning the process of scientific discovery into an algorithm, you can basically translate it into an optimization problem, but optimization variables are not discrete. They’re not zeros and ones – they’re basically infinite dimensional objects. What we have found is a new form of calculus that allows us to turn these infinite dimensional optimization problems into finite dimensional optimization problems that we can start solving on computers.

Even after reduction these optimization problems are extremely large, so that’s why we believe that we’ll probably need exascale or petascale machines for solving these kind of problems for complex systems.

HPCwire: What’s interesting in that conversation about the systems required, I recently talked to, I believe the only, quote, “quantum computer company,” D-Wave recently – and this exact type of optimization problem – this best of all worlds solution is exactly the sweet spot for quantum cmputers. Do you believe those actually exist, and if so, are they a good fit for the types of problems you’re seeking to solve?

Owhadi: This is interesting because with respect to exascale machines – some people believe that there are two ways to approach high performance computing. The first way is just to solve the same kind of problems, but bigger problems.  So for instance if you’re interested in climate modeling, you’re still going to do climate modeling. You still are going to run your model, but with a finer mesh, with a finer resolution, and hopefully instead of predicting the weather for four days ahead, you’ll be able to do it for five days out.

What we envision here is some kind of paradigm shift where instead of numerically solving a bigger model, you actually use your computer to find the model itself. So yes, if there are quantum computers out there, that would be a great thing for this new kind of framework.

Scovel: The more computing power we can have the better our success is going to be. 

Owhadi: Information doesn’t necessarily come in the form of zero and ones. If you look at the interlaying optimization problems, they involve optimization variables that are measures of probability and function and these objects live in infinite dimensional spaces. Calculus on a computer is necessarily discrete and finite. So the first step of the technology is mathematical.  You have to be able to manipulate – to come up with a new form of calculus that is able to manipulate these infinite dimensional objects. This is basically what we’ve done. This new form of calculus allows us to take these huge optimization problems and turn them into something discrete and finite that we can start solving on a computer.

Scovel: It’s also more than that in the sense that it’s not just a question of computing power. Part of this program and this paradigm shift that we’re talking about is actually coming up with formulation of what it means to be an optimal solution to these things. That’s actually a big part of the program. It’s not just, I know what I want to compute and I need more computing power. It’s actually, what do we want to compute and what does it mean to be the best.

I think the complexity involved into the formulation of these problems – and these formulations for these problems of what it means to be an optimal predictor or an optimal estimator requires communication from all levels of the effort – from the customer to the project leader to the domain expert, the material science domain experts, the statisticians – instead of for example in many places you do a bunch of runs, you do a bunch of modeling, you do a bunch of stuff, and then you hand all the results off to a statisticians and you want them to put that stuff together.

We’re saying, no you need to do that all together. You need to have everybody communicating so you’re formulating not only what the objectives that you’re interested in but you formulate what pieces of information you have good confidence about, and those establish a quantitative set of realistic paradigms. Then the optimization now proceeds as, ok, now we have this huge optimization problem, how do we reduce it analytically, how do we take those reduced analytic problems, and how do we implement them on the computer, and how do we know we’re done – that’s sort of the picture.

Owhadi: Let me give you two examples here.

The first one will be investing in the stock market. Question: is there an optimal way to invest in the stock market. Currently, it’s not clear how to turn this into an optimization problem, but with the framework that we are developing, we think that we’ll be able to turn this into an optimization problem and reduce it to something that we can start solving with a high performance cluster. The idea is to invest in an optimal way given the limited information that you have at hand.

Let me give you another example. Consider the game of chess. We have computers playing chess, and they play chess very well. Question: can you have a computer play an information game – an information war? So for instance, you have two armies and they are playing this information war – or you have two companies, and each company has limited information about the other company and they have some decisions to make with respect to what the other company could or could not know and could or could not do. If you look at this as just an information game, but the board is not just composed of 64 squares, and the moves are not discrete, they could be anything.

If you want to start addressing these kind of problems, you need some new mathematics that allows you to manipulate these pieces of information that do not live on a chess board. This is basically the first step in our technology to develop this new mathematics.

HPCwire: Where do you see this maybe revolutionizing certain approaches to computing down the road?

Owhadi: I think that eventually we will use computers to help the process of scientific discovery, but not just to solve mathematical equations, but to develop the mathematical equations themselves. We see this going into the field of machine learning. In the field of machine learning, if you want to develop an intelligent agent, you basically decompose the tasks that you ask the computer to do into small steps, and you chew the work for the computer.

What this technology that we are developing will allow to do – this is basically a long term vision – is give the ability to a computer to develop a model of reality, act on that model of reality, and update the model based on feedback that it is receiving from reality. This is basically a change in machine learning.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

World Cup is Lame Compared to This Competition

June 18, 2018

So you think World Cup soccer is a big deal? While I’m sure it’s very compelling to watch a bunch of athletes kick a ball around, World Cup misses the boat because it doesn’t include teams putting together their ow Read more…

By Dan Olds

IBM Demonstrates Deep Neural Network Training with Analog Memory Devices

June 18, 2018

From smarter, more personalized apps to seemingly-ubiquitous Google Assistant and Alexa devices, AI adoption is showing no signs of slowing down – and yet, the hardware used for AI is far from perfect. Currently, GPUs Read more…

By Oliver Peckham

Sandia to Take Delivery of World’s Largest Arm System

June 18, 2018

While the enterprise remains circumspect on prospects for Arm servers in the datacenter, the leadership HPC community is taking a bolder, brighter view of the x86 server CPU alternative. Amongst current and planned Arm HPC installations – i.e., the innovative Mont-Blanc project, led by Bull/Atos, the 'Isambard’ Cray XC50 going into the University of Bristol, and commitments from both Japan and France among others -- HPE is announcing that it will be supply the United States National Nuclear Security Administration (NNSA) with a 2.3 petaflops peak Arm-based system, named Astra. Read more…

By Tiffany Trader

HPE Extreme Performance Solutions

HPC and AI Convergence is Accelerating New Levels of Intelligence

Data analytics is the most valuable tool in the digital marketplace – so much so that organizations are employing high performance computing (HPC) capabilities to rapidly collect, share, and analyze endless streams of data. Read more…

IBM Accelerated Insights

Banks Boost Infrastructure to Tackle GDPR

As banks become more digital and data-driven, their IT managers are challenged with fast growing data volumes and lines-of-businesses’ (LoBs’) seemingly limitless appetite for analytics. Read more…

Challenges Face Astroinformatics as It Sorts Through the Stars

June 15, 2018

You might have seen one of those YouTube videos: they begin on Earth, slowly zooming out to the Moon, the Solar System, the Milky Way, beyond – and suddenly, you’re looking at trillions of stars. It’s a lot to take Read more…

By Oliver Peckham

Sandia to Take Delivery of World’s Largest Arm System

June 18, 2018

While the enterprise remains circumspect on prospects for Arm servers in the datacenter, the leadership HPC community is taking a bolder, brighter view of the x86 server CPU alternative. Amongst current and planned Arm HPC installations – i.e., the innovative Mont-Blanc project, led by Bull/Atos, the 'Isambard’ Cray XC50 going into the University of Bristol, and commitments from both Japan and France among others -- HPE is announcing that it will be supply the United States National Nuclear Security Administration (NNSA) with a 2.3 petaflops peak Arm-based system, named Astra. Read more…

By Tiffany Trader

The Machine Learning Hype Cycle and HPC

June 14, 2018

Like many other HPC professionals I’m following the hype cycle around machine learning/deep learning with interest. I subscribe to the view that we’re probably approaching the ‘peak of inflated expectation’ but not quite yet starting the descent into the ‘trough of disillusionment. This still raises the probability that... Read more…

By Dairsie Latimer

Xiaoxiang Zhu Receives the 2018 PRACE Ada Lovelace Award for HPC

June 13, 2018

Xiaoxiang Zhu, who works for the German Aerospace Center (DLR) and Technical University of Munich (TUM), was awarded the 2018 PRACE Ada Lovelace Award for HPC for her outstanding contributions in the field of high performance computing (HPC) in Europe. Read more…

By Elizabeth Leake

U.S Considering Launch of National Quantum Initiative

June 11, 2018

Sometime this month the U.S. House Science Committee will introduce legislation to launch a 10-year National Quantum Initiative, according to a recent report by Read more…

By John Russell

ORNL Summit Supercomputer Is Officially Here

June 8, 2018

Oak Ridge National Laboratory (ORNL) together with IBM and Nvidia celebrated the official unveiling of the Department of Energy (DOE) Summit supercomputer toda Read more…

By Tiffany Trader

Exascale USA – Continuing to Move Forward

June 6, 2018

The end of May 2018, saw several important events that continue to advance the Department of Energy’s (DOE) Exascale Computing Initiative (ECI) for the United Read more…

By Alex R. Larzelere

Exascale for the Rest of Us: Exaflops Systems Capable for Industry

June 6, 2018

Enterprise advanced scale computing – or HPC in the enterprise – is an entity unto itself, situated between (and with characteristics of) conventional enter Read more…

By Doug Black

Fracas in Frankfurt: ISC18 Cluster Competition Teams Unveiled

June 6, 2018

The Student Cluster Competition season heats up with the seventh edition of the ISC Student Cluster Competition, slated to begin on June 25th in Frankfurt, Germ Read more…

By Dan Olds

MLPerf – Will New Machine Learning Benchmark Help Propel AI Forward?

May 2, 2018

Let the AI benchmarking wars begin. Today, a diverse group from academia and industry – Google, Baidu, Intel, AMD, Harvard, and Stanford among them – releas Read more…

By John Russell

How the Cloud Is Falling Short for HPC

March 15, 2018

The last couple of years have seen cloud computing gradually build some legitimacy within the HPC world, but still the HPC industry lies far behind enterprise I Read more…

By Chris Downing

US Plans $1.8 Billion Spend on DOE Exascale Supercomputing

April 11, 2018

On Monday, the United States Department of Energy announced its intention to procure up to three exascale supercomputers at a cost of up to $1.8 billion with th Read more…

By Tiffany Trader

Deep Learning at 15 PFlops Enables Training for Extreme Weather Identification at Scale

March 19, 2018

Petaflop per second deep learning training performance on the NERSC (National Energy Research Scientific Computing Center) Cori supercomputer has given climate Read more…

By Rob Farber

Lenovo Unveils Warm Water Cooled ThinkSystem SD650 in Rampup to LRZ Install

February 22, 2018

This week Lenovo took the wraps off the ThinkSystem SD650 high-density server with third-generation direct water cooling technology developed in tandem with par Read more…

By Tiffany Trader

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

ORNL Summit Supercomputer Is Officially Here

June 8, 2018

Oak Ridge National Laboratory (ORNL) together with IBM and Nvidia celebrated the official unveiling of the Department of Energy (DOE) Summit supercomputer toda Read more…

By Tiffany Trader

HPE Wins $57 Million DoD Supercomputing Contract

February 20, 2018

Hewlett Packard Enterprise (HPE) today revealed details of its massive $57 million HPC contract with the U.S. Department of Defense (DoD). The deal calls for HP Read more…

By Tiffany Trader

Leading Solution Providers

SC17 Booth Video Tours Playlist

Altair @ SC17

Altair

AMD @ SC17

AMD

ASRock Rack @ SC17

ASRock Rack

CEJN @ SC17

CEJN

DDN Storage @ SC17

DDN Storage

Huawei @ SC17

Huawei

IBM @ SC17

IBM

IBM Power Systems @ SC17

IBM Power Systems

Intel @ SC17

Intel

Lenovo @ SC17

Lenovo

Mellanox Technologies @ SC17

Mellanox Technologies

Microsoft @ SC17

Microsoft

Penguin Computing @ SC17

Penguin Computing

Pure Storage @ SC17

Pure Storage

Supericro @ SC17

Supericro

Tyan @ SC17

Tyan

Univa @ SC17

Univa

Hennessy & Patterson: A New Golden Age for Computer Architecture

April 17, 2018

On Monday June 4, 2018, 2017 A.M. Turing Award Winners John L. Hennessy and David A. Patterson will deliver the Turing Lecture at the 45th International Sympo Read more…

By Staff

Google Chases Quantum Supremacy with 72-Qubit Processor

March 7, 2018

Google pulled ahead of the pack this week in the race toward "quantum supremacy," with the introduction of a new 72-qubit quantum processor called Bristlecone. Read more…

By Tiffany Trader

Google I/O 2018: AI Everywhere; TPU 3.0 Delivers 100+ Petaflops but Requires Liquid Cooling

May 9, 2018

All things AI dominated discussion at yesterday’s opening of Google’s I/O 2018 developers meeting covering much of Google's near-term product roadmap. The e Read more…

By John Russell

Nvidia Ups Hardware Game with 16-GPU DGX-2 Server and 18-Port NVSwitch

March 27, 2018

Nvidia unveiled a raft of new products from its annual technology conference in San Jose today, and despite not offering up a new chip architecture, there were still a few surprises in store for HPC hardware aficionados. Read more…

By Tiffany Trader

Pattern Computer – Startup Claims Breakthrough in ‘Pattern Discovery’ Technology

May 23, 2018

If it weren’t for the heavy-hitter technology team behind start-up Pattern Computer, which emerged from stealth today in a live-streamed event from San Franci Read more…

By John Russell

Part One: Deep Dive into 2018 Trends in Life Sciences HPC

March 1, 2018

Life sciences is an interesting lens through which to see HPC. It is perhaps not an obvious choice, given life sciences’ relative newness as a heavy user of H Read more…

By John Russell

Intel Pledges First Commercial Nervana Product ‘Spring Crest’ in 2019

May 24, 2018

At its AI developer conference in San Francisco yesterday, Intel embraced a holistic approach to AI and showed off a broad AI portfolio that includes Xeon processors, Movidius technologies, FPGAs and Intel’s Nervana Neural Network Processors (NNPs), based on the technology it acquired in 2016. Read more…

By Tiffany Trader

Google Charts Two-Dimensional Quantum Course

April 26, 2018

Quantum error correction, essential for achieving universal fault-tolerant quantum computation, is one of the main challenges of the quantum computing field and it’s top of mind for Google’s John Martinis. At a presentation last week at the HPC User Forum in Tucson, Martinis, one of the world's foremost experts in quantum computing, emphasized... Read more…

By Tiffany Trader

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This