Deep Learning Thrives in Cancer Moonshot

By John Russell

August 8, 2017

The U.S. War on Cancer, certainly a worthy cause, is a collection of programs stretching back more than 40 years and abiding under many banners. The latest is the Cancer Moonshot, launched in 2016 by then U.S. Vice President Joe Biden, and passed as part of the 21st Century Cures Act. Over the years, computational technology has become an increasingly important component in all of these cancer-fighting programs, hand-in-glove with dramatic advances in understanding cancer biology and a profusion of new experimental technologies – DNA sequencing is just one.

Recently, deep learning has emerged as a powerful tool in the arsenal and is a driving force in the CANcer Distributed Learning Environment (CANDLE) project, a joint effort between the U.S. Department of Energy (DOE) and the National Cancer Institute (NCI), that is also part of the U.S. Exascale Computing Project (ECP). (For an overview, see HPCwire article, Enlisting Deep Learning in the War on Cancer).

Think of CANDLE as an effort to develop a computational framework replete with: a broad deep learning infrastructure able to run on leadership class computers and other substantial machines; a collection of very many deep learning tools including specific networks and training data sets for use in cancer research; and potentially leading to development of new therapeutics – drugs – for use against cancer. Currently, CANDLE helps tackle the specific cancer challenges of three cancer pilots, all using deep learning as the engine. Importantly, the three pilots have begun to yield results. The first software and data set releases took place in March, and just a few weeks ago, the first stable ‘CANDLE infrastructure’ was released.

Rick Stevens, Argonne National Laboratory

In the thick of the CANDLE work is principal investigator Rick Stevens, also a co-PI of one of the pilots and associate laboratory director for the computing, environmental and life sciences directorate at Argonne National Laboratory (ANL). Stevens recently talked with HPCwire about CANDLE’s progress. The work encompasses all things deep learning – from novel network types, and algorithm development, to tuning leadership class hardware platforms (GPU and CPU) for deep learning use. Results are shared on GitHub in what is a roughly quarterly schedule.

Perhaps as important, “We are trying to plant inside the labs a critical mass of deep learning research people,” says Stevens. Just as CANDLE embodies broad hopes for applying deep learning to precision medicine, DOE has high hopes for developing deep learning tools able to run on leadership class machines and to be used in science research writ large.

“Everything is moving very fast right now,” says Stevens. “As the scientific community starts to get interesting results, two things will happen. One is we will be able to write papers and say ‘oh we took this network concept from Google and we applied it to material science and we changed it just this way and got this great result and now it is a standard way for designing materials.’ Those kinds of papers will happen.

“But then there is going to be another class of papers where I think in the labs, people that are interested in deep learning more generally and are interested in applying it to things other than driving a car or translating, but now are interested in applying it to particle physics or something – they are going to come up with some new deep learning algorithms, new network types, new layer types, new ways of doing inference for example, that are actually inspired by the problem in the scientific domain. That will take a bit longer.”

Clearly, there’s a lot going on here with CANDLE as the tip of the spear. Here’s a snapshot of the three pilots, which fall under Joint Design of Advanced Computing Solutions for Cancer (JDACS4C) program underway:

  • RAS Molecular Project. This pilot (Molecular Level Pilot for RAS Structure and Dynamics in Cellular Membranes) is intended to develop new computational approaches supporting research already being done under the RAS Initiative. Ultimately the hope is to refine our understanding of the role of the RAS (gene family) and its associated signaling pathway in cancer and to identify new therapeutic targets uniquely present in RAS protein membrane signaling complexes.
  • Pre-Clinical Screening. This pilot (Cellular Level Pilot for Predictive Modeling for Pre-clinical Screening) will develop “machine learning, large-scale data and predictive models based on experimental biological data derived from patient-derived xenografts.” The idea is to create a feedback loop, where the experimental models inform the design of the computational models. These predictive models may point to new targets in cancer and help identify new treatments. Stevens is co-PI.
  • Population Models. This pilot (Population Level Pilot for Population Information Integration, Analysis and Modeling) seeks to develop a scalable framework for efficient abstraction, curation, integration and structuring of medical record information for cancer patients. Such an ‘engine’ would be enormously powerful in many aspects of healthcare (delivery, cost control, research, etc.).

The early work has been impressive. “In March, we put out a set of seven cancer deep learning benchmarks on GitHub that represent each of the three pilot areas. They represent five or six different deep learning network types and have self-contained data that you can download. That was a first major milestone. A couple of weeks ago we pushed out our first release of reliable CANDLE infrastructure. It contains what we call the supervisor which is infrastructure for running large scale hyperparameter searches (for network development) on the leadership computing platforms,” says Stevens.

Hyperparameters are things like the network structure (number and types of layers), learning parameters (activation functions, optimizers, learning rate) and loss functions.  They are things that you set before training and are not learned during training.  For any given problem there are many combinations of hyperparameters that are possible and often it is important to search for good combinations to get improved model performance.

“It runs here at Argonne, at NERSC and Oak Ridge National Laboratory, and [with some modest modification] runs pretty much on anybody’s big machines. What it does is manage hundreds or thousands of model runs that are trying to find goods choices of model hyperparameters.” The latter is indeed a major accomplishment. (Links to CANDLE releases so far: CANDLE Version 0.1 architecture: https://github.com/ECP-CANDLE/CANDLE; CANDLE benchmarks: https://github.com/ECP-CANDLE/Benchmarks )

Already there have been surprises as illustrated here from Stevens’ pre-clinical screening project.

“We think we understand biology, and so we [were] thinking ‘we know relationships about the data and we can somehow structure the problem in a way that is going to make the biologically relevant information more accessible for the networks.’ The network surprised us by basically saying, ‘no that isn’t helping me, let me do it my own way’ in discovering the features that matter most on its own,” says Stevens.

“Of course I am anthropomorphizing but that was essentially the thought process that we went through. We were surprised by the fact that the machine could do better than we could. I think ultimately we’ll be right. It won the first round, but we’re going to get more clever in how we encode, and I think we’ll still be able to show that us working with the network will do a better job than either working alone.”

CANDLE INFRASTRUCTURE TAKES SHAPE
As described by Stevens, CANDLE is a layered architecture. At the top is a workflow engine – a Swift/T-based framework called Extreme-scale Model Exploration with Swift (EMEWS) was selected – that manages large numbers of runs on the supercomputers and can be adapted for other more moderate size machines.

“We are using technology like NoSQL database technology for managing the high level database component, and these things are running on GPU-based systems and Intel KNL-based systems. We are trying out some of the new AMD GPUs (Radeon) and are experimenting with some more NDA technology,” Stevens says without discussing which systems have demonstrated the best performance so far.

“For the next level down in terms of scripting environment, we decided early on to build CANDLE on top of Keras, a high level scripting interface language for deep learning that comes out of Google. All of our models get initially implemented in Keras. All of our model management tools and libraries are also implemented in Keras. We’re doing some trial implementation directly in neon (Intel) and MXNet (Amazon),” says Stevens, but there are no plans to switch.

One of the things CANDLE leaders liked about Keras is it sits easily on top of all the major frameworks – Tensorflow, Theano, MXNet, and Microsoft’s Cognitive Toolkit (CNTK). Stevens says, “We don’t program typically directly in those frameworks unless we have to. We program in the abstraction layer on top (Keras). That gives us a lot of productivity. It also gives us a lot of independence from specific frameworks and that is really important because the frameworks are still moving quite quickly.”

Google’s TPU, an inference engine, has not been tested, but the “TPU2 infrastructure is on our list of things to evaluate.” Google, of course, has shown no inclination to sell TPU2 on the open market beyond providing access in the Google cloud (See HPCwire article, Google Debuts TPU v2 and will Add to Google Cloud).

“I think our CANDLE environment in principle could run in the Amazon cloud or Google cloud or Azure cloud, in which case if you were going to do that we would certainly target the TPU/GPU for acceleration. Right now our focus is in CANDLE because it is part of the ECP to target the DOE Exascale platforms as our first priority target. Our second priority target are other clusters that we have in the labs,” says Stevens. CANDLE also runs on NIH’s Biowulf cluster.

Stevens says the project has been talking to Intel about the Nervana platform and the Knights Mill platform and is, of course, already using Nvidia P100s and will work with the V100 (Volta).

OLCF’s recently deployed DGX-1 artificial intelligence supercomputer by NVIDIA, featuring eight NVIDIA Tesla GPUs and NVLink technology, will offer scientists and researchers new opportunities to delve into deep learning technologies.

A major governance review took place about a month ago and CANDLE received the go-ahead for second year. Stevens says there are now about 100 people working on CANDLE from four different labs and NCI. “I would say we have good collaborations going on with Nvidia, Intel, and AMD in terms of deep learning optimization interactions with architecture groups. We are also talking to groups at IBM and Cray, and we’ve had a big set of demonstration runs on the three big open science computers.”

PRE-CLINICAL DRUG SCREENING SURPRISE
Of course the most immediate point of all this infrastructure work is to do worthwhile cancer research. Stevens is closest to the pre-clinical screening project whose premise is straight forward. Screen pairs of known drugs against cancerous cell lines and xenograft tumors, measure the effect – inhibition, growth, etc. – and build predictive models from the data. Most of the work so far has been done with cell lines grown in dishes but the effort to gather drug response data from human tumor grafts on rodents (human tumor xenograft model) is ramping up.

Says Stevens, “NCI took the 100 small molecule FDA cancer drugs and devised a high throughput screen that did all pairs of those drugs and concentration levels so we have something like 3.5 million experimental results there. We’ve been building a deep learning model that learns the relationship between the molecular properties of the tumor, in this case the cell line, and the structural properties of the pairs of drugs, and then predicts the percent growth inhibition.

“So we have millions of these pairs of drugs experiments and we have the score of how well the growth was inhibited. The machine learning model is trying to generalize from that training data, a representation that allows us to predict a result for a cell line and a drug combination that it hasn’t seen before. It’s a deep learning model. It’s a multi layered neural network and uses convolution.”

One surprise was how well a one-dimensional convolution layer neural network works on this problem.

“A priori we didn’t believe it would work, and it has better convergence than the fully connected network layers which was the way we thought we had to do it,” he says. “So that’s an interesting insight. The models are starting to become accurate enough and there is a lot of more work to do, but the initial results are encouraging. We’ve got models that are around 87 percent accurate in predicting whether the drug pair will outperform a single drug or not.

“The primary thing we are trying to do is get to a basic model architecture that has a high level of base line predictivity and then we can start optimizing it. Where we are now is encouraging. I think we can do better but we are showing that we have predictive power by building these models but we are also showing we need more data out of the tumors, not different types of data for the same tumor. We need more tumor samples from which we have highly curated molecular assay data and drug screening data,” he says.

There have already been some interesting results: “I have to be careful what I say because we are in the process of publishing those,” he says.

The RAS work is centered at Livermore. A well-known cancer promoter, the RAS family of proteins and pathways has been widely studied. It turns out ANL has the ability to synthesize “nanodiscs” with both wild type RAS and mutated RAS embedded in cell membrane and to obtain images and other biophysical measurements of the RAS proteins. That data is being used to train the simulations.

“The problem that team has been focusing on first is to reproduce computationally the phenomenon known as rafting – rafting is where more than one RAS protein aggregates [in the cell membrane] because that changes the dynamics of the signaling pathway. We have experimental data that shows there’s at least three different states of the membrane. What we are trying to do with the simulations is reproduce that basic biological behavior as a confidence builder exercise before we go to a more complex scenario,” says Stevens.

“One of the challenges is to extract the same kind of observational data from the simulation that you get from the experiment so one of the novel uses of machine learning has been to actually recognizing the different rafting states,” he says.

MAKING THE MODELS
To a large extent the network training concepts are familiar. Train lower levels of the network to basic elements progressing to more specific assemblages of those elements at higher levels. Consider training a network to recognize cars, says Stevens. The lower network levels are trained on things like edges, shadows, colors, vertical and horizontal lines. The upper portion is trained to recognize things like windshields and hubcaps, etc. There’s lots of new and familiar tools to do this. Convoluted networks are hardly new. Generative networks are hot. He and his team are sorting through the toolkit and developing novel ones as well.

Returning to the auto recognizing analogy, he says “[If] now I want it to recognize a sailboat and different types of sailboats, the part of the network that recognized hubcaps and windshields isn’t going to be very useful but the part that learned how to recognize straight edges and colors and corners and shadows is essentially going to be the same. We think the same concept can apply to these drug models, or these drug response models. We are training the network to learn the difference between tumors and regular tissue, training it to recognize the difference between brain cancer and liver cancer, between colorectal cancer and lung cancer, for instance.

“They are recognizing what cancer is, the features that distinguish between normal cells and cancerous cells, and the features that relate to drug effectiveness. The models are trained on drugs, learning drug structures, learning the basic features of how molecules get put together – the kind of basic building blocks, side chains and rings and so on that are in molecules. Those don’t change. We are training on all the known drugs, not just cancer drugs, but all the known drugs, all the chemical compounds that are in the big libraries, so we learn basic chemistry at the bottom so we can combine the networks essentially to predict drug response. They don’t have to relearn most basic features. We just have to in some sense tune them up on the specific patient populations that you are looking for. I think these models could hop from preclinical screening application to actual use and clinical practice fairly quickly.”

FANNING THE DEEP LEARNING FLAME
In some sense, deep learning, close kin to the more familiar data analytics, is a new frontier for much of science. Squeezing it under the HPC umbrella makes some people nervous. 64-bit double precision is nowhere in sight. Low precision doesn’t just work fine, it works better. How is this HPC, ask some observers in the HPC community?

Programing frameworks also present a challenge. Consider this snippet from a recent blog by Intel’s Pradeep Dubey, Intel Fellow and director of Intel’s Parallel Computing Lab, on the convergence of HPC and AI, “…unlike a traditional HPC programmer who is well-versed in low-level APIs for parallel and distributed programming, such as OpenMP or MPI, a typical data scientist who trains deep neural networks on a supercomputer is likely only familiar with high-level scripting-language based frameworks like Caffe or TensorFlow.”

Whatever your view of the computational technology being pressed into service, deep learning is spreading in the science community, and indeed, that is one of the CANDLE project’s goals. All of the major labs have deep learning programs and CANDLE has an aggressive ongoing outreach effort to share learnings.

“The CANDLE project is having a lot of impact [just] through its existence. We have kind of turned on a lot of people at the labs who say, ‘oh machine learning is real and deep learning is real and we are making sure that these future architectures we’re building now, the ones we are deploy in the 2021 timeframe are actually well-suited for running deep learning.’ That is causing the scientific community to say I should think about whether I could use deep learning in my problem,” says Stevens.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

TACC Supercomputer Delves Into Protein Interactions

September 22, 2021

Adenosine triphosphate (ATP) is a compound used to funnel energy from mitochondria to other parts of the cell, enabling energy-driven functions like muscle contractions. For ATP to flow, though, the interaction between t Read more…

The Latest MLPerf Inference Results: Nvidia GPUs Hold Sway but Here Come CPUs and Intel

September 22, 2021

The latest round of MLPerf inference benchmark (v 1.1) results was released today and Nvidia again dominated, sweeping the top spots in the closed (apples-to-apples) datacenter and edge categories. Perhaps more interesti Read more…

Why HPC Storage Matters More Now Than Ever: Analyst Q&A

September 17, 2021

With soaring data volumes and insatiable computing driving nearly every facet of economic, social and scientific progress, data storage is seizing the spotlight. Hyperion Research analyst and noted storage expert Mark No Read more…

GigaIO Gets $14.7M in Series B Funding to Expand Its Composable Fabric Technology to Customers

September 16, 2021

Just before the COVID-19 pandemic began in March 2020, GigaIO introduced its Universal Composable Fabric technology, which allows enterprises to bring together any HPC and AI resources and integrate them with networking, Read more…

What’s New in HPC Research: Solar Power, ExaWorks, Optane & More

September 16, 2021

In this regular feature, HPCwire highlights newly published research in the high-performance computing community and related domains. From parallel programming to exascale to quantum computing, the details are here. Read more…

AWS Solution Channel

Supporting Climate Model Simulations to Accelerate Climate Science

The Amazon Sustainability Data Initiative (ASDI), AWS is donating cloud resources, technical support, and access to scalable infrastructure and fast networking providing high performance computing (HPC) solutions to support simulations of near-term climate using the National Center for Atmospheric Research (NCAR) Community Earth System Model Version 2 (CESM2) and its Whole Atmosphere Community Climate Model (WACCM). Read more…

Cerebras Brings Its Wafer-Scale Engine AI System to the Cloud

September 16, 2021

Five months ago, when Cerebras Systems debuted its second-generation wafer-scale silicon system (CS-2), co-founder and CEO Andrew Feldman hinted of the company’s coming cloud plans, and now those plans have come to fruition. Today, Cerebras and Cirrascale Cloud Services are launching... Read more…

The Latest MLPerf Inference Results: Nvidia GPUs Hold Sway but Here Come CPUs and Intel

September 22, 2021

The latest round of MLPerf inference benchmark (v 1.1) results was released today and Nvidia again dominated, sweeping the top spots in the closed (apples-to-ap Read more…

GigaIO Gets $14.7M in Series B Funding to Expand Its Composable Fabric Technology to Customers

September 16, 2021

Just before the COVID-19 pandemic began in March 2020, GigaIO introduced its Universal Composable Fabric technology, which allows enterprises to bring together Read more…

Cerebras Brings Its Wafer-Scale Engine AI System to the Cloud

September 16, 2021

Five months ago, when Cerebras Systems debuted its second-generation wafer-scale silicon system (CS-2), co-founder and CEO Andrew Feldman hinted of the company’s coming cloud plans, and now those plans have come to fruition. Today, Cerebras and Cirrascale Cloud Services are launching... Read more…

AI Hardware Summit: Panel on Memory Looks Forward

September 15, 2021

What will system memory look like in five years? Good question. While Monday's panel, Designing AI Super-Chips at the Speed of Memory, at the AI Hardware Summit, tackled several topics, the panelists also took a brief glimpse into the future. Unlike compute, storage and networking, which... Read more…

ECMWF Opens Bologna Datacenter in Preparation for Atos Supercomputer

September 14, 2021

In January 2020, the European Centre for Medium-Range Weather Forecasts (ECMWF) – a juggernaut in the weather forecasting scene – signed a four-year, $89-million contract with European tech firm Atos to quintuple its supercomputing capacity. With the deal approaching the two-year mark, ECMWF... Read more…

Quantum Computer Market Headed to $830M in 2024

September 13, 2021

What is one to make of the quantum computing market? Energized (lots of funding) but still chaotic and advancing in unpredictable ways (e.g. competing qubit tec Read more…

Amazon, NCAR, SilverLining Team for Unprecedented Cloud Climate Simulations

September 10, 2021

Earth’s climate is, to put it mildly, not in a good place. In the wake of a damning report from the Intergovernmental Panel on Climate Change (IPCC), scientis Read more…

After Roadblocks and Renewals, EuroHPC Targets a Bigger, Quantum Future

September 9, 2021

The EuroHPC Joint Undertaking (JU) was formalized in 2018, beginning a new era of European supercomputing that began to bear fruit this year with the launch of several of the first EuroHPC systems. The undertaking, however, has not been without its speed bumps, and the Union faces an uphill... Read more…

Ahead of ‘Dojo,’ Tesla Reveals Its Massive Precursor Supercomputer

June 22, 2021

In spring 2019, Tesla made cryptic reference to a project called Dojo, a “super-powerful training computer” for video data processing. Then, in summer 2020, Tesla CEO Elon Musk tweeted: “Tesla is developing a [neural network] training computer called Dojo to process truly vast amounts of video data. It’s a beast! … A truly useful exaflop at de facto FP32.” Read more…

Berkeley Lab Debuts Perlmutter, World’s Fastest AI Supercomputer

May 27, 2021

A ribbon-cutting ceremony held virtually at Berkeley Lab's National Energy Research Scientific Computing Center (NERSC) today marked the official launch of Perlmutter – aka NERSC-9 – the GPU-accelerated supercomputer built by HPE in partnership with Nvidia and AMD. Read more…

Esperanto, Silicon in Hand, Champions the Efficiency of Its 1,092-Core RISC-V Chip

August 27, 2021

Esperanto Technologies made waves last December when it announced ET-SoC-1, a new RISC-V-based chip aimed at machine learning that packed nearly 1,100 cores onto a package small enough to fit six times over on a single PCIe card. Now, Esperanto is back, silicon in-hand and taking aim... Read more…

Enter Dojo: Tesla Reveals Design for Modular Supercomputer & D1 Chip

August 20, 2021

Two months ago, Tesla revealed a massive GPU cluster that it said was “roughly the number five supercomputer in the world,” and which was just a precursor to Tesla’s real supercomputing moonshot: the long-rumored, little-detailed Dojo system. “We’ve been scaling our neural network training compute dramatically over the last few years,” said Milan Kovac, Tesla’s director of autopilot engineering. Read more…

CentOS Replacement Rocky Linux Is Now in GA and Under Independent Control

June 21, 2021

The Rocky Enterprise Software Foundation (RESF) is announcing the general availability of Rocky Linux, release 8.4, designed as a drop-in replacement for the soon-to-be discontinued CentOS. The GA release is launching six-and-a-half months after Red Hat deprecated its support for the widely popular, free CentOS server operating system. The Rocky Linux development effort... Read more…

Intel Completes LLVM Adoption; Will End Updates to Classic C/C++ Compilers in Future

August 10, 2021

Intel reported in a blog this week that its adoption of the open source LLVM architecture for Intel’s C/C++ compiler is complete. The transition is part of In Read more…

Google Launches TPU v4 AI Chips

May 20, 2021

Google CEO Sundar Pichai spoke for only one minute and 42 seconds about the company’s latest TPU v4 Tensor Processing Units during his keynote at the Google I Read more…

Hot Chips: Here Come the DPUs and IPUs from Arm, Nvidia and Intel

August 25, 2021

The emergence of data processing units (DPU) and infrastructure processing units (IPU) as potentially important pieces in cloud and datacenter architectures was Read more…

Leading Solution Providers

Contributors

AMD-Xilinx Deal Gains UK, EU Approvals — China’s Decision Still Pending

July 1, 2021

AMD’s planned acquisition of FPGA maker Xilinx is now in the hands of Chinese regulators after needed antitrust approvals for the $35 billion deal were receiv Read more…

10nm, 7nm, 5nm…. Should the Chip Nanometer Metric Be Replaced?

June 1, 2020

The biggest cool factor in server chips is the nanometer. AMD beating Intel to a CPU built on a 7nm process node* – with 5nm and 3nm on the way – has been i Read more…

HPE Wins $2B GreenLake HPC-as-a-Service Deal with NSA

September 1, 2021

In the heated, oft-contentious, government IT space, HPE has won a massive $2 billion contract to provide HPC and AI services to the United States’ National Security Agency (NSA). Following on the heels of the now-canceled $10 billion JEDI contract (reissued as JWCC) and a $10 billion... Read more…

Julia Update: Adoption Keeps Climbing; Is It a Python Challenger?

January 13, 2021

The rapid adoption of Julia, the open source, high level programing language with roots at MIT, shows no sign of slowing according to data from Julialang.org. I Read more…

Quantum Roundup: IBM, Rigetti, Phasecraft, Oxford QC, China, and More

July 13, 2021

IBM yesterday announced a proof for a quantum ML algorithm. A week ago, it unveiled a new topology for its quantum processors. Last Friday, the Technical Univer Read more…

Intel Launches 10nm ‘Ice Lake’ Datacenter CPU with Up to 40 Cores

April 6, 2021

The wait is over. Today Intel officially launched its 10nm datacenter CPU, the third-generation Intel Xeon Scalable processor, codenamed Ice Lake. With up to 40 Read more…

Frontier to Meet 20MW Exascale Power Target Set by DARPA in 2008

July 14, 2021

After more than a decade of planning, the United States’ first exascale computer, Frontier, is set to arrive at Oak Ridge National Laboratory (ORNL) later this year. Crossing this “1,000x” horizon required overcoming four major challenges: power demand, reliability, extreme parallelism and data movement. Read more…

Intel Unveils New Node Names; Sapphire Rapids Is Now an ‘Intel 7’ CPU

July 27, 2021

What's a preeminent chip company to do when its process node technology lags the competition by (roughly) one generation, but outmoded naming conventions make it seem like it's two nodes behind? For Intel, the response was to change how it refers to its nodes with the aim of better reflecting its positioning within the leadership semiconductor manufacturing space. Intel revealed its new node nomenclature, and... Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire