Cycle Spins Up 50,000-core Cluster in Amazon Cloud

By Tiffany Trader

April 24, 2012

The case for utility supercomputing just got a lot bigger, literally. Cycle Computing has created a 50,000 core virtual supercomputer to assist in the development of novel drug compounds for cancer research. The cluster, codenamed Naga, sits in the Amazon infrastructure and is the biggest utility supercomputer yet. Using this mega-cluster, computational chemistry outfit Schrödinger was able to analyze 21 million drug compounds in just 3 hours for less than $4,900.

molecules

Developing a real compound, or assay, is very expensive, so before you do this you need to test all leads on a computer. Schrödinger and research partner Nimbus Discovery are working to identify important targets that have so-far been unsuccessful. They are looking for “hits” – a process that occurs very early on in the drug discovery cycle. After the hit stage comes the hit-to-lead phase and then lead optimization. Lead optimization produces a development candidate for human trials. But the process has to start somewhere, and that’s where virtual screening comes in. It’s the foundation for 2 to 5 years of discovery.

Schrödinger uses its proprietary docking application, Glide, to virtually screen different compounds against a potential cancer drug target. Out of a huge number of compounds, the computer model will whittle the initial pool down to the most worthy candidates. With Glide, as with most computer models, there’s a trade-off between accuracy and speed. Shortcuts are commonly employed to accommodate resource and time limitations.

Glide employs a series of progressive refinements, each an order of magnitude more computationally-intensive than the last. The first pass is performed with the fastest, least-accurate Glide algorithm, HTVS, which stands for, high-throughput virtual screening. About 10 percent of the initial candidates make it through to the next round, which is called SP, for standard precision. The third and final phase is XP for extra precision. This round takes 10 percent of the compounds from the previous run and outputs only the most worthy drug candidates, the ones most capable of affecting the targeted disease proteins.

The same tradeoff that allows researchers to analyze more compounds at a faster rate also leads to a significant number of false negatives and positives because the lower-quality algorithm may fail to identify good candidate compounds, while letting false positives slip by. The greater risk here is that potential blockbuster drugs are just passed over.

The utility nature of the Amazon supercomputer allows the scientists to skip the first step and move right to the second, more accurate mode. It also allows them to increase their compound set by a factor of three. So while normally they input 6 to 7 million compounds, they can now start with 21 million. Applying the higher-quality algorithm to a larger compound set reduces the problem of false negatives. The researchers are then able to identify the active compounds that would otherwise fall through the cracks.

While Schrödinger makes heavy use of its internal clusters, it requires additional resources for particularly compute-intensive workloads. With the Naga cluster, Schrödinger researchers were able to run this exceptionally large workload in record time, 21 million compounds and confirmations in just 3 hours. By comparison, running the same job on their internal 400-core cluster would take about 275 hours. Initial data sets are on the order of tens of gigabytes of molecule data, and depending on Internet bandwidth, uploading the data can take about 5 to 6 hours. Since the library of compounds is largely a static data set, it only needs to be updated once every six months or so.

“This project reflects the major trends we are seeing in medicine today. It’s the age of analytics and simulation, meaning big data and big compute,” remarks Cycle Computing CEO Jason Stowe. “We’re also seeing requirements on time-to-market and being capital-efficient. Building a 50,000 core infrastructure is a $20-30 million endeavor,” he adds.

Naga AWS deployment map
A map of the AWS compute resources harnessed during the Naga run

 

Next page >>>

A cutting-edge scientific research outfit, Schrödinger still has to face the economic realities that go along with being a small business. CEO Ramy Farid agrees that purchasing a system like this outright would be extremely expensive. Even more to the point, virtual screening is done sporadically. Farid estimates that over the course of a year, they do maybe 25 virtual screens at 3 hours each. While they do have other computational work to perform, it’s not enough to justify additional in-house resources, and certainly not a supercomputer of this ilk.

Farid points out that this dramatic increase in the number of processors lets you do better science. “It’s been like that since computing started,” he notes. As an example, Farid recalls the days when scientists had to intentionally omit hydrogen atoms on structures because computers just weren’t fast enough.

Schrödinger also uses the Cycle-based Amazon cloud to offload some of its lead optimization work, which involves doing calculations to predict binding affinity. Although not at the scale of the virtual screening process, lead optimization is still quite compute-intensive. Farid characterizes this work as the holy grail of computational chemistry, and using the Cycle setup, they’ve been able to take work that would require several months on a cluster down to a weekend on the cloud.

This speaks to the paradigm shift that Stowe is so passionate about. Despite the exponential advances in compute power driven by Moore’s Law, access to HPC resources is still one of the biggest constraints in research. Utility computing is creating a new dynamic by providing virtually unlimited computing power on-demand, and the user only has to pay for what they use. Researchers are accustomed to having to frame their questions according to the resources they have available, but the new model allows researchers to ask the most important questions, the ones that will actually move the science forward.

This 50,000-core cloud is the largest Cycle has constructed for a client, but the HPC software company has created a number of notable clusters. Last year, they did a 10,000-core run with Genentech, and a 30,000-core run with a top 5 pharmaceutical firm. Stowe points out that those organizations, however, were quite large, so theoretically-speaking, could have purchased a cluster of that size outright. What makes Schrödinger such an ideal use case, according to Stowe, is how Cycle and Amazon were able to provide a resource that otherwise would have been out of reach.

“Cycle Cloud automates the process of turning raw infrastructure into usual HPC environments,” says Stowe. “It’s like using a TOP500 supercomputer for a few hours and then turning it off.”

 Naga: Facts and Figures
 Metric  Count
 Compute Hours of Work  109,927 hours
 Compute Days of Work  4,580 days
 Compute Years of Work   12.55 years
 Ligand Count  ~21 million ligands
 Run Time  ~3 hours
 Grand Total Cost at Peak: $4,828/hour    ( $0.09 / core / hour )

 

 

 

 

 

 

 

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

UCSD Web-based Tool Tracking CA Wildfires Generates 1.5M Views

October 16, 2017

Tracking the wildfires raging in northern CA is an unpleasant but necessary part of guiding efforts to fight the fires and safely evacuate affected residents. One such tool – Firemap – is a web-based tool developed b Read more…

By John Russell

Exascale Imperative: New Movie from HPE Makes a Compelling Case

October 13, 2017

Why is pursuing exascale computing so important? In a new video – Hewlett Packard Enterprise: Eighteen Zeros – four HPE executives, a prominent national lab HPC researcher, and HPCwire managing editor Tiffany Trader Read more…

By John Russell

Intel Delivers 17-Qubit Quantum Chip to European Research Partner

October 10, 2017

On Tuesday, Intel delivered a 17-qubit superconducting test chip to research partner QuTech, the quantum research institute of Delft University of Technology (TU Delft) in the Netherlands. The announcement marks a major milestone in the 10-year, $50-million collaborative relationship with TU Delft and TNO, the Dutch Organization for Applied Research, to accelerate advancements in quantum computing. Read more…

By Tiffany Trader

HPE Extreme Performance Solutions

“Lunch & Learn” to Explore the Growing Applications of Genomic Analytics

In the digital age of medicine, healthcare providers are rapidly transforming their approach to patient care. Traditional technologies are no longer sufficient to process vast quantities of medical data (including patient histories, treatment plans, diagnostic reports, and more), challenging organizations to invest in a new style of IT to enable faster and higher-quality care. Read more…

Fujitsu Tapped to Build 37-Petaflops ABCI System for AIST

October 10, 2017

Fujitsu announced today it will build the long-planned AI Bridging Cloud Infrastructure (ABCI) which is set to become the fastest supercomputer system in Japan and will begin operation in fiscal 2018 (starts in April). A Read more…

By John Russell

Intel Delivers 17-Qubit Quantum Chip to European Research Partner

October 10, 2017

On Tuesday, Intel delivered a 17-qubit superconducting test chip to research partner QuTech, the quantum research institute of Delft University of Technology (TU Delft) in the Netherlands. The announcement marks a major milestone in the 10-year, $50-million collaborative relationship with TU Delft and TNO, the Dutch Organization for Applied Research, to accelerate advancements in quantum computing. Read more…

By Tiffany Trader

Fujitsu Tapped to Build 37-Petaflops ABCI System for AIST

October 10, 2017

Fujitsu announced today it will build the long-planned AI Bridging Cloud Infrastructure (ABCI) which is set to become the fastest supercomputer system in Japan Read more…

By John Russell

HPC Chips – A Veritable Smorgasbord?

October 10, 2017

For the first time since AMD's ill-fated launch of Bulldozer the answer to the question, 'Which CPU will be in my next HPC system?' doesn't have to be 'Whichever variety of Intel Xeon E5 they are selling when we procure'. Read more…

By Dairsie Latimer

Delays, Smoke, Records & Markets – A Candid Conversation with Cray CEO Peter Ungaro

October 5, 2017

Earlier this month, Tom Tabor, publisher of HPCwire and I had a very personal conversation with Cray CEO Peter Ungaro. Cray has been on something of a Cinderell Read more…

By Tiffany Trader & Tom Tabor

Intel Debuts Programmable Acceleration Card

October 5, 2017

With a view toward supporting complex, data-intensive applications, such as AI inference, video streaming analytics, database acceleration and genomics, Intel i Read more…

By Doug Black

OLCF’s 200 Petaflops Summit Machine Still Slated for 2018 Start-up

October 3, 2017

The Department of Energy’s planned 200 petaflops Summit computer, which is currently being installed at Oak Ridge Leadership Computing Facility, is on track t Read more…

By John Russell

US Exascale Program – Some Additional Clarity

September 28, 2017

The last time we left the Department of Energy’s exascale computing program in July, things were looking very positive. Both the U.S. House and Senate had pas Read more…

By Alex R. Larzelere

US Coalesces Plans for First Exascale Supercomputer: Aurora in 2021

September 27, 2017

At the Advanced Scientific Computing Advisory Committee (ASCAC) meeting, in Arlington, Va., yesterday (Sept. 26), it was revealed that the "Aurora" supercompute Read more…

By Tiffany Trader

How ‘Knights Mill’ Gets Its Deep Learning Flops

June 22, 2017

Intel, the subject of much speculation regarding the delayed, rewritten or potentially canceled “Aurora” contract (the Argonne Lab part of the CORAL “ Read more…

By Tiffany Trader

Reinders: “AVX-512 May Be a Hidden Gem” in Intel Xeon Scalable Processors

June 29, 2017

Imagine if we could use vector processing on something other than just floating point problems.  Today, GPUs and CPUs work tirelessly to accelerate algorithms Read more…

By James Reinders

NERSC Scales Scientific Deep Learning to 15 Petaflops

August 28, 2017

A collaborative effort between Intel, NERSC and Stanford has delivered the first 15-petaflops deep learning software running on HPC platforms and is, according Read more…

By Rob Farber

Oracle Layoffs Reportedly Hit SPARC and Solaris Hard

September 7, 2017

Oracle’s latest layoffs have many wondering if this is the end of the line for the SPARC processor and Solaris OS development. As reported by multiple sources Read more…

By John Russell

US Coalesces Plans for First Exascale Supercomputer: Aurora in 2021

September 27, 2017

At the Advanced Scientific Computing Advisory Committee (ASCAC) meeting, in Arlington, Va., yesterday (Sept. 26), it was revealed that the "Aurora" supercompute Read more…

By Tiffany Trader

Google Releases Deeplearn.js to Further Democratize Machine Learning

August 17, 2017

Spreading the use of machine learning tools is one of the goals of Google’s PAIR (People + AI Research) initiative, which was introduced in early July. Last w Read more…

By John Russell

Graphcore Readies Launch of 16nm Colossus-IPU Chip

July 20, 2017

A second $30 million funding round for U.K. AI chip developer Graphcore sets up the company to go to market with its “intelligent processing unit” (IPU) in Read more…

By Tiffany Trader

GlobalFoundries Puts Wind in AMD’s Sails with 12nm FinFET

September 24, 2017

From its annual tech conference last week (Sept. 20), where GlobalFoundries welcomed more than 600 semiconductor professionals (reaching the Santa Clara venue Read more…

By Tiffany Trader

Leading Solution Providers

Amazon Debuts New AMD-based GPU Instances for Graphics Acceleration

September 12, 2017

Last week Amazon Web Services (AWS) streaming service, AppStream 2.0, introduced a new GPU instance called Graphics Design intended to accelerate graphics. The Read more…

By John Russell

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

EU Funds 20 Million Euro ARM+FPGA Exascale Project

September 7, 2017

At the Barcelona Supercomputer Centre on Wednesday (Sept. 6), 16 partners gathered to launch the EuroEXA project, which invests €20 million over three-and-a-half years into exascale-focused research and development. Led by the Horizon 2020 program, EuroEXA picks up the banner of a triad of partner projects — ExaNeSt, EcoScale and ExaNoDe — building on their work... Read more…

By Tiffany Trader

Cray Moves to Acquire the Seagate ClusterStor Line

July 28, 2017

This week Cray announced that it is picking up Seagate's ClusterStor HPC storage array business for an undisclosed sum. "In short we're effectively transitioning the bulk of the ClusterStor product line to Cray," said CEO Peter Ungaro. Read more…

By Tiffany Trader

Delays, Smoke, Records & Markets – A Candid Conversation with Cray CEO Peter Ungaro

October 5, 2017

Earlier this month, Tom Tabor, publisher of HPCwire and I had a very personal conversation with Cray CEO Peter Ungaro. Cray has been on something of a Cinderell Read more…

By Tiffany Trader & Tom Tabor

Intel Launches Software Tools to Ease FPGA Programming

September 5, 2017

Field Programmable Gate Arrays (FPGAs) have a reputation for being difficult to program, requiring expertise in specialty languages, like Verilog or VHDL. Easin Read more…

By Tiffany Trader

IBM Advances Web-based Quantum Programming

September 5, 2017

IBM Research is pairing its Jupyter-based Data Science Experience notebook environment with its cloud-based quantum computer, IBM Q, in hopes of encouraging a new class of entrepreneurial user to solve intractable problems that even exceed the capabilities of the best AI systems. Read more…

By Alex Woodie

Intel, NERSC and University Partners Launch New Big Data Center

August 17, 2017

A collaboration between the Department of Energy’s National Energy Research Scientific Computing Center (NERSC), Intel and five Intel Parallel Computing Cente Read more…

By Linda Barney

  • arrow
  • Click Here for More Headlines
  • arrow
Share This