Alces Tries Flight on AWS and Life as an HPC SaaS Provider

By John Russell

June 9, 2016

Today, the community of HPC-in-the-cloud solution providers is fairly limited. Giant hyperscalers – notably Amazon (AWS), Google (GCP) and Microsoft (Azure) – remain at the core and keep expanding their HPC resources. Circling around them are HPC ‘services specialists’ plugging clients into cloud providers and striving to ease delivery of HPC-in-the-cloud to make good on the promise of improved efficiencies and cost reductions. Last week a new HPC Software as a Service (SaaS) player popped up on AWS marketplace – Alces Flight, a UK-based company with roots as an HPC integrator.

Predictably the company’s HPC background and the obvious opportunity drove the initiative explains Wil Mayers, director of research and development for Alces, “Organizations are starting to pivot from an on-premise first attitude to a much more cost-sensitive way of working and thinking. That means looking at public clouds. During this transition period we wanted to have a product that could give end users – scientists and engineers – the same sort of look and feel of an HPC cluster irrespective of the platform.”

For the last twelve months Alces has been working with AWS tweaking its offering which remains modest – maximum cluster deployment of around 1,000 cores (more details below) – but with plans for significant compute capacity growth. Like many such companies, the idea is to present a front-end that simplifies HPC application and workflow deployment and plays nicely with Amazon’s many powerful features. Indeed, Alces boasts its HPC cluster provisioning and SaaS offering already has a library of 750 scientific applications available for AWS users.

There are the big hitters you’d expect, GROMACS and NAMD, for example, “but also many others particularly if you look at something like bioinformatics where there are literally hundred and hundreds of different libraries and tools and utilities that people may want to use. A gene sequencing workflow might include 10 or 20 or 30 different tools, all run in sequence,” says Mayers.

The Alces Flight offering has a typical set of provisioning and SaaS components such as a job scheduler, shared file system, and management tools. Its large library of applications and HPC familiarity are important differentiators. So too, says Mayers is Gridware, its repository and orchestrator described in company literature as “applications, libraries and services you need to get working, packaged with care and attention, favoring convention over configuration.”

“The Gridware project is something we host on GitHub. It’s a big library of applications (about 800) and what Gridware is designed to do is when you invoke it and run it on a cluster, it looks at the cluster, looks at the tool you want to install, and you can tell it to go find the source code for the application and dynamically compile it for the environment that you have. It’s aware of dependencies including things like the operating system,” he says. It’s even aware of the particular MPI, interconnect types, and compilers you have chosen.

alces_flight_marketplace_badge_1One addition made to the most recent Gridware release is “a binary option as well for using those lists of instructions and there are compilations options for different packages. We have actually precompiled a lot of the applications for their instance types that are available on AWS,” he says. “So rather than having to compile on the fly – compilation gives you a lot a flexibility but it can also take a little bit of time – we can on a Alces Flight cluster, go and grab the applications and use them directly from an Amazon S3 bucket that lives on AWS.”

There are management tools including a GUI that allows multiple users to connect to the same cluster at the same time. There are also storage management tools, which not only link to an S3 account if a user has one, but also support back ends such as DropBox and Google drive.

While it’s certainly early days, Alces reports already running on the order of 20 clusters on AWS around world. Given the Euro-centricity of its customer base, complying with privacy requirements is critical especially in the bioscience and health care sectors. During the AWS beta testing a couple UK National Health Service clients were working with annonymized data and running only on systems located in the UK – that’s because of strict NHS requirements health remain on “sovereign” ground.

Even in basic research this is the case, Mayers points out. “While there’s a lot of collaborative work, they want the collaboration to be done in a controlled way. What we have found is European users usually want to stay within their region,” he says, adding the AWS’s plans to ramp up capabilities in London later this year will likely open the doors to do more business.

Given the cost pressures on hospitals there, for example, Mayers thinks the cloud will be very attractive. “Using AWS will sometimes mean not only can they reduce their costs by a factor of three or four, they can also get much bigger clusters and get diagnostic work done much faster.”

The opportunities seem enormous and worldwide thinks Alces. Others do as well. There are a few players already in the HPC-in-the-cloud market each with its own flavor of services and value proposition. Cycle Computing and UberCloud are two that come to mind. It will be interesting to watch Alces navigate the new waters. Scaling is currently modest on the AWS Marketplace version of Flight Compute but a higher ceiling is in the works; and users have other options to achieve higher core counts.

“We have a product today that scales to just over a 1,000 cores, so that’s 32 compute nodes. That limits the market,” agrees Mayers. The reason for the constraint, he says, is “We have a fairly straightforward storage deployment option at the moment. We are working with the Intel team for Lustre and the BeeGFS team in Germany and that will allow us to scale above 32 nodes up to 64, and up to some hundreds of nodes within a single cluster environment.”

Mayers stresses that it is just the AWS Marketplace version of Flight Compute which is limited to 32-nodes. The company has tested launching clusters up to 256 nodes in the AWS spot market (which is around 9,000 cores).

“The Marketplace is all about instant-access for users who want to ‘learn by doing’ rather than interacting with a vendor, so we’ve limited that to 32-nodes for the first release to ensure that new users get a good first-time experience,” explains Mayers. “For future Marketplace releases, we’re hoping to package up larger solutions with an appropriate choice of storage technologies to deliver scalable performance in all directions. Our roadmap has the 2016.3 version of Flight Compute available in Marketplace for September this year, but users can launch a Flight Cluster of any size with BeeGFS or Lustre today using appropriate AWS Cloudformation templates.”

Alces supports both Spot Instances and On-Demand instances, giving users needed choice. The company has also leveraged Amazon’s flexibility here: “So the obvious one is an automatically scaling. The job scheduler feeds into AWS and instructs the infrastructure how big it needs to be in terms of compute node.” It shrinks and expands the environment as needed.

“We try to give people the option with Flight when you launch a cluster you can request it to launch on-demand, which give you a guarantee that these instances are going to be permanent. You can also choose to launch a spot instance and Flight will allow you to enter the bid price you want. You can choose how much you want to spend. If a node does get killed because it’s blocked because your bid is too low, the job will return to a queue state on the cluster and can be submitted when the spot price falls again to a level your happy with,” says Mayers.

How all of this cloud-based HPC capability will be used is still evolving. There are, of course, the usual workflows. There is bursting to the cloud when in-house capacity is strained. That said, Mayers expects HPC in the cloud will be used several ways. For example, it might be used as a cost-effective HPC training tool in a university setting. Parking on-premise HPC capability temporarily in the cloud while the actual on-premise infrastructure undergoes change or maintenance is another. There’s no shortage of ideas.

Currently, Alces Flight’s traditional HPC integration is remains by far the largest piece of its business accounting for around 90 percent. Its integration business model, says Mayers differs from many HPC integration players in that it doesn’t buy and sell equipment; instead it works “mostly tier one” vendors and connects customers directly with them. The revenue comes from services (installation, integration, life cycle servicer, etc.).

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Q&A with Altair CEO James Scapa, an HPCwire Person to Watch in 2021

May 14, 2021

Chairman, CEO and co-founder of Altair James R. Scapa closed several acquisitions for the company in 2020, including the purchase and integration of Univa and Ellexus. Scapa founded Altair more than 35 years ago with two Read more…

HLRS HPC Helps to Model Muscle Movements

May 13, 2021

The growing scale of HPC is allowing simulation of more and more complex systems at greater detail than ever before, particularly in the biological research spheres. Now, researchers at the University of Stuttgart are le Read more…

Behind the Met Office’s Procurement of a Billion-Dollar Microsoft System

May 13, 2021

The UK’s national weather service, the Met Office, caused shockwaves of curiosity a few weeks ago when it formally announced that its forthcoming billion-dollar supercomputer – expected to be the most powerful weather and climate-focused supercomputer in the world when it launches in 2022... Read more…

AMD, GlobalFoundries Commit to $1.6 Billion Wafer Supply Deal

May 13, 2021

AMD plans to purchase $1.6 billion worth of wafers from GlobalFoundries in the 2022 to 2024 timeframe, the chipmaker revealed today (May 13) in an SEC filing. In the face of global semiconductor shortages and record-high demand, AMD is renegotiating its Wafer Supply Agreement and bumping up capacity. Read more…

Hyperion Offers Snapshot of Quantum Computing Market

May 13, 2021

The nascent quantum computer (QC) market will grow 27 percent annually (CAGR) reaching $830 million in 2024 according to an update provided today by analyst firm Hyperion Research at the HPC User Forum being held this we Read more…

AWS Solution Channel

Numerical weather prediction on AWS Graviton2

The Weather Research and Forecasting (WRF) model is a numerical weather prediction (NWP) system designed to serve both atmospheric research and operational forecasting needs. Read more…

Hyperion: HPC Server Market Ekes 1 Percent Gain in 2020, Storage Poised for ‘Tipping Point’

May 12, 2021

The HPC User Forum meeting taking place virtually this week (May 11-13) kicked off with Hyperion Research’s market update, covering the 2020 period. Although the HPC server market had been facing a 6.7 percent COVID-re Read more…

Behind the Met Office’s Procurement of a Billion-Dollar Microsoft System

May 13, 2021

The UK’s national weather service, the Met Office, caused shockwaves of curiosity a few weeks ago when it formally announced that its forthcoming billion-dollar supercomputer – expected to be the most powerful weather and climate-focused supercomputer in the world when it launches in 2022... Read more…

AMD, GlobalFoundries Commit to $1.6 Billion Wafer Supply Deal

May 13, 2021

AMD plans to purchase $1.6 billion worth of wafers from GlobalFoundries in the 2022 to 2024 timeframe, the chipmaker revealed today (May 13) in an SEC filing. In the face of global semiconductor shortages and record-high demand, AMD is renegotiating its Wafer Supply Agreement and bumping up capacity. Read more…

Hyperion Offers Snapshot of Quantum Computing Market

May 13, 2021

The nascent quantum computer (QC) market will grow 27 percent annually (CAGR) reaching $830 million in 2024 according to an update provided today by analyst fir Read more…

Hyperion: HPC Server Market Ekes 1 Percent Gain in 2020, Storage Poised for ‘Tipping Point’

May 12, 2021

The HPC User Forum meeting taking place virtually this week (May 11-13) kicked off with Hyperion Research’s market update, covering the 2020 period. Although Read more…

IBM Debuts Qiskit Runtime for Quantum Computing; Reports Dramatic Speed-up

May 11, 2021

In conjunction with its virtual Think event, IBM today introduced an enhanced Qiskit Runtime Software for quantum computing, which it says demonstrated 120x spe Read more…

AMD Chipmaker TSMC to Use AMD Chips for Chipmaking

May 8, 2021

TSMC has tapped AMD to support its major manufacturing and R&D workloads. AMD will provide its Epyc Rome 7702P CPUs – with 64 cores operating at a base cl Read more…

Fast Pass Through (Some of) the Quantum Landscape with ORNL’s Raphael Pooser

May 7, 2021

In a rather remarkable way, and despite the frequent hype, the behind-the-scenes work of developing quantum computing has dramatically accelerated in the past f Read more…

IBM Research Debuts 2nm Test Chip with 50 Billion Transistors

May 6, 2021

IBM Research today announced the successful prototyping of the world's first 2 nanometer chip, fabricated with silicon nanosheet technology on a standard 300mm Read more…

AMD Chipmaker TSMC to Use AMD Chips for Chipmaking

May 8, 2021

TSMC has tapped AMD to support its major manufacturing and R&D workloads. AMD will provide its Epyc Rome 7702P CPUs – with 64 cores operating at a base cl Read more…

Intel Launches 10nm ‘Ice Lake’ Datacenter CPU with Up to 40 Cores

April 6, 2021

The wait is over. Today Intel officially launched its 10nm datacenter CPU, the third-generation Intel Xeon Scalable processor, codenamed Ice Lake. With up to 40 Read more…

Julia Update: Adoption Keeps Climbing; Is It a Python Challenger?

January 13, 2021

The rapid adoption of Julia, the open source, high level programing language with roots at MIT, shows no sign of slowing according to data from Julialang.org. I Read more…

CERN Is Betting Big on Exascale

April 1, 2021

The European Organization for Nuclear Research (CERN) involves 23 countries, 15,000 researchers, billions of dollars a year, and the biggest machine in the worl Read more…

HPE Launches Storage Line Loaded with IBM’s Spectrum Scale File System

April 6, 2021

HPE today launched a new family of storage solutions bundled with IBM’s Spectrum Scale Erasure Code Edition parallel file system (description below) and featu Read more…

10nm, 7nm, 5nm…. Should the Chip Nanometer Metric Be Replaced?

June 1, 2020

The biggest cool factor in server chips is the nanometer. AMD beating Intel to a CPU built on a 7nm process node* – with 5nm and 3nm on the way – has been i Read more…

Saudi Aramco Unveils Dammam 7, Its New Top Ten Supercomputer

January 21, 2021

By revenue, oil and gas giant Saudi Aramco is one of the largest companies in the world, and it has historically employed commensurate amounts of supercomputing Read more…

Quantum Computer Start-up IonQ Plans IPO via SPAC

March 8, 2021

IonQ, a Maryland-based quantum computing start-up working with ion trap technology, plans to go public via a Special Purpose Acquisition Company (SPAC) merger a Read more…

Leading Solution Providers

Contributors

AMD Launches Epyc ‘Milan’ with 19 SKUs for HPC, Enterprise and Hyperscale

March 15, 2021

At a virtual launch event held today (Monday), AMD revealed its third-generation Epyc “Milan” CPU lineup: a set of 19 SKUs -- including the flagship 64-core, 280-watt 7763 part --  aimed at HPC, enterprise and cloud workloads. Notably, the third-gen Epyc Milan chips achieve 19 percent... Read more…

Can Deep Learning Replace Numerical Weather Prediction?

March 3, 2021

Numerical weather prediction (NWP) is a mainstay of supercomputing. Some of the first applications of the first supercomputers dealt with climate modeling, and Read more…

Livermore’s El Capitan Supercomputer to Debut HPE ‘Rabbit’ Near Node Local Storage

February 18, 2021

A near node local storage innovation called Rabbit factored heavily into Lawrence Livermore National Laboratory’s decision to select Cray’s proposal for its CORAL-2 machine, the lab’s first exascale-class supercomputer, El Capitan. Details of this new storage technology were revealed... Read more…

African Supercomputing Center Inaugurates ‘Toubkal,’ Most Powerful Supercomputer on the Continent

February 25, 2021

Historically, Africa hasn’t exactly been synonymous with supercomputing. There are only a handful of supercomputers on the continent, with few ranking on the Read more…

GTC21: Nvidia Launches cuQuantum; Dips a Toe in Quantum Computing

April 13, 2021

Yesterday Nvidia officially dipped a toe into quantum computing with the launch of cuQuantum SDK, a development platform for simulating quantum circuits on GPU-accelerated systems. As Nvidia CEO Jensen Huang emphasized in his keynote, Nvidia doesn’t plan to build... Read more…

New Deep Learning Algorithm Solves Rubik’s Cube

July 25, 2018

Solving (and attempting to solve) Rubik’s Cube has delighted millions of puzzle lovers since 1974 when the cube was invented by Hungarian sculptor and archite Read more…

The History of Supercomputing vs. COVID-19

March 9, 2021

The COVID-19 pandemic poses a greater challenge to the high-performance computing community than any before. HPCwire's coverage of the supercomputing response t Read more…

Microsoft to Provide World’s Most Powerful Weather & Climate Supercomputer for UK’s Met Office

April 22, 2021

More than 14 months ago, the UK government announced plans to invest £1.2 billion ($1.56 billion) into weather and climate supercomputing, including procuremen Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire