Learn how new features in AWS ParallelCluster 3 make it easier than ever to manage your workload in the cloud.
Running HPC workloads, typically involves a lot of moving parts. You need hundreds or thousands of compute cores and a job scheduler for keeping them fed. You also need a shared file system that’s tuned for the task, loads of libraries, a fast network, and a head node to make sense of all this. These are just the table stakes, because when you move to the cloud, you’re expecting to do more ambitious things – most likely because you’re a researcher with a problem to solve and a lab full of colleagues waiting for the answer.
Since 2018, AWS ParallelCluster has simplified the orchestration of HPC environments and helped researchers and engineers tackle some of the most ambitious problems facing the world today. Watching customers discover what “infrastructure as code” means in the context of HPC has really propelled us to find new ways to delight them. When a single shell command can create a complex thing like an HPC cluster, and a Lustre file system, and a visualization studio, it leads to more people trying the cloud than ever before, and they’re asking us for new functionality.
So today we’re announcing AWS ParallelCluster 3. Customers, systems integrators, and other builders have told us they want to build end-to-end “recipes” for HPC, spanning the whole gamut from infrastructure to middleware, libraries, and runtime codes. They also explained to us their need for an API interface to interact with ParallelCluster programmatically and create interfaces and services for their users. We worked backwards from this feedback, using thousands of conversations with customers to create what we’re launching today.
There are a lot of changes you’ll notice – large and small. Here’s some highlights before we dive deeper later in this post:
- A new flexible AWS ParallelCluster API – This simplifies building solutions and interfaces on top of ParallelCluster, or including your clusters lifecycle as part of a pipeline. We’ve also changed the CLI to match, so scripted or event-driven workflows are easy.
- Build custom AMIs with EC2 Image Builder – With the introduction of EC2 Image Builder, ParallelCluster now has a way to automate image production processes to align with the automation customers use across the rest of their organization.
- A new configuration file format – configurations now use a YAML format, and each one defines just one cluster. Along with several other changes we think it’ll be easier to keep your cluster configurations organized and readable.
- Simplified network configuration options – we’ve streamlined support for networking to enable the use of private, pre-existing Route 53 zones and provided some more flexibility for how we use Elastic IPs.
- Finer-grained IAM permissions – You can now specify an IAM role or an Instance Profiles, and we let you do that separately for the head node and compute nodes.
- Runtime customization scripts – you can now tweak the pre- and post-install scripts separately for the compute nodes on a live running cluster, and they’ll get updated when you issue the ‘pcluster update’ command.
This is just a short list of what’s new in today’s launch. To find out more, you can watch our HPC Tech Short video, read about it in our HPC Blog or try it out yourself in our hands-on workshop.