AWS Batch and HPC at Amazon Web Services
Ian Colle joined AWS as the General Manager for AWS Batch and HPC in November 2017. Prior to AWS, he served as the Global Director of Software Engineering for the Ceph Storage team at Red Hat. Mr. Colle joined Red Hat in April 2014 via the Inktank acquisition, where he held a similar role. Prior to Inktank, he was the Exascale Storage I/O Program Manager at Intel, which he joined as part of the Whamcloud acquisition. Prior to Whamcloud, Mr. Colle spent several years in a variety of software engineering leadership positions throughout the aerospace industry. Mr. Colle has served in the US Navy and Navy Reserve for over 25 years as an Intelligence Officer and Russian Linguist. He holds a BA Economics from the University of Illinois, MCIS & MTEL from University of Denver, MA Philosophy from Denver Seminary, and MBA from Massachusetts Institute of Technology.
HPCwire: Amazon has been building HPC cloud use cases for about a decade now, how has the landscape for HPC in the cloud changed? What role is AI playing?
Ian Colle: It’s hard to believe how far we’ve come since the initial launches of S3 and EC2 back in 2006. Each year AWS expands the number of services we offer, and increases the capabilities of our core compute, network, and storage technologies making them more performant for HPC workloads. One of the new services I’m most excited about is Amazon SageMaker – our fully managed machine learning offering. You can integrate your HPC results into a Jupyter notebook and use your favorite algorithms within TensorFlow or MXNet frameworks. The combination of HPC and AI/ML/DL (pick your favorite acronym) will drive future research and innovations in new and exciting (and probably unexpected) directions.
HPCwire: In the last 12-18 months, there’s been something of a race between the tier one cloud providers to employ cutting-edge HPC technology. What’s behind this quickening?
There’s a sense that we’ve reached a tipping point in the market. When I attended SC conferences in the past, you rarely heard people discussing cloud (unless it was dismissively). At SC17, the majority of the attendees I spoke with either had a nascent HPC in the cloud strategy, or were looking for help to create one. It has a similar feel to the early days of enterprise workloads moving off premises in 2006/2007. As we address more and more areas of concern for potential HPC customers (performance, security, data locality, etc.), we can gain their trust that we understand their needs and can satisfy their rigorous requirements.
HPCwire: What role will cloud/AWS play in exascale computing?
“Exascale computing” can be achieved in multiple ways. The less interesting path is essentially just a cluster of petascale systems. The exciting path is one of new innovative hardware architectures and interconnects, as I think we all hope to see out of Aurora. Regardless of the path taken, one could imagine the results from an exascale workload being piped into a data lake in the cloud, where they could be fed into a machine learning algorithm. The real excitement for me comes from seeing where we can allow the algorithms to learn from repeated simulations and to steer these massive computational resources in completely new and innovative directions. As an aside, it might be a fun academic exercise to quiesce all of AWS to run LINPACK and see where we come in, but I’m pretty certain our customers wouldn’t appreciate it.
HPCwire: What do you hope to see from the HPC community in the coming year?
In the coming year I hope to see a continuing growth of openness to running more intense and varied HPC workloads in the cloud. I hope that we can reach a point where scientific research and industrial advancements are no longer stuck waiting in queues on local on-premises machines for days and weeks. I get excited thinking about the positive impact that accelerating these workloads could have on our society. For example, DigitalGlobe, the global leader in commercial high-resolution satellite imagery, is using AWS ML technologies to push the boundaries of what’s possible with large data sets, allowing their customers to better understand the changing planet to save time, resources, and lives.
HPCwire: Outside of the professional sphere, what can you tell us about yourself – personal life, family, background, hobbies, etc.? Is there anything about you your colleagues might be surprised to learn?
My wife and I have lived in Colorado for over 20 years. We have four kids (three boys, one girl). We love the outdoors – hiking, snowshoeing, camping, and skiing. I enjoy reading and trail running to help me unwind. My colleagues might be surprised to learn that my first exposure to a computer as a kid was working on the PLATO system at the University of Illinois or that I spoke fluent Russian (I’m a bit rusty these days).