When weather threatens drilling rigs, refineries, and other energy facilities, oil and gas companies want to move fast to protect personnel and equipment. And for firms that trade commodity shares in oil, precious metals, crops, and livestock, the weather can significantly impact their buy-sell decisions. To limit damage, these companies need the earliest possible notice before a major storm strikes. That’s the challenge Maxar Technologies set out to solve.
Historically, many industries have relied on reports generated by the on-premises supercomputer operated by the National Oceanic and Atmospheric Administration. However, the weather predictions take an average of 100 minutes to process global data. Similar to how NASA has expanded its partnerships with private firms to acquire commercial space hardware and services, the processing and delivery of critical weather data products could also be effectively commercialized.
To resolve this issue, Maxar sought to significantly reduce the time needed to generate numerical weather predictions. So Maxar turned to Amazon Web Services.
Cloud HPC Achieves the «Impossible»
To complement the enhanced computing and networking, the application uses Amazon FSx for Lustre to accelerate the read/write throughput of the application. Maxar also takes advantage of AWS ParallelCluster, an open source cluster management tool that makes it easy to deploy HPC clusters with a simple text file that automatically models and provisions resources. Initially, Maxar designed a cloud HPC cluster with 234 Amazon EC2 instances capable of producing a numerical weather prediction forecast in roughly 53 minutes, just about half the 100 minutes that the NOAA supercomputer takes to complete the same forecast. This accomplished Maxar’s initial performance goal, so the team set its eyes on enhancing the design to reduce cost.
Using EFA networking, Maxar reduced that cluster from 234 c5.18xlarge instances to just 156 c5n. 18xlarge instances, which was driven by the ability of the C5n instances to communicate at 100 Gbps network speeds. The team’s new configuration can now produce a forecast 58%faster than NOAA’s supercomputer. Additional testing and optimization with AWS revealed Maxar could complete a forecast in under 30 minutes.
With further system tuning, Maxar projects it can cut its processing time by an additional 25%. Having achieved its performance goal, Maxar next focused on delivering the service profitably. Maxar needed to keep the cost of its weather application as low as possible to compete with the free, yet slower, service that NOAA provides. By using AWS ParallelCluster with Amazon EC2 C5n instances and EFA, Maxar generates the same computing power while decreasing the number of clustered servers by 33 percent.
The environment automatically spins up when weather data becomes available and then quickly shuts down until a new dataset is available, using numerous AWS services to orchestrate a highly scalable, redundant, and fault-tolerant workflow. Thanks to the success of the application, Maxar clients can now take proactive measures earlier when assets and personnel are threatened by extreme weather.