Organizations in weather-sensitive industries need highly accurate and near-real-time weather intelligence to make adept business decisions. Many companies in these industries rely on information from DTN, a global data, analytics, and technology company, for that information. To deliver high-level operational intelligence for weather-dependent industries, DTN deploys a suite of proprietary and supplementary weather data and models that deliver sophisticated, high-resolution outputs and require continual processing of vast amounts of data from inputs across the globe. This complexity has historically limited how often forecast engines can update. To optimize its solutions for customers worldwide, DTN sought innovative ways to efficiently increase the frequency and accuracy of its weather forecasting models.

It began testing the high-performance computing (HPC) capabilities of Amazon Web Services (AWS) and running data processing and modeling workloads on Amazon Elastic Compute Cloud (Amazon EC2), a service that provides secure, resizable compute capacity in the cloud. As a proof of concept, DTN used historical data from Hurricane Laura, a category 4 hurricane that made landfall in Louisiana in August 2020. Using HPC on AWS, the company could reliably, accurately, and consistently double the frequency with which it could generate high-resolution weather forecasts.

Delivering More Timely Weather Forecasts Using AWS

DTN specializes in the analysis and delivery of timely weather, agricultural, energy, and commodity market information. While most global weather forecasting organizations run models twice daily, DTN wanted to increase the frequency of forecast modeling to provide customers with intelligence that better reflects how changing weather could impact their operations. “In weather forecasting, we need highly elastic and scalable HPC systems to analyze huge amounts of data globally,” says Doug Chenevert, director of the forecast platform at DTN. “Because weather changes rapidly, a system that can ingest data quickly and run our models frequently is critical for delivering near-real-time insights.” DTN chose to use AWS for the capacity, flexibility, and maturity of its HPC capabilities and services. “Ideally, we want to render high-resolution global forecasts hourly,” says Chenevert. “That kind of output is uncharted territory for weather forecasting, but we’re getting closer by using AWS.”

DTN engaged the AWS team in fall 2020 to explore how to efficiently increase the frequency of forecast outputs. Starting with existing data from Hurricane Laura as a benchmark, DTN developed and tested HPC infrastructures alongside the AWS team over 18 months to optimize the throughput potential of its forecast models. “We found a lot of value in collaborating with the AWS team,” says Brent Shaw, chief weather architect and director of core content services at DTN. “As our engineers optimized our weather science workflows, AWS provided support in optimizing the HPC infrastructure. These changes led to improvements across our weather modeling technology stack.”

In January 2022 DTN began using Amazon EC2 Hpc6a Instances—which are designed specifically for compute-intensive HPC workloads in Amazon EC2—and effectively doubled its high-resolution global weather modeling capacity to four times daily. The company needed a flexible and powerful management tool to increase throughput for its range of HPC workloads, such as simultaneously running atmospheric- and oceanic wave-modeling spaces as well as handling rapid-refresh updates. It started using AWS ParallelCluster, an open-source cluster management tool that makes it easier to deploy and manage HPC clusters on AWS.

Achieving Agile HPC and Improving Performance in the Cloud

Since DTN’s successful proof of concept, the company has moved most of its weather data infrastructure to AWS. “The entire global forecasting solution currently runs on AWS,” says Chenevert. This infrastructure supports a massive amount of data input, storage, and processing; the company estimates that it processes petabytes of data per day. Running tightly coupled HPC workloads presents a challenge with intensive parallel processes running across many instances that must communicate with each other at high speeds. “Weather is the original big data problem,” says Shaw. “Each part needs to know what’s happening in the other parts of the system as it’s happening.” DTN is running HPC workloads in the cloud using Elastic Fabric Adapter (EFA), a network interface for Amazon EC2 instances that customers can use to run applications requiring high levels of internode communications at scale.

Read the full case study to learn more about how DTN Doubles Weather Forecasting Performance Using Amazon EC2 Hpc6a Instances.

Reminder: You can learn a lot from AWS HPC engineers by subscribing to the HPC Tech Short YouTube channel, and following the AWS HPC Blog channel.