In January 2020, the European Centre for Medium-Range Weather Forecasts (ECMWF) – a juggernaut in the weather forecasting scene – signed a four-year, $89-million contract with European tech firm Atos to quintuple its supercomputing capacity. With the deal approaching the two-year mark, ECMWF has announced that the datacenter that will host the center’s new supercomputers has opened in Bologna. This new facility, along with new ECMWF offices in Bonn, also marks the ECMWF’s expansion from its sole site in the UK into a multi-site organization with international presence.
Through the 2020 deal with Atos, the ECMWF is acquiring a BullSequana XH2000-based supercomputer with components from AMD, Mellanox and DDN. The system is divided into four “self-sufficient … clusters,” all of which are connected to high-performance storage. Each cluster has 1,872 compute nodes and 112 general purpose nodes spread across 22 racks (predominantly water-cooled). Each node has two AMD Epyc 7742 (64-core, Rome) processors and either 256GB of memory (compute nodes) or 512GB (general purpose nodes), for a total of over two petabytes of memory.
The high-performance storage system will contribute 700 terabytes of flash storage and 5.4 petabytes of hard disk storage for time-critical operations, as well as 13 petabytes of storage for research applications. Later, it will be joined by the ECMWF’s Data Handling System (DHS), which will be transported from Reading, UK to Bologna around the front half of 2022.
This supercomputer will be housed in the newly opened datacenter, which took over two years of construction to complete and is sited on a repurposed tobacco factory that has now become the Tecnopolo di Bologna campus. The ECMWF said that while the pandemic has hindered some of its personnel transfers, other ECMWF staff are already working in the datacenter – with more to come. The same campus will also eventually play host to EuroHPC’s Leonardo supercomputer, if all goes according to plan.
ECMWF said in its new announcement that the system “will increase sustained performance by a factor of about five” compared to its current HPC capabilities. Its existing systems were procured through a 2016 deal with Cray, which upgraded and expanded a pair of Cray systems initially acquired in 2013, delivering around 8.5 peak petaflops via a pair of XC40s. Those XC40s will continue to host the ECMWF’s operational service calculations until May 2022, when the Atos system will take over those operations for a period of four years.
Initially, the system was expected to become operational this year following installation in 2020. However, understandably, the ECMWF appears to be running a bit behind schedule: the system was installed in summer of this year and Atos is now targeting a mid-2022 start to operations for the system. Atos said that the system is “now undergoing a series of user acceptance, operational and reliability tests[.]”
The new supercomputer capacity will enable a number of important changes for the ECMWF’s operations, including crucial progress towards its goal of improving the horizontal resolution of its forecast from 18km to 10km and increasing the number of vertical resolution layers from 91 to 137. The ECMWF also has an ambitious goal of a 5km ensemble forecast set for 2025.
When was the contract was announced, Florence Rabier, ECMWF’s Director General, said that thanks to the new technology, “we will now be able to run higher resolution forecasts in under an hour, meaning better information will be shared with our Member States even faster to enable much improved weather forecasts as they combine this enhanced information with their own data and predictions. As governments and society continue to grapple with the impacts of increasingly severe weather, we are also proud to be relying on a supercomputer designed to maximise energy efficiency.”
The announcement of the datacenter’s opening was accompanied by another announcement: the opening of the ECMWF’s offices in Bonn, Germany.
“It is an important moment in the life of ECMWF as we formally become a multi-site organisation, with our head offices in the UK, our new data centre in Bologna, Italy, and our new offices in Bonn, Germany,” Rabier said. “As an intergovernmental organisation, ECMWF’s dedication and loyalty are to its Member States. Over the years, however, ECMWF has also developed a close and extremely constructive working partnership with the European Union. When you consider that most of ECMWF’s Member States are also members of the EU, this partnership makes considerable sense. It allows EU and ECMWF Member States to be efficient by avoiding duplication of efforts and spending in areas of work where synergies are paramount.”
Header image: the new ECMWF datacenter. Image courtesy of ECMWF.