Few areas of scientific research today are as important, or provoke as visceral a response as global warming. Climate scientists around the world agree that the average global temperature could rise by 1.4 to 5.8 degrees Celsius by the end of the century. And scientists — as well as government leaders, economists, and increasingly, the public at large — recognize that warming could bring about far-reaching and unpredictable environmental, social, and economic consequences.

To address these concerns, global leaders and policymakers need comprehensive, objective information on the global climate system, and the role of human activity in climate change. In 1988, the World Meteorological Organization (WMO) and the United Nations Environment Programme (UNEP) established the Intergovernmental Panel on Climate Change (IPCC) to scientifically assess and understand the global risk of human-induced climate change, its potential impact, and options for mitigation.

The Department of Energy (DOE)'s Oak Ridge National Laboratory (ORNL) has played a key role in the IPCC's fourth assessment, due in 2007. In fact, one third of the climate simulation work in the United States has been done at ORNL. These simulations, which span 11,000 years, look at climate as a function of atmospheric carbon dioxide and other greenhouse gases. Among other things, the results confirm that smaller concentrations of these gases result in less global warming.

However, IPCC scientists are continually seeking to refine their models to incorporate more atmospheric, oceanic and other processes, and to produce more accurate, higher-resolution simulations. For example, researchers now are working to incorporate dynamic ecological and chemical processes into their simulations. In preparation for the fifth IPCC assessment due in 2013, the DOE and the National Science Foundation's National Center for Atmospheric Research (NCAR) have designated a Climate Science Computational End Station at ORNL. With millions of hours of dedicated access to some of the largest, most advanced high-performance computing (HPC) systems in the world, IPCC researchers will be able to apply greater computational resources to climate problems than ever before, and develop the next generation of climate system simulations.

Understanding Climate Modeling

To prepare for the fifth IPCC assessment, a team led by NCAR researchers in Boulder, Colorado is refining the current leading climate model, called the Community Climate System Model, or CCSM. (While NCAR is leading the CCSM project, the model, as well as the larger IPCC effort, is a culmination of contributions from institutions across the country, including  six DOE National Laboratories, the National Aeronautics and Space Administration,  and a host of universities.)

CCSM is a “coupled” climate model, meaning that it integrates various component models (ocean simulations, atmospheric simulations, ice sheet modeling, etc.) into a unified picture of the Earth's climate system. To produce this unified view, the simulation must be designed so that each component model can influence and be influenced by other component models in the system. (For example, ocean models must be coupled with atmospheric models, because increases in sea surface temperature have an impact on tropical storms and El Niño events.)

Obviously, the computational requirements to perform such simulations can quickly become substantial. After all, accurately modeling atmospheric processes alone can require HPC runs of several days. Running multiple simulations simultaneously, so that the model can incorporate the ways in which component simulations impact one another over time, is an even more resource-intensive ordeal. Additionally, the model must simulate climate processes over several decades.

“The heart of CCSM is weather modeling,” says ORNL's John Drake, chief computational scientist for the End Station effort. “But whereas the weather guys give up on their models after 15 days, we plow ahead for 100 years, and then take statistical averages. The statistical properties of that solution, linked with a variety of other processes, produces what we call climate.”

Building a Better Model

Previous versions of CCSM provided a great deal of useful information. However, there were gaps in the simulations, and researchers believed many areas of the model could be improved upon with higher-resolution modeling, and by incorporating factors that had previously been omitted. In preparation for the next IPCC assessment, the research team is looking to incorporate a number of new scientific questions into CCSM.

For example, previous models had done an excellent job of simulating physical atmospheric processes, but had not incorporated some of the relevant chemical processes. The CCSM development team is now working to integrate atmospheric chemistry, including fully interactive aerosol processes, into the next simulation. Researchers also are working to incorporate biogeochemical cycles in the global climate system (such as the effect of carbon dioxide produced by phytoplankton), as well as higher-resolution glacial ice sheet processes. The new model also will gauge how dynamic vegetation processes on land affect atmospheric carbon dioxide levels. (Even though land covering is fixed when modeling short-term weather patterns, the desertification of a region over decades or centuries can have a more significant impact.)

The IPCC team also would like to incorporate much higher-resolution ocean modeling. Most current ocean models resolve to one degree. However, much of the heat transported through the oceans moves in the form of eddies, which a coarse resolution simply cannot resolve. So previous models relied on parameterization of these processes.

In addition to the problem of modeling these new processes, the team is working to improve on the original physical models themselves. This work is necessary because, as CCSM extends to incorporate new dynamic processes, the accuracy of physical process simulations becomes increasingly important. For example, past simulations were not very accurate in accounting for the specific distribution of precipitation. In past climate simulations, this was not a crucial problem, as long as the overall results for large regions (or for the global system) were correct. However, to incorporate processes such as changing vegetation, a more accurate picture of precipitation is crucial.

Ultimately, the IPCC team believes that all of this work will produce a more useful, accurate climate model. “We expect that, when the next climate model is released, we'll have options for essentially full atmospheric chemistry, dynamic vegetation processes on the land, ocean ecosystems, and more,” says Drake. “By pulling all of these processes together, we'll be able to create not only a physically coupled model, but a chemically coupled and biologically coupled climate model. That's a big stretch over where we are now.”

Bringing Leadership-Class Computing to Climate Modeling

Incorporating higher-resolution physical processes, as well as complex chemical and biological processes, demands an enormous amount of computing resources. The Climate Science End Station at ORNL, led by chief scientist Warren Washington of NCAR, provides dedicated access to a National Leadership Computing Facility (NLCF), and allows IPCC research teams in a variety of climate disciplines to collaborate and take advantage of a common infrastructure. 

At ORNL, climate researchers have access to both Cray XT3 and Cray X1E supercomputers. With thousands of processors, extremely fast interconnects, ample bandwidth to memory, and sophisticated memory hierarchies, these systems bring unprecedented performance and scalability to the team's climate simulations.

For example, the computations involved in modeling physical, ocean and atmospheric systems translate into partial differential equations (PDEs). These equations are ideally suited for vectorization — and as a result, for processing on ORNL's Cray X1E system. In fact, some models that used to run at four simulated years per day on the team's previous systems run at 20 years per day on the Cray X1E supercomputer.

ORNL's Cray XT3 system also supports much higher-resolution simulations. Adding atmospheric chemistry processes to previous physical atmospheric models, for example, increases the required computation for each gridpoint by a factor of six to 10. Only a massively parallel system with an extremely fast interconnect can provide the scalability to perform these calculations for century-long simulations in a reasonable amount of time. The Cray XT3 system also allows the IPCC team to perform one-tenth degree ocean simulations running on 2,000 processors — providing 70 times higher resolution than previous models. This fine resolution eliminates the need to parameterize the effect of ocean eddies by actually incorporating the physical processes into the model for the first time.

Without an extremely fast interconnect and exceptional bandwidth to and from memory, these advanced simulations would simply not be possible.

“It's never been about just the number of processors for us,” says Drake. “These kinds of multi-physics, hard PDE systems never look like Linpack benchmarks. Whereas theoretically you can get higher peak ratings just by adding more processors, we have to pay a lot of attention to the bandwidth, to the various network latencies, to the memory hierarchies, and so forth. The systems that have very fast memory access and good interconnects are the ones that pay off for us, and these systems have some of the fastest interconnects around. We can't even compare running our applications on some commodity system with the level of scalability and performance we have here.”

Evolving Tomorrow's Climate Models

Constituents around the globe are anxiously awaiting future IPCC assessments — and the new information that will be provided by higher-resolution, more comprehensive climate simulations. Ultimately, the leadership-class computing resources at ORNL are allowing IPCC researchers to not only produce better simulations by scaling their models to thousands of processors, but to run their simulations much faster. This speed advantage translates directly into more runs — and more refinement of IPCC climate models — than would be possible with smaller-scale computing resources.

“We will figure out how to use almost any level of computation we have available, but given that our horizon has expanded rather significantly with access to the ORNL computing facility, we're thinking much bigger,” says Drake. “We're thinking that we can use a much higher resolution and reduce some of the uncertainties associated with parameterization. We're thinking we can add a variety of processes, such as atmospheric chemistry, that are computationally very expensive. If we did not have this level of computational capability, we couldn't even try these things. We wouldn't even be able to consider them.”

—–

Per Nyberg is the marketing director for the Earth Sciences Segment at Cray Inc. Since joining the company in 2002, Mr. Nyberg has been responsible for Cray's worldwide strategic planning, business development and marketing for the earth sciences segment working extensively with many weather and climate centers worldwide. Prior to joining Cray, he worked for 12 years in NEC Corp's high performance computing business in Canada, United States and Australia, focused on earth sciences in roles ranging from software engineering to business development. He received a Bachelor of Computer Science from Concordia University in Montreal, Canada, in 1991.