Insley and his team collaborate with the ALCF user community to bring their simulations to life by creating stunning data visualizations of everything from blood flow to supernova. Their work also includes efforts to enable scientific visualization on Aurora, the ALCF’s forthcoming exascale machine.

ALCF’s Nils Heinonen spoke to Insley about the importance of scientific visualizations, as well as his team’s role in preparing for science in the exascale era.

How long have you worked in HPC and scientific visualization?

I’ve been working in various aspects of visualization since grad school, and more specifically in scientific data visualization since joining Argonne about 25 years ago. This has encompassed everything from visualizing the data movement of distributed applications across wide area networks, to volume rendering of computed microtomography data in virtual reality, to production visualizations for a wide range of scientific domains.

How does visualization aid and advance scientific research?

Humans are very visual creatures. We’re adept at visually identifying trends and patterns and outliers in an image—in principle, a much more complex task than staring at a spreadsheet, for example. With HPC resources and scientific instruments, such as light sources and telescopes, producing unprecedented amounts of data, transforming those data into images and visualizations for researchers to interactively explore is often the best, and sometimes the only, way to make sense of the reams of information and gain scientific insight.

How will scientific visualization change with exascale?

As the scale of our compute resources continues to grow, the rate at which we can compute data has greatly outpaced the rate at which we can save that data to disk or tape. While this disparity is not entirely unique to exascale, the gap between computations and saved records will continue to widen, resulting in lost data and lost science. To address this challenge the HPC community has devoted significant effort to enabling in situ visualization and analysis. This is where visualization and analysis are performed on the data that are to be discarded while those data are still in memory—even while the simulation in question is still running. Much progress has been made in this area, with a number of frameworks beginning to mature, but many challenges still remain. So while in situ capabilities have been valuable up to now, they will be a requirement for realizing the true potential of exascale systems.

What are the challenges you face in preparing for Aurora’s rollout?

As mentioned, a number of challenges remain in terms of enabling in situ visualization and analysis. In some simulations the scientist may know exactly what they are expecting to see, and where they should be looking. Other times, however, this may not be known beforehand. So how do you decide where to look so that you don’t miss something important? This is likely problem-specific, but could one leverage machine learning to help predict when and where significant events are likely to occur? An obstacle to performance optimization for visualization on Aurora relates to minimizing data movement. While applications are being readied to leverage the power of Aurora’s GPUs, Intel’s oneAPI Rendering Toolkit is also being ported to the GPU to provide maximum rendering performance. However, many of the standard visualization algorithms that are typically used to transform the computed data to a format that can be rendered are only implemented on the CPU. This means data will need to be moved from the GPU where they were computed, to the CPU for visualization transformations, and then back to the GPU for rendering. Enabling more visualization algorithms on the GPU will reduce the need for all of this data movement, resulting in better performance, and ultimately more science.

Who do you collaborate with for this work?

We work closely with Intel’s advanced rendering team, who are responsible for preparing the oneAPI Rendering Toolkit for Aurora. We are also involved in several efforts related to the development of in situ frameworks, and collaborate with a number of DOE partners, including DOE’s Lawrence Berkeley, Lawrence Livermore, Los Alamos, Oak Ridge, and Sandia National Laboratories, as well as corporate partners Kitware and Intelligent Light.  Additionally, we are working with a number of application teams to instrument their simulation codes for in situ visualization and analysis; among these is an Aurora Early Science Program project from Duke University.

We are also actively involved in the Exascale Computing Project (ECP), which is focused on delivering specific applications, software products, and outcomes on DOE computing facilities. As part of ECP Software Technologies, the VTK-m and ALPINE projects aim at implementing scientific visualization algorithms and in situ infrastructures on exascale supercomputers. We work closely with these two projects, porting their algorithms and infrastructures to Aurora. ECP also has a specific area for Application Development, with exceptional investments in key software applications. Among the Application Development projects, we collaborate with the ExaSky project, where we are preparing our tools to visualize massive cosmology datasets generated on Aurora and other exascale machines.

How has your approach to preparing for exascale evolved over time?

The design of Aurora itself has been evolving over time, so we need to maintain maximum flexibility in our approaches. We continue to investigate the use of various programming models, which have also evolved. As examples of this, the VTK-m ECP project adopted Kokkos, and we have evaluated native SYCL as well as Kokkos with various backends for our Aurora Early Science Program work.

The Aurora Software Development Series examines the range of activities and collaborations that ALCF staff undertake to guide the facility and its users into the next era of scientific computing.

The Argonne Leadership Computing Facility provides supercomputing capabilities to the scientific and engineering community to advance fundamental discovery and understanding in a broad range of disciplines. Supported by the U.S. Department of Energy’s (DOE’s) Office of Science, Advanced Scientific Computing Research (ASCR) program, the ALCF is one of two DOE Leadership Computing Facilities in the nation dedicated to open science.

Argonne National Laboratory seeks solutions to pressing national problems in science and technology. The nation’s first national laboratory, Argonne conducts leading-edge basic and applied scientific research in virtually every scientific discipline. Argonne researchers work closely with researchers from hundreds of companies, universities, and federal, state and municipal agencies to help them solve their specific problems, advance America’s scientific leadership and prepare the nation for a better future. With employees from more than 60 nations, Argonne is managed by UChicago Argonne, LLC for the U.S. Department of Energy’s Office of Science.

The U.S. Department of Energy’s Office of Science is the single largest supporter of basic research in the physical sciences in the United States and is working to address some of the most pressing challenges of our time. For more information, visit https://​ener​gy​.gov/​s​c​ience.

Source: Nils Heinonen, ALCF