Nov. 23, 2020 — An innovative collaboration between computer and computational scientists from the Data Science and Learning division (DSL) and Advanced Photon Source (APS) at the U.S. Department of Energy’s (DOE) Argonne National Laboratory and the Electrical and Computer Engineering department at the University of Illinois at Urbana-Champaign (UIUC) has garnered the Best Paper Award at SC20. SC is an annual conference that brings together the global high-performance computing community to share scientific computing advances and insights with an eye toward the upcoming exascale era.

The research team — led by Tekin Bicer of DSL and PhD candidate Mert Hidayetoglu (UIUC) and comprising Simon Garcia de Gonzalo (Barcelona Supercomputing Center), Bin Ren (College of William & Mary), Vincent De Andrade (APS), Doga Gursoy (APS), Rajkumar Kettimuthu (DSL), Ian Foster (DSL) and Wen-mei Hwu (UIUC) — is focused on using a high-performance, iterative reconstruction system for noninvasive imaging at synchrotron facilities. The new design, described in the team’s winning paper ?”Petascale XCT: 3D Image Reconstruction with Hierarchical Communications on Multi-GPU Nodes,” can be used for terabyte(s)-scale 3D volumes and is not constrained by computational requirements.

The work involves the following three novel optimizations: (1) extends the 2D memory-centric approach to 3D; (2) includes hierarchical communications by exploiting ?”fat-node” architecture with many GPUs; and (3) employs mixed-precision types while preserving convergence rate and quality.

The team extensively evaluated the proposed optimizations and scaling on the Summit supercomputer at Oak Ridge National Laboratory. Their largest reconstruction is a mouse brain volume with 9K×11K×11K voxels, where the total reconstruction time is under 3 minutes using 24,576 GPUs and reaching 65 PFLOPS, which is 34% of Summit’s peak performance.

The two facilities used in the research, the Advanced Photon Source at Argonne and the Oak Ridge Leadership Computing Facility at Oak Ridge, are DOE Office of Science User Facilities. The work was partially supported by the Office of Science, Advanced Scientific Computing Research and Basic Energy Sciences offices.

The authors wish to thank Narayanan (Bobby) Kasthuri from UChicago/Argonne and Argonne’s Rafael Vescovi, Ming Du and Vincent De Andrade.

The work was also supported by IBM-ILLINOIS Center for Cognitive Computing Systems Research (C3SR) and is based, in part, on work funded by the Center for Applications Driving Architectures (ADA), a JUMP Center co-sponsored by SRC and DARPA. Funding from the European Union’s Horizon 2020 research and innovation programme was provided under the Marie Skodowska-Curie grant agreement.

To learn more about Argonne’s participation in SC20, please visit the Argonne at SC20 website.

Argonne National Laboratory seeks solutions to pressing national problems in science and technology. The nation’s first national laboratory, Argonne conducts leading-edge basic and applied scientific research in virtually every scientific discipline. Argonne researchers work closely with researchers from hundreds of companies, universities, and federal, state and municipal agencies to help them solve their specific problems, advance America’s scientific leadership and prepare the nation for a better future. With employees from more than 60 nations, Argonne is managed by UChicago Argonne, LLC for the U.S. Department of Energy’s Office of Science.

The U.S. Department of Energy’s Office of Science is the single largest supporter of basic research in the physical sciences in the United States and is working to address some of the most pressing challenges of our time. For more information, visit https://energy.gov/science.

---

Source: DOE and Argonne