ADIOS: Providing A Framework for Scientific Data on Exascale Systems

May 30, 2018

Editor’s Note: Hardware development associated with the U.S. Exascale Computing Initiative receives the lion’s share of attention but software is at least as important. Presented here is an interview with the team working on Adaptable I/O System (ADIOS) effort as part of the Exascale Computing Project (ECP) overseeing software development and posted today on the ECP site. ADIOS tackles critical data management challenges.

The Adaptable I/O System (ADIOS) project in the ECP supports exascale applications by addressing their data management and in situ analysis needs. Led by Scott Klasky of Oak Ridge National Laboratory, ADIOS is optimizing I/O on exascale architectures and making itself easily maintainable, sustainable, and extensible, while ensuring its performance and scalability. Klasky and some of the ADIOS team members joined ECP Communications on February 6 at the ECP 2nd Annual Meeting in Knoxville, Tennessee, for a podcast episode discussion. This is an edited transcript.

What is the high-level description of your project? 

Think of ADIOS as a framework to be able to put computation in the proper place at the proper time in a data-rich environment. It provides a novel way of thinking about I/O and extreme-scale data management. And essentially it allows scientists to describe their data and talk about how they would like to use it. They don’t have to worry about different things like file formats and storage technology, so really think about it as a very simple way to get extreme-performance I/O.

You’re working with over a dozen different ECP technologies with the Fusion Whole Device Modeling [WDM] Application. Could you please clarify and tell us more about this, how this could relate to other technologies and applications?

ECP’s a very exciting project because what we’re doing is we’re talking about how we bring all these different pieces of technology together. And it’s a very important part of ECP because there’s never one single solution to everything. So one of the things that we learned in so many years of doing science, for myself, being at first a general relativist to then going to being a computer scientist, we learned that for applications—I just want to provide an easy way to this technology. I want to do my science. I don’t want to be bothered. Basically, can we actually do something simple? I/O should be simple. I want to just open. I want to write. I want to read.

For the WDM project, what we wanted to be able to do is take two codes developed by two separate teams and basically not change much of the code. Basically, you just read their file, and then make it work. Physicists can work easily with files—read and write. And then it says, well, can we make that run in situ, in memory. Don’t change your code, now it runs. Then these codes produce a lot of data. There’s ECP technology to reduce. We work with projects such as EZ, which has an SZ compression mechanism. We work with ZFP. We have another technology, MGARD, that comes out from the CODAR [Co-Design Center for Online Data Analysis and Reduction] co-design project, so think about now, when they’re reading, when they’re writing, they don’t care. They just specify reduction, then different variables are reduced. Now they want to visualize. Don’t change your code, just run a visualization service. Everything occurs in memory. Get performance turned on. Get things from TAU. So now all of a sudden, we get this.

Using technology such as DataSpaces, such as EVPath, they can just have these technologies, but for them, they’re just looking like they’re opening, reading, writing a file. And now all this real-time monitoring of the codes, the coupling, they can do their physics without being burdened by this, and they can do this in a reliable fashion. And the point is we’ve learned a lot of things about this along the way, and what we’re finding is that, yes, we have to make things more resilient; yes, we have to make things work better. But the point along the way is that they just want an easy way in, and then they can use all these separate technologies, and they can have a big win by doing this.

ADIOS project team at the ECP 2nd Annual Meeting, Knoxville, Tennessee, February 2018. From left, John Wu, Lawrence Berkeley National Laboratory; Scott Klasky, Oak Ridge National Laboratory (ORNL); Greg Eisenhauer, Georgia Tech; Norbert Podhorszki, ORNL; Qing Liu, New Jersey Institute of Technology; Chuck Atkins, Kitware; and Ruonan Wang, ORNL. Not pictured: Matthew Wolf, ORNL; and Manish Parashar, Rutgers University.

Are there certain areas of this project that you think would be especially good to elucidate, to have further insight about so that people just get a better understanding of what ADIOS is all about?

Absolutely, and what I would like to do is call on one of my colleagues, Norbert Podhorszki, who’s an expert in this area because the important thing with ECP is this is a team. It’s a team that’s built with people around the world. Norbert can now elucidate on this.

Podhorszki: Yes, thank you, Scott. So if I want to summarize in two sentences what’s all about that Scott described about these working together with so many projects is that ADIOS allows the scientist to think about the data and how they can extract the science, the knowledge from it and in an integrated way so that they are not distracted with the details of the technology. What I mean is that they can describe the data and the intent—their intent with the data in some high level. And then ADIOS is the framework that brings together all the mechanisms and the services to execute that intent in an efficient manner in an automated way.

Why is this area of research important to the overall efforts to build a capable exascale ecosystem, Scott?

That’s an excellent question, I’m going to have my good friend and colleague, Greg Eisenhauer from Georgia Tech, answer that.

Eisenhauer: I think to answer that, effectively managing large volumes of data is a key challenge that can limit the science impact of exascale. ADIOS fundamentally addresses this challenge in several ways. It is designed as a service-oriented architecture that can easily and effectively be leveraged by applications. It also enables the use of self-describing data using different file formats which are hidden from the user but is optimized depending on the patterns of the code and the data access.

A particularly key aspect of ADIOS is it allows a separation of intent from mechanisms. We want users to describe what they want to do, and ADIOS ensures that it’s efficiently implemented under the sheets. In this way, ADIOS provides an easy way for scientists to leverage state-of-the-art technologies and solutions without compromising the integrity and the stability of their code because they don’t have to change it. For example, in ADIOS, scientists only have to think about reading and writing files, and they can seamlessly leverage this code in situations that involve synchronous and asynchronous in situ coupling, data reduction, indexing, different file formats, all sorts of different technologies.

ADIOS has been around for a long time, for many years. What’s the significance of ECP to ADIOS?

 Well, again, another excellent question. I’ll say that efficient and effective data management is critical at all scales. All science is about data, and some of the challenges really become more pronounced at the exascale. So it’s really tricky to answer about some of this because we’re very passionate about this, and our view is that we’ve done a lot of research and development, but if there’s no funding in research and development, of course we can’t do this. So we do need a mechanism like anyone else, but as a scientist, I’ll say we have a passion, so we’re going to do this, but exascale really gives us this whole thing about community. And what I’ll say is that we’ve worked with dozens of students all over the world. I’ve traveled around the world talking about ADIOS, getting people involved from all these different countries, getting this passion of what we have to data, saying that we have to make it because data is the important commodity for computing. We can’t do science without it. So without ECP funding, a lot of this would have been more difficult in so many different aspects.

I’ll say one of the most important things for us is taking a lot of the research that we’ve done, that we have software we have running, but we had to make it more stable. So we have Kitware involved, where what they’re doing is using their expertise that they’ve done in their company for things like VTK and applying that to ADIOS, making it so that we have a much more sustainable infrastructure, working with brilliant researchers at, say, Rutgers that we have that can really think about, again, their research artifacts and making that hardened. So I think ECP is making it so that a lot of things that they kind of sort of work, they work normally, we can make those hardened. And other things which work really well we can make work for the newer types of technology that maybe we wouldn’t be able to do as well if we didn’t have the funding to do this.

Why was this research area selected for exascale?

You know, I’m really biased here. Science, as I said, is all about the data. If you can’t efficiently process, move, run, given all the different types of complexities that are being thrown at users in exascale, then there’s no science. Without this form of research, I don’t think there would be any science coming out. We have to really provide a capable software ecosystem to be able to handle extreme-scale data on these large-scale exascale platforms.

You are obviously passionate and your team is obviously passionate about this work. What are your accomplishments at this point that you’d really like to play up for us, really highlight as things that you’re particularly proud of?

That’s a really good question. If you remember, one of the questions that you asked me about the code coupling, we’re really proud of this. And the reason why we’re proud of this is because there were probably about 35 different scientists who’ve contributed different aspects to make it so that the physicists can actually get their science done. Those guys, the physicists, didn’t have to care about each individual technology. We’ve got that. We’re working on a science article on that for Science magazine. My good friend, C. S. Chang, is actually leading that along with the leader of the project, Amitava Bhattacharjee.

And again, it’s really motivating. I talk to other applications here. For instance, Mark Taylor leads the ECP climate community project, and when we can get them more performance, they can write out more data, they can process the data quickly, we can provide more hooks into more ECP software, so then better science can be enabled. So when we think about that, we say it’s really good. And then when we think about what is our task? We’re making software, software has bugs, so now we have to work with, again, really good software engineers like Chuck Atkins at Kitware, who can really make sure that we can make this stable so that if any one of the software technologies crashes, their physics runs can still happen. We can then have other aspects of where we can just have that crash, bring that back up. One of our postdocs, Jason Wang, has a new type of staging technology so that, again, we can bring back these sorts of services, even if they crash. That’s going to be done in many of our technologies along the way: resiliency—but making sure that the science is enabled without making the applications overburdened by the technologies.

You’ve already mentioned this some, talking about the benefits of working with other experts. Can you speak more to that, your working relationships, the ones that have resulted from your ECP collaboration?

I’ll be brief about this and say we’re leveraging a lot of wonderful research that was enabled by ASCR [the US Department of Energy’s Advanced Scientific Computing Research program in the Office of Science], and program managers such as Lucy Nowell, and other program managers such as Randall Laviolette and Ceren Susut. Now we are bringing all this research together and making this a sustainable infrastructure under ECP. We aim at building long-lasting relationships with other applications and other software technologies in ECP collaborations.

What’s next for the ADIOS project?

Everything is about performance, performance, performance. It’s ECP. So for us, performance, but reliability along the way. So just to say we are working to have more applications that can then stress other features inside of our software, making it so that we can build a community and a software ecosystem so that applications can have a very easy time with all the new challenges from exascale and beyond.

Any final comments before we wrap up the discussion today?

Yes. I’d like to thank ECP and the entire program team, along with all the facilities that we run on. And I’d like to thank everyone listening, and including you, Scott, for spending time with me today.

Link to ECP article: https://www.exascaleproject.org/adios-providing-a-framework-for-scientific-data-on-exascale-systems/

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industry updates delivered to you every week!

MLPerf Inference 4.0 Results Showcase GenAI; Nvidia Still Dominates

March 28, 2024

There were no startling surprises in the latest MLPerf Inference benchmark (4.0) results released yesterday. Two new workloads — Llama 2 and Stable Diffusion XL — were added to the benchmark suite as MLPerf continues Read more…

Q&A with Nvidia’s Chief of DGX Systems on the DGX-GB200 Rack-scale System

March 27, 2024

Pictures of Nvidia's new flagship mega-server, the DGX GB200, on the GTC show floor got favorable reactions on social media for the sheer amount of computing power it brings to artificial intelligence.  Nvidia's DGX Read more…

Call for Participation in Workshop on Potential NSF CISE Quantum Initiative

March 26, 2024

Editor’s Note: Next month there will be a workshop to discuss what a quantum initiative led by NSF’s Computer, Information Science and Engineering (CISE) directorate could entail. The details are posted below in a Ca Read more…

Waseda U. Researchers Reports New Quantum Algorithm for Speeding Optimization

March 25, 2024

Optimization problems cover a wide range of applications and are often cited as good candidates for quantum computing. However, the execution time for constrained combinatorial optimization applications on quantum device Read more…

NVLink: Faster Interconnects and Switches to Help Relieve Data Bottlenecks

March 25, 2024

Nvidia’s new Blackwell architecture may have stolen the show this week at the GPU Technology Conference in San Jose, California. But an emerging bottleneck at the network layer threatens to make bigger and brawnier pro Read more…

Who is David Blackwell?

March 22, 2024

During GTC24, co-founder and president of NVIDIA Jensen Huang unveiled the Blackwell GPU. This GPU itself is heavily optimized for AI work, boasting 192GB of HBM3E memory as well as the the ability to train 1 trillion pa Read more…

MLPerf Inference 4.0 Results Showcase GenAI; Nvidia Still Dominates

March 28, 2024

There were no startling surprises in the latest MLPerf Inference benchmark (4.0) results released yesterday. Two new workloads — Llama 2 and Stable Diffusion Read more…

Q&A with Nvidia’s Chief of DGX Systems on the DGX-GB200 Rack-scale System

March 27, 2024

Pictures of Nvidia's new flagship mega-server, the DGX GB200, on the GTC show floor got favorable reactions on social media for the sheer amount of computing po Read more…

NVLink: Faster Interconnects and Switches to Help Relieve Data Bottlenecks

March 25, 2024

Nvidia’s new Blackwell architecture may have stolen the show this week at the GPU Technology Conference in San Jose, California. But an emerging bottleneck at Read more…

Who is David Blackwell?

March 22, 2024

During GTC24, co-founder and president of NVIDIA Jensen Huang unveiled the Blackwell GPU. This GPU itself is heavily optimized for AI work, boasting 192GB of HB Read more…

Nvidia Looks to Accelerate GenAI Adoption with NIM

March 19, 2024

Today at the GPU Technology Conference, Nvidia launched a new offering aimed at helping customers quickly deploy their generative AI applications in a secure, s Read more…

The Generative AI Future Is Now, Nvidia’s Huang Says

March 19, 2024

We are in the early days of a transformative shift in how business gets done thanks to the advent of generative AI, according to Nvidia CEO and cofounder Jensen Read more…

Nvidia’s New Blackwell GPU Can Train AI Models with Trillions of Parameters

March 18, 2024

Nvidia's latest and fastest GPU, codenamed Blackwell, is here and will underpin the company's AI plans this year. The chip offers performance improvements from Read more…

Nvidia Showcases Quantum Cloud, Expanding Quantum Portfolio at GTC24

March 18, 2024

Nvidia’s barrage of quantum news at GTC24 this week includes new products, signature collaborations, and a new Nvidia Quantum Cloud for quantum developers. Wh Read more…

Alibaba Shuts Down its Quantum Computing Effort

November 30, 2023

In case you missed it, China’s e-commerce giant Alibaba has shut down its quantum computing research effort. It’s not entirely clear what drove the change. Read more…

Nvidia H100: Are 550,000 GPUs Enough for This Year?

August 17, 2023

The GPU Squeeze continues to place a premium on Nvidia H100 GPUs. In a recent Financial Times article, Nvidia reports that it expects to ship 550,000 of its lat Read more…

Shutterstock 1285747942

AMD’s Horsepower-packed MI300X GPU Beats Nvidia’s Upcoming H200

December 7, 2023

AMD and Nvidia are locked in an AI performance battle – much like the gaming GPU performance clash the companies have waged for decades. AMD has claimed it Read more…

DoD Takes a Long View of Quantum Computing

December 19, 2023

Given the large sums tied to expensive weapon systems – think $100-million-plus per F-35 fighter – it’s easy to forget the U.S. Department of Defense is a Read more…

Synopsys Eats Ansys: Does HPC Get Indigestion?

February 8, 2024

Recently, it was announced that Synopsys is buying HPC tool developer Ansys. Started in Pittsburgh, Pa., in 1970 as Swanson Analysis Systems, Inc. (SASI) by John Swanson (and eventually renamed), Ansys serves the CAE (Computer Aided Engineering)/multiphysics engineering simulation market. Read more…

Choosing the Right GPU for LLM Inference and Training

December 11, 2023

Accelerating the training and inference processes of deep learning models is crucial for unleashing their true potential and NVIDIA GPUs have emerged as a game- Read more…

Intel’s Server and PC Chip Development Will Blur After 2025

January 15, 2024

Intel's dealing with much more than chip rivals breathing down its neck; it is simultaneously integrating a bevy of new technologies such as chiplets, artificia Read more…

Baidu Exits Quantum, Closely Following Alibaba’s Earlier Move

January 5, 2024

Reuters reported this week that Baidu, China’s giant e-commerce and services provider, is exiting the quantum computing development arena. Reuters reported � Read more…

Leading Solution Providers

Contributors

Comparing NVIDIA A100 and NVIDIA L40S: Which GPU is Ideal for AI and Graphics-Intensive Workloads?

October 30, 2023

With long lead times for the NVIDIA H100 and A100 GPUs, many organizations are looking at the new NVIDIA L40S GPU, which it’s a new GPU optimized for AI and g Read more…

Shutterstock 1179408610

Google Addresses the Mysteries of Its Hypercomputer 

December 28, 2023

When Google launched its Hypercomputer earlier this month (December 2023), the first reaction was, "Say what?" It turns out that the Hypercomputer is Google's t Read more…

AMD MI3000A

How AMD May Get Across the CUDA Moat

October 5, 2023

When discussing GenAI, the term "GPU" almost always enters the conversation and the topic often moves toward performance and access. Interestingly, the word "GPU" is assumed to mean "Nvidia" products. (As an aside, the popular Nvidia hardware used in GenAI are not technically... Read more…

Shutterstock 1606064203

Meta’s Zuckerberg Puts Its AI Future in the Hands of 600,000 GPUs

January 25, 2024

In under two minutes, Meta's CEO, Mark Zuckerberg, laid out the company's AI plans, which included a plan to build an artificial intelligence system with the eq Read more…

Google Introduces ‘Hypercomputer’ to Its AI Infrastructure

December 11, 2023

Google ran out of monikers to describe its new AI system released on December 7. Supercomputer perhaps wasn't an apt description, so it settled on Hypercomputer Read more…

China Is All In on a RISC-V Future

January 8, 2024

The state of RISC-V in China was discussed in a recent report released by the Jamestown Foundation, a Washington, D.C.-based think tank. The report, entitled "E Read more…

Intel Won’t Have a Xeon Max Chip with New Emerald Rapids CPU

December 14, 2023

As expected, Intel officially announced its 5th generation Xeon server chips codenamed Emerald Rapids at an event in New York City, where the focus was really o Read more…

IBM Quantum Summit: Two New QPUs, Upgraded Qiskit, 10-year Roadmap and More

December 4, 2023

IBM kicks off its annual Quantum Summit today and will announce a broad range of advances including its much-anticipated 1121-qubit Condor QPU, a smaller 133-qu Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire