ADIOS: Providing A Framework for Scientific Data on Exascale Systems

May 30, 2018

Editor’s Note: Hardware development associated with the U.S. Exascale Computing Initiative receives the lion’s share of attention but software is at least as important. Presented here is an interview with the team working on Adaptable I/O System (ADIOS) effort as part of the Exascale Computing Project (ECP) overseeing software development and posted today on the ECP site. ADIOS tackles critical data management challenges.

The Adaptable I/O System (ADIOS) project in the ECP supports exascale applications by addressing their data management and in situ analysis needs. Led by Scott Klasky of Oak Ridge National Laboratory, ADIOS is optimizing I/O on exascale architectures and making itself easily maintainable, sustainable, and extensible, while ensuring its performance and scalability. Klasky and some of the ADIOS team members joined ECP Communications on February 6 at the ECP 2nd Annual Meeting in Knoxville, Tennessee, for a podcast episode discussion. This is an edited transcript.

What is the high-level description of your project? 

Think of ADIOS as a framework to be able to put computation in the proper place at the proper time in a data-rich environment. It provides a novel way of thinking about I/O and extreme-scale data management. And essentially it allows scientists to describe their data and talk about how they would like to use it. They don’t have to worry about different things like file formats and storage technology, so really think about it as a very simple way to get extreme-performance I/O.

You’re working with over a dozen different ECP technologies with the Fusion Whole Device Modeling [WDM] Application. Could you please clarify and tell us more about this, how this could relate to other technologies and applications?

ECP’s a very exciting project because what we’re doing is we’re talking about how we bring all these different pieces of technology together. And it’s a very important part of ECP because there’s never one single solution to everything. So one of the things that we learned in so many years of doing science, for myself, being at first a general relativist to then going to being a computer scientist, we learned that for applications—I just want to provide an easy way to this technology. I want to do my science. I don’t want to be bothered. Basically, can we actually do something simple? I/O should be simple. I want to just open. I want to write. I want to read.

For the WDM project, what we wanted to be able to do is take two codes developed by two separate teams and basically not change much of the code. Basically, you just read their file, and then make it work. Physicists can work easily with files—read and write. And then it says, well, can we make that run in situ, in memory. Don’t change your code, now it runs. Then these codes produce a lot of data. There’s ECP technology to reduce. We work with projects such as EZ, which has an SZ compression mechanism. We work with ZFP. We have another technology, MGARD, that comes out from the CODAR [Co-Design Center for Online Data Analysis and Reduction] co-design project, so think about now, when they’re reading, when they’re writing, they don’t care. They just specify reduction, then different variables are reduced. Now they want to visualize. Don’t change your code, just run a visualization service. Everything occurs in memory. Get performance turned on. Get things from TAU. So now all of a sudden, we get this.

Using technology such as DataSpaces, such as EVPath, they can just have these technologies, but for them, they’re just looking like they’re opening, reading, writing a file. And now all this real-time monitoring of the codes, the coupling, they can do their physics without being burdened by this, and they can do this in a reliable fashion. And the point is we’ve learned a lot of things about this along the way, and what we’re finding is that, yes, we have to make things more resilient; yes, we have to make things work better. But the point along the way is that they just want an easy way in, and then they can use all these separate technologies, and they can have a big win by doing this.

ADIOS project team at the ECP 2nd Annual Meeting, Knoxville, Tennessee, February 2018. From left, John Wu, Lawrence Berkeley National Laboratory; Scott Klasky, Oak Ridge National Laboratory (ORNL); Greg Eisenhauer, Georgia Tech; Norbert Podhorszki, ORNL; Qing Liu, New Jersey Institute of Technology; Chuck Atkins, Kitware; and Ruonan Wang, ORNL. Not pictured: Matthew Wolf, ORNL; and Manish Parashar, Rutgers University.

Are there certain areas of this project that you think would be especially good to elucidate, to have further insight about so that people just get a better understanding of what ADIOS is all about?

Absolutely, and what I would like to do is call on one of my colleagues, Norbert Podhorszki, who’s an expert in this area because the important thing with ECP is this is a team. It’s a team that’s built with people around the world. Norbert can now elucidate on this.

Podhorszki: Yes, thank you, Scott. So if I want to summarize in two sentences what’s all about that Scott described about these working together with so many projects is that ADIOS allows the scientist to think about the data and how they can extract the science, the knowledge from it and in an integrated way so that they are not distracted with the details of the technology. What I mean is that they can describe the data and the intent—their intent with the data in some high level. And then ADIOS is the framework that brings together all the mechanisms and the services to execute that intent in an efficient manner in an automated way.

Why is this area of research important to the overall efforts to build a capable exascale ecosystem, Scott?

That’s an excellent question, I’m going to have my good friend and colleague, Greg Eisenhauer from Georgia Tech, answer that.

Eisenhauer: I think to answer that, effectively managing large volumes of data is a key challenge that can limit the science impact of exascale. ADIOS fundamentally addresses this challenge in several ways. It is designed as a service-oriented architecture that can easily and effectively be leveraged by applications. It also enables the use of self-describing data using different file formats which are hidden from the user but is optimized depending on the patterns of the code and the data access.

A particularly key aspect of ADIOS is it allows a separation of intent from mechanisms. We want users to describe what they want to do, and ADIOS ensures that it’s efficiently implemented under the sheets. In this way, ADIOS provides an easy way for scientists to leverage state-of-the-art technologies and solutions without compromising the integrity and the stability of their code because they don’t have to change it. For example, in ADIOS, scientists only have to think about reading and writing files, and they can seamlessly leverage this code in situations that involve synchronous and asynchronous in situ coupling, data reduction, indexing, different file formats, all sorts of different technologies.

ADIOS has been around for a long time, for many years. What’s the significance of ECP to ADIOS?

 Well, again, another excellent question. I’ll say that efficient and effective data management is critical at all scales. All science is about data, and some of the challenges really become more pronounced at the exascale. So it’s really tricky to answer about some of this because we’re very passionate about this, and our view is that we’ve done a lot of research and development, but if there’s no funding in research and development, of course we can’t do this. So we do need a mechanism like anyone else, but as a scientist, I’ll say we have a passion, so we’re going to do this, but exascale really gives us this whole thing about community. And what I’ll say is that we’ve worked with dozens of students all over the world. I’ve traveled around the world talking about ADIOS, getting people involved from all these different countries, getting this passion of what we have to data, saying that we have to make it because data is the important commodity for computing. We can’t do science without it. So without ECP funding, a lot of this would have been more difficult in so many different aspects.

I’ll say one of the most important things for us is taking a lot of the research that we’ve done, that we have software we have running, but we had to make it more stable. So we have Kitware involved, where what they’re doing is using their expertise that they’ve done in their company for things like VTK and applying that to ADIOS, making it so that we have a much more sustainable infrastructure, working with brilliant researchers at, say, Rutgers that we have that can really think about, again, their research artifacts and making that hardened. So I think ECP is making it so that a lot of things that they kind of sort of work, they work normally, we can make those hardened. And other things which work really well we can make work for the newer types of technology that maybe we wouldn’t be able to do as well if we didn’t have the funding to do this.

Why was this research area selected for exascale?

You know, I’m really biased here. Science, as I said, is all about the data. If you can’t efficiently process, move, run, given all the different types of complexities that are being thrown at users in exascale, then there’s no science. Without this form of research, I don’t think there would be any science coming out. We have to really provide a capable software ecosystem to be able to handle extreme-scale data on these large-scale exascale platforms.

You are obviously passionate and your team is obviously passionate about this work. What are your accomplishments at this point that you’d really like to play up for us, really highlight as things that you’re particularly proud of?

That’s a really good question. If you remember, one of the questions that you asked me about the code coupling, we’re really proud of this. And the reason why we’re proud of this is because there were probably about 35 different scientists who’ve contributed different aspects to make it so that the physicists can actually get their science done. Those guys, the physicists, didn’t have to care about each individual technology. We’ve got that. We’re working on a science article on that for Science magazine. My good friend, C. S. Chang, is actually leading that along with the leader of the project, Amitava Bhattacharjee.

And again, it’s really motivating. I talk to other applications here. For instance, Mark Taylor leads the ECP climate community project, and when we can get them more performance, they can write out more data, they can process the data quickly, we can provide more hooks into more ECP software, so then better science can be enabled. So when we think about that, we say it’s really good. And then when we think about what is our task? We’re making software, software has bugs, so now we have to work with, again, really good software engineers like Chuck Atkins at Kitware, who can really make sure that we can make this stable so that if any one of the software technologies crashes, their physics runs can still happen. We can then have other aspects of where we can just have that crash, bring that back up. One of our postdocs, Jason Wang, has a new type of staging technology so that, again, we can bring back these sorts of services, even if they crash. That’s going to be done in many of our technologies along the way: resiliency—but making sure that the science is enabled without making the applications overburdened by the technologies.

You’ve already mentioned this some, talking about the benefits of working with other experts. Can you speak more to that, your working relationships, the ones that have resulted from your ECP collaboration?

I’ll be brief about this and say we’re leveraging a lot of wonderful research that was enabled by ASCR [the US Department of Energy’s Advanced Scientific Computing Research program in the Office of Science], and program managers such as Lucy Nowell, and other program managers such as Randall Laviolette and Ceren Susut. Now we are bringing all this research together and making this a sustainable infrastructure under ECP. We aim at building long-lasting relationships with other applications and other software technologies in ECP collaborations.

What’s next for the ADIOS project?

Everything is about performance, performance, performance. It’s ECP. So for us, performance, but reliability along the way. So just to say we are working to have more applications that can then stress other features inside of our software, making it so that we can build a community and a software ecosystem so that applications can have a very easy time with all the new challenges from exascale and beyond.

Any final comments before we wrap up the discussion today?

Yes. I’d like to thank ECP and the entire program team, along with all the facilities that we run on. And I’d like to thank everyone listening, and including you, Scott, for spending time with me today.

Link to ECP article: https://www.exascaleproject.org/adios-providing-a-framework-for-scientific-data-on-exascale-systems/

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Q&A with Google’s Bill Magro, an HPCwire Person to Watch in 2021

June 11, 2021

Last Fall Bill Magro joined Google as CTO of HPC, a newly created position, after two decades at Intel, where he was responsible for the company's HPC strategy. This interview was conducted by email at the beginning of A Read more…

A Carbon Crisis Looms Over Supercomputing. How Do We Stop It?

June 11, 2021

Supercomputing is extraordinarily power-hungry, with many of the top systems measuring their peak demand in the megawatts due to powerful processors and their correspondingly powerful cooling systems. As a result, these Read more…

Honeywell Quantum and Cambridge Quantum Plan to Merge; More to Follow?

June 10, 2021

Earlier this week, Honeywell announced plans to merge its quantum computing business, Honeywell Quantum Solutions (HQS), which focuses on trapped ion hardware, with the U.K.-based Cambridge Quantum Computing (CQC), which Read more…

ISC21 Keynoter Xiaoxiang Zhu to Deliver a Bird’s-Eye View of a Changing World

June 10, 2021

ISC High Performance 2021 – once again virtual due to the ongoing pandemic – is swiftly approaching. In contrast to last year’s conference, which canceled its in-person component with a couple months’ notice, ISC Read more…

Xilinx Expands Versal Chip Family With 7 New Versal AI Edge Chips

June 10, 2021

FPGA chip vendor Xilinx has been busy over the last several years cranking out its Versal AI Core, Versal Premium and Versal Prime chip families to fill customer compute needs in the cloud, datacenters, networks and more. Now Xilinx is expanding its reach to the booming edge... Read more…

AWS Solution Channel

Building highly-available HPC infrastructure on AWS

Reminder: You can learn a lot from AWS HPC engineers by subscribing to the HPC Tech Short YouTube channel, and following the AWS HPC Blog channel. Read more…

Space Weather Prediction Gets a Supercomputing Boost

June 9, 2021

Solar winds are a hot topic in the HPC world right now, with supercomputer-powered research spanning from the Princeton Plasma Physics Laboratory (which used Oak Ridge’s Titan system) to University College London (which used resources from the DiRAC HPC facility). One of the larger... Read more…

A Carbon Crisis Looms Over Supercomputing. How Do We Stop It?

June 11, 2021

Supercomputing is extraordinarily power-hungry, with many of the top systems measuring their peak demand in the megawatts due to powerful processors and their c Read more…

Honeywell Quantum and Cambridge Quantum Plan to Merge; More to Follow?

June 10, 2021

Earlier this week, Honeywell announced plans to merge its quantum computing business, Honeywell Quantum Solutions (HQS), which focuses on trapped ion hardware, Read more…

ISC21 Keynoter Xiaoxiang Zhu to Deliver a Bird’s-Eye View of a Changing World

June 10, 2021

ISC High Performance 2021 – once again virtual due to the ongoing pandemic – is swiftly approaching. In contrast to last year’s conference, which canceled Read more…

Xilinx Expands Versal Chip Family With 7 New Versal AI Edge Chips

June 10, 2021

FPGA chip vendor Xilinx has been busy over the last several years cranking out its Versal AI Core, Versal Premium and Versal Prime chip families to fill customer compute needs in the cloud, datacenters, networks and more. Now Xilinx is expanding its reach to the booming edge... Read more…

What is Thermodynamic Computing and Could It Become Important?

June 3, 2021

What, exactly, is thermodynamic computing? (Yes, we know everything obeys thermodynamic laws.) A trio of researchers from Microsoft, UC San Diego, and Georgia Tech have written an interesting viewpoint in the June issue... Read more…

AMD Introduces 3D Chiplets, Demos Vertical Cache on Zen 3 CPUs

June 2, 2021

At Computex 2021, held virtually this week, AMD showcased a new 3D chiplet architecture that will be used for future high-performance computing products set to Read more…

Nvidia Expands Its Certified Server Models, Unveils DGX SuperPod Subscriptions

June 2, 2021

Nvidia is busy this week at the virtual Computex 2021 Taipei technology show, announcing an expansion of its nascent Nvidia-certified server program, a range of Read more…

Using HPC Cloud, Researchers Investigate the COVID-19 Lab Leak Hypothesis

May 27, 2021

At the end of 2019, strange pneumonia cases started cropping up in Wuhan, China. As Wuhan (then China, then the world) scrambled to contain what would, of cours Read more…

AMD Chipmaker TSMC to Use AMD Chips for Chipmaking

May 8, 2021

TSMC has tapped AMD to support its major manufacturing and R&D workloads. AMD will provide its Epyc Rome 7702P CPUs – with 64 cores operating at a base cl Read more…

Intel Launches 10nm ‘Ice Lake’ Datacenter CPU with Up to 40 Cores

April 6, 2021

The wait is over. Today Intel officially launched its 10nm datacenter CPU, the third-generation Intel Xeon Scalable processor, codenamed Ice Lake. With up to 40 Read more…

Berkeley Lab Debuts Perlmutter, World’s Fastest AI Supercomputer

May 27, 2021

A ribbon-cutting ceremony held virtually at Berkeley Lab's National Energy Research Scientific Computing Center (NERSC) today marked the official launch of Perlmutter – aka NERSC-9 – the GPU-accelerated supercomputer built by HPE in partnership with Nvidia and AMD. Read more…

CERN Is Betting Big on Exascale

April 1, 2021

The European Organization for Nuclear Research (CERN) involves 23 countries, 15,000 researchers, billions of dollars a year, and the biggest machine in the worl Read more…

Google Launches TPU v4 AI Chips

May 20, 2021

Google CEO Sundar Pichai spoke for only one minute and 42 seconds about the company’s latest TPU v4 Tensor Processing Units during his keynote at the Google I Read more…

Iran Gains HPC Capabilities with Launch of ‘Simorgh’ Supercomputer

May 18, 2021

Iran is said to be developing domestic supercomputing technology to advance the processing of scientific, economic, political and military data, and to strengthen the nation’s position in the age of AI and big data. On Sunday, Iran unveiled the Simorgh supercomputer, which will deliver.... Read more…

HPE Launches Storage Line Loaded with IBM’s Spectrum Scale File System

April 6, 2021

HPE today launched a new family of storage solutions bundled with IBM’s Spectrum Scale Erasure Code Edition parallel file system (description below) and featu Read more…

Quantum Computer Start-up IonQ Plans IPO via SPAC

March 8, 2021

IonQ, a Maryland-based quantum computing start-up working with ion trap technology, plans to go public via a Special Purpose Acquisition Company (SPAC) merger a Read more…

Leading Solution Providers

Contributors

10nm, 7nm, 5nm…. Should the Chip Nanometer Metric Be Replaced?

June 1, 2020

The biggest cool factor in server chips is the nanometer. AMD beating Intel to a CPU built on a 7nm process node* – with 5nm and 3nm on the way – has been i Read more…

Julia Update: Adoption Keeps Climbing; Is It a Python Challenger?

January 13, 2021

The rapid adoption of Julia, the open source, high level programing language with roots at MIT, shows no sign of slowing according to data from Julialang.org. I Read more…

AMD Launches Epyc ‘Milan’ with 19 SKUs for HPC, Enterprise and Hyperscale

March 15, 2021

At a virtual launch event held today (Monday), AMD revealed its third-generation Epyc “Milan” CPU lineup: a set of 19 SKUs -- including the flagship 64-core, 280-watt 7763 part --  aimed at HPC, enterprise and cloud workloads. Notably, the third-gen Epyc Milan chips achieve 19 percent... Read more…

Can Deep Learning Replace Numerical Weather Prediction?

March 3, 2021

Numerical weather prediction (NWP) is a mainstay of supercomputing. Some of the first applications of the first supercomputers dealt with climate modeling, and Read more…

Livermore’s El Capitan Supercomputer to Debut HPE ‘Rabbit’ Near Node Local Storage

February 18, 2021

A near node local storage innovation called Rabbit factored heavily into Lawrence Livermore National Laboratory’s decision to select Cray’s proposal for its CORAL-2 machine, the lab’s first exascale-class supercomputer, El Capitan. Details of this new storage technology were revealed... Read more…

GTC21: Nvidia Launches cuQuantum; Dips a Toe in Quantum Computing

April 13, 2021

Yesterday Nvidia officially dipped a toe into quantum computing with the launch of cuQuantum SDK, a development platform for simulating quantum circuits on GPU-accelerated systems. As Nvidia CEO Jensen Huang emphasized in his keynote, Nvidia doesn’t plan to build... Read more…

Microsoft to Provide World’s Most Powerful Weather & Climate Supercomputer for UK’s Met Office

April 22, 2021

More than 14 months ago, the UK government announced plans to invest £1.2 billion ($1.56 billion) into weather and climate supercomputing, including procuremen Read more…

African Supercomputing Center Inaugurates ‘Toubkal,’ Most Powerful Supercomputer on the Continent

February 25, 2021

Historically, Africa hasn’t exactly been synonymous with supercomputing. There are only a handful of supercomputers on the continent, with few ranking on the Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire