ADIOS: Providing A Framework for Scientific Data on Exascale Systems

May 30, 2018

Editor’s Note: Hardware development associated with the U.S. Exascale Computing Initiative receives the lion’s share of attention but software is at least as important. Presented here is an interview with the team working on Adaptable I/O System (ADIOS) effort as part of the Exascale Computing Project (ECP) overseeing software development and posted today on the ECP site. ADIOS tackles critical data management challenges.

The Adaptable I/O System (ADIOS) project in the ECP supports exascale applications by addressing their data management and in situ analysis needs. Led by Scott Klasky of Oak Ridge National Laboratory, ADIOS is optimizing I/O on exascale architectures and making itself easily maintainable, sustainable, and extensible, while ensuring its performance and scalability. Klasky and some of the ADIOS team members joined ECP Communications on February 6 at the ECP 2nd Annual Meeting in Knoxville, Tennessee, for a podcast episode discussion. This is an edited transcript.

What is the high-level description of your project? 

Think of ADIOS as a framework to be able to put computation in the proper place at the proper time in a data-rich environment. It provides a novel way of thinking about I/O and extreme-scale data management. And essentially it allows scientists to describe their data and talk about how they would like to use it. They don’t have to worry about different things like file formats and storage technology, so really think about it as a very simple way to get extreme-performance I/O.

You’re working with over a dozen different ECP technologies with the Fusion Whole Device Modeling [WDM] Application. Could you please clarify and tell us more about this, how this could relate to other technologies and applications?

ECP’s a very exciting project because what we’re doing is we’re talking about how we bring all these different pieces of technology together. And it’s a very important part of ECP because there’s never one single solution to everything. So one of the things that we learned in so many years of doing science, for myself, being at first a general relativist to then going to being a computer scientist, we learned that for applications—I just want to provide an easy way to this technology. I want to do my science. I don’t want to be bothered. Basically, can we actually do something simple? I/O should be simple. I want to just open. I want to write. I want to read.

For the WDM project, what we wanted to be able to do is take two codes developed by two separate teams and basically not change much of the code. Basically, you just read their file, and then make it work. Physicists can work easily with files—read and write. And then it says, well, can we make that run in situ, in memory. Don’t change your code, now it runs. Then these codes produce a lot of data. There’s ECP technology to reduce. We work with projects such as EZ, which has an SZ compression mechanism. We work with ZFP. We have another technology, MGARD, that comes out from the CODAR [Co-Design Center for Online Data Analysis and Reduction] co-design project, so think about now, when they’re reading, when they’re writing, they don’t care. They just specify reduction, then different variables are reduced. Now they want to visualize. Don’t change your code, just run a visualization service. Everything occurs in memory. Get performance turned on. Get things from TAU. So now all of a sudden, we get this.

Using technology such as DataSpaces, such as EVPath, they can just have these technologies, but for them, they’re just looking like they’re opening, reading, writing a file. And now all this real-time monitoring of the codes, the coupling, they can do their physics without being burdened by this, and they can do this in a reliable fashion. And the point is we’ve learned a lot of things about this along the way, and what we’re finding is that, yes, we have to make things more resilient; yes, we have to make things work better. But the point along the way is that they just want an easy way in, and then they can use all these separate technologies, and they can have a big win by doing this.

ADIOS project team at the ECP 2nd Annual Meeting, Knoxville, Tennessee, February 2018. From left, John Wu, Lawrence Berkeley National Laboratory; Scott Klasky, Oak Ridge National Laboratory (ORNL); Greg Eisenhauer, Georgia Tech; Norbert Podhorszki, ORNL; Qing Liu, New Jersey Institute of Technology; Chuck Atkins, Kitware; and Ruonan Wang, ORNL. Not pictured: Matthew Wolf, ORNL; and Manish Parashar, Rutgers University.

Are there certain areas of this project that you think would be especially good to elucidate, to have further insight about so that people just get a better understanding of what ADIOS is all about?

Absolutely, and what I would like to do is call on one of my colleagues, Norbert Podhorszki, who’s an expert in this area because the important thing with ECP is this is a team. It’s a team that’s built with people around the world. Norbert can now elucidate on this.

Podhorszki: Yes, thank you, Scott. So if I want to summarize in two sentences what’s all about that Scott described about these working together with so many projects is that ADIOS allows the scientist to think about the data and how they can extract the science, the knowledge from it and in an integrated way so that they are not distracted with the details of the technology. What I mean is that they can describe the data and the intent—their intent with the data in some high level. And then ADIOS is the framework that brings together all the mechanisms and the services to execute that intent in an efficient manner in an automated way.

Why is this area of research important to the overall efforts to build a capable exascale ecosystem, Scott?

That’s an excellent question, I’m going to have my good friend and colleague, Greg Eisenhauer from Georgia Tech, answer that.

Eisenhauer: I think to answer that, effectively managing large volumes of data is a key challenge that can limit the science impact of exascale. ADIOS fundamentally addresses this challenge in several ways. It is designed as a service-oriented architecture that can easily and effectively be leveraged by applications. It also enables the use of self-describing data using different file formats which are hidden from the user but is optimized depending on the patterns of the code and the data access.

A particularly key aspect of ADIOS is it allows a separation of intent from mechanisms. We want users to describe what they want to do, and ADIOS ensures that it’s efficiently implemented under the sheets. In this way, ADIOS provides an easy way for scientists to leverage state-of-the-art technologies and solutions without compromising the integrity and the stability of their code because they don’t have to change it. For example, in ADIOS, scientists only have to think about reading and writing files, and they can seamlessly leverage this code in situations that involve synchronous and asynchronous in situ coupling, data reduction, indexing, different file formats, all sorts of different technologies.

ADIOS has been around for a long time, for many years. What’s the significance of ECP to ADIOS?

 Well, again, another excellent question. I’ll say that efficient and effective data management is critical at all scales. All science is about data, and some of the challenges really become more pronounced at the exascale. So it’s really tricky to answer about some of this because we’re very passionate about this, and our view is that we’ve done a lot of research and development, but if there’s no funding in research and development, of course we can’t do this. So we do need a mechanism like anyone else, but as a scientist, I’ll say we have a passion, so we’re going to do this, but exascale really gives us this whole thing about community. And what I’ll say is that we’ve worked with dozens of students all over the world. I’ve traveled around the world talking about ADIOS, getting people involved from all these different countries, getting this passion of what we have to data, saying that we have to make it because data is the important commodity for computing. We can’t do science without it. So without ECP funding, a lot of this would have been more difficult in so many different aspects.

I’ll say one of the most important things for us is taking a lot of the research that we’ve done, that we have software we have running, but we had to make it more stable. So we have Kitware involved, where what they’re doing is using their expertise that they’ve done in their company for things like VTK and applying that to ADIOS, making it so that we have a much more sustainable infrastructure, working with brilliant researchers at, say, Rutgers that we have that can really think about, again, their research artifacts and making that hardened. So I think ECP is making it so that a lot of things that they kind of sort of work, they work normally, we can make those hardened. And other things which work really well we can make work for the newer types of technology that maybe we wouldn’t be able to do as well if we didn’t have the funding to do this.

Why was this research area selected for exascale?

You know, I’m really biased here. Science, as I said, is all about the data. If you can’t efficiently process, move, run, given all the different types of complexities that are being thrown at users in exascale, then there’s no science. Without this form of research, I don’t think there would be any science coming out. We have to really provide a capable software ecosystem to be able to handle extreme-scale data on these large-scale exascale platforms.

You are obviously passionate and your team is obviously passionate about this work. What are your accomplishments at this point that you’d really like to play up for us, really highlight as things that you’re particularly proud of?

That’s a really good question. If you remember, one of the questions that you asked me about the code coupling, we’re really proud of this. And the reason why we’re proud of this is because there were probably about 35 different scientists who’ve contributed different aspects to make it so that the physicists can actually get their science done. Those guys, the physicists, didn’t have to care about each individual technology. We’ve got that. We’re working on a science article on that for Science magazine. My good friend, C. S. Chang, is actually leading that along with the leader of the project, Amitava Bhattacharjee.

And again, it’s really motivating. I talk to other applications here. For instance, Mark Taylor leads the ECP climate community project, and when we can get them more performance, they can write out more data, they can process the data quickly, we can provide more hooks into more ECP software, so then better science can be enabled. So when we think about that, we say it’s really good. And then when we think about what is our task? We’re making software, software has bugs, so now we have to work with, again, really good software engineers like Chuck Atkins at Kitware, who can really make sure that we can make this stable so that if any one of the software technologies crashes, their physics runs can still happen. We can then have other aspects of where we can just have that crash, bring that back up. One of our postdocs, Jason Wang, has a new type of staging technology so that, again, we can bring back these sorts of services, even if they crash. That’s going to be done in many of our technologies along the way: resiliency—but making sure that the science is enabled without making the applications overburdened by the technologies.

You’ve already mentioned this some, talking about the benefits of working with other experts. Can you speak more to that, your working relationships, the ones that have resulted from your ECP collaboration?

I’ll be brief about this and say we’re leveraging a lot of wonderful research that was enabled by ASCR [the US Department of Energy’s Advanced Scientific Computing Research program in the Office of Science], and program managers such as Lucy Nowell, and other program managers such as Randall Laviolette and Ceren Susut. Now we are bringing all this research together and making this a sustainable infrastructure under ECP. We aim at building long-lasting relationships with other applications and other software technologies in ECP collaborations.

What’s next for the ADIOS project?

Everything is about performance, performance, performance. It’s ECP. So for us, performance, but reliability along the way. So just to say we are working to have more applications that can then stress other features inside of our software, making it so that we can build a community and a software ecosystem so that applications can have a very easy time with all the new challenges from exascale and beyond.

Any final comments before we wrap up the discussion today?

Yes. I’d like to thank ECP and the entire program team, along with all the facilities that we run on. And I’d like to thank everyone listening, and including you, Scott, for spending time with me today.

Link to ECP article: https://www.exascaleproject.org/adios-providing-a-framework-for-scientific-data-on-exascale-systems/

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

On the Spack Track @SC19

December 5, 2019

At the annual supercomputing conference, SC19 in Denver, Colorado, there were Spack events each day of the conference. As a reflection of its grassroots heritage, nine sessions were planned by more than a dozen thought leaders from seven organizations, including three U.S. national Department of Energy (DOE) laboratories and Sylabs... Read more…

By Elizabeth Leake

Intel’s New Hyderabad Design Center Targets Exascale Era Technologies

December 3, 2019

Intel's Raja Koduri was in India this week to help launch a new 300,000 square foot design and engineering center in Hyderabad, which will focus on advanced computing technologies for the AI and exascale era. "Over th Read more…

By Tiffany Trader

AWS Debuts 7nm 2nd-Gen Graviton Arm Processor

December 3, 2019

The “x86 Big Bang,” in which market dominance of the venerable Intel CPU has exploded into fragments of processor options suited to varying workloads, has now encompassed CPUs offered by the leading public cloud serv Read more…

By Doug Black

Medical Imaging Gets an AI Boost

December 3, 2019

AI technologies incorporated into diagnostic imaging tools have proven useful in eliminating confirmation bias, often outperforming human clinicians who may bring their own prejudices. Another issue slowing progress is t Read more…

By George Leopold

Ride on the Wild Side – Squyres SC19 Mars Rovers Keynote

December 2, 2019

Reminding us of the deep and enabling connection between HPC and modern science is an important part of the SC Conference mission. And yes, HPC is a science itself. At SC19, Steve Squyres’ opening keynote recounting th Read more…

By John Russell

AWS Solution Channel

Making High Performance Computing Affordable and Accessible for Small and Medium Businesses with HPC on AWS

High performance computing (HPC) brings a powerful set of tools to a broad range of industries, helping to drive innovation and boost revenue in finance, genomics, oil and gas extraction, and other fields. Read more…

IBM Accelerated Insights

AI Needs Intelligent HPC infrastructure

Artificial Intelligence (AI) has revolutionized entire industries and enables humanity to solve some of the most daunting challenges. To accomplish this, it requires massive amounts of data from heterogeneous sources that is processed it new ways that differs significantly from HPC applications. Read more…

NSCI Update – Adapting to a Changing Landscape

December 2, 2019

It was November of 2017 when we last visited the topic of the National Strategic Computing Initiative (NSCI). As you will recall, the NSCI was started with an Executive Order (E.O. No. 13702), that was issued by President Obama in July of 2015 and was followed by a Strategic Plan that was released in July of 2016. The question for November of 2017... Read more…

By Alex R. Larzelere

On the Spack Track @SC19

December 5, 2019

At the annual supercomputing conference, SC19 in Denver, Colorado, there were Spack events each day of the conference. As a reflection of its grassroots heritage, nine sessions were planned by more than a dozen thought leaders from seven organizations, including three U.S. national Department of Energy (DOE) laboratories and Sylabs... Read more…

By Elizabeth Leake

Intel’s New Hyderabad Design Center Targets Exascale Era Technologies

December 3, 2019

Intel's Raja Koduri was in India this week to help launch a new 300,000 square foot design and engineering center in Hyderabad, which will focus on advanced com Read more…

By Tiffany Trader

AWS Debuts 7nm 2nd-Gen Graviton Arm Processor

December 3, 2019

The “x86 Big Bang,” in which market dominance of the venerable Intel CPU has exploded into fragments of processor options suited to varying workloads, has n Read more…

By Doug Black

Ride on the Wild Side – Squyres SC19 Mars Rovers Keynote

December 2, 2019

Reminding us of the deep and enabling connection between HPC and modern science is an important part of the SC Conference mission. And yes, HPC is a science its Read more…

By John Russell

NSCI Update – Adapting to a Changing Landscape

December 2, 2019

It was November of 2017 when we last visited the topic of the National Strategic Computing Initiative (NSCI). As you will recall, the NSCI was started with an Executive Order (E.O. No. 13702), that was issued by President Obama in July of 2015 and was followed by a Strategic Plan that was released in July of 2016. The question for November of 2017... Read more…

By Alex R. Larzelere

Tsinghua University Racks Up Its Ninth Student Cluster Championship Win at SC19

November 27, 2019

Tsinghua University has done it again. At SC19 last week, the eight-time gold medal-winner team took home the top prize in the 2019 Student Cluster Competition Read more…

By Oliver Peckham

SC19: IBM Changes Its HPC-AI Game Plan

November 25, 2019

It’s probably fair to say IBM is known for big bets. Summit supercomputer – a big win. Red Hat acquisition – looking like a big win. OpenPOWER and Power processors – jury’s out? At SC19, long-time IBMer Dave Turek sketched out a different kind of bet for Big Blue – a small ball strategy, if you’ll forgive the baseball analogy... Read more…

By John Russell

How the Gordon Bell Prize Winners Used Summit to Illuminate Transistors

November 22, 2019

At SC19, the Association for Computing Machinery (ACM) awarded the prestigious Gordon Bell Prize to the Swiss Federal Institute of Technology (ETH) Zurich. The Read more…

By Oliver Peckham

Supercomputer-Powered AI Tackles a Key Fusion Energy Challenge

August 7, 2019

Fusion energy is the Holy Grail of the energy world: low-radioactivity, low-waste, zero-carbon, high-output nuclear power that can run on hydrogen or lithium. T Read more…

By Oliver Peckham

Using AI to Solve One of the Most Prevailing Problems in CFD

October 17, 2019

How can artificial intelligence (AI) and high-performance computing (HPC) solve mesh generation, one of the most commonly referenced problems in computational engineering? A new study has set out to answer this question and create an industry-first AI-mesh application... Read more…

By James Sharpe

Cray Wins NNSA-Livermore ‘El Capitan’ Exascale Contract

August 13, 2019

Cray has won the bid to build the first exascale supercomputer for the National Nuclear Security Administration (NNSA) and Lawrence Livermore National Laborator Read more…

By Tiffany Trader

DARPA Looks to Propel Parallelism

September 4, 2019

As Moore’s law runs out of steam, new programming approaches are being pursued with the goal of greater hardware performance with less coding. The Defense Advanced Projects Research Agency is launching a new programming effort aimed at leveraging the benefits of massive distributed parallelism with less sweat. Read more…

By George Leopold

D-Wave’s Path to 5000 Qubits; Google’s Quantum Supremacy Claim

September 24, 2019

On the heels of IBM’s quantum news last week come two more quantum items. D-Wave Systems today announced the name of its forthcoming 5000-qubit system, Advantage (yes the name choice isn’t serendipity), at its user conference being held this week in Newport, RI. Read more…

By John Russell

Ayar Labs to Demo Photonics Chiplet in FPGA Package at Hot Chips

August 19, 2019

Silicon startup Ayar Labs continues to gain momentum with its DARPA-backed optical chiplet technology that puts advanced electronics and optics on the same chip Read more…

By Tiffany Trader

AMD Launches Epyc Rome, First 7nm CPU

August 8, 2019

From a gala event at the Palace of Fine Arts in San Francisco yesterday (Aug. 7), AMD launched its second-generation Epyc Rome x86 chips, based on its 7nm proce Read more…

By Tiffany Trader

SC19: IBM Changes Its HPC-AI Game Plan

November 25, 2019

It’s probably fair to say IBM is known for big bets. Summit supercomputer – a big win. Red Hat acquisition – looking like a big win. OpenPOWER and Power processors – jury’s out? At SC19, long-time IBMer Dave Turek sketched out a different kind of bet for Big Blue – a small ball strategy, if you’ll forgive the baseball analogy... Read more…

By John Russell

Leading Solution Providers

ISC 2019 Virtual Booth Video Tour

CRAY
CRAY
DDN
DDN
DELL EMC
DELL EMC
GOOGLE
GOOGLE
ONE STOP SYSTEMS
ONE STOP SYSTEMS
PANASAS
PANASAS
VERNE GLOBAL
VERNE GLOBAL

Cray, Fujitsu Both Bringing Fujitsu A64FX-based Supercomputers to Market in 2020

November 12, 2019

The number of top-tier HPC systems makers has shrunk due to a steady march of M&A activity, but there is increased diversity and choice of processing compon Read more…

By Tiffany Trader

Crystal Ball Gazing: IBM’s Vision for the Future of Computing

October 14, 2019

Dario Gil, IBM’s relatively new director of research, painted a intriguing portrait of the future of computing along with a rough idea of how IBM thinks we’ Read more…

By John Russell

Intel Debuts New GPU – Ponte Vecchio – and Outlines Aspirations for oneAPI

November 17, 2019

Intel today revealed a few more details about its forthcoming Xe line of GPUs – the top SKU is named Ponte Vecchio and will be used in Aurora, the first plann Read more…

By John Russell

Kubernetes, Containers and HPC

September 19, 2019

Software containers and Kubernetes are important tools for building, deploying, running and managing modern enterprise applications at scale and delivering enterprise software faster and more reliably to the end user — while using resources more efficiently and reducing costs. Read more…

By Daniel Gruber, Burak Yenier and Wolfgang Gentzsch, UberCloud

Dell Ramps Up HPC Testing of AMD Rome Processors

October 21, 2019

Dell Technologies is wading deeper into the AMD-based systems market with a growing evaluation program for the latest Epyc (Rome) microprocessors from AMD. In a Read more…

By John Russell

SC19: Welcome to Denver

November 17, 2019

A significant swath of the HPC community has come to Denver for SC19, which began today (Sunday) with a rich technical program. As is customary, the ribbon cutt Read more…

By Tiffany Trader

When Dense Matrix Representations Beat Sparse

September 9, 2019

In our world filled with unintended consequences, it turns out that saving memory space to help deal with GPU limitations, knowing it introduces performance pen Read more…

By James Reinders

With the Help of HPC, Astronomers Prepare to Deflect a Real Asteroid

September 26, 2019

For years, NASA has been running simulations of asteroid impacts to understand the risks (and likelihoods) of asteroids colliding with Earth. Now, NASA and the European Space Agency (ESA) are preparing for the next, crucial step in planetary defense against asteroid impacts: physically deflecting a real asteroid. Read more…

By Oliver Peckham

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This