New leader shares challenges and opportunities as the scientific community gears up for hybrid supercomputing

On July 1 Jack Wells became the director of science for the National Center for Computational Sciences (NCCS) at Oak Ridge National Laboratory (ORNL). The NCCS is a Department of Energy (DOE) Office of Science user facility for capability computing, which employs maximal computing power to solve in the shortest time possible problems of a size or complexity that no other computer can approach. Its Oak Ridge Leadership Computing Facility (OLCF) houses Jaguar, America’s fastest supercomputer, used by researchers to solve pressing science and energy challenges via modeling and simulation. Leveraging expertise and infrastructure, the NCCS also hosts the Gaea supercomputer, which ORNL operates on behalf of the National Oceanic and Atmospheric Administration, and the Kraken supercomputer, which is managed by the National Institute for Computational Sciences, a collaboration between the University of Tennessee and ORNL.

Wells has led ORNL groups in computational nanotechnology, computational materials science, and nanomaterials theory. He has also served as a scientific advisor to Tennessee Senator Lamar Alexander during a sabbatical. Most recently, he directed ORNL’s Office of Institutional Planning, where his responsibilities included developing a strategic plan for the lab, overseeing discretionary research and development investments, and managing Advanced Research Projects Agency–Energy (ARPA-E) programs.

He has a long history with ORNL supercomputing and was a pioneering user of the facility. Principal investigator of a project to understand the chemistry of rechargeable lithium/air batteries, which potentially can store ten times more energy than lithium/ion batteries of the same weight, he and his collaborators received 24 million processor-hours on Argonne and Oak Ridge supercomputers through the Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program.

In this interview, Wells describes his vision for executing a scientific strategy for the NCCS that ensures cost-effective, state-of-the-art computing to facilitate DOE’s scientific missions. To begin this decade’s transition to exaflop computing, capable of carrying out a million trillion floating point operations per second, plans are in the works for a staged upgrade of Jaguar, a high performance computing system employing traditional CPU microprocessors, to transform it into Titan, a hybrid system employing both CPUs and GPUs,energy-efficient number crunchers that accelerate specific types of calculations in scientific application codes. As the OLCF gears up to deliver the system, expected to have a peak performance of 10–20 petaflops, by early 2013, Wells’s challenges are many.

HPCwire: Congratulations on your new job. What is your vision for the NCCS?

Wells: My vision is a user facility that recognizes that its products are scientific discovery and technical innovation and that we will achieve this vision working with strong partners. These partnerships will be developed through our calls for proposals and our outreach efforts to the scientific, technology, and industrial communities. To broaden the scope of leadership computing, we need to engage through our networks, through our relationships, and encourage new users from new communities that can take good advantage of these resources to move us forward in scientific discovery, industrial competitiveness, and sustainability. Partnerships and alliances are a big part of what I think is important about my job.

In terms of challenges, clearly we’re at a cusp in technology moving to hybrid architectures. This implies a lot of hard work by a lot of people. But it also is a game changer, meaning that all communities are going to have to do a lot of work. No community is already prepared for these new technologies penetrating scientific computing. It’s a time when users and developers of traditional supercomputing applications will have to race to catch up too. It’s a good time for users of new applications to get in.

HPCwire: What are some examples of new users you hope to entice?

Wells: Energy technologies are clearly a topic that is interesting. We want to have impact on society, and energy technology is a compelling mission of the Department of Energy where DOE has a leadership role, and modeling and simulation can make a big impact on the rate of innovation going forward. A wonderful example today is the investment that DOE has made through the Office of Nuclear Energy in the modeling and simulation hub, the Consortium for Advanced Simulation for Light Water Reactors.

This is a huge and in some sense groundbreaking investment in modeling and simulation in energy technology. The codes that are being developed can impact nuclear energy, nuclear engineering, and many other areas in energy technology. [Accelerated innovation resulting from computing] is [also] starting to happen in areas funded by [DOE’s Office of] Energy Efficiency and Renewable Energy in battery simulation, in solar cell simulation. One can imagine a vision in which we have simulation tools for all these energy systems to accelerate innovation and design, help our industry be more competitive, and solve our societal problems.

HPCwire: As the head of an INCITE project to solve the issues that prevent electric cars from running 500 miles on a single battery charge, can you share your experience as a user of the facility you now lead?

Wells: This is a project with Argonne National Laboratory, our sister laboratory, and some major industrial partners with strong R&D capabilities, specifically IBM and Bosch, where we focused on what some have called the Mount Everest in battery chemistries, lithium/air. Just from simple chemistry arguments, it has the greatest potential for energy density storage. But by no means do these batteries work today as rechargeable cells in any sense that you would recognize. The fundamental mechanisms of reaction are not understood.

This INCITE project is trying to make significant progress in the basic chemistry of these reactions to help us understand, is there the potential that one could engineer these systems? Right now, not even [the answer to] that basic question is clear. We shouldn’t look for this battery chemistry to impact technology in the near term, and by near term I mean within the next five years. But with breakthroughs one could have disruptive change in the marketplace within 10 years. But it’s certainly not guaranteed, and other chemistries out there beyond today’s lithium/ion batteries are also exciting, like zinc/air or lithium/sulfur batteries.

Basically, these are topics to which high performance computing has not been applied historically. There have been modeling and simulation of battery systems, but typically at the level of phenomenological models. Significant progress has been [made], but over the coming years, because of the significance of the technology, many, many more people are going to be engaged in this in universities and companies, and high performance computing will play a role.

The codes we were running on the leadership computers today are state-of-the-art, first-principles chemistry and materials codes because we’re asking basic science questions about these systems. New multi-scale battery codes need to be built that can deal with systems-level issues, not just chemical reactions on anodes and cathodes, to be able to describe this as a complex engineering system. No one can do that today.

HPCwire: What was your role in ORNL’s Computing and Computational Sciences Directorate before it housed and ran a national user facility?

Wells: I came here as a [Vanderbilt] graduate student working on Office of Science-funded projects in nuclear and atomic physics. My Ph.D. was sponsored by a grand challenge project funded under a program that started with the High Performance Computing and Communications Act of 1992—that’s called the Gore Act because Senator Gore was the main sponsor in the U.S. Senate, and it’s through that, as the old story goes, he ‘invented’ the Internet. It was that program [which partnered HPC science teams from around the country with ORNL computer scientists and hardware vendor Intel] that founded the Center for Computational Sciences (CCS) originally in 1992.

After a postdoc I came back to ORNL in ’97 as a Wigner Fellow in the CCS, and Buddy Bland [project director of the OLCF-2, which built the petascale Jaguar system, and the OLCF-3, which will build the even more powerful Titan] was my first group leader. I worked in the Scientific Computing group on parallel code performance optimization and doing my science in theoretical atomic and molecular physics. I did use the CCS computers that we had in my Ph.D. thesis—the Intel iPSC/860 and Intel XP/S 5 Paragon. Then when I came back in ’97 we had the XP/S 35 Paragon and XP/S 150 then too. We transitioned to the IBM Eagle by about 1999.

The point is that we had a CCS even before we had a Leadership Computing Facility. Beginning in 1999, I worked on basic materials and engineering physics programs in DOE’s Office of Science Basic Energy Sciences. And then when the [Center for Nanophase Materials Sciences, or CNMS] was constructed at Oak Ridge, I along with my group was matrixed to form the Nanomaterials Theory Institute at the CNMS. During that time, Oak Ridge competed for and won the DOE Leadership Computing Facility in 2004. The significant thing is that CCS has been here for almost 20 years. Next year we have a 20-year anniversary.

HPCwire: What was it like to serve as an advisor to Tennessee Senator Lamar Alexander?

Wells: Since Senator Alexander has been a senator, starting in 2003, he has requested that the Office of Science provide him a Science Fellow from Oak Ridge National Laboratory, and the Office of Science has worked with the lab to provide, now, five people. This has been a relationship where Senator Alexander has benefitted from the expertise of the Office of Science and ORNL.

As Senator Alexander is fully aware, the largest federal investment in the state of Tennessee is the one that DOE makes in its facilities in and around Oak Ridge, with ORNL being one of those. And many of the Senator’s priorities align very well with our mission. Those include clean air, abundant clean energy, increased brain power as a driver for economic competitiveness, energy security. He has been an advocate for Office of Science programs within the U.S. Senate, including leadership computing. In particular, he and New Mexico Senator Jeff Bingaman were the lead authors in the senate on the DOE High-End Computing Act of 2004 that authorized funding for the leadership computing facilities.

I was not there in 2004. I went there from 2006 to 2008, and my title there was one of a legislative fellow. A fellow is someone who is working in the Senate but is not an employee of the Senate. Many scientists and engineers do this, for example through fellowships sponsored by the American Association for the Advancement of Science. While I was there I did not do politics. I did not make policy. But I informed the Senator on topics related to high performance computing, energy technology, renewable energy, nuclear energy, and science, technology, engineering, and mathematics education and its relationship to U.S. competitiveness.

HPCwire: Did directing institutional planning for ORNL provide lessons that might guide you in your new role?

Wells: What I learned from working for our laboratory director’s office from August of 2009 through June of 2011—that’s the job I was just doing before I came to the NCCS—is that both planning and science are about the future, and we need to not be constrained in our thinking by the status quo, but to try to establish a clear and compelling vision for the future for our science programs, for our institution, and ultimately, in collaboration with others, for our nation; to not always think about what is, but what could be, and why it would be an attractive future.

ARPA-E [a DOE program to spur energy innovations] is an interesting case of a good idea articulated by policymakers that was fairly rapidly put in place. It was authorized by Congress and then implemented by DOE, initially through Recovery Act funding, to bring a new approach to funding high-risk, high reward energy technology research within the Department of Energy. It’s been reviewed very well by industry and its sponsors in Congress. The ability to take risks and reach for the big payoffs is something that we should think about and try to implement when we can.