About eight months ago Bill Gropp was elevated to acting director of the National Center for Supercomputing Applications (NCSA). He is, of course, already highly accomplished. Development (with colleagues) of the MPICH implementation of MPI is one example. He was NCSA’s chief scientist, a role Gropp retains, when director Ed Seidel was tapped to serve as interim vice president for research for the University of Illinois System, and Gropp was appointed acting NCSA director.
Don’t be far misled by the “acting” and “interim” qualifiers. They are accurate but hardly diminish the aspirations of the new leaders jointly and independently. In getting ready for this interview with HPCwire, Gropp wrote, “Our goal for NCSA is nothing less than to lead the transformation of all areas scholarship in making use of advanced computing and data.” That seems an ambitious but perhaps appropriate goal for the home of Blue Waters and XSEDE.
During our interview – his first major interview since taking the job – Gropp sketched out the new challenges and opportunities he is facing. While in Gropp-like fashion emphasizing the collaborative DNA running deep throughout NCSA, he also stepped out of his comfort zone when asked what he hopes his legacy will be – a little early for that question perhaps but his response is revealing:
“If you look at us now we have three big projects. We’ve got Blue Waters. We have XSEDE. We hope to have the large synoptic survey telescope (LSST) data facility. These are really good models. They take a long time to develop and may require a fair amount of early investment. I would really like to lay the groundwork for our fourth big thing. That’s what I can contribute most to the institution.”
NCSA is a good place to think big. Blue Waters, of course, is the centerpiece of its computing infrastructure. Deployed in 2012, Blue Waters is a roughly 13-petaflops Cray XE/XK hybrid machine supported with about 1.6 PB of systems memory and 26PB of storage (usable, with an aggregate 1.1TBs). No doubt attention is turning to what’s ahead. The scope of scientific computing and industry collaboration that goes on at NCSA in concert with the University of Illinois is big by any standard.
It’s also worth remembering the NCSA and Gropp are in the thick of U.S. policy development. The most recent report from the National Academies of Sciences, Engineering, and Medicine Report – Future Directions For NSF Advanced Computing Infrastructure To Support U.S. Science In 2017-2020] – was co-chaired by Gropp and Robert Harrison of Stony Brook University; not surprisingly, it argues strongly for NSF to produce a clear long-term roadmap supporting advanced computing and was cited in testimony two weeks ago at Congressional Hearings (National Science Foundation Part II: Future Opportunities and Challenges for Science).
In preparing for the interview, Gropp sent a brief summary of his thinking about the future, which said in part:
“This is a challenging time. Technically, computing is going through a major transition as the community copes with the end of Dennard (frequency) scaling and the consequences for hardware, software, and algorithms. The rapid expansion of data science has broadened the demand for computational resources while at the same time bringing about new and disruptive ways to provide those services, as well as increasing the demand for skilled workers in all areas of computing. Funding remains tight, with support for sustaining the human and physical infrastructure still depending mostly on ad hoc and irregular competitions for hardware; in addition, while everyone talks about the long-term value of science data, there is little appetite to pay for it.
“But these challenges produce opportunities to make critical impact and are exactly why this is also an exciting time to be in computing, and to be leading NCSA. Last August, NCSA Director Ed Siedel was asked by the president of the University of Illinois System to be interim vice president of research, and to guide that office as it refocuses on innovating for economic development by building on the many strengths of the University System. Ed and I have a similar vision for NCSA, and I was honored step in as acting director to guide NCSA while Ed is helping the University system.”
HPCwire: Thanks for your time Bill. It’s hard to know where to start. NCSA has many ongoing projects – participation in the National Data Service Consortium, the Illinois Smart State Initiative, the Visualization Laboratory, and XSEDE all come to mind. Perhaps provide an example or two of NCSA projects to get a sense of the range of NCSA activities?
Bill Gropp: It’s hard to pick just one and somebody is going to be mad at me. Let me say a little a bit about the industry program. One of the things that we’ve been doing there has been focusing more on how the industry program can build on our connections with campus to provide opportunities, for example, for our students to work with companies and companies to work with our students. That’s been very attractive to our partners. I was not at all surprised that the rarest commodity is talent and we are at just a fabulous institution that has very strong, very entrepreneurial students and so that’s been a great connection.
One of the reasons for mentioning the industry program is it was really through that view of connections that we’ve really became a involved in the Smart State initiative. So it was one of the things we discussed with the state including the opportunity for students to be involved in projects, which in some cases could have significant impact improving the life of the state. We are really in the initial phases. It is going to be fun to see how it develops and what works and what doesn’t. It was exciting to see the kinds of opportunities the state was interested in pursing and their flexibility about working not just with NCSA but also with students through programs like this that mirror what we did with industry. (Interview by Illinois Public Media with Gropp on the Smart State effort)
HPCwire: There must be a list of projects?
Gropp: Of course there is but it’s not quite ready to release to the public. We’re working on developing that. I can say another thing that is interesting about this is the state is quite understanding of the fact that in many cases, the sorts of projects they are looking at are quite ambitious, and so the work is being structured as a number of reasonable steps rather than some five-year proposal to solve all the states problems. It’s being structured in much more reasonable pieces where we can perform the pieces and see where we got and figure out what are the next steps.
HPCwire: Given the attention being paid to the rise of data-driven science can you talk a little about the National Data Service consortium (NDS)? I believe NCSA is a steering committee member. What is it exactly?
Gropp: Really it’s just what is says, a consortium trying to help us find commonality and common ground in providing data services. There have been seven semi-annual meetings and there’s the NDS lab which is an effort to provide software, frameworks may not be quite the right word, but to start looking at ways you can provide support for dealing with the five Vs of big data. We sort of know how to deal with velocity and volume, I am oversimplifying it but to some extent, that’s just money. Veracity is another tricky one including provenance and so forth. You can maybe slide reproducibility under that. We have work in that area, particularly with Victoria Stodden, who’s an NCSA affiliate and one of our quite brilliant faculty.
The really tricky one is Variety. There are so many different cases to deal with. Having frameworks to discuss that and places to discuss how we deal with that as well as how we deal with making resources available over time. How do we ensure data doesn’t get lost? Having a consortium that give us a place to talk about these things, a place to start organizing and developing cooperative projects so we are working together instead of working separately – a 1,000 flowers blooming is good but at some point you need to be able to pull this together. One of the things that has been put together that is so powerful is our ability to federate different data sources and draw information out of collections.
My role as NCSA director has been more working with the NDS to ensure it has a long-term sustainable direction because NDS will only be useful if it can help deliver these resources over the time we expect the data to be valuable. I think that’s one of the biggest challenges of big data compared to big computing. When we are doing big computing, you do your computation, you have your results, and you’re done, again oversimplifying. With the data you create the data and it retains value even increasing value and so managing over long lifetimes is again going to be a challenge. It’s important to think of the national data service not as something that one institution is offering to the nation but as collaboration among some of the people who want to support data science in this country getting together to solve these problems.
HPCwire: Sort of a big data related question, can you talk a little about the large synoptic survey telescope project NCSA is planning to support. Its expected output is staggering – 10 million alerts, 1000 pairs of exposures, 15 terabytes of data every night.
Gropp: That’s an important project in our future and was really begun under Dan Reed (former NCSA director). NSF wants those projects divided into a construction project and then operations project, which has not yet been awarded but that proposal will go later this year. [The latter] will do many things; it will operate the LSST facility itself but also the other facilities including the archiving the processing centers. This is significant big data activity that we are fully expecting to be involved in and in fact be leading the data resource side of that.
I don’t have the numbers in front of me but there is a lot of data that comes out of the telescope, an 8-meter telescope. The data is filtered a little bit there and sent by network from Chile to Illinois where it gets processed and archived, and we have to be able to process it in real time. The real time requirement, I think, is in seconds or minutes if not microseconds, but very quick processing of the data to discover and send out alerts on changes. It’s a very different kind of computing than sort of the FLOPS-heavy HPC computing that we usually think about. That will most likely be one of the things that occupies our datacenter, the National Petascale Computing Facility (NPCF).
HPCwire: Given the many resources NCSA operates, what’s required to simply keep all the systems you have running. What the scope of systems supported and ongoing to support changes and maintenance activities for Blue Waters?
Gropp: We were just talking about that this morning. At no time in our history have we been operating so many HPC scale systems. There’s not just Blue Waters. There are systems for the industry program and for some of the other research groups. There’s also a campus cluster, which is officially operated by a different organization but is actually operated by our staff. [It’s an] exciting time to be running all these systems. The big thing we are waiting for is the RFP for the next track one system, and we are still quite optimistic about that.
Some of the systems reach a point of retirement and replacement so we have gone through that cycle a number of times. There was one system that we eventually broke apart and recycled parts out to various faculty members. There are things like that always going on.
For systems like Blue Waters, we have a maintenance agreement with Cray, which has actually been quite reliable. Keeping things up to date is always an issue; for example our security systems are state of the art. There’s a lot of work along those lines, which of course I can’t describe in detail. The biggest challenge for us, a big challenge for all of us in the community, is the lack of predictable schedules from our sponsors for keeping these systems up to date. So we are still waiting for the RFP for the next track one system and that remains a real challenge. That’s why the academy report called on NSF to produce a roadmap because we have to do planning for that.
We also have a lot of stuff that is going into the building (National Petascale Computing Facility) and we have a committee right now that is engaged and thinking about do we have sufficient room in the building, do we have enough power in the building, enough cooling, what do we do when we fill that building up? Those things are made much more difficult when there are so many uncertainties.
HPCwire: I probably should have started with this question. So what’s it like being director? What is the range of your responsibilities and what’s surprised you?
Gropp: I will say every day is different; that’s one of the things that is fun about the job. There are a lot of upper management sorts of things, so I spend time every day on budget and policy and personnel and implementing our strategic plan, but I also spend time interacting with more people on campus in a deeper way and also with some of our industry partners. Meeting new people from the state was really quite eye opening, both in terms of what the state is already doing but also in terms of what the opportunities are.
Last week I went to the state capital and gave some rare good news on the return on investment that they made in Blue Waters. The state provided funding for the datacenter. That was a new experience for me, going to a subcommittee hearing and being able to say that the investment you made in the University of Illinois and NCSA has paid off. Investing in science is a good thing to do and here are the numbers. It’s definitely an intense experience but I found it quite stimulating and different than what I have been doing.
On surprises, even though I have been here for quite awhile (2007), I really didn’t know the breadth and depth of all the things going on. Everyone has the tendency to see the stuff they are interested in and now I am responsible for everything so I have to be aware of everything. That was a very pleasant surprise. I wish I could say that dealing with our state budget situation was a surprise, but it wasn’t; it’s just been difficult. Really I think just coming to understand how much is going on here is more than I expected. In terms of goals, I have some very tactical things I want to get done. These sort of boring but important changes to policy to better support engagement with campus and make it easier for student to work with us, and you’ve already seen some of that in the directions we have gone with the industry program.
HPCwire: With your many director responsibilities are you still able to carry on research?
Gropp: I still have my students. I just met with one before our call. I have a number of grants. So I am still the lead PI on our Center for Exascale Simulation Plasma-coupled Combustion (XPACC). That one I find really find interesting because we are really looking at different ways of developing software for the large scale applications rather than going to new programming models. We’re trying to look at how to take the models we have, the existing code bases, and augment them with tools that help automate the task that skilled programmers find most challenging. I have another project where we have been looking at developing better algorithms and a third looking at techniques for making use of multicore processors for these large sparse linear systems and the non-linear systems they represent. That’s been fun.
Even before I took this [position], we were co-advising the students. That’s another thing I find really enjoyable here at Illinois is the faculty collaborates frequently and we have lots of joint projects. It’s fun for and I think it is good for the students because it gives several different perspectives, and they don’t run the risk of being quite so narrowly trained. One of the other things we have been doing, jumping back up to our discussion of technology, is we have always been involved in bringing in new technology and experimenting with it whether it’s hardware or software and faculty and staff and students and we are continuing to do that. In my role as director I get more involved in making decisions about which directions we are going, which projects. We have one proposal that has been submitted that involves certain kinds of deep learning. That was fun because of the tremendous upwelling of interest from campus.
So I think there will be lots of new things to do. I think if I had not been in this job I would have heard about them and said gee that sounds interesting I wish I had time to for it. In this job I say, gee that sounds great it’s my job to make it happens.
HPCwire: What I haven’t I asked that I should?
Gropp: I think the one question you didn’t ask is “what keeps me up at night.” I’m really very concerned about national support for research in general and high performance or maybe I should say advanced computing writ broadly. We see this in the delay of the RFP. We see it in a fairly modest roadmap going forward from NSF. We see it in hesitancy by other agencies to commit to the resources that are needed. I think we have so much to offer to the scientific community and the nation [and] it has been frustrating that there’s so little long-term consistent planning available. I know that we (NCSA) are not unique in this.
A trade we’ll accept is less money if you will give us consistency so we don’t have to figure out what we are getting every other year. If we had a longer term plan we’d be willing to accept a little less. So that’s the sort of thing. The uncertainty and the lack of recognition of the value of what we do at the scale that would benefit the country. That’s something that all of us spend time trying to change.
HPCwire: The new Trump administration and the economic environment generally doesn’t seem bullish on research spending, particularly basic research. How worried are you about support for advanced computing and science?
Gropp: I think we said many of these things in the academies report (summary) and I still stand behind them. I think we have lots of opportunities but I think we are …I think other countries, and that’s not just China, recognize the value of HPC, they recognize the value in fact in diversity (technologies). I was pleased to hear in the response to a question asked at the NSF hearing this week when the NSF quoted our report, saying there is a need for one or more large tightly-coupled machines and that they took that recommendation seriously.
It would be great if there were more than one NSF track one system. It would be great if there were more than a handful of track two systems. If you look at Japan, for example, they have nine large university advanced computing systems, not counting the Flagship 2020 system in their plans, and that, honestly, is more than we’ve got. So there is a concern we will not provide the level of support that will allow us to maintain broad leadership in the sciences. That’s been a continuous concern.
HPCwire: What’s your take on the race to Exascale? China has stirred up attention with its latest machine while the U.S. program has hit occasional speed bumps. Will we hit the 2022-2023 timeframe goal?
Gropp: Yes, I think the U.S. will get there in 2023. It will be a challenge but the routes that we are going down will allow us to get there. These high-end machines, they aren’t really general purpose, but they are general enough so that there are a sufficient number of science problems that they can solve. I think that will remain true. There will be some things that we will be able to accomplish on an exascale peak machine; there will be a challenge for those problems that don’t map into the sorts of architecture directions that we’re being pushed in order to meet those targets. I think that’s something that we all have to bear in mind. Reaching exascale doesn’t mean for all problems we can run them on one exascale peak system. It’s really going to be, are there enough, which I believe there are. It’s going to be a subset of problems that we can run and that set will probably shrink as we move from the pre-exascale machines to the exascale machines.