Is high performance computing in jail? Is number crunching a prisoner of policies and practices that are decades old?

The policies and practices that define the way we currently deploy and use our number-crunching HPC resources are based on assumptions that are, at best, several decades old. A lot has happened in the meantime, for example, the emergence of a sizeable and growing cadre of computational scientists and engineers, the internet and the web, continuous weakening of the coupling between work and location, the rise of data-intensive computing, commercial computing providers with staying power, and a broad consensus that the future health of our economy depends on intelligent and extensive exploitation of HPC. Perhaps it’s time to revisit those premises.

The major premises were that the engines of HPC are unique, its people are scarce, and the resources need to be held close in protected locations, its “temples.”

The engines of HPC are unique

In the early days of HPC, our “big iron” computers were crafted by hand and in small numbers. Those computers were in a class by themselves. We put them on display behind glass walls in machine rooms when they were new and, when they’d passed their prime. We retired them as museum pieces or lobby furniture. Have you ever sat on a Cray-1?

Since the late 90s, our HPC engines have been increasingly built from commodity parts and in scalable fashion. Indeed, those commodity parts are now reaching a degree of sophistication at the chip level that we previously only found at the machine level. HPC is now based on mass-produced hardware.

These computing systems have also become more modular – some to the extent of being assembled in Lego fashion from shipping-container-sized building blocks. Increasing numbers of computing floors are being operated in “lights out” mode, supported by very small numbers of on-site personnel.

The only real discriminators among HPC engines are: system balance (see: Number Crunching, Data Crunching and Energy Efficiency: the HPC Hat Trick); the size of the system one can afford; the floor space available to accommodate it; and the power bill one is prepared to pay.

The people of HPC are scarce

In those early days of HPC, there were no “computational scientists”. The term computational science was introduced by Ken Wilson around 1984. OK, maybe some of us were computational scientists but didn’t go by that name. The point is that computational science and engineering were just emerging as career fields and the number of practitioners was quite low.

Today, although there is still a supply gap, there are reasonably large numbers of computational scientists and engineers. They work in our universities, national labs and in industry. There are programs to educate them, journals to publish their work, focused meetings and conferences for them to attend, differentiated social networks for them to participate in, and human resources people who specialize in recruiting them.

HPC centers are temples

Back then, the scarcity of HPC users was complemented by a scarcity of people who were skilled at building and maintaining these specialized systems. That’s because it involved gluing various subsystems of exotic hardware and software together into functioning computers, tuning their physical environment, keeping them healthy and available, and making them accessible by remote users.

Additionally, teams of people were needed to offer training courses in the various aspects of this new thing, then called “supercomputing” and now called HPC. User support groups provided help in porting and tuning existing applications codes and expertise in writing new ones. Visualization specialists helped the end users create meaningful displays of their results, mostly by post-processing results on specialized visualization subsystems.

Given the exotic nature of supercomputing, the scarcity of the skilled people involved, and the complexity and interdependency of the tasks they performed, it made good sense to centralize HPC activities around a small number of centers. Today, there are vastly more HPC centers supporting various number-crunching applications in science and engineering.

Additionally, our definition of HPC has been enhanced to include the burgeoning field of data-intensive science. A whole new class of very large datacenters has come into existence to support the rapid growth and broad deployment of data-crunching applications in commerce, finance, marketing, social network analysis, and other fields. Furthermore, the locus of HPC activity seems to be shifting from the traditional protected locations (i.e., the temples) out into the enterprise. In concert with these developments, the operation of our computing centers has become more standardized and a larger pool of skilled people is available to support them.

Progress has overrun our premises

Depending on how you count it, we are three or four decades into this adventure we now call high performance computing. HPC has been very successful and its successes have altered the landscape of computing and its applications.

Our highest end computers are no longer unique but rather, by our own design decisions, scaled up versions of smaller and more common machines. Computational scientists and engineers are now routinely educated and employed, even if not in the numbers we might wish. HPC centers have lost a good bit of their original mystique as they have proliferated across the landscape.

Additional elements have also come into play. Ubiquitous access to the internet and world-wide web have made computing accessible to a large segment of the Earth’s population, not just those lucky few connected to some dedicated network. So, our HPC specialties can be practiced from just about anywhere. Commercial providers of computing services have reappeared, after a long hiatus , and have come back in much more resilient form (for example, Amazon, Cisco, Dell, Google, HP, IBM, Microsoft, and SGI).  It seems fair to say that commercially available computing services are here to stay.

On any given day, HPC in the Cloud is replete with news about these services. Indeed, we are told that some companies have gone so far as to compute exclusively in commercial clouds. Also, the Federal CIO has recently issued a memorandum encouraging federal agencies to develop trusted relationships with cloud service providers.

As for HPC applications, in the early days of supercomputing they were all about number crunching and we touted them as means for pushing back the frontiers of science and contributing seed corn for the future growth of science and engineering. There were no externally imposed milestones or deadlines associated with using computing in this mode for scientific discovery. Now, HPC is seen in much more immediate terms, as a problem-solving tool for economic success. Thus we desire a much swifter and more predictable transition from research into practice.

There has also been tremendous growth in data crunching applications, increasingly referred to as “big data.” Distinct from number crunching, these applications operate substantially outside the sphere of federal support. This other aspect of HPC is rapidly evolving and creating its own varieties of computing and data storage capabilities (for example, Facebook’s Open Compute Project and Open Storage Pod). Even the grande dame of supercomputing, Cray, has recently formed a Big Data Division. To follow the data-intensive computing and big data action, take a look at Datanami.

So, given the things that have changed and the new elements that have appeared, how might we adjust the premises on which we base our HPC policies and practices? Here are a few thoughts.

It’s about scalability

The elite machines in our computing ecosystem will always be, by definition, unique. We should take their care and feeding seriously. But the apex has now been pushed so high that computing seems really to be about “whateverscale.”

While are rightfully advocating for the development of exascale computers, most of our end-user applications compute at the terascale level, at best. They are thus “behind the computing curve” by a factor of 1,000,000, or soon will be. So, perhaps we should encourage innovation and growth of computing applications at whatever scale is appropriate to their immediate needs.

To the extent that we focus on moving “whateverscale” applications forward, their eventual transition to “apexscale” should be easier and come sooner. Furthermore, apexscale machines will be scaled-up versions of the smaller, whateverscale ones. Thus, the code development and architecture exploitation lessons learned at whateverscale will carry over to those top systems.

It’s about empowering people

In the early days of supercomputing, there were very few computational scientists , that is, people who were well versed in both their science field and in plying their science on our highest-end computing hardware. So, for allocating public resources, we created a dual review system. A traditional merit peer review was conducted to inform the process of allocating money and a separate “computational readiness” review was conducted to ensure that any compute cycles allocated would be used wisely.

Over the past several decades, we have been successful in producing lots of computational scientists and engineers – people who are very knowledgeable about both aspects of their profession. However, this dual review process is still in widespread use, especially by federal agencies. Thus, there is an opportunity to remove this form of “double jeopardy” by reverting to a single, comprehensive, review of computational science and engineering proposals. Doing this would also acknowledge our current generation of computational scientists and engineers for the skills they have worked hard to acquire. Emancipating and empowering computational scientists to better control their own fate can only serve to strengthen and grow the field.

It’s about the economy

When supercomputing was in its infancy, it was important to hold it close, to protect and nurture it. Consequently, our HPC resources were concentrated at federally funded research and development centers. Also, although there was a desire for these resources to have an economic impact, the kind of compulsion to do so that we see today was largely absent.

To be sure, there were a few companies that were early adopters of number-crunching HPC and continue to use it to great benefit. They generally employ their own internal computing resources. You can find some of them listed in the Top500 list, while others see competitive advantage in remaining invisible. What was missing then was an open, generally available, and well-supported commercial HPC infrastructure that could be used by much broader segments of the economy.

A few decades ago, there was an attempt to stand up commercial computer service bureaus and make supercomputing generally available. This attempt was less than successful, serving to reinforce the notion that custody of resources should be left to the government, its contractors, and perhaps a few other stalwarts.

Today, the computing and networking environment is vastly improved and our economic incentives to compute are intense. Data sciences and analytics have led the way in clearly establishing that commercial HPC is a viable enterprise. In many dimensions, this commercial form of HPC is now larger than our traditional government-sponsored form.

This presents all of us, including the federal HPC establishment, with the opportunity to embrace commercial HPC and to strengthen and grow it into a fabric to support the innovation in products and services that we’ve identified as necessary for the future health of our economy.

To be clear, embracing and using commercial HPC resources would not serve as a replacement for the vitally important work done by our government HPC complex. Rather, it would free this complex to pursue its essential role in HPC: pathfinding our computing future and making it real.

If HPC resources and services that are currently provided by the government complex can be provided commercially, then we should avail ourselves of this opportunity to build out our national computing infrastructure and actively support economic growth. If aspects of the current commercial offerings fall short of our number- and data-crunching needs, then we should aggressively transfer the knowledge and technologies necessary to remedy those shortcomings. Perhaps most importantly, we should use these commercial resources to the maximal extent feasible, to help grow and strengthen the marketplace and its offerings.

Rethinking things…

The premises on which we have built our HPC enterprise have clearly changed over the past several decades. While those premises served us well in getting HPC established, they may now be limiting its further development. Perhaps it’s time to jailbreak HPC.

—–

About the author

Gary M. Johnson is the founder of Computational Science Solutions, LLC, whose mission is to develop, advocate, and implement solutions for the global computational science and engineering community.

Dr. Johnson specializes in management of high performance computing, applied mathematics, and computational science research activities; advocacy, development, and management of high performance computing centers; development of national science and technology policy; and creation of education and research programs in computational engineering and science.

He has worked in Academia, Industry and Government. He has held full professorships at Colorado State University and George Mason University, been a researcher at United Technologies Research Center, and worked for the Department of Defense, NASA, and the Department of Energy.

He is a graduate of the U.S. Air Force Academy; holds advanced degrees from Caltech and the von Karman Institute; and has a Ph.D. in applied sciences from the University of Brussels.