It’s been a rocky couple of years for the United States government on all fronts, with dwindling budgets tripping over increased demand for maintenance and growth across the board. Despite these concerns, the growth of high performance computing as a dominant part of federal and state organizations remains steady, according to Patrick Dreher, DRC’s Chief Scientist of their High Performance Technologies Group.

“We are seeing strong demand for HPC from state and federal departments and agencies where it is almost universally recognized that HPC is a cost effective tool for not only simulations in science and engineering, but also for development and prototyping,” said Dreher. “HPC has been branching beyond the domain of chemists, physicists and mathematician—and the development is focused on new types of applications that are finding their way into many key areas.”

Few other companies have the insight that DRC has about the actual constraints, concerns and trends for government-fed high performance computing across agencies. DRC, which has been in the government technology consulting and training business since the black and white era, has an insider view on system decisions for the Department of Defense, Department of Energy, as well as a number of military, health, and regulatory agencies.

Given their observations in recent years, DRC has found that the real issue for government organizations when it comes to HPC is dealing with the “perfect storm” of system and application needs. On the one hand, agencies and departments are seeking new architectures that will lend to better efficiency, higher productivity, and better results—all of which need to be wrapped around a very compelling, proven financial story. However, with new architectures comes the need for advancements on the software side, which are far more time consuming and costly than some realize, says Dreher.

While power consumption, hardware and system upgrades are costly, one of the most profound costs agencies face is on the software side, even if it tends to be the hardware upgrades and shifts in architecture that tend to garner the most attention from those watching spending. For instance, a federal department or agency may decide that the moving from a system with around 1000 cores to one with 10,000 or even 100,000 might make good sense for productivity, the software costs of making sure that the application runs smoothly by retuning the file system and addressing I/O and other issues adds significantly to the project’s bottom line, notes Dreher. As another example, while the model for GPUs might be evaluated in terms of hardware costs, Dreher says it’s surprising what such a move can mean cost-wise when entire, stable applications need to be reworked entirely, but it’s a factor they always consider when evaluating the cost-effectiveness of a proposal.

The hardware and software “storm” has another heavy cloud on the horizon, says Dreher. Add the architectural and software concerns to another item on the government HPC agenda—data. Whether agencies are creating new, more robust models or making use of new machines (genomic sequencers, for example) or simply adding cores to boost an application’s capability, this all means the creation (and subsequent storage, processing and curating) of even more data. The biggest problem for agencies is finding ways to manage and extract valuable insight from the influx, Dreher notes. While there are some noteworthy technologies for solving these problems (he pointed to YarcData’s systems as a prime example) this in itself creates another factor to consider.

“One of the biggest concerns among users now, and I imagine in the future, is that they are going to be left with different machines for different problems.” In other words, Dreher says that agencies are finding that they will need to dedicate specific systems to processing data, certain machines for simple number crunching, and others that are specifically geared toward their research. Users want these capabilities in one system but a solution to that problem has yet to emerge.

The storm, which has elements of data, new architectures, the need for more cores and better performance, as well as the inherent software tweaks required that add time and expense, continues to float overhead. Interestingly, this same set of considerations isn’t much different than those that are coming from large companies who are facing similar challenges in relation to cost, productivity, and a constant need for more compute and data handling capability. If one is taking a market view of this from an HPC perspective, this is not just the perfect storm of concerns for users—it’s a perfect moment to capitalize on an undisputed set of needs for a system that considers these elements as a whole to provide an integrated solution.

“The U.S. government is clearly under budget pressure and at the same time, there is pressure from the users and agencies to use HPC to get better productivity and make new discoveries. Up until about 10 years ago, the agencies and departments could count on the vendors to come out with a new and advanced archicture. The users would then count on the fact that they would just make a few small change– they’d up the clock frequencies, up the transistor count. But that trend came to an end since you can’t keep upping the clock frequency, for example—they need to find other ways to get better performance,” said Dreher.