In 2007 Tabor Research established the supply-side and demand-side research methodologies to determine the size and dynamics of the High Productivity Computing (HPC) market. From a technology perspective, we are investigating the products and services that complement the server or cluster portion of the HPC market, in order to establish the true market size beyond servers. From a usage perspective, we are looking beyond traditional HPC markets to new “threshold” application areas that have the potential for significant continued growth.

In mid-January Tabor Research will release preliminary 2007 revenue figures for the traditional HPC server market. This estimate will encompass all systems sold within traditional HPC markets, across industry, government and academia, including application areas such as manufacturing, oil and gas exploration, bio and life sciences, financial analytics, digital content creation or distribution, healthcare, utilities, academic and national research, and national defense.

We are currently validating our final figures with the HPC vendor community, but at this point we can state that the traditional HPC server market amounts to between $7.3 billion and $7.8 billion in 2007, which will represent single-digit growth rates from 2006. A critical basis to our methodology is that HPC servers represent about 15 percent to 17 percent of all server sales by revenue. Complete breakdowns of the traditional HPC server market – by product class (entry-level, midrange, high-end server, supercomputer), user sector (academia, government, industry), geography (U.S. and Canada, Latin America, EMEA, Asia Pacific), and vendor share – will be made available to subscribers to the Tabor Research InterSect360 market advisory service and to the user members on the HPC User Views Advisory Council who contribute to our demand-side research modules.

Prediction #1: Look for Technology Innovation outside the Server

The HPC market goes well beyond servers. In our first demand-side report published in 2007, users revealed how they allocate their HPC budgets among and within major spending categories: hardware, software, services, facilities, staffing, and outsourcing. A major finding of this study was that users spend only about one-third of their total HPC budgets on servers. They spent roughly an equal amount on other HPC products and services, with the remainder going to internal spending categories, such as staffing, power consumption, and cooling.

Counting servers as half of the total external spending in the HPC market means we can at least double the server revenue number to come up with the total size of the traditional HPC market, including items such as storage, interconnects, and services, to approximately $15 billion. In truth, this figure is slightly higher because of “not in budget” effects for some categories of spending, especially storage, in which additional monies come from IT budgets that support HPC but are not HPC-specific.

Among the non-server product spending categories, storage is perhaps the most critical. Not only does it command a significant percentage of the HPC budget, but it also is a technology area that can have enormous influence over the true productivity of a system. Furthermore there are technology shifts in play in file systems and storage interconnects that complicate users’ buying decisions. A comprehensive research study conducted by Tabor Research in 2007 concluded that in addition to simple metrics such as increased capacity and higher bandwidth, users are interested in storage solutions that drive improvements in application performance.

Interconnects are another major component of the overall solution, and they are another area that is undergoing a technology change. The growth in InfiniBand adoption has been a hot topic in 2006 and 2007, but at SC07 we noticed many companies showing their new 10 Gigabit Ethernet solutions. What interested us most about this was that rather than the newest generation of Ethernet simply appearing as a matter-of-fact “it’s here, and we have it” announcement, many vendors were touting differentiation in their 10GbE solutions. If users view Ethernet as something more than a check-box item, the dynamics of the interconnect market will be important to watch in 2008.

Prediction #2: Multi-Core Drives Further Node Changes

As multi-core becomes the standard option for new HPC purchases, users will continue to grapple with the introduction of parallelism at the socket level. In addition, memory and cache contention will become major bottlenecks for certain applications. The rapid shift to multi-core will drive innovation in several related areas within the node.

With increasing numbers of processing cores per socket, users will need to rebalance node configurations to optimize data movement between processors, memory, and storage. The challenge here is not to overload a node with processing cores that will be idle for most of the time waiting for data from memory or disk. For some users it may prove more cost-effective to reduce processor chip count and spend their money on larger memory and additional I/O ports.

As another option for increasing node-level performance, 2008 will see widespread experimentation with various accelerator technologies. We expect to see an earnest effort made to employ the full panoply of accelerator options (e.g. FPGAs, GPGPUs, game chips, purpose-built accelerators, etc.) across virtually all applications segments. However, these efforts will generally be in the “proof of concept” phase, with relatively small numbers of accelerators deployed. We expect this experimentation to continue into 2009.

Prediction #3: Software Strives to Catch Up

The demand for software tools that can reduce the complexity of programming and managing multi-node, multi-processor, multi-core systems will reach an all-time high during the year. Users will seek software innovations across the entire range of HPC activities.

• Responding to multi-core — Users will need to implement job scheduling and optimizing software to maximize the efficiency of clusters based on multicore processors. In addition, users will look for software solutions to help them deal with power and cooling challenges within the computer room.
• Programming complexity — Like death and taxes, programming complexity will always be with us (and just as welcome). 2008 will test the limitations of MPI and current programming methods, while opportunities open up for extensions of existing programming tools and development of new approaches.
• Support for accelerator implementation — Programmability is the most limiting factor to accelerator adoption. The ultimate winners in the accelerator race will be companies that can provide both efficient processor-to-accelerator operations and effective programming tools for accelerator use.

Prediction #4: Productivity Measurements Get More Sophisticated

Productivity encompasses more than price/performance. This becomes more evident with the adoption of multi-core systems with high peak performance and potential programming challenges. Users have embraced the concept of evaluating the usefulness of HPC systems based on productivity in theory, but putting it in practice for actual acquisitions has been difficult, because the “productivity” concept lacks simple metrics and tools.

In 2008 Tabor Research will introduce tools for users to better evaluate end-to-end workflow optimization – a major step in supporting its emphasis on productivity beyond performance. In addition, several users have already designed their own internal productivity metrics, and many HPC system and solution vendors are showing interest in productivity metrics as well. For example, Microsoft highlighted productivity as the centerpiece of its HPC++ campaign unveiled at SC07.

Prediction #5: Threshold Applications Are the Major Growth Area

Advancements in technology, improvements in price/performance, and true productivity gains have combined to make HPC an attractive investment in non-traditional application areas. Tabor Research refers to these as “threshold applications.” A threshold HPC application is one that:

• Does not fit into established, traditional definitions of HPC applications, or is a distinct subset within a category that has not previously benefited from HPC.
• Is enabled by recent advancements in HPC technology, price/performance, or productivity.
• Reflects traditional HPC applications in its overall workflow or type of configuration required.
• Has the potential for significant growth in HPC, i.e., computational methods could become the new paradigm.

For example, in the finance industry, HPC systems have traditionally been deployed for financial analytics or portfolio risk analysis. Recently some users have begun to use HPC system for real-time trading solutions. This is a new application subset with an HPC workflow that shows potential for industry-wide adoption.

Another important category within threshold applications is “ultra-scale business computing.” This is our catch-all designation for applications that are not HPC in of themselves, but which require supercomputing levels of scale for certain users, thus resulting in the use of proprietary or HPC-specific solutions or technologies. For example, web-based transactions or searches are not HPC applications per se, but eBay and Google use supercomputing technologies to run them at extreme scale. Tabor Research tracks this usage as part of the threshold supercomputing market.

These threshold applications represent a significant opportunity in the market, and they have the potential for significant growth. We are currently modeling market sizes for these applications (a comprehensive research study is pending in 2008), and at present estimate an additional 35 percent of revenue to the HPC market.

Conclusions

When we consider the traditional HPC server market, additional spending on technologies outside the server, and the impact of non-traditional, threshold applications, the total HPC market is on the order of $20 billion to $28 billion. HPC overall is growing, but the fastest growth rates are in the threshold application areas.

Tabor Research predicts that the focus for high productivity will shift away from the server in 2008 and toward the overall computing environment, including storage, interconnects, software and accelerators. This focus will come from the drive to achieve true productivity, and users will strive to quantify and optimize the productivity of their HPC investments.