The HPC conference season started in earnest this Tuesday with the HPCC event, a two-and-a-half day conference that attracts a rather elite lineup of speakers and attendees. HPCC, aka the Newport conference (after its Rhode Island locale), is celebrating its 25th anniversary this year, and as someone pointed out, that makes it a couple of years older than the much better-known (and much larger) SC conference of November.

HPCC is geared for the government supercomputing crowd, but topic coverage is broadly spread across the HPC application spectrum. The first day was no exception; it included everything from climate modeling and computational engineering to exascale computing.

Two of the sessions on Tuesday had to do with the infamous "missing middle" problem in HPC. That termed, coined by the Council on Competitiveness, refers to the group of HPC users between low-end, mostly workstation-bound HPC users, and the kind of high-end HPC typically performed at national labs and some universities. The problem is that transitioning from desktop HPC to server-based HPC is filled with roadblocks, especially for commercial users looking to make the leap to small clusters.

At HPCC, we got two perspectives on the problem -- one from a vendor, the other from a user. The user in this case was Caterpillar, a company that is fairly well along in the adoption of HPC. In that sense, they're no longer missing, but they're definitely in the middle. According to Keven Hoffstetter, Caterpillar's research program manager for their Virtual Product Development group, they use HPC-based virtual tools to drive much of their product development.

Caterpillar is the largest manufacturer of construction and mining equipment, which may make it seem like they're not a "middle" HPC user at all. But keep in mind that the term refers to a capability level, not to company size or R&D budget, which in the case of Caterpillar is significant on both counts (93K employees and $1 billion-plus for R&D). Even though they use a number of virtual tools to model things like engines, cooling systems, and linkage systems, they do so at a relatively low capability level.

For example, Caterpillar does modeling of a differential system using a machine with just 64 cores. That simulation takes 4 weeks. According to Hoffstetter, they would like to increase the performance by at least an order of magnitude so that they can run that model much faster and do hundreds of them at a time. The idea is to go through many iterations as possible in order to optimize the design.

Despite the modest level of use, they've managed to realize significant gains from these tools. They're no longer tied to a 10-year product development cycle, which used to be the accepted timeline at Caterpillar. The virtual modeling speeds up all development phases: concept, design, and testing.

But for Caterpillar, there are still major hurdles to get to greater levels of HPC including getting software to scale to greater level, software licensing costs (which Hoffstetter said are already greater than hardware costs), software usability, multi-physics integration and dealing with increasing hardware complexity (including divergent technologies like CPUs and GPUs).

But Hoffstetter contends the biggest problem there is the cultural change required to move to a complete virtual design paradigm. Basically the fear of doing something differently and the concern that the virtual models do not accurately represent the physical product hinder the acceptance of these tools internally. Besides, the models don't yet simulate everything, so the engineers end up having to build a physical prototype anyway. In such an environment, the adoption of this technology is by degrees, rather than en masse.

From the vendor side, we had Intel's Stephen Wheat talking up the democratization of HPC. Intel, of course, has a big stake in broadening the HPC market as far and as fast as it can go. Wheat said HPC currently represents north of 25 percent of their processor revenue and volume. Obviously, expanding that market means big bucks for the chipmaker.

To Intel, this missing middle appears to be low-hanging fruit. A specific focus lately has been the manufacturing industry, which represents a particularly large group of users who have a quantified need for HPC-level simulation and modeling. Intel estimates of that there are about 280,000 small and mid-sized manufacturers in the US alone, and nearly half would use advanced simulation and modeling. That represents a market nearly as large as the entire global HPC segment today.

To that end, Intel has been a big driver behind the newfound Alliance for High Performance Digital Manufacturing (AHPDM), a group devoted to bringing high performance computing to manufacturers. It is especially targeted to these smaller players without the resources or expertise to swallow HPC whole. The alliance consists of industry players, ISVs, academia, and HPC centers of various stripes. The idea is to leverage the existing HPC expertise and infrastructure in order to connect to dots for the manufacturing companies.

Intel is not the only vendor in this group. Microsoft, HP, Dell, NVIDIA, ANSYS and Cray (to name a few) are also part of this. (Disclaimer: HPCwire's publisher, Tabor Communications, is an Alliance member too.) In any case, I suspect we'll be hearing a lot more about AHPDM in the future as some of the efforts get going.

As Wheat said, the definition of success if fairly simple: "When the middle is no longer missing."