Today, the United States Postal Service is on its third generation of supercomputers, with each generation more capable than its predecessor. IDC believes the USPS embrace of HPC exemplifies an important, accelerating IT trend: Leading organizations in the private and public sectors are increasingly turning to high-performance computing to tackle challenging big data analytics workloads that traditional enterprise IT technology alone cannot handle effectively.
The USPS case history is instructive. Each HPC generation employed by the USPS has been based on an SGI UV supercomputer provided by prime contractor FedCentric Technologies, based in Chevy Chase, Md. The SGI UV system stands out from the cluster crowd primarily because of its large, NUMA shared memory space that is designed to scale as high as 64TB. In this instance, architecture matters.
The Postal Service reports that the ability to process very large data problems entirely in memory can boost performance by three to six orders of magnitude. Conversely, having to process data outside of a shared memory can severely cut performance — by about 50 percent when moving data from one blade to a neighboring blade; by two-thirds when moving from the top blade in a rack to the middle blade; by a factor of 25 when moving data from the top blade to the bottom blade in the same rack, and by a far greater magnitude when transferring data between racks.
(This recognition gave rise to the USPS-coined term “high-density supercomputing,” which refers to the notion of keeping memory and processing in tight physical proximity —”affinity,” in USPS terminology — and using an in-memory database to attack big data problems.)
IDC uses the term high-performance data analysis (HPDA) to refer to workloads that are daunting enough to require HPC technology. The primary factors driving the HPDA trend are the complexity and time criticality of the most challenging big data workloads. HPC can enable organizations to aim more complex questions at their data infrastructures and obtain answers faster, even with more variables included. IDC forecasts the global market for HPDA servers and external storage will grow robustly from $1.4 billion in 2013 to $4.3 billion in 2018.
Click here to continue reading the rest of this article on our sister site, EnterpriseTech.