At all times in the technology industry, new innovations emerge that either disrupt or enhance the capabilities of computer systems. Innovate or be replaced – that’s the ongoing dynamic. In the world of high-end data storage, hyper-convergence is “The Great Unifier,” enabling the full potential of processor, networking and storage technologies to work in concert at extreme scale, synthesizing within a single, integrated, optimized and comprehensive whole a unified framework that will serve as the storage platform for data-intensive organizations of the future.

At SC15 in Austin, TX, DDN will roll out SFA14K, the most powerful hyper-converged platform on the market. SFA14 addresses three major storage challenges:

• Closing the gap between processing power and storage capabilities in high performance and advanced scale systems.
• Overcoming integration and performance problems of storage solutions that comprise disparate components across a network.
• Delivering on the growing demand in the enterprise and in higher education to leverage data as an analytical, modeling, simulation and predictive tool – that is, to utilize data as it has been used in HPC for many years.

While the processing power of high-end systems has generally adhered to Moore’s Law, storage performance lags behind. SSDs, flash memory and other new storage technologies of recent years have improved performance, but throwing together fast but unmatched components results in latency problems, idle compute capacity and slower time-to-solution – along with systems management headaches.    An optimized hyper-convergence solution that incorporates advanced storage technology within a single box significantly reduces latency while expanding memory footprint and IOPS, maximizing performance in data intensive computing environments. It also stretches IT budgets. And because it is supported by a single vendor, it allows single system management with a common toolset.

An Advanced Approach

DataDirect Networks (DDN) is at the forefront of hyper-convergence for storage, bringing compute, storage, networking and server virtualization into one chassis that can be centrally managed. The vendor of choice at two-thirds of the world’s most demanding supercomputing centers, DDN has more than a decade of experience integrating solutions that combine the newest and most advanced storage technologies.

DDN’s new SFA14K is the building block for upcoming DDN storage solutions. In just 4U of rack space, it delivers:

• More than 60 GB/sec throughput, processing power that remove storage bottlenecks while maximizing velocity and compute ROI, and nearly double that of any competing system
• 6 million IOPS, three times faster than any other storage solution
• Latency reduction, on average, from 10x to 1000x

The system can hold 1,752 hard disk drives and solid-state drives (SSDs) delivering industry-leading density of over 7 PB/ rack, and supports OpenStack, Lustre, GPFS and a wide range of market specific applications that a user may specify.

“It’s all about integration of high-performance storage within a unified fabric,” said Molly Rector, DDN’s CMO and EVP of Product Management & Worldwide Marketing. “You have your applications, networks, memory and processing all built into a cohesive system able to transparently connect to the archive as well to bring all those pieces under tight management, within a consolidated high-performance network.”

DDN’s performance superiority is the result of the company’s unique product development strategy pursued over the past several years, according to Rector, one that diverged from the prevailing commodity hardware strategy adopted by many competitors.

“Other companies fell into using cheap, commodity hardware, stringing together a large white box storage array taking up significantly more space, components and networking hops to integrate,” said Rector. “But that’s like herding cats. It involves complex systems integration and systems management, it has many fail points (network ports, switches, cables), high latency, high power consumption, endless troubleshooting, high TCO and lower performance.

“If you’re talking one or two boxes then the white box approach can work, but not in an extreme scale environment with petabytes of data running against high-end analytics, modeling and Big Data applications,” she said.

Traditionally, storage systems are sized based on anticipated peak performance requirements, not sustained requirements – which drives the need to purchase additional hardware and increases CAPEX. DDN has taken on this problem with a new software technology that extends the memory footprint to whatever bandwidth is required by the customer. Called IME (Infinite Memory Engine), its ability to expand memory bandwidth means IME absorbs “bursty” workflows, Rector explained, decoupling performance from capacity, enabling systems to continue to process jobs while systems are most actively used and reducing storage hardware requirements by up to 70 percent.

“IME pushes the content, the data being created, into a high-performance storage cache that accepts data as fast as the compute environment can create it,” she said. In addition, IME intelligently reorders the IO so that it writes data to the file system in the most efficient way possible.

The result: IME can accelerate file system performance by 100x and I/O performance by 1000x, critical functionality as the HPC industry evolves toward exascale compute environments and the enterprise tackles massive Big Data projects.

SFA14K and other DDN platforms utilize DDN’s Web Object Scaler (WOS) for fully integrated data archive and content distribution.  WOS is a hyperscale object storage platform that enables high-performance storage, web-scale storage clouds, cost-efficient active archives and real-time global collaboration infrastructures. WOS addresses the massive scalability, security and distribution requirements of Big Data. The second most widely used object storage in the world (Amazon Web Services is No. 1), it simplifies and automates archiving, lowering the cost of long-term storage vs. public cloud alternatives, by 2 to 3x.

While DDN is the most widely used provider to the HPC industry, the company’s storage solutions also are seeing rapid adoption in mid-range HPC and in data-intensive enterprise environments.  With systems that start under 100 TB and a few GB/s, DDN entry-level offerings deliver the best price/performance and the smallest footprints to achieve capacity requirements, meeting the growing needs in these sectors that traditional storage solutions vendors simply cannot provide.

The three markets have differing needs and dynamics. While HPC tends to place a premium on extreme scale performance at the lowest possible price, the enterprise values data security and data management, including inline compression and encryption. Meanwhile, higher education tends to run jobs requiring relatively smaller datasets, but it requires fast data sharing among fellow researchers across university systems and research consortia. Both of the mid-market sectors need support for a wide variety of applications and file types. DDN tailors its hyper-converged offerings for the varying storage environments of the three markets.

Although adoption of the technologies and techniques DDN leverages in building hyper-converged storage solutions is relative new to the enterprise and higher ed, HPC has served as a proving ground, enabling low-risk early adoption of the technology.

“I suppose you could say the enterprise and higher education are ‘crossing the chasm,’ ” explained Rector, who said DDN’s biggest source of growth since 2012 has been from Fortune 100 companies, the financial services sector, cloud storage and web companies. “But our technology has already been proven out in most extreme use cases in the world over last 10 years. HPC has tested the technology at scale, at performance, at the quality of what the enterprise needs – more than what it needs.  Our hyper-converged technology is actually quite mature. HPC is a large-scale test case for enterprise market.”