Introduction

Many companies today want to make faster, more intelligent business decisions by analyzing larger volumes and new types of data. Storing and staging this data for analysis introduces many new storage management challenges.

Unfortunately, the traditional approach of simply adding more raw storage capacity, often on a per application basis, does not scale well as Big Data analytics volumes grow. If the status quo is maintained and the management challenges are not addressed, storage administration costs will rise, data silos will persist, Big Data analysis workflows will suffer, and organizations might miss business opportunities.

What’s needed is a storage management approach that allows companies to easily add capacity, scale volumes and performance, make efficient use of storage resources, optimize Big Data analysis workflows, and help plan and meet the requirements of new analytics efforts.

Big Data’s management Challenges 

With Big Data analytics, storage management chores keep growing. More drives and devices are needed to house the data. And to ensure high performance server CPUs are satiated, data must be selectively stored and moved to different storage tiers to meet the varying I/O and throughput performance characteristics of each analytics application.

At the same time, the ever-increasing variety of analytics applications now needed to derive business intelligence from new types and growing volumes of data has led companies to use dedicated systems optimized for the different business operations the analytics efforts support.

This approach can make inefficient use of storage. Spare capacity in one effort goes idle, while another group’s effort requires buying additional capacity. This increases CAPEX spending and its impact on IT budgets is compounded by an associated increase in OPEX costs since the added devices must be managed and maintained, take data center space, and must be powered and cooled.

This approach also results in siloing. This further complicates matters. Siloing also prevents organizations from realizing the advantages of a whole-company view of its data. Additionally, since some of the same data (a customer sales database or stock market indices, for example) might be used by different groups for different purposes, having multiple versions of this data increases the total data volume. It also increases the need for multiple data entry, which contributes to multiple versions of the truth.

Making matters worse, by using an optimized storage solution to match the various analytics application’s performance needs, there are often different storage product lines used throughout the organizations. Each type of storage system would typically have its own storage management system. This is often the case even if all of the systems come from one vendor. For instance, some high performance storage solution providers have recently rounded out their product lines through acquisition. And typically, the management systems of the two companies are not integrated for years after an acquisition.

What’s needed?

To keep Big Data analytics workflows running smoothly requires marrying high performance severs with an appropriate performance storage solution. Selection of the solution should take a number of factors into account to help simplify storage management.

To start, organizations need scale-out storage solutions with a file system that supports the volumes of data encountered in businesses today. In this way, all of the data used within an organization can be managed as a single data volume.

A storage solution must also have a management solution that allows centralized configuration of all storage devices, as well as centralized monitoring of device status and capacity. These capabilities would provide a great deal of help in reducing storage management chores for the IT staff versus managing storage devices separately.

Also desirable is management solution that provides performance information. This can be used to help identify potential workflow bottlenecks. IT managers might also use this information to optimize their storage to improve analytics workflows.

DDN as your technology partner

Traditional storage solutions can introduce major management problems when scaled to meet today’s increased requirements for Big Data analytics. In particular, managing the growing volumes of data, combined with the complexity of storage infrastructures, is creating significant challenges for IT managers.

This is driving the costs of maintaining storage infrastructures up. All at a time when budgets and staffing levels remain flat.

That’s where DataDirect™ Networks (DDN) can help. DDN offers an array of storage solutions with different I/O and throughput capabilities to meet the performance requirements of any Big Data analytics workflow. And they are extremely scalable in capacity and density. Based on its Storage Fusion Architecture, the DDN SFA 12K line offers a number of firsts including 15 GB/s host throughput for reads AND writes, 120 Terabytes of storage per drawer (1.8 PB per rack), and the ability to scale to 1,200 drives per array for 3.6 PB per system. Furthermore, DDN lets organizations optimize performance versus cost offering SSD, SAS and SATA Intermix drives.

To manage its line of storage solutions, DDN offers DirectMon™, an advanced storage configuration and monitoring solution. DirectMon works across DDN’s line of DDN SFATM Storage Arrays, as well as GRIDScaler™ file system solutions. (EXAScaler™ support is expected in November 2012.)

All of these DDN systems can be managed from a single console. To that end, DirectMon lets IT managers discover, configure, manage, and optimize complete systems. This leads to great time savings over solutions that require each device or type of storage system to be managed separately.

DirectMon lets IT managers set thresholds and be notified of performance and hardware or software problems to prevent downtime. IT managers can customizable graphs of current and historical trends to get insight into the cause of bottlenecks, resolve problems, and optimize analytics workflows. Additionally, DirectMon regularly collects system configuration and logs as well as works as a central Syslog server. All of these features help frees up IT staff time and helps lower storage management costs.

The DDN solutions thus offer the needed scalability without compromising on performance to help eliminate islands of analytics. By using DirectMon, all DDN systems can be easily managed and monitored. This can reduce OPEX costs by reducing storage management chores, helps organizations optimize analytics workflows, allows for faster troubleshooting and problem resolution, and provides storage capacity insight across the company to help better scale capacity.

To recap, with a unified interface designed to handle all aspects of Big Data storage infrastructure administration, DDN’s DirectMon minimizes IT management workloads and provides a comprehensive framework for both real-time and predictive systems management and tuning of SAN, NAS and parallel file storage environments.

For more information about DDN storage solutions and its DirectMon storage resources manager, visit http://scale.ddn.com/DirectMonPromoAug2012