[Connect with HPC users and learn new skills in the IBM Spectrum LSF User Community.]
Rapid rates of change sometimes result in unexpected bedfellows.
Take mainframes for example. Today, high-performance computing (HPC) mainframes are as important to research and business as ever before.1 87% of all credit card transactions and nearly $8 trillion in payments annually are processed on mainframes. These HPC systems manage 30 billion ATM transactions each year, equivalent to nearly $5 billion per day. 2
At the same time, more than 85% of enterprises around the world now operate in IT environments that incorporate multiple public cloud services providers, quite often integrated with private cloud solutions. These are called hybrid, multicloud environments. And in the next few years nearly all enterprises will move in this direction. 3
More accurately, enterprises that leverage the power and efficiency of on-premises mainframe servers are incorporating both public and private cloud-based resources into their overall data processing solutions.
This unexpected combination of technologies is driving the need for complementary systems that can keep pace and even help accelerate innovation. Hybrid, multicloud capabilities and functionality are cropping up on both the compute and the storage sides of mainframe data processing.
[Read also which technologies are helping companies to boost HPC cloud performance.]
Advancements in Cloud Cluster Management and Storage Technology
On the compute side, cluster management solutions have emerged that enable mainframe environments to leverage the cloud during times of peak activity. To accommodate spikes in demand, traditional HPC environments often divide up jobs and stretch out scheduling – but this can lengthen time to insight. New cluster management capabilities such as those found in IBM Spectrum LSF help address this challenge by enabling dynamic access to cloud resources.
Not to be out done, data storage technology has also evolved quickly to better participate in this significant shift away from isolated on-premises computing. These developments mirror the basic trajectory of storage over the past decade, which has seen flash media move into the role of handling active data, pushing disk further toward the role of cool data management. This trend results in a natural storage tiering effect where tape still plays an active role in data archiving and cost reduction. Cloud storage resources can be incorporated to supplement or even replace the lower performance and cost tiers, both in the form of standard block storage and now as cloud-based object storage.
As flash media has taken control of active data storage duties, dynamic tiering functionality has become a standard feature of enterprise and HPC-grade storage systems. Dynamic tiering algorithms monitor the activity or “heat” of data sets and automatically move them to the appropriate type of storage to optimize established business priorities such as performance or cost.
[Also learn how to avoid becoming trapped in the Cloud.]
The Importance of Cloud Workload Management
IBM Spectrum LSF is a workload and resource management platform for demanding, distributed HPC environments. It supports traditional HPC and high-throughput environments, as well as big data, artificial intelligence, GPU, and containerized workloads, among many others.
Two of the most common IBM Spectrum LSF cluster solutions are the LSF Stretch Cluster and the LSF Multi Cluster configurations. In the LSF Stretch Cluster architecture, the master scheduler and other core functionality remain with the on-premises IBM Spectrum LSF cluster, but the cluster resources can be dynamically “stretched” over a WAN to include cloud resources.
The Multi Cluster configuration, on the other hand, essentially creates two clusters, one on-premises and one in the cloud. This architecture can simplify communication and coordination between the on-premises and cloud-based clusters.
Each cluster configuration has its advantages and disadvantages, but both accomplish the main goal – making mainframes “cloudier.”
Integrating On-Prem Storage with Cloud
DS8000-family storage arrays make this functionality cloudy by incorporating off-premises storage targets such as AWS or IBM Cloud.
The IBM DS8000 family of data systems and the IBM TS7700 family of virtual tape libraries (VTL) offer excellent examples of the new storage architectures. Both families can support distributed computing environments, but both are more focused on mainframes. They benefit from years of trailblazing research and deep collaboration between the IBM Storage and IBM Z teams. Most importantly for this particular discussion, both storage families offer transparent cloud tiering functionality.
VTL solutions like IBM TS7700 provide a layer of disk storage for cooler data sets. Because the DS8000 and TS7700 families are even closer than first cousins, they easily connect to mainframe processors on the frontend, and to true tape – and thanks to their transparent cloud tiering capabilities –to various cloud resources on the backend. The result is a multicloud network of storage resources connected by intelligent tools that transparently move data around the data center – and around the world – to maximize performance and efficiency.
And by the way, don’t assume that security is compromised as all this data flies into and out of the clouds under its own recognizance. Solutions made from the IBM DS8000 and TS7700 families integrate seamlessly into the IBM Z “pervasive encryption” regime so that data is protected both at rest and on the fly.
It’s an inevitable phenomenon – mainframes are getting cloudy. But just like every big thunderhead results from the interplay of many complex underlying currents and forces, so too the shift away from isolated on-premises computing, especially in mainframe environments, has been driven by plenty of on-going innovation and hard work.
 Compuware: Inside Tech Talk: New BMC Survey: Digital Business Impacts Mainframe, Is Here to Stay, November 2016 (https://compuware.com/new-bmc-survey/)
2 IBM Mainframe Ushers in New Era of Data Protection, July 2017, https://www-03.ibm.com/press/us/en/pressrelease/52805.wss
3 IBM Institute for Business Value: Assembling Your Cloud Orchestra, October 2018 https://www.ibm.com/downloads/cas/EXLAL23W