According to findings from Hyperion Research, simulation is primarily responsible for expanding the global HPC market from $2 billion in 1990 to a projected $38 billion in 2022. And one of the fastest-growing components of that forecast is high performance data analysis — using HPC systems for data-intensive simulation and analytics.
HPC simulation began in government and academic research organizations, to tackle daunting problems in the “hard sciences”: physics, chemistry, biology, astronomy/cosmology and geology. HPC soon spread to a broad spectrum of private sector firms, from large global enterprises to 25-person SMBs. Even within academia, the use of HPC simulation now extends to disciplines including cultural anthropology and archeology, historical linguistics and the social sciences.
The use of HPC systems primarily for integer-based, data-intensive computing, as opposed to floating point-based simulation, began in the intelligence/defense community in the 1960s, at the start of the supercomputer era, and spread to large investment banks in the financial services industry in the 1980s. Today, HPDA (high-performance data analysis) and AI methods are being added to the computing mix of traditional HPC users, most of whom pursue upstream R&D using dedicated HPC data centers. These methods are also motivating a growing number of commercial firms to integrate HPC systems into enterprise data centers to advance business operations, especially fraud detection and cyber security, business intelligence, affinity marketing, ERP and sales planning. Commercial firms are being driven to do this by competitive forces, especially the need to direct more complex questions at their data structures in near-real time.
Simulation remains by far the most popular problem-solving method associated with HPC, but simulation and HPDA-AI data analytics will increasingly join forces to tackle problems faster and more thoroughly than either approach could alone. Intelligent simulation is the favored term for this potent combination. Here are examples of the mutually beneficial relationship between AI and HPC –
- Analytics Helping Simulation. Climate research historically has been one of the most daunting HPC simulation problems, especially with the expansion of ensemble models, but for the past decade researchers have been advancing analytics-based “climate knowledge discovery algorithms” to provide additional insight. The first IEEE workshop on this topic, held in 2008, was called “Data Mining for Climate Change and Impacts.”1
- Simulation Helping Analytics. The development of automated driving systems, on the other hand, is primarily an analytics problem, but experts say fully automated vehicles will need to be test-driven for several billion miles to establish consumer trust. Completing that many physical test-miles is impractical, so HPC simulation, already a mainstay at major automakers, will come to the rescue. In this case, a virtual vehicle that looks and behaves like an identical “digital twin” of its physical counterpart, is guided by an AI model through millions of test drives representing a comprehensive range of real-world situations. The output from these simulations continually refines the analytics-based AI model and vehicle design.
Intelligent simulation promises to create competitive advantages for HPC users, with the following five benefits:
- Increased accuracy, by making it possible to review more data and explore more of the problem space in the given timeframe
- Solution Identification. Domain-specific cognitive models and knowledge graphs can curate huge volumes of information from disparate sources, allowing you to draw on a vast pool of expertise to quickly focus on the most promising lines of investigation and dismiss the less promising ones
- Faster solutions, by using intelligent algorithms to zero in quickly on the input data that is most valuable for insights and innovation
- Improved cost-efficiency and TCO, by boosting the productivity of the HPC system and leaving less of the data unanalyzed
- Greater trustworthiness, by analyzing more possibilities and increasing the transparency of machine learning/deep learning operations (e.g., the automated driving system example, above)
Intelligent simulation promises to create competitive advantages for HPC users who employ it and competitive disadvantages for those who don’t.
Read the complete Hyperion Research paper: Intelligent Simulation Exploits AI to Improve HPC Results to discover the nature and beneficial uses of intelligent simulation, as well as IBM’s integrated software solutions designed to address challenging AI workloads.
Explore technical blogs & how-to videos covering a broad range of topics focused on IBM Spectrum LSF and other relevant technologies for HPC at the online IBM Spectrum LSF user community.