After five annual workshops on Big Data and Extreme-Scale Computing (BDEC), a group of international HPC heavyweights including Jack Dongarra (University of Tennessee), Satoshi Matsuoka (Tokyo Institute of Technology), William Gropp (National Center for Supercomputing Applications), and Thomas Schulthess (Swiss National Supercomputing Centre), among others, has issued a comprehensive Big Data and Extreme-Scale Computing Pathways to Convergence Report. Not surprisingly it’s a large work not easily plumbed in a single sitting.

Convergence – harmonizing computational infrastructures to accommodate HPC and big data – isn’t a new topic. Recently, big data’s close cousin, machine learning, has become part of the discussion. Moreover, the accompanying rise of cyberinfrastructure as a dominant force in science computing has complicated convergence efforts.

The central premise of this study is that a ‘data-driven’ upheaval is exacerbating divisions – technical, cultural, political, economic – in the cyberecosystem of science. The report tackles in some depth a narrower slice of the problem. Big data, say the authors, has caused or worsened two ‘paradigm splits’: 1) one between the traditional ‘HPC and High-end Data Analysis (HDA)’ and 2) another between ‘stateless networks and stateful services’ provided by end systems. The report lays out a roadmap for mending these fissures.

This snippet from the report’s executive summary does a nice job of summing up the challenge:

“Looking toward the future of cyberinfrastructure for science and engineering through the lens of these two bifurcations made it clear to the BDEC community that, in the era of Big Data, the most critical problems involve the logistics of wide-area, multistage workflows—the diverse patterns of when, where, and how data is to be produced, transformed, shared, and analyzed. Consequently, the challenges involved in codesigning software infrastructure for science have to be reframed to fully take account of the diversity of workflow patterns that different application communities want to create. For the HPC community, all the imposing design and development issues of creating an exascale-capable software stack remain; but the supercomputers that need this stack must now be viewed as the nodes (perhaps the most important nodes) in the very large network of computing resources required to process and explore rivers of data flooding in from multiple sources.”

There’s a lot to digest here, including a fair amount of technical guidance. Issued at the end of 2017, the report is the result of workshops held in the U.S. (2013), Japan (2014), Spain (2015), Germany (2016), and China (2017); it grew out of prior efforts of the International Exascale Software Project (IESP). Descriptions and results of the five workshops (agendas, white papers, presentations, attendee lists) are available at the BDEC site (http://www.exascale.org/bdec/).

Commenting on the work, Dongarra said, “Computing is at a profound inflection point, economically and technically. The end of Dennard scaling and its implications for continuing semiconductor-design advances, the shift to mobile and cloud computing, the explosive growth of scientific, business, government, and consumer data and opportunities for data analytics and machine learning, and the continuing need for more-powerful computing systems to advance science and engineering are the context for the debate over the future of exascale computing and big data analysis.”

The broad hope is that the ideas presented in the report will guide community efforts. Dongarra emphasized “High-end data analytics (big data) and high-end computing (exascale) are both essential elements of an integrated computing research-and-development agenda; neither should be sacrificed or minimized to advance the other.” Shown below are typical differences in the BDEC software ecosystem.

There’s too much in the report to adequately cover here. Here are the report’s summary recommendations:

“Our major, global recommendation is to address the basic problem of the two paradigm splits: the HPC/HDA software ecosystem split and the wide area data logistics split. For this to be achieved, there is a need for new standards that will govern the interoperability between data and compute, based on a new, common and open Distributed Services Platform (DSP), that offers programmable access to shared processing, storage and communication resources, and that can serve as a universal foundation for the component interoperability that novel services and applications will require.

“We make five recommendations for decentralized edge and peripheral ecosystems:

• Converge on a new hourglass architecture for a Common Distributed Service Platform (DSP).
• Target workflow patterns for improved data logistics.
• Design cloud stream processing capabilities for HPC.
• Promote a scalable approach to Content Delivery/Distribution Networks.
• Develop software libraries for common intermediate processing tasks.

“We make five actionable conclusions for centralized facilities:

• Energy is an overarching challenge for sustainability.
• Data reduction is a fundamental pattern.
• Radically improved resource management is required.
• Both centralized and decentralized systems share many common software challenges and opportunities: 
(a) Leverage HPC math libraries for HDA.
(b) More efforts for numerical library standards.
(c) New standards for shared memory parallel processing.
(d) Interoperability between programming models and data formats.
• Machine learning is becoming an important component of scientific workloads, and HPC architectures must be adapted to accommodate this evolution.”

Link to BDEC Report: http://www.exascale.org/bdec/