New Blueprint for Converging HPC, Big Data

By John Russell

January 18, 2018

After five annual workshops on Big Data and Extreme-Scale Computing (BDEC), a group of international HPC heavyweights including Jack Dongarra (University of Tennessee), Satoshi Matsuoka (Tokyo Institute of Technology), William Gropp (National Center for Supercomputing Applications), and Thomas Schulthess (Swiss National Supercomputing Centre), among others, has issued a comprehensive Big Data and Extreme-Scale Computing Pathways to Convergence Report. Not surprisingly it’s a large work not easily plumbed in a single sitting.

Convergence – harmonizing computational infrastructures to accommodate HPC and big data – isn’t a new topic. Recently, big data’s close cousin, machine learning, has become part of the discussion. Moreover, the accompanying rise of cyberinfrastructure as a dominant force in science computing has complicated convergence efforts.

The central premise of this study is that a ‘data-driven’ upheaval is exacerbating divisions – technical, cultural, political, economic – in the cyberecosystem of science. The report tackles in some depth a narrower slice of the problem. Big data, say the authors, has caused or worsened two ‘paradigm splits’: 1) one between the traditional ‘HPC and High-end Data Analysis (HDA)’ and 2) another between ‘stateless networks and stateful services’ provided by end systems. The report lays out a roadmap for mending these fissures.

 

This snippet from the report’s executive summary does a nice job of summing up the challenge:

“Looking toward the future of cyberinfrastructure for science and engineering through the lens of these two bifurcations made it clear to the BDEC community that, in the era of Big Data, the most critical problems involve the logistics of wide-area, multistage workflows—the diverse patterns of when, where, and how data is to be produced, transformed, shared, and analyzed. Consequently, the challenges involved in codesigning software infrastructure for science have to be reframed to fully take account of the diversity of workflow patterns that different application communities want to create. For the HPC community, all the imposing design and development issues of creating an exascale-capable software stack remain; but the supercomputers that need this stack must now be viewed as the nodes (perhaps the most important nodes) in the very large network of computing resources required to process and explore rivers of data flooding in from multiple sources.”

There’s a lot to digest here, including a fair amount of technical guidance. Issued at the end of 2017, the report is the result of workshops held in the U.S. (2013), Japan (2014), Spain (2015), Germany (2016), and China (2017); it grew out of prior efforts of the International Exascale Software Project (IESP). Descriptions and results of the five workshops (agendas, white papers, presentations, attendee lists) are available at the BDEC site (http://www.exascale.org/bdec/).

Jack Dongarra

Commenting on the work, Dongarra said, “Computing is at a profound inflection point, economically and technically. The end of Dennard scaling and its implications for continuing semiconductor-design advances, the shift to mobile and cloud computing, the explosive growth of scientific, business, government, and consumer data and opportunities for data analytics and machine learning, and the continuing need for more-powerful computing systems to advance science and engineering are the context for the debate over the future of exascale computing and big data analysis.”

The broad hope is that the ideas presented in the report will guide community efforts. Dongarra emphasized “High-end data analytics (big data) and high-end computing (exascale) are both essential elements of an integrated computing research-and-development agenda; neither should be sacrificed or minimized to advance the other.” Shown below are typical differences in the BDEC software ecosystem.

 

There’s too much in the report to adequately cover here. Here are the report’s summary recommendations:

“Our major, global recommendation is to address the basic problem of the two paradigm splits: the HPC/HDA software ecosystem split and the wide area data logistics split. For this to be achieved, there is a need for new standards that will govern the interoperability between data and compute, based on a new, common and open Distributed Services Platform (DSP), that offers programmable access to shared processing, storage and communication resources, and that can serve as a universal foundation for the component interoperability that novel services and applications will require.

“We make five recommendations for decentralized edge and peripheral ecosystems:

  • Converge on a new hourglass architecture for a Common Distributed Service Platform (DSP).
  • Target workflow patterns for improved data logistics.
  • Design cloud stream processing capabilities for HPC.
  • Promote a scalable approach to Content Delivery/Distribution Networks.
  • Develop software libraries for common intermediate processing tasks.

“We make five actionable conclusions for centralized facilities:

  • Energy is an overarching challenge for sustainability.
  • Data reduction is a fundamental pattern.
  • Radically improved resource management is required.
  • Both centralized and decentralized systems share many common software challenges and opportunities: 
(a) Leverage HPC math libraries for HDA.
(b) More efforts for numerical library standards.
(c) New standards for shared memory parallel processing.
(d) Interoperability between programming models and data formats.
  • Machine learning is becoming an important component of scientific workloads, and HPC architectures must be adapted to accommodate this evolution.”

Link to BDEC Report: http://www.exascale.org/bdec/

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

D-Wave Breaks New Ground in Quantum Simulation

July 16, 2018

Last Friday D-Wave scientists and colleagues published work in Science which they say represents the first fulfillment of Richard Feynman’s 1982 notion that simulating physical systems could be done most effectively Read more…

By John Russell

RIKEN and CEA Mark One Year of Exascale-focused Collaboration

July 16, 2018

RIKEN in Japan and the French Alternative Energies and Atomic Energy Commission (CEA) formed a five-year cooperative research effort on January 11, 2017, to advance HPC and prepare for exascale computing (see HPCwire co Read more…

By Nishi Katsuya

AI Thought Leaders on Capitol Hill

July 14, 2018

On Thursday, July 12, the House Committee on Science, Space, and Technology heard from four academic and industry leaders – representatives from Berkeley Lab, Argonne Lab, GE Global Research and Carnegie Mellon University – on the opportunities springing from the intersection of machine learning and advanced-scale computing. Read more…

By Tiffany Trader

HPE Extreme Performance Solutions

Introducing the First Integrated System Management Software for HPC Clusters from HPE

How do you manage your complex, growing cluster environments? Answer that big challenge with the new HPC cluster management solution: HPE Performance Cluster Manager. Read more…

IBM Accelerated Insights

Are Your Software Licenses Impeding Your Productivity?

In my previous article, Improving chip yield rates with cognitive manufacturing, I highlighted the costs associated with semiconductor manufacturing, and how cognitive methods can yield benefits in both design and manufacture.  Read more…

HPC Serves as a ‘Rosetta Stone’ for the Information Age

July 12, 2018

In an age defined and transformed by its data, several large-scale scientific instruments around the globe might be viewed as a ‘mother lode’ of precious data. With names seemingly created for a ‘techno-speak’ glossary, these interferometers, cyclotrons, sequencers, solenoids, satellite altimeters, and cryo-electron microscopes are churning out data in previously unthinkable and seemingly incomprehensible quantities -- billions, trillions and quadrillions of bits and bytes of electro-magnetic code. Read more…

By Warren Froelich

D-Wave Breaks New Ground in Quantum Simulation

July 16, 2018

Last Friday D-Wave scientists and colleagues published work in Science which they say represents the first fulfillment of Richard Feynman’s 1982 notion that Read more…

By John Russell

AI Thought Leaders on Capitol Hill

July 14, 2018

On Thursday, July 12, the House Committee on Science, Space, and Technology heard from four academic and industry leaders – representatives from Berkeley Lab, Argonne Lab, GE Global Research and Carnegie Mellon University – on the opportunities springing from the intersection of machine learning and advanced-scale computing. Read more…

By Tiffany Trader

HPC Serves as a ‘Rosetta Stone’ for the Information Age

July 12, 2018

In an age defined and transformed by its data, several large-scale scientific instruments around the globe might be viewed as a ‘mother lode’ of precious data. With names seemingly created for a ‘techno-speak’ glossary, these interferometers, cyclotrons, sequencers, solenoids, satellite altimeters, and cryo-electron microscopes are churning out data in previously unthinkable and seemingly incomprehensible quantities -- billions, trillions and quadrillions of bits and bytes of electro-magnetic code. Read more…

By Warren Froelich

Tsinghua Powers Through ISC18 Field

July 10, 2018

Tsinghua University topped all other competitors at the ISC18 Student Cluster Competition with an overall score of 88.43 out of 100. This gives Tsinghua their s Read more…

By Dan Olds

HPE, EPFL Launch Blue Brain 5 Supercomputer

July 10, 2018

HPE and the Ecole Polytechnique Federale de Lausannne (EPFL) Blue Brain Project yesterday introduced Blue Brain 5, a new supercomputer built by HPE, which displ Read more…

By John Russell

Pumping New Life into HPC Clusters, the Case for Liquid Cooling

July 10, 2018

High Performance Computing (HPC) faces some daunting challenges in the coming years as traditional, industry-standard systems push the boundaries of data center Read more…

By Scott Tease

Meet the ISC18 Cluster Teams: Up Close & Personal

July 6, 2018

It’s time to meet your ISC18 Student Cluster Competition teams. While I was able to film them live at the ISC show, the trick was finding time to edit the vid Read more…

By Dan Olds

PRACEdays18 Keynote Allan Williams (Australia/NCI): We’re Open for Business Down Under!

July 5, 2018

The University of Ljubljana in Slovenia hosted the third annual EHPCSW18 and fifth annual PRACEdays18 events which opened with a plenary session on May 29, 2018 Read more…

By Elizabeth Leake (STEM-Trek for HPCwire)

Leading Solution Providers

SC17 Booth Video Tours Playlist

Altair @ SC17

Altair

AMD @ SC17

AMD

ASRock Rack @ SC17

ASRock Rack

CEJN @ SC17

CEJN

DDN Storage @ SC17

DDN Storage

Huawei @ SC17

Huawei

IBM @ SC17

IBM

IBM Power Systems @ SC17

IBM Power Systems

Intel @ SC17

Intel

Lenovo @ SC17

Lenovo

Mellanox Technologies @ SC17

Mellanox Technologies

Microsoft @ SC17

Microsoft

Penguin Computing @ SC17

Penguin Computing

Pure Storage @ SC17

Pure Storage

Supericro @ SC17

Supericro

Tyan @ SC17

Tyan

Univa @ SC17

Univa

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This