10

Convergence between HPC and Big Data: Convergence between HPC and Big Data: The Day After Tomorrow Dan Reed [email protected] www.hpcdan.org

Convergence between HPC and Big Data: The Day After …sc18.supercomputing.org/proceedings/panel/panel...Convergence between HPC and Big Data: Convergence between HPC and Big Data:

Download PDF Report

Upload
others
View
8
Download
0

Embed Size (px)

Citation preview

Page 1: Convergence between HPC and Big Data: The Day After …sc18.supercomputing.org/proceedings/panel/panel...Convergence between HPC and Big Data: Convergence between HPC and Big Data:

ConvergencebetweenHPCandBigData:ConvergencebetweenHPCandBigData:

TheDayAfterTomorrow

DanReed

[email protected]

Page 2: Convergence between HPC and Big Data: The Day After …sc18.supercomputing.org/proceedings/panel/panel...Convergence between HPC and Big Data: Convergence between HPC and Big Data:

Thepurposeofcomputingisinsight,notnumbers.

RichardHamming

Page 3: Convergence between HPC and Big Data: The Day After …sc18.supercomputing.org/proceedings/panel/panel...Convergence between HPC and Big Data: Convergence between HPC and Big Data:

ReducedModel

ComputationalModel

ReducedModel

Page 4: Convergence between HPC and Big Data: The Day After …sc18.supercomputing.org/proceedings/panel/panel...Convergence between HPC and Big Data: Convergence between HPC and Big Data:

Meetings Roadmap

Meetings Questions

Participants• The“usualsuspects”

IoT/Edge,Biginstruments BigdataAI,extremecomputing,clouds

Missingparticipants• Deeplearningcommunity• Newsciencedemands• InfrastructureprovidersShiftinglandscape• Infrastructurecapabilities• Scienceexpectations

Page 5: Convergence between HPC and Big Data: The Day After …sc18.supercomputing.org/proceedings/panel/panel...Convergence between HPC and Big Data: Convergence between HPC and Big Data:

ShortenedversionoftheJanuaryreport:https://bit.ly/2qOIOBj

Page 6: Convergence between HPC and Big Data: The Day After …sc18.supercomputing.org/proceedings/panel/panel...Convergence between HPC and Big Data: Convergence between HPC and Big Data:

HPC

DeepLearning

Edge&AI

Page 7: Convergence between HPC and Big Data: The Day After …sc18.supercomputing.org/proceedings/panel/panel...Convergence between HPC and Big Data: Convergence between HPC and Big Data:

Size Nano Micro Milli Server Fog Campus Facility

Example AdafruitTrinket

Particle.ioBoron

ArrayofThings LinuxBox Co-located

Blades1000-nodecluster

Datacenter&Exascale

Memory 0.5KB 256KB 8GB 32GB 256 GB 32TB 16PB

Network BLE WiFi/LTE WiFi/LTE 1GigE 10GigE 40GigE N*100GigE

Cost $5 $30 $600 $3K $50K $2M $1000M

Edge HPC/Cloud/InstrumentFog

CountxComplexity=~Constant Statefulvs.Stateless

Page 8: Convergence between HPC and Big Data: The Day After …sc18.supercomputing.org/proceedings/panel/panel...Convergence between HPC and Big Data: Convergence between HPC and Big Data:

CommonDigitalContinuumPlatformforBigDataandExtremeScaleComputing

Sixinternationalmeetingsspreadovertwoyearshttps://bit.ly/2RVxSgR

Broaddiscussions

Page 9: Convergence between HPC and Big Data: The Day After …sc18.supercomputing.org/proceedings/panel/panel...Convergence between HPC and Big Data: Convergence between HPC and Big Data:

Page 10: Convergence between HPC and Big Data: The Day After …sc18.supercomputing.org/proceedings/panel/panel...Convergence between HPC and Big Data: Convergence between HPC and Big Data:

1010

This image cannot currently be displayed.

ThankYou

[email protected]

The convergence of HPC and BigData...FOSDEM 2019 – Feb 03, 2019 – Brussels | damien.francois@uclouvain.be The convergence of HPC and BigData What does it mean for HPC sysadmins?

The convergence of HPC and BigData...FOSDEM 2019 – Feb 03, 2019 – Brussels | [email protected] The convergence of HPC and BigData What does it mean for HPC sysadmins?

Documents

The Convergence of Big Compute and Big Data in Cloud-Based HPC

The Convergence of Big Compute and Big Data in Cloud-Based HPC

Documents

Big Data Meets HPC - Exploiting HPC Technologies for Accelerating Big Data Processing

Big Data Meets HPC - Exploiting HPC Technologies for Accelerating Big Data Processing

Technology

The Convergence of HPC and Big Data - ncic.ac.cn · The Convergence of HPC and Big Data ... , Rajeev Thakur, Kazutomo Yoshii LLNL: ... Smart Sensors + Supercomputers/Cloud Compu6ng

The Convergence of HPC and Big Data - ncic.ac.cn · The Convergence of HPC and Big Data ... , Rajeev Thakur, Kazutomo Yoshii LLNL: ... Smart Sensors + Supercomputers/Cloud Compu6ng

Documents

STORAGE-BASED CONVERGENCE BETWEEN HPC AND CLOUD …bigstorage-project.eu/images/Deliverables/BigStorage-D2.1.pdf · Deliverable number D2.1 Deliverable title Intermediate Report Big

STORAGE-BASED CONVERGENCE BETWEEN HPC AND CLOUD …bigstorage-project.eu/images/Deliverables/BigStorage-D2.1.pdf · Deliverable number D2.1 Deliverable title Intermediate Report Big

Documents

JST-CREST “Extreme Big Data” Project (2013 -2018)workshops.cs.georgetown.edu/BDPI-2016/slides... · 2016-05-03 · Not enough data scientists Convergence of top-tier HPC and Big

JST-CREST “Extreme Big Data” Project (2013 -2018)workshops.cs.georgetown.edu/BDPI-2016/slides... · 2016-05-03 · Not enough data scientists Convergence of top-tier HPC and Big

Documents

The INRIA Project LAB HPC-BigData: …...The INRIA Project LAB HPC-BigData: Addressingthe HPC/Big-Data/IA Convergence Bruno Raffin, INRIA Grenoble Rhône-Alpes Lyon, October 2018 The

The INRIA Project LAB HPC-BigData: …...The INRIA Project LAB HPC-BigData: Addressingthe HPC/Big-Data/IA Convergence Bruno Raffin, INRIA Grenoble Rhône-Alpes Lyon, October 2018 The

Documents

HPC & Big Data Convergence

HPC & Big Data Convergence

Documents

Big Data/AI and HPC Convergence Towards Post-K - Post H2020...Big Data/AI and HPC Convergence Towards Post-K Director, Riken Center for Computational Science / Professor, Tokyo Institute

Big Data/AI and HPC Convergence Towards Post-K - Post H2020...Big Data/AI and HPC Convergence Towards Post-K Director, Riken Center for Computational Science / Professor, Tokyo Institute

Documents

Big Data, Simulations and HPC Convergence - About DSCdsc.soic.indiana.edu/publications/HPCBigDataConvergence.pdf · Big Data, Simulations and HPC Convergence 5 for streaming data

Big Data, Simulations and HPC Convergence - About DSCdsc.soic.indiana.edu/publications/HPCBigDataConvergence.pdf · Big Data, Simulations and HPC Convergence 5 for streaming data

Documents

New Era of High Performance Computing (convergence of AI ......The Convergence of Big Data, AI and HPC Modeling The World Data-Intensive Processing Hybrid workflows with a mix of simulation

New Era of High Performance Computing (convergence of AI ......The Convergence of Big Data, AI and HPC Modeling The World Data-Intensive Processing Hybrid workflows with a mix of simulation

Documents

Big data, Clouds & HPC

Big data, Clouds & HPC

Technology

Big Data Meets HPC: Exploiting HPC Technologies …mvapich.cse.ohio-state.edu/.../dk_bigdata_convergence.pdfConvergence of HPC, Big Data, and Deep Learning!!! Network Based Computing

Big Data Meets HPC: Exploiting HPC Technologies …mvapich.cse.ohio-state.edu/.../dk_bigdata_convergence.pdfConvergence of HPC, Big Data, and Deep Learning!!! Network Based Computing

Documents

Big PanDA on HPC/LCF Update

Big PanDA on HPC/LCF Update

Documents

Research & Innovation funding opportunities for Big Data in … · 2017-11-06 · algorithms for key applications and support to the HPC ecosystem • Convergence of HPC, Big Data

Research & Innovation funding opportunities for Big Data in … · 2017-11-06 · algorithms for key applications and support to the HPC ecosystem • Convergence of HPC, Big Data

Documents

Big Data, Simulations and HPC Convergence

Big Data, Simulations and HPC Convergence

Documents

HPC & Big Data @ Atos

HPC & Big Data @ Atos

Documents

HPC. Big Data. Convergence....HPC. Big Data. Convergence. System Physical Status Front Panel Storage System cooling Power Supply Form Factor Chassis Dimension Support MB Size Buttons

HPC. Big Data. Convergence....HPC. Big Data. Convergence. System Physical Status Front Panel Storage System cooling Power Supply Form Factor Chassis Dimension Support MB Size Buttons

Documents

Convergence Lessons: Big Data and HPC - Post H2020...Convergence Lessons: Big Data and HPC dan.reed@utah.edu Flammarion engraving Meetings Roadmap Meetings Questions Participants •

Convergence Lessons: Big Data and HPC - Post H2020...Convergence Lessons: Big Data and HPC [email protected] Flammarion engraving Meetings Roadmap Meetings Questions Participants •

Documents

Pathways to Convergence - European eXtreme Data and ... · BDEC = Big Data & Exascale Computing. European HPC Technology Projects 3 • Successor to the IESP (International Exascale

Pathways to Convergence - European eXtreme Data and ... · BDEC = Big Data & Exascale Computing. European HPC Technology Projects 3 • Successor to the IESP (International Exascale

Documents

HPC & Big Data Convergence - IXPUG...HPC and Big Data • HPC technologies are needed by Big Data to deal with the ever increasing Vs of data in order to forecast and extract insights

HPC & Big Data Convergence - IXPUG...HPC and Big Data • HPC technologies are needed by Big Data to deal with the ever increasing Vs of data in order to forecast and extract insights

Documents

The Convergence of HPC and Big DataThe Convergence of HPC and Big Data Pete Beckman Senior Scien6st, Argonne Na6onal Laboratory Co-Director, Northwestern / Argonne Ins6tute for Science

The Convergence of HPC and Big DataThe Convergence of HPC and Big Data Pete Beckman Senior Scien6st, Argonne Na6onal Laboratory Co-Director, Northwestern / Argonne Ins6tute for Science

Documents

HPC and Big data convergence - events.science-japon.orgevents.science-japon.org/hpc17/slides/Serge Petiton... · HPC and Big data convergence Serge G. Petiton, UniversitéLille 1

HPC and Big data convergence - events.science-japon.orgevents.science-japon.org/hpc17/slides/Serge Petiton... · HPC and Big data convergence Serge G. Petiton, UniversitéLille 1

Documents

HPC. Big Data. Convergence. Performance-optimized EPYC 7000 Series_201811.pdf · ‧Arti˜cial Intelligence ‧Machine Learning ‧HPC ‧Virtualization Application 7000 EPYC Series

HPC. Big Data. Convergence. Performance-optimized EPYC 7000 Series_201811.pdf · ‧Arti˜cial Intelligence ‧Machine Learning ‧HPC ‧Virtualization Application 7000 EPYC Series

Documents

The Convergence of HPC and Deep Learning

The Convergence of HPC and Deep Learning

Technology

After HPC, it is Big Data!

After HPC, it is Big Data!

Technology

Convergence of Big Data, AI and HPC · 2020-01-15 · As of today, service orchestrators start containers unaware of shared systems HPC workload schedulers start userland processes

Convergence of Big Data, AI and HPC · 2020-01-15 · As of today, service orchestrators start containers unaware of shared systems HPC workload schedulers start userland processes

Documents

STORAGE-BASED CONVERGENCE BETWEEN HPC AND CLOUD …bigstorage-project.eu/images/Deliverables/BigStorage-D3.1.pdf · STORAGE-BASED CONVERGENCE BETWEEN HPC AND CLOUD TO HANDLE BIG DATA

STORAGE-BASED CONVERGENCE BETWEEN HPC AND CLOUD …bigstorage-project.eu/images/Deliverables/BigStorage-D3.1.pdf · STORAGE-BASED CONVERGENCE BETWEEN HPC AND CLOUD TO HANDLE BIG DATA

Documents

HPC Technologies for Big Data - HPC Advisory Council - A community

HPC Technologies for Big Data - HPC Advisory Council - A community

Documents

Deep Learning: Convergence of HPC and Hyperscale

Deep Learning: Convergence of HPC and Hyperscale

Technology