• Home

  • Documents

  • TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES...
30
Mitul Tiwari and Deepak Bobbarjung TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES

TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

  • Upload
    others

  • View
    7

  • Download
    0

Embed Size (px)

Citation preview

Page 1: TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

Mitul Tiwari and Deepak Bobbarjung

TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES

Page 2: TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

Introductions

Page 3: TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

Outline

Conversational AI and Deep Learning

Need for a Jobs framework on Kubernetes

Our Jobs architecture

Page 4: TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

Bot Training#1 AI/NLP Model.

Bot DeploymentBuild Once, Deploy Everywhere.

Our Conversational AI Platform

Bot BuilderNo Coding Required.

AI/NLP

Intents

Entities

Attributes

Speech to text

Language Translation

Sentiment Knowledge Base

Page 5: TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

How to Make a Bot Intelligent?

• Natural Language Understanding

• Information Extraction

• Entities

• Intents

• Actions

• Natural Language Generation

• Generating Response

Page 6: TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

Deep Learning

• Traditional Machine Learning

• Human designed features and representations

• Optimize weights to combine

• Deep Learning

• Deep Neural Network

• Learn good features and multiple levels of representations

Page 7: TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

Deep Learning for NLP• Language Translation

• Image Captioning

• Text Summarization

• Parts-of-speech Tagging

• Named Entity Recognition

• Natural Language Generation

• Question-Answering

• Optical Character Recognition

• Speech Recognition

• Machine Reading Comprehension

Page 8: TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

Neural Network for Word Embedding

• Word Embedding: Word2Vec

• Embed words in continues vector space

• Semantically similar words are mapped to nearby points

• Enables powerful operations

• “King”-“Man”+”Woman” -> “Queen”

Page 9: TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

Bag of Words - Curse of Dimensionality

• Before word embeddings - Bag of words

• Dictionary of words & counts in the text

• Easy feature generation technique

• Limitations

• Hard to capture order of words

• Curse of dimensionality - limited vocabulary - similar words don’t match

Page 10: TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

Word Embeddings Cont’d

• Word Embedding: mapping words to a higher dimensional space, typically 200-500, e.g.,

• W(‘King’) = (0.2, -0.4, 0.9, …)

• W(‘Queen’) = (0.1, -0.3, 0.8, …)

• Learn representations of words

• How: two layer NN to learn word representations by predicting validity of phrases

Example of similar word vectors

Page 11: TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

Sequence Learning: Response Generation

• Automated Response Generation

• Sequence 2 Sequence Model

• Recurrent Neural Network (RNN)

• Long Short Term Memory Network (LSTM)

• Example: GMail Smart Reply

• Automated Response Suggestions

Page 12: TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

Sequence Learning: RNN And LSTM

• Recurrent Neural Network

• Output of a module go into a module of same type (recurrent)

• Good for capturing a sequence

• Long Short Term Memory Network

• Long running cell state: forget & add new values

• Output: combination of cell state, previous output, and new input

Page 13: TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

Training Deep Learning Models for NLP

• Intent Classification• Deep Learning — LSTM

• Information Extraction• Named Entity Recognition (NER)• Slot attributes

• Sentiment and Complaint Classifier

• Knowledge Base & Semantic Search

• Machine Reading Comprehension

Natural Language Understanding &

GenerationAnalysis for Complaints

Targeted Personalized Timely

Notification

Automatic Speech Recognition &

Generation

Deep Learning

Entity Graph & Knowledge Base

Text Sentiment Notifications Speech

Page 14: TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

Scaling Training Deep Learning Models For NLP

•Off line: Started with a script for training models

•Run Time: A service for prediction during runtime

•However, the number of models are reaching in thousands

•Hard to manage model training script for each of the bot

Conversation Plane (Run Time)

Control Plane(Offline)

Users

Interfaces

NLP Prediction (for example, Intent classification …)

Train Models

Store Models

Training Data Load Models

Page 15: TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

Outline

Conversational AI and Deep Learning

Need for a Jobs framework on Kubernetes

Our Jobs architecture

Page 16: TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

Training plane

Conversation plane Control planeData

User Data

Config Data

Orchestration

AI/NLPAPI Service

Configure Bots

Train Bottraining job

Hi

Classify (Hi)

model

Greetings intentWelcome to..

Passage AI Architecture

Page 17: TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

Passage AI Architecture

Training plane

(Jobs)

Page 18: TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

When do we train a new model for a bot ?

• When a new bot is created

• When a bot is changed

• utterances are added or modified

• New training data is available

Page 19: TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

# of Bots

0

125

250

375

500

August September October November

# of Bot changes per day

0

25

50

75

100

August September October November

Page 20: TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

Why do we need a Jobs framework?• Run jobs at scale

• Eliminate out of band scripts that tend to become ‘tribal’.

• APIs and UI for exposing jobs to our customers in our Bot Builder UI.

• Reporting and auditing around jobs.

Control plane

API Service

Create job specificationRun a jobShow last 10 job runs

Page 21: TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

Conversation plane

Why Kubernetes (K8S) For Our Microservices?

• Scale and availability of our microservices

Orchestration Service

K8s service K8s hpaK8s deployment

Pod Pod Pod Pod Pod

Nginx

Pod

Pod PodPod PodPod Pod

Page 22: TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

On-Prem

Why Kubernetes?(Contd)

• Cloud Agnostic and On-prem ready

Staging

Conversation plane

Control plane

ProductionConversation

planeControl plane

IntegrationConversation

planeControl plane

Conversation plane

Control plane

$helm install passage-ai

StandbyConversation

planeControl plane

Page 23: TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

Why Create The Jobs Framework In Kubernetes?

• Handle scale and availability in the same way as our microservices Jobs Plane

Pod Pod Pod Pod Pod Pod

Pod PodPod PodPod Pod

• Jobs should also be cloud-agnostic and on-prem ready

• Same set of tools for monitoring, logging and auditing.

Page 24: TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

Outline

Conversational AI and Deep Learning

Need for a Jobs framework on Kubernetes

Our Jobs architecture

Page 25: TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

Example Job types in our system

• Training deep learning models

• Extracting and indexing knowledge base articles

• Nightly testing of our bots

Page 26: TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

Control planeAPI

Job Specification

Create a job spec

Schedule: “Every Monday at 1 AM”Job type: “Training”

- bot ID, training dataTraining specific params:

priority

Job ObjectControl plane

API

Trigger a job from a job spec

progress: 25 job_spec_id: <id>

- data: < confusion matrix>

description: performing training

state: in_progress

Page 27: TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

Jobs plane

Control planeAPI Service

jobs job_specs

Jobs Service

train

ing

Q

KB In

dex

Q

Bot T

estin

g Q

K8s deployment of training pods

Pod1

Pod4

Pod2

Pod3

Upda

te job

prog

ress

Jobs Architecture

Pod1

Pod2

Pod1

Pod2

Trigger training job from job spec

Create a Job (params)

Add item on Q

Pickup item

gpu

gpu

Page 28: TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

Jobs plane

Control plane

API Servicejobs job_specs

Jobs Service

train

ingQ

KB In

dex

Q

K8s deployment of training pods

Pod1

Pod4Pod3

Pod2

Upda

te job

prog

ressTrigger job from job spec

Jobs Architecture (scheduled jobs)

Pod1

Pod2

Pod1

Pod2

Scheduler

CronJob

Get job status

Bot T

estin

g Q

Page 29: TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

Alternatives that we considered

Apache Airflow Azkaban

Page 30: TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

Thank You

Mitul Tiwari Deepak Bobbarjung

@Bobbarjung @[email protected] [email protected]

https://twitter.com/bobbarjung
https://twitter.com/mitultiwari
mailto:[email protected]
mailto:[email protected]

Large-scale Deep Unsupervised Learning using Graphics Processors

Large-scale Deep Unsupervised Learning using Graphics Processors

Documents
Large-Scale Deep Learninghelper.ipam.ucla.edu/publications/gss2012/gss2012_10783.pdf · Large-Scale Deep Learning IPAM SUMMER SCHOOL Marc'Aurelio Ranzato- ... deep learning methods?

Large-Scale Deep Learninghelper.ipam.ucla.edu/publications/gss2012/gss2012_10783.pdf · Large-Scale Deep Learning IPAM SUMMER SCHOOL Marc'Aurelio Ranzato- ... deep learning methods?

Documents
Multi-scale deep learning for gesture detection and ... · Multi-scale deep learning for gesture detection and localization 1;2Natalia Neverova 1;2Christian Wolf 3Graham W. Taylor

Multi-scale deep learning for gesture detection and ... · Multi-scale deep learning for gesture detection and localization 1;2Natalia Neverova 1;2Christian Wolf 3Graham W. Taylor

Documents
Deep Learning Performance Comparing Scale-out vs Scale-up · Deep Learning Performance: Scale-up vs Scale-out Architectures & Technologies Dell EMC | Infrastructure Solutions Group

Deep Learning Performance Comparing Scale-out vs Scale-up · Deep Learning Performance: Scale-up vs Scale-out Architectures & Technologies Dell EMC | Infrastructure Solutions Group

Documents
Build, Scale, and Deploy Deep Learning Pipelines Using Apache Spark

Build, Scale, and Deploy Deep Learning Pipelines Using Apache Spark

Software
Democratizing Production-Scale Distributed Deep Learning...arXiv:1811.00143v2 [cs.CV] 3 Nov 2018 Democratizing Production-Scale Distributed Deep Learning additional overhead. Examples

Democratizing Production-Scale Distributed Deep Learning...arXiv:1811.00143v2 [cs.CV] 3 Nov 2018 Democratizing Production-Scale Distributed Deep Learning additional overhead. Examples

Documents
Multi-Scale Deep Learning Architectures for Person Re ......Multi-scale Deep Learning Architectures for Person Re-identification Xuelin Qian1 Yanwei Fu2,5,* Yu-Gang Jiang1,3 Tao Xiang4

Multi-Scale Deep Learning Architectures for Person Re ......Multi-scale Deep Learning Architectures for Person Re-identification Xuelin Qian1 Yanwei Fu2,5,* Yu-Gang Jiang1,3 Tao Xiang4

Documents
Deep Learning based Large Scale Visual Recommendation … · Deep Learning based Large Scale Visual Recommendation and Search for E-Commerce Devashish Shankar, Sujay Narumanchi, Ananya

Deep Learning based Large Scale Visual Recommendation … · Deep Learning based Large Scale Visual Recommendation and Search for E-Commerce Devashish Shankar, Sujay Narumanchi, Ananya

Documents
Scalable Machine Learning - Centrum Wiskunde & … Machine...• hardware for deep learning –CPUs (SIMD), GPUs, TPUs • parallel training: does deep learning scale? –Trivially

Scalable Machine Learning - Centrum Wiskunde & … Machine...• hardware for deep learning –CPUs (SIMD), GPUs, TPUs • parallel training: does deep learning scale? –Trivially

Documents
Benchmarking Deep Learning Workloads on Large-scale HPC ... · Benchmarking Deep Learning Workloads on Large-scale HPC Systems AmmarAhmad Awan and Dhabaleswar K. Panda awan.10@osu.edu,

Benchmarking Deep Learning Workloads on Large-scale HPC ... · Benchmarking Deep Learning Workloads on Large-scale HPC Systems AmmarAhmad Awan and Dhabaleswar K. Panda [email protected],

Documents
Accelerating Large Scale Deep Learning Inference through ... · Accelerating Large Scale Deep Learning Inference through DeepCPU at Microsoft Minjia Zhang, Samyam Rajbandari, Wenhan

Accelerating Large Scale Deep Learning Inference through ... · Accelerating Large Scale Deep Learning Inference through DeepCPU at Microsoft Minjia Zhang, Samyam Rajbandari, Wenhan

Documents
Intelligent Computer Systems Large-Scale Deep Learning forstatic.googleusercontent.com/media/research.google.com/en//people/... · Large-Scale Deep Learning for Intelligent Computer

Intelligent Computer Systems Large-Scale Deep Learning forstatic.googleusercontent.com/media/research.google.com/en//people/... · Large-Scale Deep Learning for Intelligent Computer

Documents
Large Scale Deep Learning - … Scale Deep Learning Jeff Dean. ... representation conveyed by the IT cortex. Thus, compared with early visual representations, object manifolds are

Large Scale Deep Learning - … Scale Deep Learning Jeff Dean. ... representation conveyed by the IT cortex. Thus, compared with early visual representations, object manifolds are

Documents
Applications: Large-Scale Deep Learningsrihari/CSE676/12.1 LargeScaleSystems.pdf · Deep Learning Srihari Large Scale Deep Learning •Philosophy of connectionism –While an individual

Applications: Large-Scale Deep Learningsrihari/CSE676/12.1 LargeScaleSystems.pdf · Deep Learning Srihari Large Scale Deep Learning •Philosophy of connectionism –While an individual

Documents
Deep Learning Performance Scale-Out · 4 Deep Learning Performance Scale-Out Motivation With the recent advances in the field of Machine Learning and especially Deep Learning, it’s

Deep Learning Performance Scale-Out · 4 Deep Learning Performance Scale-Out Motivation With the recent advances in the field of Machine Learning and especially Deep Learning, it’s

Documents
Large Scale Deep Learning Jeff Dean

Large Scale Deep Learning Jeff Dean

Technology
Large-Scale Deep Learning for Intelligent Computer Systems

Large-Scale Deep Learning for Intelligent Computer Systems

Documents
Person Re-Identification by Deep Learning Multi-Scale ...sgg/papers/ChenEtAl_ICCV2017WK_CHI.pdf · Deep Pyramidal Feature Learning (DPFL) CNN architec-ture for learning explicitly

Person Re-Identification by Deep Learning Multi-Scale ...sgg/papers/ChenEtAl_ICCV2017WK_CHI.pdf · Deep Pyramidal Feature Learning (DPFL) CNN architec-ture for learning explicitly

Documents
Deep Learning on Hadoop Scale out Deep Learning on YARN

Deep Learning on Hadoop Scale out Deep Learning on YARN

Documents
Initial Characterization of I/O in Large-Scale Deep Learning … · 2018-11-11 · Deep Learning (DL) applications demand large-scale computing facilities. DL applications require

Initial Characterization of I/O in Large-Scale Deep Learning … · 2018-11-11 · Deep Learning (DL) applications demand large-scale computing facilities. DL applications require

Documents
Modular Deep Learning Analysis of Galaxy-Scale Strong

Modular Deep Learning Analysis of Galaxy-Scale Strong

Documents
Deep Learning Performance Comparing Scale-out vs Scale-up · Deep Learning consists of two phases: Training and inference. As illustrated in Figure 2, training involves learning a

Deep Learning Performance Comparing Scale-out vs Scale-up · Deep Learning consists of two phases: Training and inference. As illustrated in Figure 2, training involves learning a

Documents
Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Documents
Large-Scale Deep Learning - Hebrew University of Jerusalem · 2015-10-25 · Large-Scale Deep Learning Tal Ben-Nun ... •Algorithm: •Divide training data into subsets and run a

Large-Scale Deep Learning - Hebrew University of Jerusalem · 2015-10-25 · Large-Scale Deep Learning Tal Ben-Nun ... •Algorithm: •Divide training data into subsets and run a

Documents
Named Entity Recognition at Scale with Deep Learning

Named Entity Recognition at Scale with Deep Learning

Documents
Toward Accelerating Deep Learning at Scale Using ... · Toward Accelerating Deep Learning at Scale Using Specialized Hardware in the Datacenter . 2 . 3 1Case study: CNN-based Image

Toward Accelerating Deep Learning at Scale Using ... · Toward Accelerating Deep Learning at Scale Using Specialized Hardware in the Datacenter . 2 . 3 1Case study: CNN-based Image

Documents
Deep Learning at Scale

Deep Learning at Scale

Technology
Learning Deep Structured Multi-Scale Features using

Learning Deep Structured Multi-Scale Features using

Documents
Large-scale Deep Unsupervised Learning using Graphics ...€¦ · Rajat Raina Anand Madhavan Andrew Y. Ng Stanford University Large-scale Deep Unsupervised Learning using Graphics

Large-scale Deep Unsupervised Learning using Graphics ...€¦ · Rajat Raina Anand Madhavan Andrew Y. Ng Stanford University Large-scale Deep Unsupervised Learning using Graphics

Documents
Quoc le tera-scale deep learning

Quoc le tera-scale deep learning

Education