Machine Learning Systems (294-162) - GitHub PagesWaves of AI Research Ø1950 to 1974:Birth of AI...

Joseph E. GonzalezCo-director of the RISE Lab

jegonzal@cs.berkeley.edu

Machine Learning Systems(294-162)

About Me

Ø Co-director of the RISE Lab

Ø Co-founder of Turi Inc.

ResearchØ Machine Learning Ø Distributed Systems Ø Computer VisionØ Autonomous DrivingØ Secure Learning Ø …

riselabUC BerkeleyAbout Me

My story …

MachineLearning

LearningSystems

Back in 2007

Back in 2007I started studying the use of Gaussian Process (GP) models for wireless link quality estimation

Research was slowed by the speed of training… and the fact that link quality is difficult to predict

Back in 2008I started studying parallel inference algorithms and systems

I designed and implemented parallel learning algorithms on top of low-level primitives …

ApplicationIdea Model +

Algorithm

SerialPrototype

ParallelPrototype

Debug

Optimize

Evaluate

DistributedPrototype

OptimizeDebug

Evaluate

MLPaper

MLPaper

Low-Level Primitives

Ø Shared Memory Parallelism: pThreads & OpenMP

Ø Distributed Communication: Actors & MPIØ Yes,… I built an RPC framework and then an actor framework

Ø Hardware APIs + Languages: GPU Programming (CUDA)

Low-level parallel primitives offer fine grained control over parallel execution.

Advantages of theLow-Level ApproachØ Extract maximum performance from hardware

Ø Enable exploration of more complex algorithms• Fine grained locking• Atomic data-structures• Distributed coordination protocols

My implementation is better than your implementation.

Limitations of theLow-Level ApproachØ Repeatedly address the same system challenges

Ø Algorithm conflates learning and system logic

Ø Difficult to debug and extend

Ø Typically does not address issues at scale: hardware failure, stragglers, …

Months of tuning and engineering for one problem.

ApplicationIdea Model +

Algorithm

SerialPrototype

ParallelPrototype

Debug

Optimize

Evaluate

DistributedPrototype

OptimizeDebug

Evaluate

MLPaper

MLPaper

Interesting Machine Learning

ResearchInteresting

Systems ResearchAbstraction

Design Complexity

Model

Accuracy

MachineLearning

Parallelism

Locality

Scheduling

Fault Tolerance

Network

StragglersLarge-Scale

Systems

Training

Design Complexity

Training

MachineLearning

Locality

Scheduling

Fault Tolerance

Network

StragglersLarge-Scale

Systems

Model Parallelism

Accuracy

Interaction

Design Complexity

MachineLearning

Large-ScaleSystems

Model Parallelism

Interaction

Learning systems combine the complexities of machine learning with system design

Scheduling

Fault Tolerance

Stragglers

LocalityNetwork

Training Accuracy

Managing ComplexityThrough Abstraction

Learning AlgorithmCommon Patterns

System

Abstraction (API)

1. Parallelism2. Data Locality3. Network

4. Scheduling5. Fault-tolerance6. Stragglers

Identify common patterns

Define a narrowinterface

Exploit limited abstractionto address system

design challenges

PhD in Machine Learning from CMU 2013

ScalableSystemsAbstractions

Y2

Y1

Y3

X2

X3

X1

� 1,3

�1,2

�2,3(x2, x3)

�1(x1)

�2(x2)

�3(x3)

Graphical Model Inference

Vertex Program

Machine 1 Machine 2

GraphLab/GraphXSystem

MachineLearning

Looking Back on AI SystemsHow did the field evolve during the time I was doing my PhD

Machine learning communityhas had an evolving focus on AI Systems

2009 2019

FastAlgorithms

ML forSystems

DistributedAlgorithms

Deep LearningFrameworks

Integration ofCommunities

Machine LearningFrameworks

RL forSystems

Systems and the Third Wave of AI

Waves of AI ResearchØ 1950 to 1974: Birth of AI

Ø 1950 Alan Turing publishes the Imitation GameØ 1951 Marvin Minsky builds first neural network machine (SNARC)Ø DARPA starts to pour money into AI Research (limited oversight)Ø Incredible optimism

“Within ten years a digital computer will be the world’s chess champion” -- Herbert A. Simon and Allen Newell (1958)

“In from three to eight years we will have a machine with the general intelligence of an average human being”

– Marvin Minksy (1970)

Waves of AI ResearchØ 1950 to 1974: Birth of AI

Ø 1974 to 1980: First AI WinterØ Technology was unable to meet the high expectations

Ø Insufficient computer powerØ Over emphasis on combinatorial search Ø Insufficient knowledge (data)

Ø 1969 Book “Perceptrons” by Minsky and PapertØ Showed that certain basic functions (e.g., parity) cannot be computed using

local connections Ø Strong argument against connectionist approaches to AI (neural networks)

Ø Government funding driesØ Not meeting objectivesØ DARPA focuses more on immediate impact

Waves of AI ResearchØ 1950 to 1974: Birth of AI

Ø 1974 to 1980: First AI Winter

Ø 1980 to 1987: Second Wave of AIØ Expert systems start to solve real-world problems

Ø CMU developed XCON (AI for Systems) for DEC à saves $40M annually Ø Combines “knowledge” (expert rules “data”) with logic

Ø Japanese government invests $850M in AI ResearchØ Other governments respond by also investing heavily

Ø Emergence of the AI hardware and software industry to

Waves of AI ResearchØ 1950 to 1974: Birth of AI

Ø 1974 to 1980: First AI Winter

Ø 1980 to 1987: Second Wave of AI

Ø 1987 to 1993: Second AI WinterØ Expert systems too brittle to maintainØ Collapse of the specialized AI hardware market

Ø Commodity hardware was improving to rapidlyØ Ultimately over 300 AI companies would vanish from the market

Ø Massive government funding cuts

Waves of AI ResearchØ 1950 to 1974: Birth of AI

Ø 1974 to 1980: First AI Winter

Ø 1980 to 1987: Second Wave of AI

Ø 1987 to 1993: Second AI Winter

Ø 1993 to 2011: AI Goes Stealth Mode (aka Machine Learning)Ø Hardware becomes fast enough, and AI techniques start to work

Ø Deep Blue beats Garry Kasparov (1997)Ø OCR, Speech Recognition, Google Search, …

Ø Confluence of ideas and techniques: optimization, statistics, probability theory, and information theory

Waves of AI ResearchØ 1950 to 1974: Birth of AI

Ø 1974 to 1980: First AI Winter

Ø 1980 to 1987: Second Wave of AI

Ø 1987 to 1993: Second AI Winter

Ø 1993 to 2011: AI Goes Stealth Mode (aka Machine Learning)Ø Hardware becomes fast enough, and AI techniques start to work

Ø Deep Blue beats Garry Kasparov (1997)Ø OCR, Speech Recognition, Google Search, …

Ø Confluence of ideas and techniques: optimization, statistics, probability theory, and information theory

Waves of AI ResearchØ 1950 to 1974: Birth of AI

Ø 1974 to 1980: First AI Winter

Ø 1980 to 1987: Second Wave of AI

Ø 1987 to 1993: Second AI Winter

Ø 1993 to 2011: AI Goes Stealth Mode (aka Machine Learning)

Ø 2011 to 2019: Third Wave (AI Goes Deep)Ø Large quantities of data in conjunction with advances in hardware

and software enable the design and training of complex modelsØ New applications emerge

Ø Autonomous driving, home automation, Ø AI Market frenzy à IDC predicts 77.6B Market for AI-Systems in 2022Ø …

“A New Golden Age in Computer Architecture: Empowering the Machine-Learning Revolution”,https://ieeexplore.ieee.org/document/8259424

Why?

New Forces Driving AI Revolution

Compute AbstractionsData

Benchmarks

Advances inAlgorithms and

Models

Stochastic Gradient Descent

✓1<latexit sha1_base64="cbM0eSmGxMtQkVBDiiBAEOk4Olo=">AAAJx3icpZZfb9s2EMDVrtti70/b7bEvwowALZAGlrcg2VuxIkD/BIvb1Wkw0zAo+iQTpSiBpFO7jB72FfbafYJ+o32bUZScSFQLaBsB22fe/e5OJ95JYcaoVMPh3zdufnbr8y++3On1v/r6m29v37n73ZlMV4LAhKQsFechlsAoh4miisF5JgAnIYPX4ZvHhf71BQhJU/5KbTKYJTjmNKIEK7N1jtQSFJ4H8zuD4f7QLr8tBJUw8Ko1nt/d+YAWKVklwBVhWMppMMzUTGOhKGGQ99FKQobJGxzD1IgcJyBn2iac+7tmZ+FHqTAfrny7Wyc0TqTcJKGxTLBaSldXbH5MN12p6GimKc9WCjgpA0Ur5qvUL67eX1ABRLGNETAR1OTqkyUWmChTo34jTJlq3yzE4S1JkwTzhUY4lPk0mGnEIFKXgwAJGi/VZd60IkDZtRlixX9/EPilNRJW30Qilpp4NcZu1KHSwk0oS9mGpXE+HRlyDxU1CSN9QvO5HgS59XXfH4wqLw+cqObik5RT4uDLT9JN/CTlkOuTuQ5yx/EJ5ZHVICOoTUv9lEe/piKp1bL8vr7gy+rnky5OOXR1EbTKJja5NvepuMlTEYczPdw/Ojw8ODjY2x77QvhxNDo8yluhxbo7vG7Btue6Oyh7tOUFMklZyv+Fny3RPtQxvQDjCe1doj0njlQLM4XKGleHA2U097dHY1vqB74DjkXYoMbdmGM3mKJsAYbugB/zJvukS0h6lagRdGGZuzbPQXBr8rxDDutGCsdVCtNrZNZGZNl+V4xtvitwVANd8q1QndkGeoYFxZxAI1uz2SHfp3zRoKo5UUcc4pdtAZGCtdL2ry1iUb7+7q7/8L+uAv4NlM9TZZ9obpNgJvMq0VC/dJuIcgWxmfzXJr+7JhJUbSjr7cBHrt0FEGkeHVC5IpjpM9dmwmnRsCikMVllraqaZ1ClxK5yVZFtjFZYm5H0HdRmY20qFgPg/xb9cZpkDNZUbZxbTeNTG/UULW1B9MMgU3mrb9wTQuNXdszZhK3Y7rU2c5pAXDFW/BjT75u3m8B9l2kLk9H+z/vBi58Gj8bVa86Od8/7wbvvBd6h98h74o29iUc85v3pvff+6j3rZb2L3ro0vXmjYr73Gqv3xz/lK1WD</latexit><latexit sha1_base64="cbM0eSmGxMtQkVBDiiBAEOk4Olo=">AAAJx3icpZZfb9s2EMDVrtti70/b7bEvwowALZAGlrcg2VuxIkD/BIvb1Wkw0zAo+iQTpSiBpFO7jB72FfbafYJ+o32bUZScSFQLaBsB22fe/e5OJ95JYcaoVMPh3zdufnbr8y++3On1v/r6m29v37n73ZlMV4LAhKQsFechlsAoh4miisF5JgAnIYPX4ZvHhf71BQhJU/5KbTKYJTjmNKIEK7N1jtQSFJ4H8zuD4f7QLr8tBJUw8Ko1nt/d+YAWKVklwBVhWMppMMzUTGOhKGGQ99FKQobJGxzD1IgcJyBn2iac+7tmZ+FHqTAfrny7Wyc0TqTcJKGxTLBaSldXbH5MN12p6GimKc9WCjgpA0Ur5qvUL67eX1ABRLGNETAR1OTqkyUWmChTo34jTJlq3yzE4S1JkwTzhUY4lPk0mGnEIFKXgwAJGi/VZd60IkDZtRlixX9/EPilNRJW30Qilpp4NcZu1KHSwk0oS9mGpXE+HRlyDxU1CSN9QvO5HgS59XXfH4wqLw+cqObik5RT4uDLT9JN/CTlkOuTuQ5yx/EJ5ZHVICOoTUv9lEe/piKp1bL8vr7gy+rnky5OOXR1EbTKJja5NvepuMlTEYczPdw/Ojw8ODjY2x77QvhxNDo8yluhxbo7vG7Btue6Oyh7tOUFMklZyv+Fny3RPtQxvQDjCe1doj0njlQLM4XKGleHA2U097dHY1vqB74DjkXYoMbdmGM3mKJsAYbugB/zJvukS0h6lagRdGGZuzbPQXBr8rxDDutGCsdVCtNrZNZGZNl+V4xtvitwVANd8q1QndkGeoYFxZxAI1uz2SHfp3zRoKo5UUcc4pdtAZGCtdL2ry1iUb7+7q7/8L+uAv4NlM9TZZ9obpNgJvMq0VC/dJuIcgWxmfzXJr+7JhJUbSjr7cBHrt0FEGkeHVC5IpjpM9dmwmnRsCikMVllraqaZ1ClxK5yVZFtjFZYm5H0HdRmY20qFgPg/xb9cZpkDNZUbZxbTeNTG/UULW1B9MMgU3mrb9wTQuNXdszZhK3Y7rU2c5pAXDFW/BjT75u3m8B9l2kLk9H+z/vBi58Gj8bVa86Od8/7wbvvBd6h98h74o29iUc85v3pvff+6j3rZb2L3ro0vXmjYr73Gqv3xz/lK1WD</latexit><latexit sha1_base64="cbM0eSmGxMtQkVBDiiBAEOk4Olo=">AAAJx3icpZZfb9s2EMDVrtti70/b7bEvwowALZAGlrcg2VuxIkD/BIvb1Wkw0zAo+iQTpSiBpFO7jB72FfbafYJ+o32bUZScSFQLaBsB22fe/e5OJ95JYcaoVMPh3zdufnbr8y++3On1v/r6m29v37n73ZlMV4LAhKQsFechlsAoh4miisF5JgAnIYPX4ZvHhf71BQhJU/5KbTKYJTjmNKIEK7N1jtQSFJ4H8zuD4f7QLr8tBJUw8Ko1nt/d+YAWKVklwBVhWMppMMzUTGOhKGGQ99FKQobJGxzD1IgcJyBn2iac+7tmZ+FHqTAfrny7Wyc0TqTcJKGxTLBaSldXbH5MN12p6GimKc9WCjgpA0Ur5qvUL67eX1ABRLGNETAR1OTqkyUWmChTo34jTJlq3yzE4S1JkwTzhUY4lPk0mGnEIFKXgwAJGi/VZd60IkDZtRlixX9/EPilNRJW30Qilpp4NcZu1KHSwk0oS9mGpXE+HRlyDxU1CSN9QvO5HgS59XXfH4wqLw+cqObik5RT4uDLT9JN/CTlkOuTuQ5yx/EJ5ZHVICOoTUv9lEe/piKp1bL8vr7gy+rnky5OOXR1EbTKJja5NvepuMlTEYczPdw/Ojw8ODjY2x77QvhxNDo8yluhxbo7vG7Btue6Oyh7tOUFMklZyv+Fny3RPtQxvQDjCe1doj0njlQLM4XKGleHA2U097dHY1vqB74DjkXYoMbdmGM3mKJsAYbugB/zJvukS0h6lagRdGGZuzbPQXBr8rxDDutGCsdVCtNrZNZGZNl+V4xtvitwVANd8q1QndkGeoYFxZxAI1uz2SHfp3zRoKo5UUcc4pdtAZGCtdL2ry1iUb7+7q7/8L+uAv4NlM9TZZ9obpNgJvMq0VC/dJuIcgWxmfzXJr+7JhJUbSjr7cBHrt0FEGkeHVC5IpjpM9dmwmnRsCikMVllraqaZ1ClxK5yVZFtjFZYm5H0HdRmY20qFgPg/xb9cZpkDNZUbZxbTeNTG/UULW1B9MMgU3mrb9wTQuNXdszZhK3Y7rU2c5pAXDFW/BjT75u3m8B9l2kLk9H+z/vBi58Gj8bVa86Od8/7wbvvBd6h98h74o29iUc85v3pvff+6j3rZb2L3ro0vXmjYr73Gqv3xz/lK1WD</latexit><latexit sha1_base64="cbM0eSmGxMtQkVBDiiBAEOk4Olo=">AAAJx3icpZZfb9s2EMDVrtti70/b7bEvwowALZAGlrcg2VuxIkD/BIvb1Wkw0zAo+iQTpSiBpFO7jB72FfbafYJ+o32bUZScSFQLaBsB22fe/e5OJ95JYcaoVMPh3zdufnbr8y++3On1v/r6m29v37n73ZlMV4LAhKQsFechlsAoh4miisF5JgAnIYPX4ZvHhf71BQhJU/5KbTKYJTjmNKIEK7N1jtQSFJ4H8zuD4f7QLr8tBJUw8Ko1nt/d+YAWKVklwBVhWMppMMzUTGOhKGGQ99FKQobJGxzD1IgcJyBn2iac+7tmZ+FHqTAfrny7Wyc0TqTcJKGxTLBaSldXbH5MN12p6GimKc9WCjgpA0Ur5qvUL67eX1ABRLGNETAR1OTqkyUWmChTo34jTJlq3yzE4S1JkwTzhUY4lPk0mGnEIFKXgwAJGi/VZd60IkDZtRlixX9/EPilNRJW30Qilpp4NcZu1KHSwk0oS9mGpXE+HRlyDxU1CSN9QvO5HgS59XXfH4wqLw+cqObik5RT4uDLT9JN/CTlkOuTuQ5yx/EJ5ZHVICOoTUv9lEe/piKp1bL8vr7gy+rnky5OOXR1EbTKJja5NvepuMlTEYczPdw/Ojw8ODjY2x77QvhxNDo8yluhxbo7vG7Btue6Oyh7tOUFMklZyv+Fny3RPtQxvQDjCe1doj0njlQLM4XKGleHA2U097dHY1vqB74DjkXYoMbdmGM3mKJsAYbugB/zJvukS0h6lagRdGGZuzbPQXBr8rxDDutGCsdVCtNrZNZGZNl+V4xtvitwVANd8q1QndkGeoYFxZxAI1uz2SHfp3zRoKo5UUcc4pdtAZGCtdL2ry1iUb7+7q7/8L+uAv4NlM9TZZ9obpNgJvMq0VC/dJuIcgWxmfzXJr+7JhJUbSjr7cBHrt0FEGkeHVC5IpjpM9dmwmnRsCikMVllraqaZ1ClxK5yVZFtjFZYm5H0HdRmY20qFgPg/xb9cZpkDNZUbZxbTeNTG/UULW1B9MMgU3mrb9wTQuNXdszZhK3Y7rU2c5pAXDFW/BjT75u3m8B9l2kLk9H+z/vBi58Gj8bVa86Od8/7wbvvBd6h98h74o29iUc85v3pvff+6j3rZb2L3ro0vXmjYr73Gqv3xz/lK1WD</latexit>

✓2<latexit sha1_base64="YwKmc936ERHJ7kguU7c+8BuB0wk=">AAAJx3icpZZfb9s2EMDVrtti70/b7bEvwowALZAGlrcg2VuxIkD/BIvb1Wkw0zAo+iQTpSiBpFO7jB72FfbafYJ+o32bUZScSFQLaBsB22fe/e5OJ95JYcaoVMPh3zdufnbr8y++3On1v/r6m29v37n73ZlMV4LAhKQsFechlsAoh4miisF5JgAnIYPX4ZvHhf71BQhJU/5KbTKYJTjmNKIEK7N1jtQSFJ6P5ncGw/2hXX5bCCph4FVrPL+78wEtUrJKgCvCsJTTYJipmcZCUcIg76OVhAyTNziGqRE5TkDOtE0493fNzsKPUmE+XPl2t05onEi5SUJjmWC1lK6u2PyYbrpS0dFMU56tFHBSBopWzFepX1y9v6ACiGIbI2AiqMnVJ0ssMFGmRv1GmDLVvlmIw1uSJgnmC41wKPNpMNOIQaQuBwESNF6qy7xpRYCyazPEiv/+IPBLaySsvolELDXxaozdqEOlhZtQlrINS+N8OjLkHipqEkb6hOZzPQhy6+u+PxhVXh44Uc3FJymnxMGXn6Sb+EnKIdcncx3kjuMTyiOrQUZQm5b6KY9+TUVSq2X5fX3Bl9XPJ12ccujqImiVTWxybe5TcZOnIg5nerh/dHh4cHCwtz32hfDjaHR4lLdCi3V3eN2Cbc91d1D2aMsLZJKylP8LP1uifahjegHGE9q7RHtOHKkWZgqVNa4OB8po7m+PxrbUD3wHHIuwQY27McduMEXZAgzdAT/mTfZJl5D0KlEj6MIyd22eg+DW5HmHHNaNFI6rFKbXyKyNyLL9rhjbfFfgqAa65FuhOrMN9AwLijmBRrZms0O+T/miQVVzoo44xC/bAiIFa6XtX1vEonz93V3/4X9dBfwbKJ+nyj7R3CbBTOZVoqF+6TYR5QpiM/mvTX53TSSo2lDW24GPXLsLINI8OqByRTDTZ67NhNOiYVFIY7LKWlU1z6BKiV3lqiLbGK2wNiPpO6jNxtpULAbA/y364zTJGKyp2ji3msanNuopWtqC6IdBpvJW37gnhMav7JizCVux3Wtt5jSBuGKs+DGm3zdvN4H7LtMWJqP9n/eDFz8NHo2r15wd7573g3ffC7xD75H3xBt7E494zPvTe+/91XvWy3oXvXVpevNGxXzvNVbvj38A7odVhA==</latexit><latexit sha1_base64="YwKmc936ERHJ7kguU7c+8BuB0wk=">AAAJx3icpZZfb9s2EMDVrtti70/b7bEvwowALZAGlrcg2VuxIkD/BIvb1Wkw0zAo+iQTpSiBpFO7jB72FfbafYJ+o32bUZScSFQLaBsB22fe/e5OJ95JYcaoVMPh3zdufnbr8y++3On1v/r6m29v37n73ZlMV4LAhKQsFechlsAoh4miisF5JgAnIYPX4ZvHhf71BQhJU/5KbTKYJTjmNKIEK7N1jtQSFJ6P5ncGw/2hXX5bCCph4FVrPL+78wEtUrJKgCvCsJTTYJipmcZCUcIg76OVhAyTNziGqRE5TkDOtE0493fNzsKPUmE+XPl2t05onEi5SUJjmWC1lK6u2PyYbrpS0dFMU56tFHBSBopWzFepX1y9v6ACiGIbI2AiqMnVJ0ssMFGmRv1GmDLVvlmIw1uSJgnmC41wKPNpMNOIQaQuBwESNF6qy7xpRYCyazPEiv/+IPBLaySsvolELDXxaozdqEOlhZtQlrINS+N8OjLkHipqEkb6hOZzPQhy6+u+PxhVXh44Uc3FJymnxMGXn6Sb+EnKIdcncx3kjuMTyiOrQUZQm5b6KY9+TUVSq2X5fX3Bl9XPJ12ccujqImiVTWxybe5TcZOnIg5nerh/dHh4cHCwtz32hfDjaHR4lLdCi3V3eN2Cbc91d1D2aMsLZJKylP8LP1uifahjegHGE9q7RHtOHKkWZgqVNa4OB8po7m+PxrbUD3wHHIuwQY27McduMEXZAgzdAT/mTfZJl5D0KlEj6MIyd22eg+DW5HmHHNaNFI6rFKbXyKyNyLL9rhjbfFfgqAa65FuhOrMN9AwLijmBRrZms0O+T/miQVVzoo44xC/bAiIFa6XtX1vEonz93V3/4X9dBfwbKJ+nyj7R3CbBTOZVoqF+6TYR5QpiM/mvTX53TSSo2lDW24GPXLsLINI8OqByRTDTZ67NhNOiYVFIY7LKWlU1z6BKiV3lqiLbGK2wNiPpO6jNxtpULAbA/y364zTJGKyp2ji3msanNuopWtqC6IdBpvJW37gnhMav7JizCVux3Wtt5jSBuGKs+DGm3zdvN4H7LtMWJqP9n/eDFz8NHo2r15wd7573g3ffC7xD75H3xBt7E494zPvTe+/91XvWy3oXvXVpevNGxXzvNVbvj38A7odVhA==</latexit><latexit sha1_base64="YwKmc936ERHJ7kguU7c+8BuB0wk=">AAAJx3icpZZfb9s2EMDVrtti70/b7bEvwowALZAGlrcg2VuxIkD/BIvb1Wkw0zAo+iQTpSiBpFO7jB72FfbafYJ+o32bUZScSFQLaBsB22fe/e5OJ95JYcaoVMPh3zdufnbr8y++3On1v/r6m29v37n73ZlMV4LAhKQsFechlsAoh4miisF5JgAnIYPX4ZvHhf71BQhJU/5KbTKYJTjmNKIEK7N1jtQSFJ6P5ncGw/2hXX5bCCph4FVrPL+78wEtUrJKgCvCsJTTYJipmcZCUcIg76OVhAyTNziGqRE5TkDOtE0493fNzsKPUmE+XPl2t05onEi5SUJjmWC1lK6u2PyYbrpS0dFMU56tFHBSBopWzFepX1y9v6ACiGIbI2AiqMnVJ0ssMFGmRv1GmDLVvlmIw1uSJgnmC41wKPNpMNOIQaQuBwESNF6qy7xpRYCyazPEiv/+IPBLaySsvolELDXxaozdqEOlhZtQlrINS+N8OjLkHipqEkb6hOZzPQhy6+u+PxhVXh44Uc3FJymnxMGXn6Sb+EnKIdcncx3kjuMTyiOrQUZQm5b6KY9+TUVSq2X5fX3Bl9XPJ12ccujqImiVTWxybe5TcZOnIg5nerh/dHh4cHCwtz32hfDjaHR4lLdCi3V3eN2Cbc91d1D2aMsLZJKylP8LP1uifahjegHGE9q7RHtOHKkWZgqVNa4OB8po7m+PxrbUD3wHHIuwQY27McduMEXZAgzdAT/mTfZJl5D0KlEj6MIyd22eg+DW5HmHHNaNFI6rFKbXyKyNyLL9rhjbfFfgqAa65FuhOrMN9AwLijmBRrZms0O+T/miQVVzoo44xC/bAiIFa6XtX1vEonz93V3/4X9dBfwbKJ+nyj7R3CbBTOZVoqF+6TYR5QpiM/mvTX53TSSo2lDW24GPXLsLINI8OqByRTDTZ67NhNOiYVFIY7LKWlU1z6BKiV3lqiLbGK2wNiPpO6jNxtpULAbA/y364zTJGKyp2ji3msanNuopWtqC6IdBpvJW37gnhMav7JizCVux3Wtt5jSBuGKs+DGm3zdvN4H7LtMWJqP9n/eDFz8NHo2r15wd7573g3ffC7xD75H3xBt7E494zPvTe+/91XvWy3oXvXVpevNGxXzvNVbvj38A7odVhA==</latexit><latexit sha1_base64="YwKmc936ERHJ7kguU7c+8BuB0wk=">AAAJx3icpZZfb9s2EMDVrtti70/b7bEvwowALZAGlrcg2VuxIkD/BIvb1Wkw0zAo+iQTpSiBpFO7jB72FfbafYJ+o32bUZScSFQLaBsB22fe/e5OJ95JYcaoVMPh3zdufnbr8y++3On1v/r6m29v37n73ZlMV4LAhKQsFechlsAoh4miisF5JgAnIYPX4ZvHhf71BQhJU/5KbTKYJTjmNKIEK7N1jtQSFJ6P5ncGw/2hXX5bCCph4FVrPL+78wEtUrJKgCvCsJTTYJipmcZCUcIg76OVhAyTNziGqRE5TkDOtE0493fNzsKPUmE+XPl2t05onEi5SUJjmWC1lK6u2PyYbrpS0dFMU56tFHBSBopWzFepX1y9v6ACiGIbI2AiqMnVJ0ssMFGmRv1GmDLVvlmIw1uSJgnmC41wKPNpMNOIQaQuBwESNF6qy7xpRYCyazPEiv/+IPBLaySsvolELDXxaozdqEOlhZtQlrINS+N8OjLkHipqEkb6hOZzPQhy6+u+PxhVXh44Uc3FJymnxMGXn6Sb+EnKIdcncx3kjuMTyiOrQUZQm5b6KY9+TUVSq2X5fX3Bl9XPJ12ccujqImiVTWxybe5TcZOnIg5nerh/dHh4cHCwtz32hfDjaHR4lLdCi3V3eN2Cbc91d1D2aMsLZJKylP8LP1uifahjegHGE9q7RHtOHKkWZgqVNa4OB8po7m+PxrbUD3wHHIuwQY27McduMEXZAgzdAT/mTfZJl5D0KlEj6MIyd22eg+DW5HmHHNaNFI6rFKbXyKyNyLL9rhjbfFfgqAa65FuhOrMN9AwLijmBRrZms0O+T/miQVVzoo44xC/bAiIFa6XtX1vEonz93V3/4X9dBfwbKJ+nyj7R3CbBTOZVoqF+6TYR5QpiM/mvTX53TSSo2lDW24GPXLsLINI8OqByRTDTZ67NhNOiYVFIY7LKWlU1z6BKiV3lqiLbGK2wNiPpO6jNxtpULAbA/y364zTJGKyp2ji3msanNuopWtqC6IdBpvJW37gnhMav7JizCVux3Wtt5jSBuGKs+DGm3zdvN4H7LtMWJqP9n/eDFz8NHo2r15wd7573g3ffC7xD75H3xBt7E494zPvTe+/91XvWy3oXvXVpevNGxXzvNVbvj38A7odVhA==</latexit>

✓̂<latexit sha1_base64="6WifpBZdSRyY2pUiuBMF5KWI8MI=">AAAJzHicpZZfb9s2EMDV7l/s/Wm7Pe5FmBGgBdLAcuMleStWBGjRoPG2Ji1mGgFFn2SiFCWQdGqP0Ws/Ql+3536jfptRlJxIVANoGwHbJ9797s4n3klhxqhUw+HHW7c/+/yLL7/a6vW//ubb7+7cvff9mUyXgsApSVkqXodYAqMcThVVDF5nAnASMngVvnlS6F9dgJA05S/VOoNZgmNOI0qwMlsILbDSSC1A4fz87mC4ezgMDoM93wiPxsNxYISDw73Rz4Ef7A7tGnjVmpzf2/qA5ilZJsAVYVjKaTDM1ExjoShhkPfRUkKGyRscw9SIHCcgZ9omnfvbZmfuR6kwH658u1snNE6kXCehsUywWkhXV2x+SjddquhgpinPlgo4KQNFS+ar1C8q4M+pAKLY2giYCGpy9ckCC0yUqVO/EaZMtW8W4vCWpEmC+VwjHMp8Gsw0YhCpy0GABI0X6jJvWhGg7NoMseLaHwR+aY2E1TeRiKUmXo2xG3WotHATylK2ZmmcT0eG3EFFTcJIH9P8XA+C3Pq67w9GlZcHTlTz55OUU+LgixvpJn6ccsj18bkOcsfxMeWR1SAjqHVL/YxHL1KR1GpZfl//4cvq50YXJxy6ughaZRPrXJv7VNzkqYjDmTanfX9/PB7vDKvTXgiPRqP9g7wVWqy6w6sWXLZcZwdVi7peIJOUpfxf+NkQ7UMd0wswntDOJdpx4kg1N5OorHF1OFBGc39zNDalfuA74ESEDWrSjTlygynK5mDoDvgRb7JPu4SkV4kaQReWuWvzHAS3Js875LBqpHBUpTC9RmZtRJbtd8XY5rsCRzXQJd8K1ZltoGdYUMwJNLI1mx3yfcbnDaqaE3XEIX7ZFBApWCltL20Ri/L1t7f9h/91FfDvoHyeKvtUc5sEM5lXiYb6N7eJKFcQm8l/bfKHayJB1Yay3gx85NpdAJHm0QGVK4KZPnNtTjktGhaFNCbLrFVV8wyqlNhVLiuyjdEKazOS/gm12VibisUA+L9Ff5ImGYMVVWvnVtP4xEY9QQtbEP0wyFTe6hv3hND4pR1zNmErtnutzZwkEFeMFT/F9Pvm7WbzCuPfLJyNdoPhbvDr3uDxpHrP2fJ+9H7y7nuBt+899p56E+/UI17mvff+8v7uveipnu7lpentWxXzg9dYvXf/AJL+WCU=</latexit><latexit sha1_base64="6WifpBZdSRyY2pUiuBMF5KWI8MI=">AAAJzHicpZZfb9s2EMDV7l/s/Wm7Pe5FmBGgBdLAcuMleStWBGjRoPG2Ji1mGgFFn2SiFCWQdGqP0Ws/Ql+3536jfptRlJxIVANoGwHbJ9797s4n3klhxqhUw+HHW7c/+/yLL7/a6vW//ubb7+7cvff9mUyXgsApSVkqXodYAqMcThVVDF5nAnASMngVvnlS6F9dgJA05S/VOoNZgmNOI0qwMlsILbDSSC1A4fz87mC4ezgMDoM93wiPxsNxYISDw73Rz4Ef7A7tGnjVmpzf2/qA5ilZJsAVYVjKaTDM1ExjoShhkPfRUkKGyRscw9SIHCcgZ9omnfvbZmfuR6kwH658u1snNE6kXCehsUywWkhXV2x+SjddquhgpinPlgo4KQNFS+ar1C8q4M+pAKLY2giYCGpy9ckCC0yUqVO/EaZMtW8W4vCWpEmC+VwjHMp8Gsw0YhCpy0GABI0X6jJvWhGg7NoMseLaHwR+aY2E1TeRiKUmXo2xG3WotHATylK2ZmmcT0eG3EFFTcJIH9P8XA+C3Pq67w9GlZcHTlTz55OUU+LgixvpJn6ccsj18bkOcsfxMeWR1SAjqHVL/YxHL1KR1GpZfl//4cvq50YXJxy6ughaZRPrXJv7VNzkqYjDmTanfX9/PB7vDKvTXgiPRqP9g7wVWqy6w6sWXLZcZwdVi7peIJOUpfxf+NkQ7UMd0wswntDOJdpx4kg1N5OorHF1OFBGc39zNDalfuA74ESEDWrSjTlygynK5mDoDvgRb7JPu4SkV4kaQReWuWvzHAS3Js875LBqpHBUpTC9RmZtRJbtd8XY5rsCRzXQJd8K1ZltoGdYUMwJNLI1mx3yfcbnDaqaE3XEIX7ZFBApWCltL20Ri/L1t7f9h/91FfDvoHyeKvtUc5sEM5lXiYb6N7eJKFcQm8l/bfKHayJB1Yay3gx85NpdAJHm0QGVK4KZPnNtTjktGhaFNCbLrFVV8wyqlNhVLiuyjdEKazOS/gm12VibisUA+L9Ff5ImGYMVVWvnVtP4xEY9QQtbEP0wyFTe6hv3hND4pR1zNmErtnutzZwkEFeMFT/F9Pvm7WbzCuPfLJyNdoPhbvDr3uDxpHrP2fJ+9H7y7nuBt+899p56E+/UI17mvff+8v7uveipnu7lpentWxXzg9dYvXf/AJL+WCU=</latexit><latexit sha1_base64="6WifpBZdSRyY2pUiuBMF5KWI8MI=">AAAJzHicpZZfb9s2EMDV7l/s/Wm7Pe5FmBGgBdLAcuMleStWBGjRoPG2Ji1mGgFFn2SiFCWQdGqP0Ws/Ql+3536jfptRlJxIVANoGwHbJ9797s4n3klhxqhUw+HHW7c/+/yLL7/a6vW//ubb7+7cvff9mUyXgsApSVkqXodYAqMcThVVDF5nAnASMngVvnlS6F9dgJA05S/VOoNZgmNOI0qwMlsILbDSSC1A4fz87mC4ezgMDoM93wiPxsNxYISDw73Rz4Ef7A7tGnjVmpzf2/qA5ilZJsAVYVjKaTDM1ExjoShhkPfRUkKGyRscw9SIHCcgZ9omnfvbZmfuR6kwH658u1snNE6kXCehsUywWkhXV2x+SjddquhgpinPlgo4KQNFS+ar1C8q4M+pAKLY2giYCGpy9ckCC0yUqVO/EaZMtW8W4vCWpEmC+VwjHMp8Gsw0YhCpy0GABI0X6jJvWhGg7NoMseLaHwR+aY2E1TeRiKUmXo2xG3WotHATylK2ZmmcT0eG3EFFTcJIH9P8XA+C3Pq67w9GlZcHTlTz55OUU+LgixvpJn6ccsj18bkOcsfxMeWR1SAjqHVL/YxHL1KR1GpZfl//4cvq50YXJxy6ughaZRPrXJv7VNzkqYjDmTanfX9/PB7vDKvTXgiPRqP9g7wVWqy6w6sWXLZcZwdVi7peIJOUpfxf+NkQ7UMd0wswntDOJdpx4kg1N5OorHF1OFBGc39zNDalfuA74ESEDWrSjTlygynK5mDoDvgRb7JPu4SkV4kaQReWuWvzHAS3Js875LBqpHBUpTC9RmZtRJbtd8XY5rsCRzXQJd8K1ZltoGdYUMwJNLI1mx3yfcbnDaqaE3XEIX7ZFBApWCltL20Ri/L1t7f9h/91FfDvoHyeKvtUc5sEM5lXiYb6N7eJKFcQm8l/bfKHayJB1Yay3gx85NpdAJHm0QGVK4KZPnNtTjktGhaFNCbLrFVV8wyqlNhVLiuyjdEKazOS/gm12VibisUA+L9Ff5ImGYMVVWvnVtP4xEY9QQtbEP0wyFTe6hv3hND4pR1zNmErtnutzZwkEFeMFT/F9Pvm7WbzCuPfLJyNdoPhbvDr3uDxpHrP2fJ+9H7y7nuBt+899p56E+/UI17mvff+8v7uveipnu7lpentWxXzg9dYvXf/AJL+WCU=</latexit><latexit sha1_base64="6WifpBZdSRyY2pUiuBMF5KWI8MI=">AAAJzHicpZZfb9s2EMDV7l/s/Wm7Pe5FmBGgBdLAcuMleStWBGjRoPG2Ji1mGgFFn2SiFCWQdGqP0Ws/Ql+3536jfptRlJxIVANoGwHbJ9797s4n3klhxqhUw+HHW7c/+/yLL7/a6vW//ubb7+7cvff9mUyXgsApSVkqXodYAqMcThVVDF5nAnASMngVvnlS6F9dgJA05S/VOoNZgmNOI0qwMlsILbDSSC1A4fz87mC4ezgMDoM93wiPxsNxYISDw73Rz4Ef7A7tGnjVmpzf2/qA5ilZJsAVYVjKaTDM1ExjoShhkPfRUkKGyRscw9SIHCcgZ9omnfvbZmfuR6kwH658u1snNE6kXCehsUywWkhXV2x+SjddquhgpinPlgo4KQNFS+ar1C8q4M+pAKLY2giYCGpy9ckCC0yUqVO/EaZMtW8W4vCWpEmC+VwjHMp8Gsw0YhCpy0GABI0X6jJvWhGg7NoMseLaHwR+aY2E1TeRiKUmXo2xG3WotHATylK2ZmmcT0eG3EFFTcJIH9P8XA+C3Pq67w9GlZcHTlTz55OUU+LgixvpJn6ccsj18bkOcsfxMeWR1SAjqHVL/YxHL1KR1GpZfl//4cvq50YXJxy6ughaZRPrXJv7VNzkqYjDmTanfX9/PB7vDKvTXgiPRqP9g7wVWqy6w6sWXLZcZwdVi7peIJOUpfxf+NkQ7UMd0wswntDOJdpx4kg1N5OorHF1OFBGc39zNDalfuA74ESEDWrSjTlygynK5mDoDvgRb7JPu4SkV4kaQReWuWvzHAS3Js875LBqpHBUpTC9RmZtRJbtd8XY5rsCRzXQJd8K1ZltoGdYUMwJNLI1mx3yfcbnDaqaE3XEIX7ZFBApWCltL20Ri/L1t7f9h/91FfDvoHyeKvtUc5sEM5lXiYb6N7eJKFcQm8l/bfKHayJB1Yay3gx85NpdAJHm0QGVK4KZPnNtTjktGhaFNCbLrFVV8wyqlNhVLiuyjdEKazOS/gm12VibisUA+L9Ff5ImGYMVVWvnVtP4xEY9QQtbEP0wyFTe6hv3hND4pR1zNmErtnutzZwkEFeMFT/F9Pvm7WbzCuPfLJyNdoPhbvDr3uDxpHrP2fJ+9H7y7nuBt+899p56E+/UI17mvff+8v7uveipnu7lpentWxXzg9dYvXf/AJL+WCU=</latexit>

1951

1998Convolutional Networks

What is ML/AI Systems Research Today?

What is AI-Systems Research?

Ø Good AI and Systems researchØ Provides insights to both communities

Ø Leverages understanding of both domainsØ Examples:

Ø Studies tradeoff between statistical and computational efficiencyØ Identify essential abstractions in DNN designØ Leverages framing of indexes to exploit overfitting

Ø More than just great software!Ø Builds on Big Ideas in AI and Systems Research

Big Ideas in Systems Research

Ø Problem FramingØ Identifying the right problem and solution requirements

Ø Abstraction & Managing ComplexityØ Reducing complex problems into smaller parts

Ø TradeoffsØ Understanding fundamental constraints

Ø Details: System design and Implementation

Big Ideas in ML Research

Ø GeneralizationØ What is being “learned”?

Ø Inductive Biases and RepresentationsØ What assumptions about domain enable efficient learning?

Ø Efficiency (Data and Computation)Ø How much data and time are needed to learn?

Ø Details: Objectives/Models/Algorithms

Kinds of AI-Systems Research

AI + Systems

Advances in AI are being used to address fundamental challenges in systems.

AI + SystemsØ Reinforcement Learning for

Ø Pandas code generationØ SQL join planningØ Network packet

classificationØ Autoscaling

Ø Bandit Algorithms for radio link adaptation

Ø Wireless link quality estimation

Ø Multi-task learning for straggler mitigation

Ø VM Selection using Trees ..

AI + SystemsAdvances in systems are enabling substantial progress in AI

AI + SystemsDeveloping Systems for:Ø Autonomous Vehicles Ø Reinforcement LearningØ Secure Machine LearningØ Prediction ServingØ Experiment Management

Advancing AIØ Dynamic Neural NetsØ Prediction on

Compressed DataØ Distributed TrainingØ Distributed Auto-ML

AI + SystemsBerkeley is the place to be for

ResearchWhy?

riselabUC Berkeley

AI + SystemsBerkeley is the place to be for

Researchriselab

UC Berkeley

About You?

Background RequirementsØ You have taken previous courses in either

Ø Systems (for example CS162) à comfortable building systemsand familiar with big ideas in system design

Ø Machine Learning (for example CS189) à comfortable training neural networks and familiar with big ideas in ML

Ø You are actively involved in research in either systems or machine learning (or both)

If this does not describe you, please visit my office hourson Wednesdays from 4-5 in 773 Soda Hall.

Goals for the ClassWhat do you expect from this class?

My Goals for the Class

Ø Identify and solve impactful problems at the intersection of AI and Systems

Ø Learn about the big ideas and key results in AI systems

Ø Learn how to read, evaluate, and peer review papers

Ø Learn how to write papers that get high review scores

How will we achieve these goals?

Ø Lectures and Reading: study developments in AI SystemsØ Identify the key open problems and research opportunities

Ø Projects: collaboration between Systems and AI studentsØ Explore open problems and produce top tier publications

Ø In Class Reviews: weekly oral reviews of papersØ Learn both how to evaluate papers and understand how

papers are reviewed

Reasons not to take this class … LØ If you want to learn how to train models and

use TensorFlow, PyTorch, and SkLearnØ Take: CS C100, CS182, CS189

Ø If you want learn how to use big data systemsØ Take: CS162, CS186

Ø You are not interested in learning how to evaluate, conduct, and communicate researchØ Why not?

Ø You are unable to attend lecture

If you plan to drop this class, please drop it soon so others can

enroll!

ProblemsWhat makes a good problem?

What makes a good problem?

Ø Impact: People care about the solutionØ … and progress advances our understanding (research)

Ø Metrics: You know when you have succeededØ Can you measure progress on the solution?

Ø Divisible: The problem can be divided into smaller problemsØ You can identify the first sub-problem.

Ø Your Edge: Why is it a good problem for you?Ø Leverage your strengths and imagine a new path.

Can you Solve a Solved Problem?

Ø Ideally you want to solve a new and important problem

Ø A new solution to a solved problem can be impactful if:Ø It supports a broader set of applications (users)Ø It reveals a fundamental trade-off orØ Provides a deeper understanding of the problem spaceØ 10x Better?

Ø Often publishable…Ø Should satisfy one of the three above conditions.

Class Organization

Weekly Topic OrganizationØ Each week we will cover a new topic

Ø There will be 3 required papers to read each week

Ø Monday I will cover the background for the topic and an overview of the reading for the week

Ø Friday you will participate in an in class “Program Committee Meeting” to discuss the reading

• Big Ideas in ML/Sys• ML Life-cycle• ML in DBs• Model Dev. Frameworks• Distributed Training• Prediction Serving

• Autonomous Driving• Model Compilation• Hardware Acceleration• ML à Systems• Secure ML• Debugging and Interpretability

I may make changes to topics and

format based on class

participation.

In Class PC Meeting Format (V.0)

For each of the three papers: (30 Minutes per paper)

Ø Neutral: recap of the paper (neutral opinion) [5 Minutes]

Ø Advocate: Strengths of the paper [5 Minutes]

Ø Critic: Weaknesses of the paper [5 Minutes]

Ø Class will discuss rebuttal and improvements [10 Minutes]

Ø Brief in-class vote for acceptance into the AI-Sys prelim

The Neutral Presenter will Summarize

Ø What is the problem being solved?

Ø What was the solution? (Summary!)

Ø What metrics did they use to evaluate their solution?Ø What was the Baselines of comparison?

Ø What was the key insight or enabling idea?

Ø What are the claimed technical contributions?

Advocate and Critic Will DiscussØ Novelty and Impact

Ø Are the problem and solution novel and how will the solution affect future research?

Ø Technical QualitiesØ Are the problem framing and assumptions reasonableØ Discuss merits of the technical contributionsØ Does the evaluation support claims and reveal limitations of the

proposed approach?

Ø PresentationØ Discuss the writing clarity and presentation of resultsØ Positioning of related work

Simple Grading Policy

20% Class ParticipationØ Attend lecture and participate in class discussionsØ Lead 2 to 3 in class discussions

20% Weekly ReviewsØ Submit 3 weekly paper reviews (google form) per week!Ø Time consuming but important!

60% Class ProjectsØ Do great research!

Action Items

Ø Go to website: https://ucbrise.github.io/cs294-ai-sys-fa19/

Ø Make sure you are on the course PiazzaØ Needed for announcements

https://tinyurl.com/aisysfa19signup

Ø Signup for 3 discussion slots as different roles here: