Embed Size (px)
Citation preview
About Me
Ø Co-director of the RISE Lab
Ø Co-founder of Turi Inc.
ResearchØ Machine Learning Ø Distributed Systems Ø Computer VisionØ Autonomous DrivingØ Secure Learning Ø …
riselabUC BerkeleyAbout Me
My story …
MachineLearning
LearningSystems
Back in 2007
Back in 2007I started studying the use of Gaussian Process (GP) models for wireless link quality estimation
Research was slowed by the speed of training… and the fact that link quality is difficult to predict
Back in 2008I started studying parallel inference algorithms and systems
I designed and implemented parallel learning algorithms on top of low-level primitives …
ApplicationIdea Model +
Algorithm
SerialPrototype
ParallelPrototype
Debug
Optimize
Evaluate
DistributedPrototype
OptimizeDebug
Evaluate
MLPaper
MLPaper
Low-Level Primitives
Ø Shared Memory Parallelism: pThreads & OpenMP
Ø Distributed Communication: Actors & MPIØ Yes,… I built an RPC framework and then an actor framework
Ø Hardware APIs + Languages: GPU Programming (CUDA)
Low-level parallel primitives offer fine grained control over parallel execution.
Advantages of theLow-Level ApproachØ Extract maximum performance from hardware
Ø Enable exploration of more complex algorithms• Fine grained locking• Atomic data-structures• Distributed coordination protocols
My implementation is better than your implementation.
Limitations of theLow-Level ApproachØ Repeatedly address the same system challenges
Ø Algorithm conflates learning and system logic
Ø Difficult to debug and extend
Ø Typically does not address issues at scale: hardware failure, stragglers, …
Months of tuning and engineering for one problem.
ApplicationIdea Model +
Algorithm
SerialPrototype
ParallelPrototype
Debug
Optimize
Evaluate
DistributedPrototype
OptimizeDebug
Evaluate
MLPaper
MLPaper
Interesting Machine Learning
ResearchInteresting
Systems ResearchAbstraction
Design Complexity
Model
Accuracy
MachineLearning
Parallelism
Locality
Scheduling
Fault Tolerance
Network
StragglersLarge-Scale
Systems
Training
Design Complexity
Training
MachineLearning
Locality
Scheduling
Fault Tolerance
Network
StragglersLarge-Scale
Systems
Model Parallelism
Accuracy
Interaction
Design Complexity
MachineLearning
Large-ScaleSystems
Model Parallelism
Interaction
Learning systems combine the complexities of machine learning with system design
Scheduling
Fault Tolerance
Stragglers
LocalityNetwork
Training Accuracy
Managing ComplexityThrough Abstraction
Learning AlgorithmCommon Patterns
System
Abstraction (API)
1. Parallelism2. Data Locality3. Network
4. Scheduling5. Fault-tolerance6. Stragglers
Identify common patterns
Define a narrowinterface
Exploit limited abstractionto address system
design challenges
PhD in Machine Learning from CMU 2013
ScalableSystemsAbstractions
Y2
Y1
Y3
X2
X3
X1
� 1,3
�1,2
�2,3(x2, x3)
�1(x1)
�2(x2)
�3(x3)
Graphical Model Inference
Vertex Program
Machine 1 Machine 2
GraphLab/GraphXSystem
MachineLearning
Looking Back on AI SystemsHow did the field evolve during the time I was doing my PhD
Machine learning communityhas had an evolving focus on AI Systems
2009 2019
FastAlgorithms
ML forSystems
DistributedAlgorithms
Deep LearningFrameworks
Integration ofCommunities
Machine LearningFrameworks
RL forSystems
Systems and the Third Wave of AI
Waves of AI ResearchØ 1950 to 1974: Birth of AI
Ø 1950 Alan Turing publishes the Imitation GameØ 1951 Marvin Minsky builds first neural network machine (SNARC)Ø DARPA starts to pour money into AI Research (limited oversight)Ø Incredible optimism
“Within ten years a digital computer will be the world’s chess champion” -- Herbert A. Simon and Allen Newell (1958)
“In from three to eight years we will have a machine with the general intelligence of an average human being”
– Marvin Minksy (1970)
Waves of AI ResearchØ 1950 to 1974: Birth of AI
Ø 1974 to 1980: First AI WinterØ Technology was unable to meet the high expectations
Ø Insufficient computer powerØ Over emphasis on combinatorial search Ø Insufficient knowledge (data)
Ø 1969 Book “Perceptrons” by Minsky and PapertØ Showed that certain basic functions (e.g., parity) cannot be computed using
local connections Ø Strong argument against connectionist approaches to AI (neural networks)
Ø Government funding driesØ Not meeting objectivesØ DARPA focuses more on immediate impact
Waves of AI ResearchØ 1950 to 1974: Birth of AI
Ø 1974 to 1980: First AI Winter
Ø 1980 to 1987: Second Wave of AIØ Expert systems start to solve real-world problems
Ø CMU developed XCON (AI for Systems) for DEC à saves $40M annually Ø Combines “knowledge” (expert rules “data”) with logic
Ø Japanese government invests $850M in AI ResearchØ Other governments respond by also investing heavily
Ø Emergence of the AI hardware and software industry to
Waves of AI ResearchØ 1950 to 1974: Birth of AI
Ø 1974 to 1980: First AI Winter
Ø 1980 to 1987: Second Wave of AI
Ø 1987 to 1993: Second AI WinterØ Expert systems too brittle to maintainØ Collapse of the specialized AI hardware market
Ø Commodity hardware was improving to rapidlyØ Ultimately over 300 AI companies would vanish from the market
Ø Massive government funding cuts
Waves of AI ResearchØ 1950 to 1974: Birth of AI
Ø 1974 to 1980: First AI Winter
Ø 1980 to 1987: Second Wave of AI
Ø 1987 to 1993: Second AI Winter
Ø 1993 to 2011: AI Goes Stealth Mode (aka Machine Learning)Ø Hardware becomes fast enough, and AI techniques start to work
Ø Deep Blue beats Garry Kasparov (1997)Ø OCR, Speech Recognition, Google Search, …
Ø Confluence of ideas and techniques: optimization, statistics, probability theory, and information theory
Waves of AI ResearchØ 1950 to 1974: Birth of AI
Ø 1974 to 1980: First AI Winter
Ø 1980 to 1987: Second Wave of AI
Ø 1987 to 1993: Second AI Winter
Ø 1993 to 2011: AI Goes Stealth Mode (aka Machine Learning)Ø Hardware becomes fast enough, and AI techniques start to work
Ø Deep Blue beats Garry Kasparov (1997)Ø OCR, Speech Recognition, Google Search, …
Ø Confluence of ideas and techniques: optimization, statistics, probability theory, and information theory
Waves of AI ResearchØ 1950 to 1974: Birth of AI
Ø 1974 to 1980: First AI Winter
Ø 1980 to 1987: Second Wave of AI
Ø 1987 to 1993: Second AI Winter
Ø 1993 to 2011: AI Goes Stealth Mode (aka Machine Learning)
Ø 2011 to 2019: Third Wave (AI Goes Deep)Ø Large quantities of data in conjunction with advances in hardware
and software enable the design and training of complex modelsØ New applications emerge
Ø Autonomous driving, home automation, Ø AI Market frenzy à IDC predicts 77.6B Market for AI-Systems in 2022Ø …
“A New Golden Age in Computer Architecture: Empowering the Machine-Learning Revolution”,https://ieeexplore.ieee.org/document/8259424
Why?
New Forces Driving AI Revolution
Compute AbstractionsData
Benchmarks
Advances inAlgorithms and
Models
Stochastic Gradient Descent
✓1<latexit sha1_base64="cbM0eSmGxMtQkVBDiiBAEOk4Olo=">AAAJx3icpZZfb9s2EMDVrtti70/b7bEvwowALZAGlrcg2VuxIkD/BIvb1Wkw0zAo+iQTpSiBpFO7jB72FfbafYJ+o32bUZScSFQLaBsB22fe/e5OJ95JYcaoVMPh3zdufnbr8y++3On1v/r6m29v37n73ZlMV4LAhKQsFechlsAoh4miisF5JgAnIYPX4ZvHhf71BQhJU/5KbTKYJTjmNKIEK7N1jtQSFJ4H8zuD4f7QLr8tBJUw8Ko1nt/d+YAWKVklwBVhWMppMMzUTGOhKGGQ99FKQobJGxzD1IgcJyBn2iac+7tmZ+FHqTAfrny7Wyc0TqTcJKGxTLBaSldXbH5MN12p6GimKc9WCjgpA0Ur5qvUL67eX1ABRLGNETAR1OTqkyUWmChTo34jTJlq3yzE4S1JkwTzhUY4lPk0mGnEIFKXgwAJGi/VZd60IkDZtRlixX9/EPilNRJW30Qilpp4NcZu1KHSwk0oS9mGpXE+HRlyDxU1CSN9QvO5HgS59XXfH4wqLw+cqObik5RT4uDLT9JN/CTlkOuTuQ5yx/EJ5ZHVICOoTUv9lEe/piKp1bL8vr7gy+rnky5OOXR1EbTKJja5NvepuMlTEYczPdw/Ojw8ODjY2x77QvhxNDo8yluhxbo7vG7Btue6Oyh7tOUFMklZyv+Fny3RPtQxvQDjCe1doj0njlQLM4XKGleHA2U097dHY1vqB74DjkXYoMbdmGM3mKJsAYbugB/zJvukS0h6lagRdGGZuzbPQXBr8rxDDutGCsdVCtNrZNZGZNl+V4xtvitwVANd8q1QndkGeoYFxZxAI1uz2SHfp3zRoKo5UUcc4pdtAZGCtdL2ry1iUb7+7q7/8L+uAv4NlM9TZZ9obpNgJvMq0VC/dJuIcgWxmfzXJr+7JhJUbSjr7cBHrt0FEGkeHVC5IpjpM9dmwmnRsCikMVllraqaZ1ClxK5yVZFtjFZYm5H0HdRmY20qFgPg/xb9cZpkDNZUbZxbTeNTG/UULW1B9MMgU3mrb9wTQuNXdszZhK3Y7rU2c5pAXDFW/BjT75u3m8B9l2kLk9H+z/vBi58Gj8bVa86Od8/7wbvvBd6h98h74o29iUc85v3pvff+6j3rZb2L3ro0vXmjYr73Gqv3xz/lK1WD</latexit><latexit sha1_base64="cbM0eSmGxMtQkVBDiiBAEOk4Olo=">AAAJx3icpZZfb9s2EMDVrtti70/b7bEvwowALZAGlrcg2VuxIkD/BIvb1Wkw0zAo+iQTpSiBpFO7jB72FfbafYJ+o32bUZScSFQLaBsB22fe/e5OJ95JYcaoVMPh3zdufnbr8y++3On1v/r6m29v37n73ZlMV4LAhKQsFechlsAoh4miisF5JgAnIYPX4ZvHhf71BQhJU/5KbTKYJTjmNKIEK7N1jtQSFJ4H8zuD4f7QLr8tBJUw8Ko1nt/d+YAWKVklwBVhWMppMMzUTGOhKGGQ99FKQobJGxzD1IgcJyBn2iac+7tmZ+FHqTAfrny7Wyc0TqTcJKGxTLBaSldXbH5MN12p6GimKc9WCjgpA0Ur5qvUL67eX1ABRLGNETAR1OTqkyUWmChTo34jTJlq3yzE4S1JkwTzhUY4lPk0mGnEIFKXgwAJGi/VZd60IkDZtRlixX9/EPilNRJW30Qilpp4NcZu1KHSwk0oS9mGpXE+HRlyDxU1CSN9QvO5HgS59XXfH4wqLw+cqObik5RT4uDLT9JN/CTlkOuTuQ5yx/EJ5ZHVICOoTUv9lEe/piKp1bL8vr7gy+rnky5OOXR1EbTKJja5NvepuMlTEYczPdw/Ojw8ODjY2x77QvhxNDo8yluhxbo7vG7Btue6Oyh7tOUFMklZyv+Fny3RPtQxvQDjCe1doj0njlQLM4XKGleHA2U097dHY1vqB74DjkXYoMbdmGM3mKJsAYbugB/zJvukS0h6lagRdGGZuzbPQXBr8rxDDutGCsdVCtNrZNZGZNl+V4xtvitwVANd8q1QndkGeoYFxZxAI1uz2SHfp3zRoKo5UUcc4pdtAZGCtdL2ry1iUb7+7q7/8L+uAv4NlM9TZZ9obpNgJvMq0VC/dJuIcgWxmfzXJr+7JhJUbSjr7cBHrt0FEGkeHVC5IpjpM9dmwmnRsCikMVllraqaZ1ClxK5yVZFtjFZYm5H0HdRmY20qFgPg/xb9cZpkDNZUbZxbTeNTG/UULW1B9MMgU3mrb9wTQuNXdszZhK3Y7rU2c5pAXDFW/BjT75u3m8B9l2kLk9H+z/vBi58Gj8bVa86Od8/7wbvvBd6h98h74o29iUc85v3pvff+6j3rZb2L3ro0vXmjYr73Gqv3xz/lK1WD</latexit><latexit sha1_base64="cbM0eSmGxMtQkVBDiiBAEOk4Olo=">AAAJx3icpZZfb9s2EMDVrtti70/b7bEvwowALZAGlrcg2VuxIkD/BIvb1Wkw0zAo+iQTpSiBpFO7jB72FfbafYJ+o32bUZScSFQLaBsB22fe/e5OJ95JYcaoVMPh3zdufnbr8y++3On1v/r6m29v37n73ZlMV4LAhKQsFechlsAoh4miisF5JgAnIYPX4ZvHhf71BQhJU/5KbTKYJTjmNKIEK7N1jtQSFJ4H8zuD4f7QLr8tBJUw8Ko1nt/d+YAWKVklwBVhWMppMMzUTGOhKGGQ99FKQobJGxzD1IgcJyBn2iac+7tmZ+FHqTAfrny7Wyc0TqTcJKGxTLBaSldXbH5MN12p6GimKc9WCjgpA0Ur5qvUL67eX1ABRLGNETAR1OTqkyUWmChTo34jTJlq3yzE4S1JkwTzhUY4lPk0mGnEIFKXgwAJGi/VZd60IkDZtRlixX9/EPilNRJW30Qilpp4NcZu1KHSwk0oS9mGpXE+HRlyDxU1CSN9QvO5HgS59XXfH4wqLw+cqObik5RT4uDLT9JN/CTlkOuTuQ5yx/EJ5ZHVICOoTUv9lEe/piKp1bL8vr7gy+rnky5OOXR1EbTKJja5NvepuMlTEYczPdw/Ojw8ODjY2x77QvhxNDo8yluhxbo7vG7Btue6Oyh7tOUFMklZyv+Fny3RPtQxvQDjCe1doj0njlQLM4XKGleHA2U097dHY1vqB74DjkXYoMbdmGM3mKJsAYbugB/zJvukS0h6lagRdGGZuzbPQXBr8rxDDutGCsdVCtNrZNZGZNl+V4xtvitwVANd8q1QndkGeoYFxZxAI1uz2SHfp3zRoKo5UUcc4pdtAZGCtdL2ry1iUb7+7q7/8L+uAv4NlM9TZZ9obpNgJvMq0VC/dJuIcgWxmfzXJr+7JhJUbSjr7cBHrt0FEGkeHVC5IpjpM9dmwmnRsCikMVllraqaZ1ClxK5yVZFtjFZYm5H0HdRmY20qFgPg/xb9cZpkDNZUbZxbTeNTG/UULW1B9MMgU3mrb9wTQuNXdszZhK3Y7rU2c5pAXDFW/BjT75u3m8B9l2kLk9H+z/vBi58Gj8bVa86Od8/7wbvvBd6h98h74o29iUc85v3pvff+6j3rZb2L3ro0vXmjYr73Gqv3xz/lK1WD</latexit><latexit sha1_base64="cbM0eSmGxMtQkVBDiiBAEOk4Olo=">AAAJx3icpZZfb9s2EMDVrtti70/b7bEvwowALZAGlrcg2VuxIkD/BIvb1Wkw0zAo+iQTpSiBpFO7jB72FfbafYJ+o32bUZScSFQLaBsB22fe/e5OJ95JYcaoVMPh3zdufnbr8y++3On1v/r6m29v37n73ZlMV4LAhKQsFechlsAoh4miisF5JgAnIYPX4ZvHhf71BQhJU/5KbTKYJTjmNKIEK7N1jtQSFJ4H8zuD4f7QLr8tBJUw8Ko1nt/d+YAWKVklwBVhWMppMMzUTGOhKGGQ99FKQobJGxzD1IgcJyBn2iac+7tmZ+FHqTAfrny7Wyc0TqTcJKGxTLBaSldXbH5MN12p6GimKc9WCjgpA0Ur5qvUL67eX1ABRLGNETAR1OTqkyUWmChTo34jTJlq3yzE4S1JkwTzhUY4lPk0mGnEIFKXgwAJGi/VZd60IkDZtRlixX9/EPilNRJW30Qilpp4NcZu1KHSwk0oS9mGpXE+HRlyDxU1CSN9QvO5HgS59XXfH4wqLw+cqObik5RT4uDLT9JN/CTlkOuTuQ5yx/EJ5ZHVICOoTUv9lEe/piKp1bL8vr7gy+rnky5OOXR1EbTKJja5NvepuMlTEYczPdw/Ojw8ODjY2x77QvhxNDo8yluhxbo7vG7Btue6Oyh7tOUFMklZyv+Fny3RPtQxvQDjCe1doj0njlQLM4XKGleHA2U097dHY1vqB74DjkXYoMbdmGM3mKJsAYbugB/zJvukS0h6lagRdGGZuzbPQXBr8rxDDutGCsdVCtNrZNZGZNl+V4xtvitwVANd8q1QndkGeoYFxZxAI1uz2SHfp3zRoKo5UUcc4pdtAZGCtdL2ry1iUb7+7q7/8L+uAv4NlM9TZZ9obpNgJvMq0VC/dJuIcgWxmfzXJr+7JhJUbSjr7cBHrt0FEGkeHVC5IpjpM9dmwmnRsCikMVllraqaZ1ClxK5yVZFtjFZYm5H0HdRmY20qFgPg/xb9cZpkDNZUbZxbTeNTG/UULW1B9MMgU3mrb9wTQuNXdszZhK3Y7rU2c5pAXDFW/BjT75u3m8B9l2kLk9H+z/vBi58Gj8bVa86Od8/7wbvvBd6h98h74o29iUc85v3pvff+6j3rZb2L3ro0vXmjYr73Gqv3xz/lK1WD</latexit>
✓2<latexit sha1_base64="YwKmc936ERHJ7kguU7c+8BuB0wk=">AAAJx3icpZZfb9s2EMDVrtti70/b7bEvwowALZAGlrcg2VuxIkD/BIvb1Wkw0zAo+iQTpSiBpFO7jB72FfbafYJ+o32bUZScSFQLaBsB22fe/e5OJ95JYcaoVMPh3zdufnbr8y++3On1v/r6m29v37n73ZlMV4LAhKQsFechlsAoh4miisF5JgAnIYPX4ZvHhf71BQhJU/5KbTKYJTjmNKIEK7N1jtQSFJ6P5ncGw/2hXX5bCCph4FVrPL+78wEtUrJKgCvCsJTTYJipmcZCUcIg76OVhAyTNziGqRE5TkDOtE0493fNzsKPUmE+XPl2t05onEi5SUJjmWC1lK6u2PyYbrpS0dFMU56tFHBSBopWzFepX1y9v6ACiGIbI2AiqMnVJ0ssMFGmRv1GmDLVvlmIw1uSJgnmC41wKPNpMNOIQaQuBwESNF6qy7xpRYCyazPEiv/+IPBLaySsvolELDXxaozdqEOlhZtQlrINS+N8OjLkHipqEkb6hOZzPQhy6+u+PxhVXh44Uc3FJymnxMGXn6Sb+EnKIdcncx3kjuMTyiOrQUZQm5b6KY9+TUVSq2X5fX3Bl9XPJ12ccujqImiVTWxybe5TcZOnIg5nerh/dHh4cHCwtz32hfDjaHR4lLdCi3V3eN2Cbc91d1D2aMsLZJKylP8LP1uifahjegHGE9q7RHtOHKkWZgqVNa4OB8po7m+PxrbUD3wHHIuwQY27McduMEXZAgzdAT/mTfZJl5D0KlEj6MIyd22eg+DW5HmHHNaNFI6rFKbXyKyNyLL9rhjbfFfgqAa65FuhOrMN9AwLijmBRrZms0O+T/miQVVzoo44xC/bAiIFa6XtX1vEonz93V3/4X9dBfwbKJ+nyj7R3CbBTOZVoqF+6TYR5QpiM/mvTX53TSSo2lDW24GPXLsLINI8OqByRTDTZ67NhNOiYVFIY7LKWlU1z6BKiV3lqiLbGK2wNiPpO6jNxtpULAbA/y364zTJGKyp2ji3msanNuopWtqC6IdBpvJW37gnhMav7JizCVux3Wtt5jSBuGKs+DGm3zdvN4H7LtMWJqP9n/eDFz8NHo2r15wd7573g3ffC7xD75H3xBt7E494zPvTe+/91XvWy3oXvXVpevNGxXzvNVbvj38A7odVhA==</latexit><latexit sha1_base64="YwKmc936ERHJ7kguU7c+8BuB0wk=">AAAJx3icpZZfb9s2EMDVrtti70/b7bEvwowALZAGlrcg2VuxIkD/BIvb1Wkw0zAo+iQTpSiBpFO7jB72FfbafYJ+o32bUZScSFQLaBsB22fe/e5OJ95JYcaoVMPh3zdufnbr8y++3On1v/r6m29v37n73ZlMV4LAhKQsFechlsAoh4miisF5JgAnIYPX4ZvHhf71BQhJU/5KbTKYJTjmNKIEK7N1jtQSFJ6P5ncGw/2hXX5bCCph4FVrPL+78wEtUrJKgCvCsJTTYJipmcZCUcIg76OVhAyTNziGqRE5TkDOtE0493fNzsKPUmE+XPl2t05onEi5SUJjmWC1lK6u2PyYbrpS0dFMU56tFHBSBopWzFepX1y9v6ACiGIbI2AiqMnVJ0ssMFGmRv1GmDLVvlmIw1uSJgnmC41wKPNpMNOIQaQuBwESNF6qy7xpRYCyazPEiv/+IPBLaySsvolELDXxaozdqEOlhZtQlrINS+N8OjLkHipqEkb6hOZzPQhy6+u+PxhVXh44Uc3FJymnxMGXn6Sb+EnKIdcncx3kjuMTyiOrQUZQm5b6KY9+TUVSq2X5fX3Bl9XPJ12ccujqImiVTWxybe5TcZOnIg5nerh/dHh4cHCwtz32hfDjaHR4lLdCi3V3eN2Cbc91d1D2aMsLZJKylP8LP1uifahjegHGE9q7RHtOHKkWZgqVNa4OB8po7m+PxrbUD3wHHIuwQY27McduMEXZAgzdAT/mTfZJl5D0KlEj6MIyd22eg+DW5HmHHNaNFI6rFKbXyKyNyLL9rhjbfFfgqAa65FuhOrMN9AwLijmBRrZms0O+T/miQVVzoo44xC/bAiIFa6XtX1vEonz93V3/4X9dBfwbKJ+nyj7R3CbBTOZVoqF+6TYR5QpiM/mvTX53TSSo2lDW24GPXLsLINI8OqByRTDTZ67NhNOiYVFIY7LKWlU1z6BKiV3lqiLbGK2wNiPpO6jNxtpULAbA/y364zTJGKyp2ji3msanNuopWtqC6IdBpvJW37gnhMav7JizCVux3Wtt5jSBuGKs+DGm3zdvN4H7LtMWJqP9n/eDFz8NHo2r15wd7573g3ffC7xD75H3xBt7E494zPvTe+/91XvWy3oXvXVpevNGxXzvNVbvj38A7odVhA==</latexit><latexit sha1_base64="YwKmc936ERHJ7kguU7c+8BuB0wk=">AAAJx3icpZZfb9s2EMDVrtti70/b7bEvwowALZAGlrcg2VuxIkD/BIvb1Wkw0zAo+iQTpSiBpFO7jB72FfbafYJ+o32bUZScSFQLaBsB22fe/e5OJ95JYcaoVMPh3zdufnbr8y++3On1v/r6m29v37n73ZlMV4LAhKQsFechlsAoh4miisF5JgAnIYPX4ZvHhf71BQhJU/5KbTKYJTjmNKIEK7N1jtQSFJ6P5ncGw/2hXX5bCCph4FVrPL+78wEtUrJKgCvCsJTTYJipmcZCUcIg76OVhAyTNziGqRE5TkDOtE0493fNzsKPUmE+XPl2t05onEi5SUJjmWC1lK6u2PyYbrpS0dFMU56tFHBSBopWzFepX1y9v6ACiGIbI2AiqMnVJ0ssMFGmRv1GmDLVvlmIw1uSJgnmC41wKPNpMNOIQaQuBwESNF6qy7xpRYCyazPEiv/+IPBLaySsvolELDXxaozdqEOlhZtQlrINS+N8OjLkHipqEkb6hOZzPQhy6+u+PxhVXh44Uc3FJymnxMGXn6Sb+EnKIdcncx3kjuMTyiOrQUZQm5b6KY9+TUVSq2X5fX3Bl9XPJ12ccujqImiVTWxybe5TcZOnIg5nerh/dHh4cHCwtz32hfDjaHR4lLdCi3V3eN2Cbc91d1D2aMsLZJKylP8LP1uifahjegHGE9q7RHtOHKkWZgqVNa4OB8po7m+PxrbUD3wHHIuwQY27McduMEXZAgzdAT/mTfZJl5D0KlEj6MIyd22eg+DW5HmHHNaNFI6rFKbXyKyNyLL9rhjbfFfgqAa65FuhOrMN9AwLijmBRrZms0O+T/miQVVzoo44xC/bAiIFa6XtX1vEonz93V3/4X9dBfwbKJ+nyj7R3CbBTOZVoqF+6TYR5QpiM/mvTX53TSSo2lDW24GPXLsLINI8OqByRTDTZ67NhNOiYVFIY7LKWlU1z6BKiV3lqiLbGK2wNiPpO6jNxtpULAbA/y364zTJGKyp2ji3msanNuopWtqC6IdBpvJW37gnhMav7JizCVux3Wtt5jSBuGKs+DGm3zdvN4H7LtMWJqP9n/eDFz8NHo2r15wd7573g3ffC7xD75H3xBt7E494zPvTe+/91XvWy3oXvXVpevNGxXzvNVbvj38A7odVhA==</latexit><latexit sha1_base64="YwKmc936ERHJ7kguU7c+8BuB0wk=">AAAJx3icpZZfb9s2EMDVrtti70/b7bEvwowALZAGlrcg2VuxIkD/BIvb1Wkw0zAo+iQTpSiBpFO7jB72FfbafYJ+o32bUZScSFQLaBsB22fe/e5OJ95JYcaoVMPh3zdufnbr8y++3On1v/r6m29v37n73ZlMV4LAhKQsFechlsAoh4miisF5JgAnIYPX4ZvHhf71BQhJU/5KbTKYJTjmNKIEK7N1jtQSFJ6P5ncGw/2hXX5bCCph4FVrPL+78wEtUrJKgCvCsJTTYJipmcZCUcIg76OVhAyTNziGqRE5TkDOtE0493fNzsKPUmE+XPl2t05onEi5SUJjmWC1lK6u2PyYbrpS0dFMU56tFHBSBopWzFepX1y9v6ACiGIbI2AiqMnVJ0ssMFGmRv1GmDLVvlmIw1uSJgnmC41wKPNpMNOIQaQuBwESNF6qy7xpRYCyazPEiv/+IPBLaySsvolELDXxaozdqEOlhZtQlrINS+N8OjLkHipqEkb6hOZzPQhy6+u+PxhVXh44Uc3FJymnxMGXn6Sb+EnKIdcncx3kjuMTyiOrQUZQm5b6KY9+TUVSq2X5fX3Bl9XPJ12ccujqImiVTWxybe5TcZOnIg5nerh/dHh4cHCwtz32hfDjaHR4lLdCi3V3eN2Cbc91d1D2aMsLZJKylP8LP1uifahjegHGE9q7RHtOHKkWZgqVNa4OB8po7m+PxrbUD3wHHIuwQY27McduMEXZAgzdAT/mTfZJl5D0KlEj6MIyd22eg+DW5HmHHNaNFI6rFKbXyKyNyLL9rhjbfFfgqAa65FuhOrMN9AwLijmBRrZms0O+T/miQVVzoo44xC/bAiIFa6XtX1vEonz93V3/4X9dBfwbKJ+nyj7R3CbBTOZVoqF+6TYR5QpiM/mvTX53TSSo2lDW24GPXLsLINI8OqByRTDTZ67NhNOiYVFIY7LKWlU1z6BKiV3lqiLbGK2wNiPpO6jNxtpULAbA/y364zTJGKyp2ji3msanNuopWtqC6IdBpvJW37gnhMav7JizCVux3Wtt5jSBuGKs+DGm3zdvN4H7LtMWJqP9n/eDFz8NHo2r15wd7573g3ffC7xD75H3xBt7E494zPvTe+/91XvWy3oXvXVpevNGxXzvNVbvj38A7odVhA==</latexit>
✓̂<latexit sha1_base64="6WifpBZdSRyY2pUiuBMF5KWI8MI=">AAAJzHicpZZfb9s2EMDV7l/s/Wm7Pe5FmBGgBdLAcuMleStWBGjRoPG2Ji1mGgFFn2SiFCWQdGqP0Ws/Ql+3536jfptRlJxIVANoGwHbJ9797s4n3klhxqhUw+HHW7c/+/yLL7/a6vW//ubb7+7cvff9mUyXgsApSVkqXodYAqMcThVVDF5nAnASMngVvnlS6F9dgJA05S/VOoNZgmNOI0qwMlsILbDSSC1A4fz87mC4ezgMDoM93wiPxsNxYISDw73Rz4Ef7A7tGnjVmpzf2/qA5ilZJsAVYVjKaTDM1ExjoShhkPfRUkKGyRscw9SIHCcgZ9omnfvbZmfuR6kwH658u1snNE6kXCehsUywWkhXV2x+SjddquhgpinPlgo4KQNFS+ar1C8q4M+pAKLY2giYCGpy9ckCC0yUqVO/EaZMtW8W4vCWpEmC+VwjHMp8Gsw0YhCpy0GABI0X6jJvWhGg7NoMseLaHwR+aY2E1TeRiKUmXo2xG3WotHATylK2ZmmcT0eG3EFFTcJIH9P8XA+C3Pq67w9GlZcHTlTz55OUU+LgixvpJn6ccsj18bkOcsfxMeWR1SAjqHVL/YxHL1KR1GpZfl//4cvq50YXJxy6ughaZRPrXJv7VNzkqYjDmTanfX9/PB7vDKvTXgiPRqP9g7wVWqy6w6sWXLZcZwdVi7peIJOUpfxf+NkQ7UMd0wswntDOJdpx4kg1N5OorHF1OFBGc39zNDalfuA74ESEDWrSjTlygynK5mDoDvgRb7JPu4SkV4kaQReWuWvzHAS3Js875LBqpHBUpTC9RmZtRJbtd8XY5rsCRzXQJd8K1ZltoGdYUMwJNLI1mx3yfcbnDaqaE3XEIX7ZFBApWCltL20Ri/L1t7f9h/91FfDvoHyeKvtUc5sEM5lXiYb6N7eJKFcQm8l/bfKHayJB1Yay3gx85NpdAJHm0QGVK4KZPnNtTjktGhaFNCbLrFVV8wyqlNhVLiuyjdEKazOS/gm12VibisUA+L9Ff5ImGYMVVWvnVtP4xEY9QQtbEP0wyFTe6hv3hND4pR1zNmErtnutzZwkEFeMFT/F9Pvm7WbzCuPfLJyNdoPhbvDr3uDxpHrP2fJ+9H7y7nuBt+899p56E+/UI17mvff+8v7uveipnu7lpentWxXzg9dYvXf/AJL+WCU=</latexit><latexit sha1_base64="6WifpBZdSRyY2pUiuBMF5KWI8MI=">AAAJzHicpZZfb9s2EMDV7l/s/Wm7Pe5FmBGgBdLAcuMleStWBGjRoPG2Ji1mGgFFn2SiFCWQdGqP0Ws/Ql+3536jfptRlJxIVANoGwHbJ9797s4n3klhxqhUw+HHW7c/+/yLL7/a6vW//ubb7+7cvff9mUyXgsApSVkqXodYAqMcThVVDF5nAnASMngVvnlS6F9dgJA05S/VOoNZgmNOI0qwMlsILbDSSC1A4fz87mC4ezgMDoM93wiPxsNxYISDw73Rz4Ef7A7tGnjVmpzf2/qA5ilZJsAVYVjKaTDM1ExjoShhkPfRUkKGyRscw9SIHCcgZ9omnfvbZmfuR6kwH658u1snNE6kXCehsUywWkhXV2x+SjddquhgpinPlgo4KQNFS+ar1C8q4M+pAKLY2giYCGpy9ckCC0yUqVO/EaZMtW8W4vCWpEmC+VwjHMp8Gsw0YhCpy0GABI0X6jJvWhGg7NoMseLaHwR+aY2E1TeRiKUmXo2xG3WotHATylK2ZmmcT0eG3EFFTcJIH9P8XA+C3Pq67w9GlZcHTlTz55OUU+LgixvpJn6ccsj18bkOcsfxMeWR1SAjqHVL/YxHL1KR1GpZfl//4cvq50YXJxy6ughaZRPrXJv7VNzkqYjDmTanfX9/PB7vDKvTXgiPRqP9g7wVWqy6w6sWXLZcZwdVi7peIJOUpfxf+NkQ7UMd0wswntDOJdpx4kg1N5OorHF1OFBGc39zNDalfuA74ESEDWrSjTlygynK5mDoDvgRb7JPu4SkV4kaQReWuWvzHAS3Js875LBqpHBUpTC9RmZtRJbtd8XY5rsCRzXQJd8K1ZltoGdYUMwJNLI1mx3yfcbnDaqaE3XEIX7ZFBApWCltL20Ri/L1t7f9h/91FfDvoHyeKvtUc5sEM5lXiYb6N7eJKFcQm8l/bfKHayJB1Yay3gx85NpdAJHm0QGVK4KZPnNtTjktGhaFNCbLrFVV8wyqlNhVLiuyjdEKazOS/gm12VibisUA+L9Ff5ImGYMVVWvnVtP4xEY9QQtbEP0wyFTe6hv3hND4pR1zNmErtnutzZwkEFeMFT/F9Pvm7WbzCuPfLJyNdoPhbvDr3uDxpHrP2fJ+9H7y7nuBt+899p56E+/UI17mvff+8v7uveipnu7lpentWxXzg9dYvXf/AJL+WCU=</latexit><latexit sha1_base64="6WifpBZdSRyY2pUiuBMF5KWI8MI=">AAAJzHicpZZfb9s2EMDV7l/s/Wm7Pe5FmBGgBdLAcuMleStWBGjRoPG2Ji1mGgFFn2SiFCWQdGqP0Ws/Ql+3536jfptRlJxIVANoGwHbJ9797s4n3klhxqhUw+HHW7c/+/yLL7/a6vW//ubb7+7cvff9mUyXgsApSVkqXodYAqMcThVVDF5nAnASMngVvnlS6F9dgJA05S/VOoNZgmNOI0qwMlsILbDSSC1A4fz87mC4ezgMDoM93wiPxsNxYISDw73Rz4Ef7A7tGnjVmpzf2/qA5ilZJsAVYVjKaTDM1ExjoShhkPfRUkKGyRscw9SIHCcgZ9omnfvbZmfuR6kwH658u1snNE6kXCehsUywWkhXV2x+SjddquhgpinPlgo4KQNFS+ar1C8q4M+pAKLY2giYCGpy9ckCC0yUqVO/EaZMtW8W4vCWpEmC+VwjHMp8Gsw0YhCpy0GABI0X6jJvWhGg7NoMseLaHwR+aY2E1TeRiKUmXo2xG3WotHATylK2ZmmcT0eG3EFFTcJIH9P8XA+C3Pq67w9GlZcHTlTz55OUU+LgixvpJn6ccsj18bkOcsfxMeWR1SAjqHVL/YxHL1KR1GpZfl//4cvq50YXJxy6ughaZRPrXJv7VNzkqYjDmTanfX9/PB7vDKvTXgiPRqP9g7wVWqy6w6sWXLZcZwdVi7peIJOUpfxf+NkQ7UMd0wswntDOJdpx4kg1N5OorHF1OFBGc39zNDalfuA74ESEDWrSjTlygynK5mDoDvgRb7JPu4SkV4kaQReWuWvzHAS3Js875LBqpHBUpTC9RmZtRJbtd8XY5rsCRzXQJd8K1ZltoGdYUMwJNLI1mx3yfcbnDaqaE3XEIX7ZFBApWCltL20Ri/L1t7f9h/91FfDvoHyeKvtUc5sEM5lXiYb6N7eJKFcQm8l/bfKHayJB1Yay3gx85NpdAJHm0QGVK4KZPnNtTjktGhaFNCbLrFVV8wyqlNhVLiuyjdEKazOS/gm12VibisUA+L9Ff5ImGYMVVWvnVtP4xEY9QQtbEP0wyFTe6hv3hND4pR1zNmErtnutzZwkEFeMFT/F9Pvm7WbzCuPfLJyNdoPhbvDr3uDxpHrP2fJ+9H7y7nuBt+899p56E+/UI17mvff+8v7uveipnu7lpentWxXzg9dYvXf/AJL+WCU=</latexit><latexit sha1_base64="6WifpBZdSRyY2pUiuBMF5KWI8MI=">AAAJzHicpZZfb9s2EMDV7l/s/Wm7Pe5FmBGgBdLAcuMleStWBGjRoPG2Ji1mGgFFn2SiFCWQdGqP0Ws/Ql+3536jfptRlJxIVANoGwHbJ9797s4n3klhxqhUw+HHW7c/+/yLL7/a6vW//ubb7+7cvff9mUyXgsApSVkqXodYAqMcThVVDF5nAnASMngVvnlS6F9dgJA05S/VOoNZgmNOI0qwMlsILbDSSC1A4fz87mC4ezgMDoM93wiPxsNxYISDw73Rz4Ef7A7tGnjVmpzf2/qA5ilZJsAVYVjKaTDM1ExjoShhkPfRUkKGyRscw9SIHCcgZ9omnfvbZmfuR6kwH658u1snNE6kXCehsUywWkhXV2x+SjddquhgpinPlgo4KQNFS+ar1C8q4M+pAKLY2giYCGpy9ckCC0yUqVO/EaZMtW8W4vCWpEmC+VwjHMp8Gsw0YhCpy0GABI0X6jJvWhGg7NoMseLaHwR+aY2E1TeRiKUmXo2xG3WotHATylK2ZmmcT0eG3EFFTcJIH9P8XA+C3Pq67w9GlZcHTlTz55OUU+LgixvpJn6ccsj18bkOcsfxMeWR1SAjqHVL/YxHL1KR1GpZfl//4cvq50YXJxy6ughaZRPrXJv7VNzkqYjDmTanfX9/PB7vDKvTXgiPRqP9g7wVWqy6w6sWXLZcZwdVi7peIJOUpfxf+NkQ7UMd0wswntDOJdpx4kg1N5OorHF1OFBGc39zNDalfuA74ESEDWrSjTlygynK5mDoDvgRb7JPu4SkV4kaQReWuWvzHAS3Js875LBqpHBUpTC9RmZtRJbtd8XY5rsCRzXQJd8K1ZltoGdYUMwJNLI1mx3yfcbnDaqaE3XEIX7ZFBApWCltL20Ri/L1t7f9h/91FfDvoHyeKvtUc5sEM5lXiYb6N7eJKFcQm8l/bfKHayJB1Yay3gx85NpdAJHm0QGVK4KZPnNtTjktGhaFNCbLrFVV8wyqlNhVLiuyjdEKazOS/gm12VibisUA+L9Ff5ImGYMVVWvnVtP4xEY9QQtbEP0wyFTe6hv3hND4pR1zNmErtnutzZwkEFeMFT/F9Pvm7WbzCuPfLJyNdoPhbvDr3uDxpHrP2fJ+9H7y7nuBt+899p56E+/UI17mvff+8v7uveipnu7lpentWxXzg9dYvXf/AJL+WCU=</latexit>
1951
1998Convolutional Networks
What is ML/AI Systems Research Today?
What is AI-Systems Research?
Ø Good AI and Systems researchØ Provides insights to both communities
Ø Leverages understanding of both domainsØ Examples:
Ø Studies tradeoff between statistical and computational efficiencyØ Identify essential abstractions in DNN designØ Leverages framing of indexes to exploit overfitting
Ø More than just great software!Ø Builds on Big Ideas in AI and Systems Research
Big Ideas in Systems Research
Ø Problem FramingØ Identifying the right problem and solution requirements
Ø Abstraction & Managing ComplexityØ Reducing complex problems into smaller parts
Ø TradeoffsØ Understanding fundamental constraints
Ø Details: System design and Implementation
Big Ideas in ML Research
Ø GeneralizationØ What is being “learned”?
Ø Inductive Biases and RepresentationsØ What assumptions about domain enable efficient learning?
Ø Efficiency (Data and Computation)Ø How much data and time are needed to learn?
Ø Details: Objectives/Models/Algorithms
Kinds of AI-Systems Research
AI + Systems
Advances in AI are being used to address fundamental challenges in systems.
AI + SystemsØ Reinforcement Learning for
Ø Pandas code generationØ SQL join planningØ Network packet
classificationØ Autoscaling
Ø Bandit Algorithms for radio link adaptation
Ø Wireless link quality estimation
Ø Multi-task learning for straggler mitigation
Ø VM Selection using Trees ..
AI + SystemsAdvances in systems are enabling substantial progress in AI
AI + SystemsDeveloping Systems for:Ø Autonomous Vehicles Ø Reinforcement LearningØ Secure Machine LearningØ Prediction ServingØ Experiment Management
Advancing AIØ Dynamic Neural NetsØ Prediction on
Compressed DataØ Distributed TrainingØ Distributed Auto-ML
AI + SystemsBerkeley is the place to be for
ResearchWhy?
riselabUC Berkeley
AI + SystemsBerkeley is the place to be for
Researchriselab
UC Berkeley
About You?
Background RequirementsØ You have taken previous courses in either
Ø Systems (for example CS162) à comfortable building systemsand familiar with big ideas in system design
Ø Machine Learning (for example CS189) à comfortable training neural networks and familiar with big ideas in ML
Ø You are actively involved in research in either systems or machine learning (or both)
If this does not describe you, please visit my office hourson Wednesdays from 4-5 in 773 Soda Hall.
Goals for the ClassWhat do you expect from this class?
My Goals for the Class
Ø Identify and solve impactful problems at the intersection of AI and Systems
Ø Learn about the big ideas and key results in AI systems
Ø Learn how to read, evaluate, and peer review papers
Ø Learn how to write papers that get high review scores
How will we achieve these goals?
Ø Lectures and Reading: study developments in AI SystemsØ Identify the key open problems and research opportunities
Ø Projects: collaboration between Systems and AI studentsØ Explore open problems and produce top tier publications
Ø In Class Reviews: weekly oral reviews of papersØ Learn both how to evaluate papers and understand how
papers are reviewed
Reasons not to take this class … LØ If you want to learn how to train models and
use TensorFlow, PyTorch, and SkLearnØ Take: CS C100, CS182, CS189
Ø If you want learn how to use big data systemsØ Take: CS162, CS186
Ø You are not interested in learning how to evaluate, conduct, and communicate researchØ Why not?
Ø You are unable to attend lecture
If you plan to drop this class, please drop it soon so others can
enroll!
ProblemsWhat makes a good problem?
What makes a good problem?
Ø Impact: People care about the solutionØ … and progress advances our understanding (research)
Ø Metrics: You know when you have succeededØ Can you measure progress on the solution?
Ø Divisible: The problem can be divided into smaller problemsØ You can identify the first sub-problem.
Ø Your Edge: Why is it a good problem for you?Ø Leverage your strengths and imagine a new path.
Can you Solve a Solved Problem?
Ø Ideally you want to solve a new and important problem
Ø A new solution to a solved problem can be impactful if:Ø It supports a broader set of applications (users)Ø It reveals a fundamental trade-off orØ Provides a deeper understanding of the problem spaceØ 10x Better?
Ø Often publishable…Ø Should satisfy one of the three above conditions.
Class Organization
Weekly Topic OrganizationØ Each week we will cover a new topic
Ø There will be 3 required papers to read each week
Ø Monday I will cover the background for the topic and an overview of the reading for the week
Ø Friday you will participate in an in class “Program Committee Meeting” to discuss the reading
• Big Ideas in ML/Sys• ML Life-cycle• ML in DBs• Model Dev. Frameworks• Distributed Training• Prediction Serving
• Autonomous Driving• Model Compilation• Hardware Acceleration• ML à Systems• Secure ML• Debugging and Interpretability
I may make changes to topics and
format based on class
participation.
In Class PC Meeting Format (V.0)
For each of the three papers: (30 Minutes per paper)
Ø Neutral: recap of the paper (neutral opinion) [5 Minutes]
Ø Advocate: Strengths of the paper [5 Minutes]
Ø Critic: Weaknesses of the paper [5 Minutes]
Ø Class will discuss rebuttal and improvements [10 Minutes]
Ø Brief in-class vote for acceptance into the AI-Sys prelim
The Neutral Presenter will Summarize
Ø What is the problem being solved?
Ø What was the solution? (Summary!)
Ø What metrics did they use to evaluate their solution?Ø What was the Baselines of comparison?
Ø What was the key insight or enabling idea?
Ø What are the claimed technical contributions?
Advocate and Critic Will DiscussØ Novelty and Impact
Ø Are the problem and solution novel and how will the solution affect future research?
Ø Technical QualitiesØ Are the problem framing and assumptions reasonableØ Discuss merits of the technical contributionsØ Does the evaluation support claims and reveal limitations of the
proposed approach?
Ø PresentationØ Discuss the writing clarity and presentation of resultsØ Positioning of related work
Simple Grading Policy
20% Class ParticipationØ Attend lecture and participate in class discussionsØ Lead 2 to 3 in class discussions
20% Weekly ReviewsØ Submit 3 weekly paper reviews (google form) per week!Ø Time consuming but important!
60% Class ProjectsØ Do great research!
Action Items
Ø Go to website: https://ucbrise.github.io/cs294-ai-sys-fa19/
Ø Make sure you are on the course PiazzaØ Needed for announcements
https://tinyurl.com/aisysfa19signup
Ø Signup for 3 discussion slots as different roles here:
