• Home

  • Documents

  • LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning...
52
X: The Moonshot Factory Copyright 2019 X Development LLC LEARNING TO GRASP USING SIMULATION Jun 23, 2019 Yunfei Bai [email protected] Collaboration between X (Formally Google[X]), Robotics at Google (Research), and DeepMind 1

LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

  • Upload
    others

  • View
    7

  • Download
    0

Embed Size (px)

Citation preview

Page 1: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

X: The Moonshot FactoryCopyright 2019 X Development LLC

LEARNING TO GRASP USING SIMULATION

Jun 23, 2019

Yunfei [email protected]

Collaboration between X (Formally Google[X]), Robotics at Google (Research), and DeepMind

1

Page 2: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

X: The Moonshot FactoryCopyright 2019 X Development LLC

● Learning Vision Based Grasping

○ Self-Supervised Learning

○ Deep Reinforcement Learning

● Improving Data Efficiency

○ With Sim-to-Real

○ With Sim-to-Sim

● Learning New Tasks

● Learning New Object Representation

Outline

2

Page 3: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

X: The Moonshot FactoryCopyright 2019 X Development LLC

—1Learning Vision Based Grasping with Self-Supervised Learning

3

“Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection”,

Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz, Deirdre Quillen

Page 4: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

X: The Moonshot FactoryCopyright 2019 X Development LLC

Goal: Learn to grasp arbitrary objects

● RGB monocular camera input● Camera positioned “over the shoulder”● Poor / non-existent camera calibration

Assumptions:● Overhead grasps

Reward function:● Gripper angle● Image subtraction

4

Page 5: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

X: The Moonshot FactoryCopyright 2019 X Development LLC 5

http://www.youtube.com/watch?v=iaF43Ze1oeI
Page 6: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

X: The Moonshot FactoryCopyright 2019 X Development LLC

1,100 objects used for training

6

Page 7: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

X: The Moonshot FactoryCopyright 2019 X Development LLC

● Inference using Cross-Entropy Method (CEM)● Replan every 300 to 500ms● Hand-eye coordination

7

Grasp Success Prediction Model

Grasp Prediction Loss

Real World Grasping Data

Initial Image Current Image Motor Command

Page 8: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

X: The Moonshot FactoryCopyright 2019 X Development LLC 8

http://www.youtube.com/watch?v=l8zKZLqkfII&t=35
Page 9: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

X: The Moonshot FactoryCopyright 2019 X Development LLC

Grasp Success Rates with Increasing Amounts of Data

9

Page 10: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

Copyright 2019 X Development LLC X: The Moonshot Factory

Two Directions for Improvement

● Break the ceiling of the grasp success rate to make it close to 100%.

● Improve real world data efficiency and reduce the data collection time.

10

Page 11: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

X: The Moonshot FactoryCopyright 2019 X Development LLC

—2Learning Vision Based Grasping with Deep Reinforcement Learning

11

“QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly,

Mrinal Kalakrishnan, Vincent Vanhoucke, Sergey Levine

Page 12: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

Copyright 2019 X Development LLC X: The Moonshot Factory

Q Learning for Grasping

● Supervised learning

○ Optimize for the next step

● Reinforcement learning

○ Predict a few steps ahead

12

Page 13: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

Copyright 2019 X Development LLC X: The Moonshot Factory

Replay Buffersoff-policy

on-policy

train

Training Worker

Offline data 580K grasps

Model weights

Bellman Updater

Cross Entropy Method

Replay buffers

13

Q Learning for Grasping: QT-OPT

Page 14: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

Copyright 2019 X Development LLC X: The Moonshot Factory

Q Network: Action Space Includes Open/Close Gripper and Termination

14

Grasping Value Function Q(s, a)

Real World Grasping Data

Initial Image Current Image Motor Command Gripper Open/Close

Episode Terminate

Page 15: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

Copyright 2019 X Development LLC X: The Moonshot Factory 15

https://docs.google.com/file/d/19SBJNndspsI2A7R4MwpEdedBHRZLbofT/preview
Page 16: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

Copyright 2019 X Development LLC X: The Moonshot Factory

QT-Opt achieves 96% grasp success, with higher data efficiency

16

less data,78% -> 96% grasp success

Page 17: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

X: The Moonshot FactoryCopyright 2019 X Development LLC

—3Leveraging Sim-to-Real for Data Efficiency

17

“Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping”,

Konstantinos Bousmalis, Alex Irpan, Paul Wohlhart, Yunfei Bai, Matthew Kelcey, Mrinal Kalakrishnan, Laura Downs,

Julian Ibarz, Peter Pastor, Kurt Konolige, Sergey Levine, Vincent Vanhoucke

Page 18: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

Copyright 2019 X Development LLC X: The Moonshot Factory

Motivation for Using Simulation

18

● 608,000 real-world grasps to achieve best performance○ 7 KUKA robots running for 2-3 months

● Use simulation!○ Easy to parallelize, reset, safe exploration, access

to ground-truth for exploration policy, …Real world Simulation

Page 19: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

Copyright 2019 X Development LLC X: The Moonshot Factory

Real world

Simulation

Dynamic model discrepancy

Uncertainenvironment

Erroneoussensing

Numericalerror

Latency

Others

Reality Gap

19

Page 20: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

X: The Moonshot FactoryCopyright 2019 X Development LLC

System Identification Use statistical methods to build mathematical models of dynamical systems from measured data.

Methods for Sim-to-Real Transfer

Domain Randomization

Feature-level Domain Adaptation

Pixel-level Domain Adaptation

Vary texture, background, lighting, color, object shape, and dynamics in simulation.

Train features to be domain-invariant yet expressive, by using an adversarial loss.

Train a generator network that converts simulated images to real images, by using an adversarial loss.

20

Page 21: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

Copyright 2019 X Development LLC X: The Moonshot Factory

realsim

reality gap

Sim-to-Real Transfer for Physics - System Identification

21

Page 22: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

Copyright 2019 X Development LLC X: The Moonshot Factory

sim real

Sim-to-Real Transfer for Physics - System Identification

22

Page 23: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

Copyright 2019 X Development LLC X: The Moonshot Factory

sim real

Sim-to-Real Transfer for Perception - Domain Randomization

23

Page 24: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

X: The Moonshot FactoryCopyright 2019 X Development LLC

● 51,300 object models from ShapeNet.

Chang et al., CoRR 2015

24

Page 25: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

X: The Moonshot FactoryCopyright 2019 X Development LLC

● 1,000 procedurally generated object models.

25

Page 26: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

X: The Moonshot FactoryCopyright 2019 X Development LLC

● Procedural objects with random textures.

26

Page 27: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

Copyright 2019 X Development LLC X: The Moonshot Factory

AdversarialLoss

Domain Classifier

Grasp Prediction Loss

Grasp Prediction Loss

Real World Grasping Data

Initial Image Current Image Motor Command

Simulated Grasping Data

Initial Image Current Image Motor Command

● Domain Adversarial Neural Network. Train features to be domain-invariant yet expressive, by using an adversarial loss. Learn features that confuse the domain classifier.

Based on Ganin et al., JMLR 2016

Sim-to-Real Transfer for Perception - Feature Level Domain Adaptation

27

Page 28: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

Copyright 2019 X Development LLC X: The Moonshot Factory

● Learn a generator (G) which converts synthetic images to real looking (fake) images.

Sim-to-Real Transfer for Perception - Pixel Level Domain Adaptation

28

Page 29: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

Copyright 2019 X Development LLC X: The Moonshot Factory

Sim-to-Real Transfer for Perception - Pixel Level Domain Adaptation

29

Page 30: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

X: The Moonshot FactoryCopyright 2019 X Development LLC

GraspGAN outperforms other techniques, providing more than 50x data efficiency.

30

Page 31: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

X: The Moonshot FactoryCopyright 2019 X Development LLC

GraspGAN outperforms other techniques, providing more than 50x data efficiency.

2% of the real world data,same performance

31

Page 32: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

X: The Moonshot FactoryCopyright 2019 X Development LLC

—4Solve Sim-to-Real via Sim-to-Sim

32

“Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks”,

Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz,

Sergey Levine, Raia Hadsell, Konstantinos Bousmalis

Page 33: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

Copyright 2019 X Development LLC X: The Moonshot Factory

Randomized-to-Canonical Adaptation Networks

33

● RCAN is a real-to-sim image translator trained with domain randomization:○ We define a “canonical” version of simulation and randomizations

canonical

canonical realrandomized

Page 34: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

Copyright 2019 X Development LLC X: The Moonshot Factory

Randomized-to-Canonical Adaptation Networks

34

● RCAN is a real-to-sim image translator trained with domain randomization:○ We define a “canonical” version of simulation and randomizations○ We train a pix2pix model to convert randomized sim images to equivalent

canonical versions

G

Randomized

D adapted/canonical

Adapted

Canonical

Page 35: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

Copyright 2019 X Development LLC X: The Moonshot Factory

Randomized-to-Canonical Adaptation Networks

35

● RCAN is a real-to-sim image translator trained with domain randomization:○ We define a “canonical” version of simulation and randomizations○ We train a pix2pix model to convert randomized sim images to equivalent

canonical versions○ In the real world, RCAN will then also be able to translate real images to

canonical sim versions

G

Real Adapted

Page 36: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

Copyright 2019 X Development LLC X: The Moonshot Factory

Randomized-to-Canonical Adaptation Networks

36

https://docs.google.com/file/d/15lhGa7L2Ft3m11-pwVPpUdBcfWGZqWSe/preview
Page 37: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

X: The Moonshot FactoryCopyright 2019 X Development LLC

RCAN achieves the similar success rate (94% vs 96%), but without using 580k real word data

37

No 580k off-policy data,similar grasp success

Page 38: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

X: The Moonshot FactoryCopyright 2019 X Development LLC

—5Learning New Tasks

38

“Multi-Task Domain Adaptation for Deep Learning of Instance Grasping from Simulation”,

Kuan Fang, Yunfei Bai, Stefan Hinterstoisser, Silvio Savarese, Mrinal Kalakrishnan

Page 39: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

X: The Moonshot FactoryCopyright 2019 X Development LLC

Sim-to-Real Transfer: Applying to a More Challenging Task

39

Page 40: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

X: The Moonshot FactoryCopyright 2019 X Development LLC

Instance Grasping Framework: Multi-Task Domain Adaptation

AdversarialLoss

Domain Classifier

Indiscriminate Grasp Prediction Loss

Indiscriminate Grasp Prediction Loss

Real World Indiscriminate Grasping Data

Initial Image Current Image Motor Command

Simulated Indiscriminate Grasping Data

Initial Image Current Image Motor Command

40

Page 41: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

X: The Moonshot FactoryCopyright 2019 X Development LLC

Instance Grasping Framework: Multi-Task Domain Adaptation

AdversarialLoss

Domain Classifier

Indiscriminate Grasp Prediction Loss

Indiscriminate Grasp Prediction Loss

Real World Indiscriminate Grasping Data

Initial Image Current Image Motor Command

Simulated Indiscriminate Grasping Data

Initial Image Current Image Motor Command

Instance Grasp Prediction Loss

Simulated Instance Grasping Data

Initial Image Current Image Target Mask Motor Command

41

Page 42: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

X: The Moonshot FactoryCopyright 2019 X Development LLC

Instance Grasping Framework: Multi-Task Domain Adaptation

AdversarialLoss

Domain Classifier

Indiscriminate Grasp Prediction Loss

Indiscriminate Grasp Prediction Loss

Real World Indiscriminate Grasping Data

Initial Image Current Image Constant Mask Motor Command

Simulated Indiscriminate Grasping Data

Initial Image Current Image Constant Mask Motor Command

Instance Grasp Prediction Loss

Simulated Instance Grasping Data

Initial Image Current Image Target Mask Motor Command

42

Page 43: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

Copyright 2019 X Development LLC X: The Moonshot Factory

Evaluation in the Real World

43

Page 44: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

X: The Moonshot FactoryCopyright 2019 X Development LLC

—6Learning New Object Representation

44

“Data-Efficient Learning for Sim-to-Real Robotic Grasping using Deep Point Cloud Prediction Networks”,

Xinchen Yan, Mohi Khansari, Jasmine Hsu, Yuanzheng Gong, Yunfei Bai, Soren Pirk, Honglak Lee

Page 45: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

Copyright 2019 X Development LLC X: The Moonshot Factory

Shape Prediction

visualizations of point clouds generated with our pointprediction network

45

Page 46: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

Copyright 2019 X Development LLC X: The Moonshot Factory

Learning to Grasp Using Object Point Cloud

visualizations of point clouds generated with our pointprediction network

Gripper Transform

Interaction Outcome

3D Point Cloud (camera frame)

3D Point Cloud (gripper frame)

Grasping Critic Network

46

Page 47: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

Copyright 2019 X Development LLC X: The Moonshot Factory

Grasping Evaluations

47

Page 48: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

X: The Moonshot FactoryCopyright 2019 X Development LLC

—7Conclusion

48

Page 49: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

X: The Moonshot FactoryCopyright 2019 X Development LLC

Summary

● Learn vision based grasping through self-supervised learning and deep reinforcement learning.

● Simulation helps reduce real world data requirements by 100x, by solving sim-to-real transfer and sim-to-sim transfer.

● Simulation also enables us to learn new related tasks, and good object representation can facilitate sim-to-real transfer.

49

Page 50: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

X: The Moonshot FactoryCopyright 2019 X Development LLC

1. Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data CollectionS. Levine et al. IJRR 2017.

2. QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic ManipulationD. Kalashnikov et al. CoRL 2018.

3. Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic GraspingK. Bousmalis et al. ICRA 2018.

4. Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation NetworksS. James et al. CVPR 2019.

5. Multi-Task Domain Adaptation for Deep Learning of Instance Grasping from SimulationK. Fang et al. ICRA 2018.

6. Data-Efficient Learning for Sim-to-Real Robotic Grasping using Deep Point Cloud Prediction NetworksUnder review

Reference

50

https://research.googleblog.com/2016/03/deep-learning-for-robots-learning-from.html
https://arxiv.org/abs/1806.10293
https://arxiv.org/abs/1709.07857
https://arxiv.org/abs/1812.07252
https://arxiv.org/abs/1710.06422
https://arxiv.org/abs/1906.08989
Page 51: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

X: The Moonshot FactoryCopyright 2019 X Development LLC

X

Yunfei Bai

Kuan Fang

Alexander Herzog

Stefan Hinterstoisser

Stephen James

Mrinal Kalakrishnan

Peter Pastor

Paul Wohlhart

Google Brain

Laura Downs

Ethan Holly

Julian Ibarz

Alex Irpan

Eric Jang

Dmitry Kalashnikov

Matthew Kelcey

Kurt Konolige

Alex Krizhevsky

Sergey Levine

Deirdre Quillen

Vincent Vanhoucke51

Credits

DeepMind

Konstantinos Bousmalis

Raia Hadsell

Page 52: LEARNING TO GRASP USING SIMULATION...Learning Vision Based Grasping with Deep Reinforcement Learning 11 “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”,

X: The Moonshot FactoryCopyright 2019 X Development LLC

—Thank you!

Yunfei [email protected]

52

http://www.youtube.com/watch?v=c57fh_a9k6M

Deep Reinforcement Learning for General Game Playing ...cs229.stanford.edu/proj2016/report/ArthursBirnbaum-Deep... · Deep Reinforcement Learning for General Game Playing ... approximates

Deep Reinforcement Learning for General Game Playing ...cs229.stanford.edu/proj2016/report/ArthursBirnbaum-Deep... · Deep Reinforcement Learning for General Game Playing ... approximates

Documents
Deep Reinforcement Learning An Introduction

Deep Reinforcement Learning An Introduction

Technology
Deep Reinforcement Learning

Deep Reinforcement Learning

Science
Deep Reinforcement Learning in a Handful of Trials …papers.nips.cc/paper/7725-deep-reinforcement-learning-in...Deep Reinforcement Learning in a Handful of Trials using Probabilistic

Deep Reinforcement Learning in a Handful of Trials …papers.nips.cc/paper/7725-deep-reinforcement-learning-in...Deep Reinforcement Learning in a Handful of Trials using Probabilistic

Documents
10703 Deep Reinforcement Learning and Controlrsalakhu/10703/Lecture_Exploration.pdf · 10703 Deep Reinforcement Learning and Control Russ Salakhutdinov Machine Learning Department

10703 Deep Reinforcement Learning and Controlrsalakhu/10703/Lecture_Exploration.pdf · 10703 Deep Reinforcement Learning and Control Russ Salakhutdinov Machine Learning Department

Documents
Towards Deep Symbolic Reinforcement Learning

Towards Deep Symbolic Reinforcement Learning

Documents
Deep Learning with Reinforcement Learning · about statistics, information theory, machine learning, reinforcement learning and finally deep learning. Finally, an honest and special

Deep Learning with Reinforcement Learning · about statistics, information theory, machine learning, reinforcement learning and finally deep learning. Finally, an honest and special

Documents
Deep Reinforcement Learning with Double Q-Learning

Deep Reinforcement Learning with Double Q-Learning

Documents
Deep Reinforcement Learning from Human Preferencespapers.nips.cc/...deep-reinforcement-learning-from-human-preferences.pdf · Deep Reinforcement Learning from Human Preferences Paul

Deep Reinforcement Learning from Human Preferencespapers.nips.cc/...deep-reinforcement-learning-from-human-preferences.pdf · Deep Reinforcement Learning from Human Preferences Paul

Documents
Tutorial: Deep Reinforcement Learning - Machine Learning ...hunch.net/~beygel/deep_rl_tutorial.pdfTutorial: Deep Reinforcement Learning - Machine Learning

Tutorial: Deep Reinforcement Learning - Machine Learning ...hunch.net/~beygel/deep_rl_tutorial.pdfTutorial: Deep Reinforcement Learning - Machine Learning

Documents
Deep Reinforcement Learning for Robotics

Deep Reinforcement Learning for Robotics

Documents
Deep Reinforcement Learning - Cross Entropy

Deep Reinforcement Learning - Cross Entropy

Documents
Deep Learning and Reinforcement Learning Workflows in A.I. · Deep Learning and Reinforcement Learning Workflows in A.I. Jon Cherrie Software Engineering Manager Deep Learning Toolbox

Deep Learning and Reinforcement Learning Workflows in A.I. · Deep Learning and Reinforcement Learning Workflows in A.I. Jon Cherrie Software Engineering Manager Deep Learning Toolbox

Documents
Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Documents
(Deep) Reinforcement Learning - Computer Vision Lab. …cvlab.postech.ac.kr/.../cse703r_2016s/Deep_Reinforcement_Learning… · (Deep) Reinforcement Learning Kee-Eung Kim School of

(Deep) Reinforcement Learning - Computer Vision Lab. …cvlab.postech.ac.kr/.../cse703r_2016s/Deep_Reinforcement_Learning… · (Deep) Reinforcement Learning Kee-Eung Kim School of

Documents
Reinforcement Learning - Deep Reinforcement Learningmmartin/URL/Lecture4.pdfReinforcement Learning Deep Reinforcement Learning Mario Martin CS-UPC May 17, 2018 ... Deep Q-Network algorithm

Reinforcement Learning - Deep Reinforcement Learningmmartin/URL/Lecture4.pdfReinforcement Learning Deep Reinforcement Learning Mario Martin CS-UPC May 17, 2018 ... Deep Q-Network algorithm

Documents
Deep Reinforcement Learning - Environments Tour€¦ · Deep Reinforcement Learning - Environments Tour Machine Learning Gdańsk, 02.10.2017 - Adam Wróbel Starcraft 2 DeepMind toolset

Deep Reinforcement Learning - Environments Tour€¦ · Deep Reinforcement Learning - Environments Tour Machine Learning Gdańsk, 02.10.2017 - Adam Wróbel Starcraft 2 DeepMind toolset

Documents
Deep Reinforcement Learning in System Optimization · arXiv:1908.01275v2 [cs.LG] 7 Aug 2019. Deep Reinforcement Learning in System Optimization Figure 1. Reinforcement learning. By

Deep Reinforcement Learning in System Optimization · arXiv:1908.01275v2 [cs.LG] 7 Aug 2019. Deep Reinforcement Learning in System Optimization Figure 1. Reinforcement learning. By

Documents
Deep Learning for Reinforcement Learning in · PDF fileDeep Learning for Reinforcement Learning in ... Deep Learning for Reinforcement Learning in Pacman Deep Learning für ... Während

Deep Learning for Reinforcement Learning in · PDF fileDeep Learning for Reinforcement Learning in ... Deep Learning for Reinforcement Learning in Pacman Deep Learning für ... Während

Documents
From Reinforcement Learning to Deep Reinforcement …fagostin/assets/files/...Keywords: Machine learning · Reinforcement learning Deep learning · Deep reinforcement learning 1 Introduction

From Reinforcement Learning to Deep Reinforcement …fagostin/assets/files/...Keywords: Machine learning · Reinforcement learning Deep learning · Deep reinforcement learning 1 Introduction

Documents
Deep Reinforcement Learning - KAISTalinlab.kaist.ac.kr/resource/AI602_Lec12_Deep_Reinforcement_Learni… · What is Reinforcement Learning? 2. Value-based Methods •Q-learning •Deep

Deep Reinforcement Learning - KAISTalinlab.kaist.ac.kr/resource/AI602_Lec12_Deep_Reinforcement_Learni… · What is Reinforcement Learning? 2. Value-based Methods •Q-learning •Deep

Documents
Hierarchical Deep Reinforcement Learning: Integrating ...papers.nips.cc/paper/6233-hierarchical-deep-reinforcement-learning... · work that integrates deep reinforcement learning

Hierarchical Deep Reinforcement Learning: Integrating ...papers.nips.cc/paper/6233-hierarchical-deep-reinforcement-learning... · work that integrates deep reinforcement learning

Documents
Reinforcement Learning: CNNs and Deep Q Learning

Reinforcement Learning: CNNs and Deep Q Learning

Documents
Reinforcement and deep reinforcement learning for wireless

Reinforcement and deep reinforcement learning for wireless

Documents
Human-level control through deep reinforcement … control through deep reinforcement learning ... We demon- stratethatthedeepQ ... Human-level control through deep reinforcement learning

Human-level control through deep reinforcement … control through deep reinforcement learning ... We demon- stratethatthedeepQ ... Human-level control through deep reinforcement learning

Documents
Tutorial: Deep Reinforcement Learning · Outline Introduction to Deep Learning Introduction to Reinforcement Learning Value-Based Deep RL Policy-Based Deep RL Model-Based Deep RL

Tutorial: Deep Reinforcement Learning · Outline Introduction to Deep Learning Introduction to Reinforcement Learning Value-Based Deep RL Policy-Based Deep RL Model-Based Deep RL

Documents
Towards Deeper Deep Reinforcement Learning

Towards Deeper Deep Reinforcement Learning

Documents
Exploration in Deep Reinforcement Learning · 2017-11-10 · Exploration in Deep Reinforcement Learning Exploration in Deep Reinforcement Learning Vorgelegte Bachelor-Thesis von Markus

Exploration in Deep Reinforcement Learning · 2017-11-10 · Exploration in Deep Reinforcement Learning Exploration in Deep Reinforcement Learning Vorgelegte Bachelor-Thesis von Markus

Documents
Deep Reinforcement Learning that Matters

Deep Reinforcement Learning that Matters

Documents
Deep Reinforcement Learning of Video Games...the eld of machine learning and reinforcement learning, in particular deep reinforcement learning. 1.2.1 Architectural neural network design

Deep Reinforcement Learning of Video Games...the eld of machine learning and reinforcement learning, in particular deep reinforcement learning. 1.2.1 Architectural neural network design

Documents