Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
1© 2019 The MathWorks, Inc.
A Mosaic of Models as the Driver for
Domain-Specific Modeling in the Robotics Industry
Advanced Research & Technology Office
MathWorks
Anastasia Mavrommati, PhD
stacymav.github.io
2
Robotics and DSL in Research Conferences
A. Nordmann, N. Hochgeschwender, D. Wigand and S. Wrede, “A Survey on Domain-Specific Modeling Languages in Robotics”, Journal of Software Engineering for Robotics (JOSER), 2016
Robotics
Software
Hybrid
3
What about Robotics in the manufacturing industry?
4
Where are we and how did we get
here?
A mosaic of models for Robot System
Design
Robot System Design for pick-and-place applications
Summary
Outline
5
Source: Kotsanas Museum of Ancient
Greek Technology, Athens, Greece
www.kotsanas.com
The Automatic Servant of Philon
~300 BC
6
UNIMATE robot (serving liquor)
1954 AD
G. DevolJ. Engelberger
~300 BC
Source: https://robots.ieee.org/robots/unimate/
7
When did the need for programming languages in Robotics appear?
1954
First industrial robotFirst robot programming
language in industry
1979
First robot programming
language in academia
VALWAVE
1973
Teach the robot by guiding it through motion
• Physical guide through
• Using the teaching pendant
Source: “Introduction to Robotics: Mechanics and Control”, John J. Craig, 1985
PUMA Teaching Pendant
~20 years
8
First desideratum: Move robot!
Move to location PART.
Then, move to location BOX.
ROBOT USERS
9
First solution: Textual programming languages
Variable Assembly Language
1979
ROBOT USERS
10
First solution: Textual programming languages
Teaching Pendant Language
1979 2019
>> L P[1:PART] 100%
>> J P[2:BOX] 50%
40 years!
✓ Programmed on the robot
computer
✓ Robot operation stopped
✓ Similar semantics
(assembler-like)
✓ More GPL-like syntax
ROBOT USERS
11
Target groups for Robotics applications
ROBOT USERS
ROBOT BUILDERS
SOFTWARE
DEVELOPERS
12
Target groups for Robotics applications
ROBOT USERS
ROBOT BUILDERS
ROBOTICS
RESEARCHERS
SOFTWARE
DEVELOPERS
13
Target groups for Robotics applications
ROBOT USERS
ROBOT
BUILDERS
SOFTWARE
DEVELOPERS
ROBOTICS
RESEARCHERS
14
New desideratum: Move robot at the office!
ROBOT USERSMove to location PART. Then,
move to location BOX.
SOFTWARE
DEVELOPERS
15
Solution: Off-line programming languages
✓ No access to robot required
✓ Similar to General Purpose Languages with abstractions for Robotics
L P[1:PART] 100%
J P[2:BOX] 50%
Assignment
Declaration
1979 2019 2019
TP VAL KAREL
ROBOT USERS
SOFTWARE
DEVELOPERS
16
Solution: Off-line programming languages
ROBOT COMPANY PENDANT TEACHING OFF-LINE
FANUC TP KAREL
KUKA KUKA Robot Language (KRL) KRL
ABB RAPID RAPID
Universal Robots URScript URScript
Staubli VAL3 VAL3
YASKAWA INFORM MotoCom C++ Library
DENSO PacScript PacScript
More than 30 different robot programming languages
ROBOT USERS
SOFTWARE
DEVELOPERS
17
All languages are specific to robot manufacturers.
“A Model-Driven Approach to Support Engineering Changes in Industrial Robotics Software”
Yu Sun, Jeff Gray, Karlheinz Bulheller, Nicolaus von Baillou
18
Robotics developers need non-robot specific tools
…circa 2007
Not a modeling language but an enabling platform for domain-specific applications
Publisher-subscriber communication middleware
Source: https://robohub.org/ros-101-intro-to-the-robot-operating-system/
SOFTWARE
DEVELOPERS
19
Development of ROS
…circa 2007
From languages specific to robots, to languages specific to Robotics.
…circa 2012
SOFTWARE
DEVELOPERS
20
Where are we and how did we get
here?
A mosaic of models for Robot System Design
Outline
21
Actuation
Dynamics
Motor Control
Sensing
Kinematics
Component Modeling
Rigid Body Modeling
Motion Planning
Motion Modeling
World Modeling
Collision modeling Vision modeling
Perception Modeling
Camera Depth LiDAR
Intelligence Modeling
Deep Learning Reinforcement Learning
Task Modeling
Task Planning Task Scheduling Task Description
Layers of technologies for Robotic System Design
Trajectory Optimization
Physics
One modeling language for all?
Human-Machine Interface Modeling
Speech Recognition Gesture Recognition Graphical User Interface
22
Physics
Actuation
Dynamics
Motor Control
Sensing
Kinematics
Component Modeling
Rigid Body Modeling
Motion Planning Motion Modeling
World ModelingCollision modeling Vision modeling
Perception ModelingCamera Depth LiDAR
Intelligence ModelingDeep Learning Reinforcement Learning
Task ModelingTask Planning Task Scheduling Task Description
Layers of technologies for Robotic System Design
Trajectory Optimization
Or a pandaisia* of languages?
Speech Recognition Gesture Recognition Graphical User Interface Human-Machine Interface Modeling
* Pronounced p-aa-n-dh-eh-s-EE-aa, loosely translated as “feast”
23
Layers of language abstractions for Robotic System Design
Robot interface DSL KINOVA robots Users
Robotics System Toolbox DSL Robotics Developers
MATLAB DSL Math/Engineering
Researchers
Builders
C/C++ GPL
Example technology: Simulate robot motion
Source: www.kinovarobotics.com
24
Robot interface DSL KINOVA robots Users
Robotics System Toolbox DSL Robotics Developers
MATLAB DSL Math/Engineering
Researchers
Builders
C/C++ GPL
Languages applied to Robotics
Languages adapted to Robotics
Languages inherent to robots
Layers of language abstractions for Robotic System Design
25
Example: Modeling robot kinematics
“Model the kinematic structure of a robot arm with 1 rigid link and 1 rotational joint”
26
Example: Languages for robot kinematics
GAZEBO / ROS Robotics System Toolbox
✓ Textual
✓ Declarative✓ Textual
✓ Imperative
27
More languages for robot kinematics
Robotics System Toolbox
✓ Graphical
✓ Declarative
GAZEBO / ROS
Simscape Multibody
28
More languages for robot kinematics
Unreal Engine
✓ Graphical
✓ Declarative
GAZEBO / ROS
Use third party 3D computer graphics software*
Robotics System Toolbox
Simscape Multibody
* Source: Christopher Yonge – YouTube ChannelSource: https://developer.nvidia.com/isaac-sdk
29
Where are we and how did we get
here?
A mosaic of models for Robot System
Design
Robot System Design for pick-and-place applications
Outline
30
Pick and place workflow
31
Pick and place workflowspeed x12
What formalisms do we need to implement this workflow?
32
Task ModelingRigid Body Modeling
Rigid Body & Motion Modeling
HMI Modeling
Environment Modeling
Perception & Intelligence
Modeling
A pandaisia of modeling formalisms!
33
Task ModelingRigid Body Modeling
Rigid Body & Motion Modeling
HMI Modeling
Environment Modeling
Perception & Intelligence
Modeling
A pandaisia of modeling formalisms!
❑ A single developer must be able to switch among modeling
formalisms with intuitive semantics
❑ There must be mechanisms in place to integrate all
modeling formalisms into one system
What are some formalisms that enable these requirements?
34
Task ModelingRigid Body Modeling
A pandaisia of modeling formalisms!
35
Task Modeling
Object detection
Initialize
Pick
Place
Home position
Use Stateflow as high-level task-planner
The task modeling language:
✓ Allows the developer to visually match their
hand-drawn flowchart to syntactic states
36
Pick
Place
Task Modeling
Use Stateflow as high-level task-planner
The task modeling language:
✓ Allows the developer to visually match their
hand-drawn flowchart to syntactic states
✓ renders the task complexity manageable
37
Rigid Body Modeling
Environment Modeling
A pandaisia of modeling formalisms!
38
Environment Modeling
GAZEBO is a simulation platform
essential for vision and contact modeling
Source: gym-gazebo
Simulink is a simulation platform
essential for systems modeling
Co-simulation allows
systems integration for
Robotics with combination of
most intuitive languages.
Simulation results are
✓ Synchronous
✓ Reproducible
39
Environment modeling: Co-simulation necessities
Composite ports• width
• height
• data
• data_type
Data structure transformation• From GAZEBO to Simulink
Multi-formalism semantics• Integrate with MATLAB scripts for data processing
40
Rigid Body Modeling
A pandaisia of modeling formalisms!
Perception & Intelligence
Modeling
41
Intelligence Modeling: System Object
AI algorithms can be embedded to execution systems to be used in the real world.
Inherits from MATLAB System Object
Constructor definition
Step implementation
42
Reinforcement Learning Example: Co-simulation and System Object
Reinforcement Learning Agent as System Object• Inputs
• Observations
• Rewards
• Output
• Action
GAZEBO Co-simulation
43
Where are we and how did we get
here?
A mosaic of models for Robot System
Design
Robot System Design for pick-and-place applications
Summary
Outline
44
Robotics models span a large range of modeling formalisms.
From languages specific to robots, to languages specific to Robotics.
“Robotics is a science of integration rather
than a fundamental science, and the
integration becomes ever more complex.”Herman Bruyninckx
Bruyninckx, H. , “Robotics Software: The Future Should Be Open,” IEEE Robotics & Automation Magazine, vol. 15(1), pp. 9–11, 2008.
45
46
47
Can I order a “Robotics-specific modeling language”?
Pick…
1. …. a target group
Robot Users
Software
Developers
Robotics
Researchers
Robot
Builders
2. …. a technology
Actuation
Dynamics
Sensing
Kinematics
Motion Planning
Collision modeling Vision modeling
Depth Camera
Deep Learning Reinforcement Learning
Task Scheduling Task Description
Trajectory Optimization
3. …. an abstraction
Languages applied to Robotics
Languages adapted to Robotics
Languages inherent to robots
Speech Recognition Gesture Recognition
4. …. a target
ROS node
Embedded MCU
48
A mosaic of models and languages
Actuation
Dynamics
Motor Control
Sensing
Kinematics
Motion Planning
Collision modeling Vision modeling
Camera Depth LiDAR
Deep Learning Reinforcement Learning
Task Planning Task Scheduling Task Description
Trajectory Optimization
Robot Users
Software
Developers
Robotics
Researchers
Robot Builders
Languages inherent to robots
Languages applied to RoboticsLanguages adapted to Robotics
Speech Recognition Gesture Recognition Graphical User Interface
ROS nodeEmbedded MCU
49
A mosaic of models and languages!
Actuation
Dynamics
Motor Control
Sensing
Kinematics
Motion Planning
Collision modeling Vision modeling
Camera Depth LiDAR
Deep Learning Reinforcement Learning
Task Planning Task Scheduling Task Description
Trajectory Optimization
Robot Users
Software
Developers
Robotics
Researchers
Robot Builders
Languages inherent to robots
Languages applied to RoboticsLanguages adapted to Robotics
Speech Recognition Gesture Recognition Graphical User Interface
ROS nodeEmbedded MCU
50
Thank you!