Art & Robotics: Toward Robot artists - Through Learning-from-observation - Katsushi Ikeuchi...
Preview:
Citation preview
- Slide 1
- Art & Robotics: Toward Robot artists - Through
Learning-from-observation - Katsushi Ikeuchi University of
Tokyo
- Slide 2
- Robot dancer Can we make a robot dancer through Programming-by-
Demonstration?
- Slide 3
- Dancing robot Through observing human dance Can a robot
dance
- Slide 4
- Dancing robot: Learning-from-observation Observing dance
Representing dance Demonstrating representation Humanoid robot
- Slide 5
- Observation
- Slide 6
- One of eight sequences
- Slide 7
- Background subtraction Video imageBackground Human area -
- Slide 8
- Obtained 3D Sequence
- Slide 9
- Stick image unfortunately still, unstable Motion capture
system
- Slide 10
- Observation: Motion capture system Joint angles obtained
Theoretically, a robot can imitate the same dance???
- Slide 11
- But, . AIST dynamic simulator
- Slide 12
- Worse with steps!!!
- Slide 13
- Learning from observation Observation Performance
Relation-1Relation-2 Action Representation Not direct imitation
Top-down approach
- Slide 14
- Three issues Representation What does the dancer perform? How
is the dancer performing? Demonstration How does a robot perform
using his/her body?
- Slide 15
- History: learning from observation 1988
- Slide 16
- Learning from observation Top-down approach Ikeuchi, Reddy,
Tanguy 89
- Slide 17
- Object Recognition
- Slide 18
- Task Recognition
- Slide 19
- Relation transition = Task AB BA A B B A Ikeuchi, Reddy, Tanguy
89 Put A on top B Put A side of B
- Slide 20
- Later system 1988 1990 1995 2002
- Slide 21
- Possible contact relations in polyhedral world From Kuhn Tucker
Theory
- Slide 22
- Relation transition = Tasks
- Slide 23
- Task and skill parameters object start configuration object
approach configuration object approach direction gripper start
configuration gripper approach configuration gripper approach
direction 3d-s3d-a Move-to-contact Skill parameters
- Slide 24
- Observation
- Slide 25
- Real-time stereo hardware
- Slide 26
- Observation in CAD world
- Slide 27
- Task Recognition based on contact transition Make-contact in
translation Slide in translation
(20010000)(11010010)(20010020)(11010010)
(02010020)(20010021)(10100111)(01100121) Make-contact in
translation Make-contact in translation Slide in rotation
- Slide 28
- Demonstration Takamatsu, Kimura, Ikeuchi 2002
- Slide 29
- How about other domains? Task models in contact operation
Polyhedral objects Mechanical parts Flexible objects (Rope) Task
models in non-contact operation Hand motion Whole body with
dynamics
- Slide 30
- Task models primitive1primitive2 movement StateS1S2S3
- Slide 31
- Mathematical background How to describe a state of a knot What
kind of motion primitives to be used? 1. State: P-data 2. Motion
primitives: Reidemeister moves+ Cross Knot theory
- Slide 32
- P-data 1 2 3 4 5 6 4 5 6 1 2 3 over 1 under 2 over 3 under 4
value 3 1 2 4 2 1 1 2 3 4 5 6 sign vertical OOUUUO
- Slide 33
- Necessary & Sufficient From a line drawing of a rope, we
can obtain a unique P-data representation. [Inverse]: From a P-data
representation, we can reconstruct a line drawing of a rope.
- Slide 34
- Three Types of Reidemeister moves Reidemeister move
Reidemeister move Reidemeister move
- Slide 35
- Sufficiency to cover all possible moves Two equivalent knots
convert to each other by a finite number of Reidemeister moves
Proof provided by Reidemeister [Reidemeister 32]
- Slide 36
- Observation Observation Convert to P-Data Transition of P-Data
-> Reidemeister move
- Slide 37
- Task models One P-data transition One Reidemeister move
- Slide 38
- Transition of P-Data Rep 1 2 3 4 5 6 4 5 6 1 2 3 3 1 2 4 2 1 1
2 3 4 5 6 4 5 6 1 2 3 3 1 2 4 2 1 1 2 5 6 7 8 6 7 8 1 2 5 3 1 2 4 2
1 34 4 3 Reidemeister Move I
- Slide 39
- Demonstration Takamatsu, Morita, Ikeuchi 2006
- Slide 40
- Dancing robot 06 1988 1990 1995 2002 2006
- Slide 41
- Dancing robot Through observing human dance Can a robot
dance
- Slide 42
- Task model design Foot supporting upper body Upper body
representing a dance
- Slide 43
- Foot Task models: what a human does? Left step Right step
standing Foot contact Squat Waist position
- Slide 44
- Recognizer Step contact states speed of foot Squat speed of
waist
- Slide 45
- Motion-capture dataResults:what a human does Recognition
results
- Slide 46
- Task: What a dancer does Skill: How a dancer is doing? Standing
Squat Step Period Foot width depth Foot width Highest point What
How From motion-capture data
- Slide 47
- Skill reconstruction Skill prototype Skill parameters from
observation New trajectory
- Slide 48
- Start point End point Generated trajectory Foot Width Highest
points Skill parameters observed
- Slide 49
- Foot trajectory Whole leg motion
- Slide 50
- Upper body tasks: Teacher s sketch on how to dance What is
this? How can we extract?
- Slide 51
- Key pose extraction Assumption brief stops of body parts z x y
Body centered coordinates time Vel. Brief stops
- Slide 52
- Key poses extracted (based on only motion) Segmentation based
on the assumption Over-segmentation New assumption rhythm + brief
stop
- Slide 53
- Rhythm analysis Estimated Beat Interval 0.704 [sec] Music with
inserted Beep = 84
- Slide 54
- Key poses extracted (rhythm and motion) R. Hand L. Foot R. Foot
L. Hand
- Slide 55
- Comparison Teachers key-poses from her sketch Extracted
key-poses from motion capture data
- Slide 56
- Interpolation Hierarchical B-spline Key point Teachers
motion
- Slide 57
- Adjustment of whole body ZMP = Zero Moment Point
- Slide 58
- ZMP control Calculate current ZMP Compare with desired ZMP
Adjust waist position to reduce the difference
- Slide 59
- Costarring with the dance teacher Nakaoka 2006, Shiratori 2007
With cooperation of AIST, Kawata
- Slide 60
- Beyond current dancing robot able to imitate dance motions
Need: Listening capability Self-dancing
- Slide 61
- Synthesis of New Dance from Music Motion reservoir: motion
segments with code and intensity Analyze current music: music code
and intensity Search motion segments to match music code and
intensity Connect segments seamlessly for generating a new
dance
- Slide 62
- Generated new dance (Shiratori 07) Input Music: Kansho Motion
reservoir: Six Japanese folk dances
- Slide 63
- Beyond current dancing robot able to imitate dance motions
Need: Listening capability Self-dancing Adjusting to music
Tempo
- Slide 64
- Motion difference due to tempi Faster tempo: detailed motions
omitted Green Original Tempo Yellow 1.3 faster (Synchronized)
- Slide 65
- Hierarchical B-spline Input t B-splin t B-spline t Half knot
intervals t Difference
- Slide 66
- Remove higher layers within speed limit constraint 1 2 3 4 w
limit
- Slide 67
- Generated 1.5 faster motion Human Generated Simple fast
forward
- Slide 68
- Beyond current dance robot able to mimic dance motions Need:
listening capability Need: ego-desire of a robot dance to perform
to improve cannot observe its own dance painting robot
- Slide 69
- Toward a robot painter System design Self-judgment on painting
results beautiful satisfy ugly dissatisfy
- Slide 70
- Paining robot Relation-1Relation-2 Action Representation
Observation Painting
- Slide 71
- Structure of painting Representation for painting
Representation for painting Model acquisition Painting by a
robot
- Slide 72
- Observation 3D model of an apple
- Slide 73
- Why 3D model? Represent as we do (assumption) Obtain painting
features from the representation Arbitrary view Superimposed views
Abstract view
- Slide 74
- How to paint? Acquired model Representation Painting by a
robot
- Slide 75
- Arbitrary view
- Slide 76
- Contour features from an apple model 11 segments
- Slide 77
- How to paint? Acquired model Representation Painting by a robot
Do we need a robot body for painting?
- Slide 78
- Painting with a brush Grasp a brush Paint a line with the brush
Paint a line with the brush Verify painting results Verify painting
results Re-painting
- Slide 79
- Grasping a brush
- Slide 80
- Verifying the results Green OK Purple need re-painting
- Slide 81
- Re-painting through visual feedback
- Slide 82
- Painting an apple
- Slide 83
- Paintings by the robot Due to three- finger grasping
- Slide 84
- Future Plan: hierarchical painting
- Slide 85
- Future plan: motion representation Marcel Duchamp
- Slide 86
- Remaining Issues Self-judgment on painting results beautiful
satisfy ugly dissatisfy Robot s ego-desire to paint Mind and
desire
- Slide 87
- Science in dancing and painting Learning from observation Task
recognition what he/she does Skill recognition how he/she does Body
recognition: what and how to do Robot artist through
learning-from-observation embodying artist mind ???
- Slide 88
- Art and Science Middle ages: University = Art + Science 20 th
century Divorce between Art and Science 21 st century Let s
remarriage between art and science through Robot Artist based
Learning-from-observation paradigm
- Slide 89
- Misc Info Creation of 21 st Century Digital Art Under JST-CREST
program Web: http://www.cvl.iis.u-tokyo.ac.jp cvl: Computer Vision
Lab iis: Institute of Industrial Science u-tokyo: The University of
Tokyo