MindRACES, First Review Meeting, Lund, 11/01/2006
Fovea-Based Robot Control for Anticipation Studies in Various
Scenarios
Alexander Förster, Daan Wierstra, Jürgen Schmidhuber
IDSIA - Lugano - Switzerland
MindRACES, First Review Meeting, Lund, 11/01/2006 2
Overview
• Real world scenarios
• Fovea
• Software Framework
• Robertino Robot
• Simulated 3D scenarios
• Integration and Future Work
MindRACES, First Review Meeting, Lund, 11/01/2006 3
Instances of scenario 1: find an object
Still Image
Robot Lab
Office Room
Movie (LUCS)
2D E
nviro
nmen
ts3D
Environm
ents
MindRACES, First Review Meeting, Lund, 11/01/2006 4
Fovea Centralis Simulation
Three regions with increasing resolutions
MindRACES, First Review Meeting, Lund, 11/01/2006 5
Fovea Centralis Simulation
Three regions with increasing resolutions
Subsample each region to 13x11 pixels
MindRACES, First Review Meeting, Lund, 11/01/2006 6
Fovea Centralis Simulation
Three regions with increasing resolutions
Subsample each region to 13x11 pixels
Combine the subsampled image
MindRACES, First Review Meeting, Lund, 11/01/2006 7
?? ? ? ? ? ? ? ? ?? ?
Simulated Vision: 1-D Fovea Simulation
Task: Find the sad smiley
Bins Objects
1-D world
Transformation
Fovea
Memory
State 0
MindRACES, First Review Meeting, Lund, 11/01/2006 8
?? ? ? ? ? ? ? ? ?? ?
Simulated Vision: 1-D Fovea Simulation
Task: Find the sad smiley State 1
Bins Objects
1-D world
Transformation
Fovea
Memory
MindRACES, First Review Meeting, Lund, 11/01/2006 9
?? ? ? ? ? ? ? ? ?? ?
Simulated Vision: 1-D Fovea Simulation
Task: Find the sad smiley State 2
Bins Objects
1-D world
Transformation
Fovea
Memory
MindRACES, First Review Meeting, Lund, 11/01/2006 10
?? ? ? ? ? ? ? ? ?? ?
Simulated Vision: 1-D Fovea Simulation
Task: Find the sad smiley State 3
Bins Objects
1-D world
Transformation
Fovea
Memory
MindRACES, First Review Meeting, Lund, 11/01/2006 11
? ? ? ? ? ?
Simulated Vision: 1-D Fovea Simulation
Task: Find the sad smiley State 4
Bins Objects
1-D world
Transformation
Fovea
Memory
MindRACES, First Review Meeting, Lund, 11/01/2006 12
? ? ? ? ? ?
Simulated Vision: 1-D Fovea Simulation
Task: Find the sad smiley State 5
Bins Objects
1-D world
Transformation
Fovea
Memory
MindRACES, First Review Meeting, Lund, 11/01/2006 13
? ? ? ? ?
Simulated Vision: 1-D Fovea Simulation
Task: Find the sad smiley State 6
Bins Objects
1-D world
Transformation
Fovea
Memory
MindRACES, First Review Meeting, Lund, 11/01/2006 14
? ? ? ? ?
Simulated Vision: 1-D Fovea Simulation
Task: Find the sad smiley State 7
Bins Objects
1-D world
Fovea
Memory
Transformation
MindRACES, First Review Meeting, Lund, 11/01/2006 15
Simulated Vision: 1-D Fovea Simulation
Task: Find the sad smiley State 8
Bins Objects
1-D world
Fovea
Memory
Transformation
MindRACES, First Review Meeting, Lund, 11/01/2006 16
Simulated Vision: 1-D Fovea Simulation
Task: Find the sad smiley State 9
Bins Objects
1-D world
Fovea
Memory
Transformation
MindRACES, First Review Meeting, Lund, 11/01/2006 17
Simulated Vision: abstract 2-D Simulation
• Extended 1-D Simulation• Only the centered sensor can differentiate between
objects
MindRACES, First Review Meeting, Lund, 11/01/2006 18
Simulated Vision: 2-D Simulation
Original image with 2 objects
MindRACES, First Review Meeting, Lund, 11/01/2006 19
Simulated Vision: 2-D Simulation
Original image with 2 objects
Simulated fovea images
Center of the fovea
Step 1
MindRACES, First Review Meeting, Lund, 11/01/2006 20
Simulated Vision: 2-D Simulation
Original image with 2 objects
Simulated fovea images
Center of the fovea
Detected!!!
Step 1 Step n
MindRACES, First Review Meeting, Lund, 11/01/2006 21
Software Framework
Server
Robertino
•Debian/GNU Linux operating system•CAN bus interface •FireWire (IEEE 1394) interface•Video4Linux library•Fovea simulation
Client
Robosim
•Linux or Windows•Ogre framework•Fovea simulation•Simple physical and collision detection system
•Same interface as the real robot
Robomon
•Linux or Windows•Experiment management•Interface for learning algorithms•Full remote control of the robot/simulation
TCP/IP
Previously recorded data
MindRACES, First Review Meeting, Lund, 11/01/2006 24
Robertino – Client Software
Manual control of the robot
Learning control interface
Fovea image
MindRACES, First Review Meeting, Lund, 11/01/2006 25
Robertino - Overview
• Diameter: 40 cm• Height: 43 cm• Weight: 6.5 kg. • Holonomic three wheeled drive• PC-103 (industry standard)
with a 500MHz Intel Mobile-Pentium II processor on-board
• WLAN (IEEE 802.11a)• 2 cameras, used to simulate
the fovea• Actuators: the three wheels
and the simulated foveawww.openrobertino.org
MindRACES, First Review Meeting, Lund, 11/01/2006 26
Omnidirectional Camera
Web cam
Omnidirectional mirror
Image of an office environment, as seen by the robot
MindRACES, First Review Meeting, Lund, 11/01/2006 27
Image Transformation
• Used only for human convenience for navigation• Simple transformation algorithm
MindRACES, First Review Meeting, Lund, 11/01/2006 28
High Quality Camera
Imaging Source DFK21 AF04Image of an office environment, as seen by the robot
MindRACES, First Review Meeting, Lund, 11/01/2006 29
Robot Lab Environment
• Robot lab 550 cm x 280 cm• Robot can navigate through the world and move the fovea• Robot can be observed from a top view camera and local camera
MindRACES, First Review Meeting, Lund, 11/01/2006 30
Robot Lab Environment
Camera image Fovea data Composition
No video for his presentation!
MindRACES, First Review Meeting, Lund, 11/01/2006 31
3D Simulation of the Robot
• Environment design in 3D Studio MAX or G-MAX. • Simulation in Ogre3D• Same objects as in the real world• Shadows (optional)
MindRACES, First Review Meeting, Lund, 11/01/2006 32
Simulated vs. Real Vision (Fovea)
Simulation Real Robot
• Fovea transformation brings both world more together• Artificial noise simulates camera noise (optional)
MindRACES, First Review Meeting, Lund, 11/01/2006 33
Integration and Future Work
• Develop and evaluate anticipatory learning algorithms in the simulated environments (UW, ÖFAI); compare them also to non-anticipatory ones
• Share scenarios for attentive vision (LUCS, NBU)• Real robot with simulation as anticipation for surprise studies (CNR) • Simulation-based learning of anticipation (UW)• Share fovea simulation (IST)• Evaluate transfer of learned behavior to the real robot• Systematically increase the complexity of simulated and real
environments
• Possibly use a movable camera and zoom lens mounted on the robot