Institute of Systems and Robotics

University of Coimbra

Institute of Systems and Robitcs

http://paloma.isr.uc.pt

Institute of Systems and Robotics

TRIDENT2 - Meeting, Dec. 28th

Teleconf. Meeting, Dec. 28th Presentation by Prof. Jorge Dias


Institute of Systems and Robotics, ISR-UC

•The Institute of Systems and Robotics, has activities in the areas of:

• robotics vision,• autonomous systems,• multi-sensor fusion and integration,• tele-operation,• sensor development,• soft-control and motors and drives.


Institute of Systems and Robotics, ISR-UC

PARTNER PRESENTATION:

•The Institute of Systems and Robotics (ISR-UC, www.isr.uc.pt) is an associated university research institution of the University of Coimbra. ISR-UC promotes advanced multidisciplinary R&D in the areas Robotic Manipulation, Medical Robotics, Assistive Systems, Autonomous Mobile Robotics, Intelligent Transportation Systems, Computer Vision, Biomedical Engineering, Automation, Control Theory, Operations Management, Sustainable Energy Systems, and cooperates with local neurosciences groups.

• ISR- UC is involved in over 25 national projects and over a dozen international projects. More recently in BACS - Bayesian Approach to Cognitive Systems (FP6-IST-027140), PROMETHEUS - Prediction and interpretation of human behaviour based on probabilistic structures and heterogeneous sensors (FP7 - 214901) and HANDLE - Developmental pathway towards autonomy and dexterity in robot in-hand manipulation (FP7-2008– 231640), both applying Bayesian learning and inference techniques, resulting in 2 completed PhD thesis and 3 under work.

• ISR-UC received the outstanding grade "Excellent" as a result of the last R&D Portuguese Unit Evaluation, being the only Electrical and Computer Engineering (among a total of 25 units) to receive that distinction.


Mobile Robotics LabSome of the researches topics in the Lab (Most of them dealing with uncertainty using probabilistic approaches):

Multi-sensor fusion for 3D reconstruction; Bayesian Multi-perception Mobile Robot Sensing Human Machine Interfaces Human Motion Modeling Human and Robotics Dexterous Manipulation Human Behavior Understanding Multi-sensor 3D Data Registration Health Care Visual Navigation and Tracking for Human Machine Interactionhttp://paloma.isr.uc.pt

Some past and ongoing projects which our group have been collaborating with human data extraction, learning, Bayesian inference, 3D reconstruction, etc.: more projects at http://paloma.isr.uc.pt

Project Contribution description

HANDLE(Developmental Pathway towards Autonomy and Dexterity in Robot In-Hand Manipulation)

-Modeling and learning of the dexterous manipulation strategies performed by Humans in order to endow robotic platforms with equivalent skills.

PROMETHEUS(Prediction and interpretation of human

behaviour based on probabilistic structures and heterogeneous sensors)

-Modeling and learning of potentially dangerous Human behaviors in high sensitive environments based on multimodal sensor data.


Our Group within MRLab for TRIDENT2

Dr. Jorge DiasHead of the Lab

Dr. Jorge Lobo

Jorge Manuel Miranda Dias born on March 7, 1960, in Coimbra, Portugal and has a Ph.D. degree on Electrical Engineering at University of Coimbra, specialisation in Control and Instrumentation, November 1994. Jorge Dias conducts his research activities at the Institute of Systems and Robotics (ISR-Instituto de Sistemas e Robótica) at University of Coimbra. Jorge Dias’ research area is Computer Vision and Robotics, with activities and contributions on the field since 1984. He has several publications on Scientific Reports, Conferences, Journals and Book Chapters. Jorge Dias teaches several engineering courses at the Electrical Engineering and Computer Science Department, Faculty of Science and Technology, University of Coimbra. He is responsible for courses on Computer Vision, Robotics, Industrial Automation, Microprocessors and Digital Systems. He is also responsible for the supervision of Master and Ph.D. students on the field of Computer Vision and Robotics.

Jorge Lobo (Jorge Nuno de Almeida e Sousa Almada Lobo) was born on the 23rd of September 1971, in Cambridge, UK. In 1995, he completed his five year course in Electrical Engineering at Coimbra University. In April 2002, he received the M.Sc degree, and in June 2007 he received the Ph.D degree from the University of Coimbra. He was a junior teacher in the Computer Science Department of the Coimbra Polytechnic School, and later joined the Electrical and Computer Engineering Department of the Faculty of Science and Technology at the University of Coimbra, where he currently works as Assistant Professor. He is responsible for courses on Digital Design, Microprocessors and Computer Architecture. His current research is carried out at the Institute of Systems and Robotics, University of Coimbra, working in the field of computer vision, sensor fusion, and mobile robotics. Current research interests focus on inertial sensor data integration in computer vision systems, Bayesian models for multimodal perception of 3D structure and motion, and real-time performance using GPUs and reconfigurable hardware. He has participated in several national and European projects, most recently in BACS, Bayesian Approach to Cognitive Systems, and HANDLE.


Dr. Paulo Menezes

Dr. Rui Rocha

Rui Rocha was born on 13 May 1973, in Castelo de Paiva, north of Portugal. He completed his Electrical and Computer Engineering degree (specialization on Automation, Control and Instrumentation) on July 1996, his M.Sc. degree (specialization on Industrial Informatics) on March 1999, and his Ph.D. degree on May 2006, all by the Faculty of Engineering of the University of Porto. Between February 2000 and May 2006, he was a Teaching Assistant at the Department of Electrical and Computer Engineering, in the Faculty of Sciences and Technology of the University of Coimbra. Currently he is an Assistant Professor at the Department of Electrical and Computer Engineering and a researcher at the Institute of Systems and Robotics, in the Faculty of Sciences and Technology University of Coimbra. His main research topics are cooperative multi-robot systems, 3-D map building, distributed architectures and Intelligent Transportation Systems.

Paulo Menezes is an Assistant Professor at the Department of Electrical and Computer Engineering of the Faculty of Sciences and Technology of the University of Coimbra, Portugal. He received his Ph.D. degree from the University of Coimbra for his dissertation on "Multi-Cue Visual Tracking for Human-Robot Interaction." He also holds M.S. and B.S. degrees from the University of Coimbra in Electrical Engineering.He is a senior researcher of the Institute of Systems and Robotics and belongs to the Mobile Robotics Laboratory team. His research interests are: robotics, computer vision, human-robot interaction, augmented reality and new technologies for health care and life quality support. He is involved in several European and National funded projects on these fields.



Diego Faria

Ricardo Martins

Diego Resende Faria was born on Aug.17th , 1979 in Londrina (state of Parana), Brazil. He is a Ph.D. student at the University of Coimbra, Portugal. He is Researcher at the Institute of Systems and Robotics - Department of Electrical and Computer Engineering - University of Coimbra. He is under the supervision of Prof. Jorge Dias (advisor) and Prof. Jorge Lobo (co-advisor). He is sponsored by a Ph.D. scholarship from the Portuguese Foundation for Technology and Sciences. He has graduated in Information Systems Technology in 2000 and has finished a Computer Science Specialisation Course in 2002 at the State University of Londrina, Brazil. He holds an M.Sc. degree in Computer Science from the Federal University of Parana, Brazil, since 2005. Currently, Diego Faria is collaborating as researcher on the European Project HANDLE within the 7º framework FP7. His research interest is Robotic Grasping, Multimodal Perception, Imitation Learning, Computer Vision and Pattern Recognition.

Ricardo Filipe Alves Martins was born on the 15th of October 1984 in Proença-a-Nova, Portugal. Ricardo has an M.Sc. degree in Biomedical Engineering from the University of Coimbra, Portugal obtained in 2008. Currently, he is a Ph.D. student and researcher at the Institute of Systems and Robotics, Department of Electrical Engineering and Computers, University of Coimbra, Portugal. He is sponsored by a Ph.D. scholarship from the Portuguese Foundation for Technology and Sciences. He is collaborating as researcher on the European Project HANDLE within the 7° framework FP7. His research interests are Robotic Grasping/Haptics, Multimodal Perception and Imitation Learning.



TRIDENT Project overview

-The ASC/I-AUV team gathers navigation data for geo-referencing the measurements (seafloor images and multibeam bathymetry profiles).

-Finally, the I-AUV surfaces (3) and contacts to the end user to set-up and acoustic/optical map of the surveyed area.

-Using this map, the en user selects a target object (an object of interest) as well as a suitable intervention task (grasping, hooking, etc...).

PHASE I (Survey)

-The Autonomous Surface Craft (ASC) is launched to carry the Intervention Autonomous Underwater Vehicle (I-AUV) towards the area to be surveyed.

-Then, the I-AUV is deployed (1) and both vehicles start a coordinated survey path (2) to explore the area.

- ISR-UC potential contribution on this topic


TRIDENT Project overview

-After selecting the target, the ASC/I-AUV team navigates towards the target position. Then, the ASC performs dynamic position (4) while keeping the I-AUV inside the USBL cone of coverage.

PHASE II (Intervention)

-Then, the I-AUV performs a search (5) looking for the Target of Interest (ToI). When the object appears in the robot field of view, it is identified and the I-AUV switches to free floating mode using its robotic arm as well as the dexterous hand to do the smart manipulation (6).

-Finally (7), the I-AUV docks to the ASC before recovery.

- ISR-UC potential contribution on this topic


Timeline -->future

Can be used as input for Learning Models

(TRIDENT-2)


Scientific Objectives (SO) & Technical Objectives (TO)

Learning by Imitation and Decision (SO)

Technical Objectives (TO) Description

TO1 –Environment exploration

(Phase I) Algorithms to estimate the exploration strategy parameters for the sensed environment context. Estimation based on a previously learned association between different environment contexts and exploration strategy parameters.

TO2 –Trajectory learning

(Phase II) Algorithms for probabilistic generalization of trajectories from training sets (simulator and/or tele-operation)

TO3 -Trajectory library

(Phase II) Build a library of trajectories that act as memory of previous successful tested movements. The sensorial context for specific tasks, represented probabilistically, will be recorded.

Description




Technical Objectives (TO) Description

TO4 –Trajectory generation

(Phase II) For a specific question/task to develop algorithms to generate a set of hypothesis for trajectories. These will be generated by indexing algorithms based on sensor and context analysis.

TO5 –Decision and Short-Path Mission

(Phase II) The interface layer will remain in the mission control modules in the experimental TRIDENT set-up. This module should provide the best short-path decision and its probabilistic confidence.

TO6 –Cooperative manipulation - object assembly strategies

(Phase II) Algorithms to estimate at symbolic level the dual-arm cooperative strategies required assemble a perceived multi-element object configuration.

Description




Technical Objectives (TO)

Scenarios

Known Context(Learned sessions by

simulation)

Unknown Context(Autonomous Operation by

Inference)

TO1 –Environment exploration

Input: Perceived environment context

-Selection of the cooperative exploration strategy parameters trained for that environment context

Input: Perceived environment context

-Inference: Description of the perceived environment context as a combination of the trainned environment contextsAdaptation of the cooperative exploration parameters to the new context.

Scenarios





Scenarios


simulation)


Inference)

TO2 –Trajectory learning

-Fully known object 6D pose relative to the UAV and environmental conditions.-Selection of the sub-set of trajectories demonstrated in the simulator correspondent to the object state and environmental conditions.-Application of the trajectory generalization algorithms to the subset of trajectories.

-If the trajectory selected in T05 (decision and short-path mission) provides a successful outcome, the UAV trajectory as well as the object pose, shape and contact characteristics are stored in the T03 (trajectory library).

Scenarios





Scenarios


simulation)


Inference)

TO3 –Trajectory Library

-Fully known object 6D pose relative to the UAV and environmental conditions.-If the trajectory selected in T05 (decision and short-path mission) provides a successful outcome, the trajectory learned through the simulator is considered validated.

-If the trajectory selected in T05 (decision and short-path mission) provides a successful outcome, the UAV trajectory as well as the object pose, shape and contact characteristics are stored in the T03 (trajectory library).

Scenarios





Scenarios


simulation)


Inference)

TO4 –Trajectory generation

-Fully known object 6D pose relative to the UAV.

-Selection from TO3 (trajectory library) the sub-set of robotic UAV arm trajectory demonstrated in the simulator for the mission.

-Inputs from object identification: Target object pose and shape primitives) and the stable grasp configuration to successfully interact with the object.Other environmental and internal perceived status variables.

-Inference: Generation of a set of hypothesis for robotic UAV arm trajectories.

Scenarios





Scenarios


simulation)


Inference)

TO5 –Decision and short-path mission

-Fully known object 6D pose relative to the UAV and environmental conditions.

-Selection of the robotic UAV arm trajectory demonstrated in the simulator for the mission objectives, environmental context and mission constraints.

-Inputs from the action TO4: set of hypothesis for trajectories;-Other external configured mission constrains: e.g: time to execution, minimum confidence level threshold, etc.

-Inference: Provide the best (most probable) robotic UAV arm trajectory for the mission objectives and pre-defined minimum confidence level (cost functions)

Scenarios





Scenarios


simulation)


Inference)

TO6 –Cooperative manipulation - object assembly strategies

-Input: Perceived multi-element object configuration

-Selection of the dual-arm multi-element object assemble strategy described by a sequence of primitives

-Input: Perceived multi-element object configuration

-Inference: Estimation of the dual-arm element object assemble strategy, based on primitive dictionary grammar rules defined for that type of task/object.

Scenarios

University of Coimbra 2/29

Motivation Both vehicles are deployed in the mission environment; Collaborative exploration strategy to map the unknown environment;

Approach Adapt the collaborative trained exploration strategies parameters to unknown environment contexts;

Simulation environment – Learning phase Environment context A –>Demo: AUV1 & AUV2 exploration strategy parameters A Environment context B–>Demo: AUV1 & AUV2 exploration strategy parameters B

Real world experimental environment– Inference phase Environment context analysis

-Description of the perceived environment context as a probabilistic combination of previously known/learned environment contexts.

Estimation of the ponderated exploration strategy parameters of the new unknown environmental context.

TRIDENT2: Example of ApplicationPhase I - TO1


TRIDENT2: Example of Application

Flow chart that encloses two cases:-the first one is a known context where the object (target) is known a priori

- Imitation of previous knowledge (trajectories) taking into consideration the object pose (to generate hypothesis of possible grasps that influences also the trajectories;

-the second, the object is not known (during exploration, rocks/stones, tubes, object that was not used in the learning phase)

- *in this case a generalization process by similarities is used. The object is segmented in such way that is possible to generate known shapes (primitives) that composes the object, so that the grasp planning can be applied to that shape, including the trajectories.

* Example of similarities: Given a unknown object (not learned before), then its shape is decomposed in primitives (superquadrics) to approximate the shape into known geometric shapes to be possible to generate a specific grasp for some of the segmented parts:

Known geometrical shapes and its pose can generates known (learned) grasps and trajectories

Phase II - T02, T03, T04, T05

University of Coimbra 2/29

Motivation Both vehicles working together and under coordination with the two robotic arms to assemble multi-elements objects;

Approach Cooperative manipulation movements and assembly strategy described at a symbolic kevel

Simulation environment – Learning phase Multi-elements object configuration A Object assembly demo 1, 2 ... NExtraction of the finite set of primitives – task dictionary

-Characterization of each primitive using lower level signals (trajectories, control signals, …)

Extraction of a generalized task structure (general primitive sequence)-Extraction of some of the primitive sequence rules (required precedence, …) - grammar rules of the task primitives dictionary

Real world experimental environment– Inference phase Multi-element object configuration recognition Primitive sequence generation

TRIDENT2: Example of ApplicationPhase III – TO6


TRIDENT2: GeneralizationMethods adopted in previous works ISR-UC

Objectives Techniques Ref.

TO2, TO3, TO4, T05

Interpolation via Splines (using the extracted motion pattern found in similar trajectories)

Polynomial regression applied locally in a set of similar trajectories

Symbolic representation? (our previous work on tactile sensing can be extended for trajectories) finding signatures and then by probabilistic classification is achieved the generalization

D. Faria, R. Martins, J. Lobo, J. Dias - Extracting Data from Human Manipulation of Objects Towards Improving Autonomous Robotic Grasping- Robotics and Autonomous Systems, Elsevier (Accepted manuscript, In Press): Special Issue on Autonomous Grasping, 2011.

R. Martins, D. R. Faria, J. Dias - Symbolic Level Generalization of In-hand Manipulation Tasks from Human Demonstrations using Tactile Data Information - to appear in IEEE/RSJ IROS'2010: Workshop on Grasping Planning and Task Learning by Imitation


TRIDENT2: Generalization

Possible alternative methods that can be studied for TRIDENT2

Objectives Techniques Examples of some related work that can serve as support for this studies

TO2, TO3, TO4, T05

Clustering of Trajectories (GMM, or HMM for segmentation and temporal clustering of features data) later applying regression models on the clustering (e.g. splines)

Calinon, S., Guenter, F. and Billard, A. (2007). On Learning, Representing and Generalizing a Task in a Humanoid Robot. IEEE Transactions on Systems, Man and Cybernetics, Part B, Special issue on robot learning by observation, demonstration and imitation, 37:2, 286-298.

Andrew Irish, Iraj Mantegh, and Farrokh Janabi-Sharifi. A PbD approach for learning pseudo-periodic robot trajectories over curved surfaces. IEEE/ASME International Conference on Advanced Intelligent Mechatronics, 2010


Open Issues for Discussion

2/29

What are the signals provided by the other project modules that can be used for learning in T01, T02, T03 ? :

Combined trajectories of the Arm and UAV ?Combined trajectories/pose of object, the gripper, the Arm and UAV?Controls signals and trajectories accomplished in the demonstrations ?

What are the object detection features provided by the environment mapping module?

Target object tracking - 2D and 3D cases? Object Detection (template matching, shape approximation) – Known and unknown objects? Automatic Segmentation (floor, 3D object map) ? Intervention for 3D case (already tested?)


Institute of Systems and Robitcs

http://paloma.isr.uc.pt

Institute of Systems and Robotics

TRIDENT2 - Meeting, Dec. 28th

Thank You!END.

Documents

Institute of Systems and Robotics