Chapter 1. Cognitive Systems Introductionin Cognitive Systems, Christensen et al.

Course: Robots Learning from Humans

Park, Sae-Rom

Lee, Woo-Jin

Statistical Learning & Computational Finance Laboratory

Industrial Engineering

Seoul National University

http://slcf.snu.ac.kr

2

Contents Introduction

Objective of Project

Motivating example Organization of the Research/Research Questions

Architecture Representations Learning Perception-Action Modelling Continuous Planning and Acting Models of Action and Communication Multi-Modal Recognition and Categrization Scenario Driven Research

Organization of the Book

CoSy project

The assumption of the visionary FP6 “To construct physically instantiated … systems that can

perceive, understand … and interact with their environ-ment, and evolve in order to achieve human-like perfor-mance in activities requiring context- (situation and task) specific knowledge”

Requirements Architectures, forms of representation, perceptual mech-

anisms, learning, planning, reasoning, motivation, action, and communication

To validate science progress using test scenarios

4

Contents Introduction

Objective of Project

Motivating example Organization of the Research/Research Questions

Architecture Representations Learning Perception-Action Modelling Continuous Planning and Acting Models of Action and Communication Multi-Modal Recognition and Categrization Scenario Driven Research

Organization of the Book

Objective of Project

Problem Most systems able to perform complex tasks that hu-

mans and other animals can perform easily, for instance robot manipulators, or intelligent advisers, have to be carefully crafted

The way to forward Combining many different capabilities in a coherent

manner -> 4-5 year child Generic capabilities

Steps to Success

Achievable sub-goals Theory deliverables Implementation deliverables

Theory deliverables The notion of an architecture combining components

Reactive Deliberative Self-reflective, meta management

Different learning processes Different varieties of communication and social interac-

tion

Steps to Success

Implementation Deliverables

nature

nurture

vs

Linguistic

Visual

ReasoningPlanning

Motor skills

Integration

8

Contents Introduction

Objective of Project

Motivating example Organization of the Research/Research Questions

Architecture Representations Learning Perception-Action Modelling Continuous Planning and Acting Models of Action and Communication Multi-Modal Recognition and Categrization Scenario Driven Research

Organization of the Book

Motivating Example

10

Contents Introduction

Objective of Project

Motivating example Organization of the Research/Research Questions

Architecture Representations Learning Perception-Action Modelling Continuous Planning and Acting Models of Action and Communication Multi-Modal Recognition and Categrization Scenario Driven Research

Organization of the Book

Organization of the Research Research challenges

Two scenarios for study of integrated systems Two major milestones

Using intermodality and affordances for the acquisition of concepts, categories and language

Introspection of models & representations; planning for autonomy – goal seeking

architectures

Represen-tations

learningPerception-

action model-ing

Communi-cation

Planning & failure han-

dling

Architecture

Putting pieces together into a complex functional system Perception, action, reasoning and communicating

Representation

The representation should Enable integration of representations of objects, scenes,

actions, events, causal relations and affordances Allow incremental updating or sometimes correction Allow different types of learning (supervised, unsuper-

vised, reinforcement) Allow integration of various modalities, of very different

input signals Be suitable for recognition and categorization in the

presence of a cluttered background and variable illumi-nation

Be scalable

Representation

Representation

Specific vs General Representations

Learning

Modes of learning

Tutor Driven A user (tutor) shows to the system an object or an ac-

tion and explains to the cognitive system what he/she is showing or doing

Tutor Supervised A cognitive system detects a new object, an action,

event, affordance or a scene by itself and builds its representation in an unsupervised manner.

Exploratory Updates the representation autonomously

Learning Example

Continuous Learning

Representations employed allow the learning to be a continuous, open-ended, life-long process Continuously updated over time, adapting to the change

in environment, new tasks, user reactions, user prefer-ences, …

Reliable continuous learning Representations have to be carefully chosen How new data is extracted and prepared

Perception-Action Modelling

Abstract relation model General, non-task specific Observability of the world hand-constructed abstraction

Probability relational representation Capture uncertainty in both action and observation Tractable for localization and path planning in continuous

space Sensor-dependence

Reinforcement learning Identifying features that are relevant to predicting the outcome

on the task

20

Continuous Planning

Difficulties Dynamic nature, partial observability

Conditional planning Probabilistic planning

21

Continuous Planning

Active Failure Diagnosis In most approaches it is typically assumed that the sensors and ac-

tuators of the robot are reliable in the sense that their input always corresponds to the expected input and that there is no malfunction of the sensors or actuators

These approaches do not exploit the actuators of the ro-bot to identify potential faults

Once a fault has been identified, the high-level system is notified so that appropriate actions can be generated at the planning level.

Continuous Planning

Collaborative Planning and Acting

Cooperation is at the heart of the Cosy project

Common language protocol Dialogue

Continuous Planning

Models of Action and Communication for Em-bodied Cognitive Agents

Natural Language

Integration of communication and action Recognition of intention, attention, and grounding/under-

standing Mixed-initiative Embodiment in an unknown environment

Models of Action and Communication for Em-bodied Cognitive Agents

Multi-Modal Recognition and Categorization

Recognize Categorize Entry level categorization vs Recognition

Recognition of objects Categorization Multi-cue

Scenario Driven Research

System level

Exploration/Mapping of Space Models of objects and concepts

Exploration / Mapping of space

Where am I? How do I get to my destination? How do I detect that I have arrived at the destination?

Perception and action Localization in the World Construction of a map of the environment Plan a sequence of actions

Affordances and Newer Approaches Space Object

Robustness

Wall, Door, Table

The World as an Outside Memory

Mapping of the Environment

Encoding of position of objects/places Encoding of environmental topology Invariant to changes to perception system Invariant to changes in action system Facilitate spatial reasoning

Models for Object and Concepts

Representation Continuous Learning Robustness Categorization Architecture Communication and Language

33

Contents Introduction

Objective of Project

Motivating example Organization of the Research/Research Questions

Architecture Representations Learning Perception-Action Modelling Continuous Planning and Acting Models of Action and Communication Multi-Modal Recognition and Categrization Scenario Driven Research

Organization of the Book

Organization

Chapter 2 Architecture design, representation Chapter 3 perception - action Chapter 4 spatial maps Chapter 5 visual perception Chapter 6 planning recovery Chapter 7 adaptation & learning Chapter 8 Human-robot interaction Chapter 9 & 10 Demonstration

Thank you for Listening