5th September 2003

Foresight Cognitive Systems Project

InterAction Conference

“The Connective Tissue”Richard Morris and Lionel Tarassenko

Cognitive Systems A working definition

“ Cognitive systems are natural or artificial information processing systems, including those responsible for perception, learning, reasoning, and decision-making, and both communication and action ”

Foresight Cognitive Systems IAC (LT and RGMM)

What this Foresight Project has been about...

It is a project that has been run by scientists and supported by Foresight who have provided the scope for events and activities designed by the communities themselves

Interaction and collaboration between the physical and life sciences communities has been one of the key strengths of the project

The main aim has been to provide a vision for the future of research in cognitive systems, both for the design of artificial cognitive systems (applications) and to further our scientific understanding of biological systems

The biologically-inspired approach focuses on the scientific study and, where appropriate, exploitation of naturally-occurring cognitive systems

The pure engineering approach seeks to create artificial systems that exhibit some desired level of cognitive performance or behaviour

Can we achieve more by combining the biologically inspired approach with the mathematical models of the engineering approach?

Overview

Speech & language (Grand Challenge) 3-D vision (Grand Challenge) How brains wire themselves (Keynote lecture) Brain rhythms (Grand Challenge) Memory & forgetting (Debate) Framework for the future?

Not covered here but some very important themes to emerge in discussion

Robotics Agents Learning Levels of analysis

Not covered here but some very important themes to emerge in discussion

Robotics Agents Learning Levels of analysis and emergent

properties Sensor fusion Affective cognition

To understand and emulate human capabilityfor robust communication and interaction.

To understand and emulate human capabilityfor robust communication and interaction.

Grand Challenge Language and speech

To construct a neuro-biologically realistic, computationally specific account of human language processing.

To construct functionally accurate models of human interaction based on and consistent with real-world data.

To build and demonstrate human-computer interfaces which demonstrate human levels of robustness and flexibility.

Goals:

10Grand Challenge

0

5

10

15

20

25

30

35

40

SpeakerIndependentDictationBroadcastNews

TelephoneConversations

11

Progress in Automatic Speech Recognition

State of the Art: Speech Recognition

Easy

Hard

Wor

d E

rror

Rat

e

What Humans Do that Today’s Systems Don’t

Use context to interpret and respond to questions Ask for clarification Relate new information to what’s already been said Avoid repetition Use linguistic and prosodic cues to convey meaning

Distinguish what’s new or interesting Signal misunderstanding, lack of agreement, rejection

Adapt to their conversational partners Manage the conversational turn Learn from experience

12State of the Art: Computational Language Systems

700 ms720 ms740 ms750 ms760 ms770 ms780 ms790 ms800 ms

13State of the Art: Cognitive Neuroscience of Speech and Language

Demonstration using MEG to track cortical activity related to spoken word recognition

Demonstration using MEG to track cortical activity related to spoken word recognition

1. Greater scientific understanding of human cognition and communication

2. Significant advances in noise-robust speech recognition, understanding, and generation technology

3. Dialogue systems capable of adapting to their users and learning on-line

4. Improved treatment and rehabilitation of disorders in language function; novel language prostheses

Summary of Benefits

To understand and emulate human capabilityfor robust communication and interaction.

To understand and emulate human capabilityfor robust communication and interaction.

14Grand Challenge

Grand Challenge - 3D Vision

Measurements

3D Structure using computational geometry

A 3D Visual Task

Biological systems can perform 3D tasks using visual information

Pointing to remembered objects Ants homing

Real-time 3D localisation in natural scenes using the engineering approach will not be possible simply as a result of increase in computational power

“computer vision hitting a computational wall” The design of a biologically plausible model of 3D

localisation is a Grand Challenge which would take us through the wall

Overview

Speech & language (Grand Challenge) 3-D vision (Grand Challenge) How brains wire themselves (Keynote lecture)

How brains wire themselves

(Keynote lecture – Mriganka Sur)

Specificity

Plasticity

Optic tract

Lateralgeniculate

n.Optic

radiations

Primaryvisualcortex

The visual cortex has specific processing networks

Orientation selectivity in V1: How do orientation networks form?

D.H. Hubel T.N. Wiesel

Optical imaging of cortical activity

Cortical vasculature

Single orientation images Composite orientation map

Light guide

Stimuluscomputer

Video dataacquisition

Ca

me

ra

Amplifier

Normal

Rewired

Visually responsive auditory cortex

MGN

LGN

Visual cortex

Auditory cortex

Visual cortex

SuperiorcolliculusLGN

Inferior colliculi

MGN

Rewiring alters the pattern of activity to the developing cortex

M. Sur and C. Leamey, Nature Reviews Neurosci, 2001

J. Sharma, A. Angelucci, M. Sur, Nature, 2000

Orientation maps arise in rewired A1

How brains wire themselves

Specificity

Plasticity

Some aspects of

arise by virtue of

Implications : should engineering systems incorporate self-organisation and, if so, how?

Brain rhythms (Grand Challenge)

How are representations of perceptual events given the correct temporal organisation for storage and recall?

The rhythmic activities of different groups of neurons in the brain may play a fundamental role in helping us to do this.

Computer scientists working in the area of asynchronous

computing are interested in any insight into how complex asynchronous natural systems can deliver coherent behaviours

Hypothesis is that each period of the fast gamma rhythm underlies a specific representation.

Gamma is superimposed on a slower rhythm (alpha or theta) that effectively multiplexes the representations

This mechanism could explain how the interactions between neuronal rhythms participate in shaping the holding in short-term (working) memory of perceptual events: the fast wave representations would constitute the contents of each discrete snapshot, the entire percept being mediated by the slow waves.

Gamma-theta interaction

From VanRullen & Koch, 2003

Memory and Forgetting (Debate)

• New “intelligent” information processing software should take more account of advances in our growing understanding of human memory systems.

But should they compensate for, or mimic their known failings, including forgetting?

Synaptic potentiation shows varying persistence

Memory also fails for psychological reasons….

The “seven sins” of memory The sin of transience The sin of absent-

mindedness The sin of blocking The sin of misattribution The sin of suggestibility

The sin of bias The sin of persistence

Weakening or loss Breakdown of attention Thwarted memory

search Assigning to wrong

source Implanted by a leading

question Editing and rewriting Repeated recall of

disturbing informationAfter Schacter (2001)After Schacter (2001)

The “seven sins” of memory (Dan Schacter)

• Schacter argues that they are not really “failures” at all, but reflect the proper operation of a finely tuned system…. not vices but virtues.

• Should we build into artificial devices for storing and retrieving information the same “trade-offs” between memory and forgetting that we see in human systems?

Overview

Speech & language (Grand Challenge) 3-D vision (Grand Challenge) How brains wire themselves (Keynote lecture) Brain rhythms (Grand Challenge) Memory & forgetting (Debate) Framework for the future?

Framework for the Future

In the study of “cognitive systems”, can the interaction between life and physical sciences be better promoted by setting up an appropriate infrastructure?

Critical mass of scientists

Technology and instrumentation

Dedicated research programs

Relevant training programs

If yes….

Momentum generated by this Foresight Project must be maintained

The following might be fruitful areas of interaction: 3D-Vision for natural scenes Speech and language – natural dialogue Memory systems Action and robotics Social cognition

Other topics also offer important and tractable problems.

Possible research themes (not an exhaustive list)

How do humans recognise objects? How could the new paradigm of generative models help us

to understand sensory processing in mammalian brains? The use of context in both artificial and natural systems Can appreciation of the social context of speech get us

beyond the apparent limits of machine-learning? How does the brain encode and remember time? How are we to understand intentionality in action? What are the principles and functional consequences of

self-organisation?