EE141

How to Motivate How to Motivate Machines to Learn and Machines to Learn and Help Humans in Making Help Humans in Making Water Decisions?Water Decisions? Janusz StarzykSchool of Electrical Engineering and Computer Science, Ohio University, USA

www.ent.ohiou.edu/~starzyk

EE141

Embodied Intelligence (EI) Embodiment of Mind EI Interaction with Environment How to Motivate a Machine Goal Creation Hierarchy Goal Creation Experiment Promises of EI

To economy To society

OutlineOutline

EE141

“…Perhaps the last frontier of science – its ultimate challenge- is to understand the biological basis of consciousness and the mental process by which we perceive, act, learn and remember..” from Principles of Neural Science by E. R. Kandel et al. E. R. Kandel won Nobel Price in 2000 for his work on physiological

basis of memory storage in neurons. “…The question of intelligence is the last great

terrestrial frontier of science...” from Jeff Hawkins On Intelligence. Jeff Hawkins founded the Redwood Neuroscience Institute devoted

to brain research

IntelligenceIntelligence

AI’s holy grailFrom Pattie Maes MIT Media Lab

EE141

Traditional AITraditional AI Embodied Intelligence Embodied Intelligence Abstract intelligence

attempt to simulate “highest” human faculties:

– language, discursive reason, mathematics, abstract problem solving

Environment model Condition for problem

solving in abstract way “brain in a vat”

Embodiment knowledge is implicit in the

fact that we have a body– embodiment is a

foundation for brain development

Intelligence develops through interaction with environment Situated in a specific

environment Environment is its best

model

EE141

Design principles of intelligent systemsDesign principles of intelligent systemsfrom Rolf Pfeifer “Understanding of Intelligence”, 1999

Interaction with complex environment

cheap design ecological balance redundancy principle parallel, loosely

coupled processes asynchronous sensory-motor

coordination value principle

Agent

Drawing by Ciarán O’Leary- Dublin Institute of Technology

EE141

Embodied Intelligence Embodied Intelligence

Definition Embodied Intelligence (EI) is a mechanism that learns

how to survive in a hostile environment

– Mechanism: biological, mechanical or virtual agent

with embodied sensors and actuators– EI acts on environment and perceives its actions– Environment hostility is persistent and stimulates EI to act– Hostility: direct aggression, pain, scarce resources, etc– EI learns so it must have associative self-organizing memory– Knowledge is acquired by EI

EE141

Embodiment

Actuators

Sensors

Intelligence core

channel

channel

Embodiment

Sensors

Intelligence core

Environment

channel

channelActuators

Embodiment

Actuators

Sensors

Intelligence core

channel

channel

Embodiment

Sensors

Intelligence core

Environment

channel

channelActuators

Embodiment of a MindEmbodiment of a Mind

Embodiment contains intelligence core and sensory motor interfaces under its control to interact with environment

Necessary for development of intelligence

Not necessarily constant or in the form of a physical body

Boundary transforms modifying brain’s self-determination

EE141

Brain learns own body’s dynamic Self-awareness is a result of

identification with own embodiment Embodiment can be extended by

using tools and machines Successful operation is a function

of correct perception of environment and own embodiment

Embodiment of a MindEmbodiment of a Mind

EE141

INPUT OUTPUT

Simulation or

Real-World System

TaskEnvironment

Agent Architecture

Long-term Memory

Short-term Memory

Reason

ActPerceive

RETRIEVAL LEARNING

EI Interaction with EnvironmentEI Interaction with Environment

From Randolph M. Jones, P : www.soartech.com

EE141

How to Motivate a MachineHow to Motivate a Machine ? ?

The fundamental question is how to motivate a machine to do anything, in particular to increase its “brain” complexity?

How to motivate it to explore the environment and learn how to effectively work in this environment?

Can a machine that only implements externally given goals be intelligent?If not how these goals can be created?

EE141

I suggest that hostility of environment motivates us. It is the pain that moves us. Our intelligence that tries to minimize this pain motivates our actions,

learning and development

We need both the environment hostility and the mechanism that learns how to reduce inflicted by the environment pain

How to Motivate a MachineHow to Motivate a Machine ? ?

I propose based on the pain mechanism that motivates the machine to act, learn and develop.

So the pain is good.Without the pain there will be no intelligence. Without the pain there will be no motivation to develop.

EE141

Pain-center and Goal CreationPain-center and Goal Creation

Simple Mechanism Creates hierarchy of

values Leads to formulation of

complex goals Reinforcement :

• Pain increase• Pain decrease

Forces exploration

+

-

Environment

Sensor

MotorPain level

Dual pain levelPain increase

Pain decrease

(-)

(+)

Excitation

(-)

(-)

(+)

(+)

Wall-E’s goal is to keep his plants from dying

EE141

Primitive Goal CreationPrimitive Goal Creation

- +

Pain

Dry soilPrimitive

level

opentank

sit on garbage

refillfaucet

w. can water

Dual pain

EE141

Abstract Goal CreationAbstract Goal Creation The goal is to reduce the primitive pain level Abstract goals are created to reduce abstract pains in order to satisfy the primitive goals Abstract pain center

- +

PainDual pain

+

Dry soil

Abstract pain

“water can” – sensory input

to abstract pain center

Sensory pathway(perception, sense)

Motor pathway(action, reaction)

Primitive Level

Level I

Level IIfaucet

-

w. can

open

water

ActivationStimulationInhibitionReinforcementEchoNeedExpectation

EE141

Abstract Goal HierarchyAbstract Goal Hierarchy

A hierarchy of abstract goals is created - they satisfy the lower level goals

ActivationStimulationInhibitionReinforcementEchoNeedExpectation

- +

+

Dry soilPrimitive Level

Level I

Level IIfaucet

-

w. can

open

water

+

Sensory pathway(perception, sense)

Motor pathway(action, reaction)

Level IIItank

-

refill

EE141

GCS vs. Reinforcement LearningGCS vs. Reinforcement Learning

Environment

CriticStates

Value Function

Policy

reward

action

Environment

CriticStates

Value Function

Policy

reward

action

Actor-critic design Goal creation system

Case study: “How can Wall-E water his plants if the water resources are limited and hard to find?”

Sensorypathway

Motorpathway

GCS

Environment

Pain

States

Gate control

Desired action &state

Action decision

Action

EE141

Goal Creation ExperimentGoal Creation Experiment

Sensory-motor pairs and their effect on the environment

PAIR #SENSORY MOTOR INCREASES DECREASES

1 water can water the plant moisture water in can

8 faucet open water in can water in tank

15 tank refill water in tank reservoir water

22 pipe open reservoir water lake water

29 rain fall lake water -

EE141

Results from GCS schemeResults from GCS scheme

0 100 200 300 400 500 6000

2

4pa

in

Dry soil

0 100 200 300 400 500 6000

1

2

pain

No water in can

0 100 200 300 400 500 6000

1

2

pain

No water in tank

0 100 200 300 400 500 6000

0.5

1

pain

No water in reservoir

0 100 200 300 400 500 6000

2

4

pain

No water in lake

EE141

Averaged performance over 10 trials:

GCS:

RL:0 100 200 300 400 500 600

0

0.5

1

pain

Primitive pain

0 100 200 300 400 500 6000

0.5

1

pain

Lack of food

0 100 200 300 400 500 6000

0.2

0.4

pain

Lack of money

0 100 200 300 400 500 6000

0.2

0.4

pain

Lack of bank savings

0 100 200 300 400 500 6000

0.2

0.4

pain

Lack of job opportunity

0 100 200 300 400 500 600-1

0

1

pain

Lack of school opportunity

Machine using GCS learns to control all abstract pains and maintains the primitive pain signal on a low level in

demanding environment conditions.

0 100 200 300 400 500 6000

10

20

30

GCS vs. Reinforcement LearningGCS vs. Reinforcement Learning

EE141

Goal Creation ExperimentGoal Creation Experiment

Action scatters in 5 CGS simulations

0 100 200 300 400 500 6000

5

10

15

20

25

30

35

40Goal Scatter Plot

Go

al I

D

Discrete time

EE141

Goal Creation ExperimentGoal Creation Experiment

The average pain signals in 100 CGS simulations

0 100 200 300 400 500 6000

0.5

Primitive pain – dry soil

Pa

in

0 100 200 300 400 500 6000

0.10.2

Lack of water in can

Pa

in

0 100 200 300 400 500 6000

0.10.2

Lack of water in tank

Pa

in

0 100 200 300 400 500 6000

0.10.2

Lack of water in reservoir

Pa

in

0 100 200 300 400 500 6000

0.050.1

Lack of water in lake

Pa

in

Discrete time

EE141

Promises of embodied intelligencePromises of embodied intelligence To society

Advanced use of technology– Robots– Tutors– Intelligent gadgets

Intelligence age follows– Industrial age– Technological age– Information age

Society of minds– Superhuman intelligence– Progress in science– Solution to societies’ ills

To industry Technological development New markets Economical growth

ISAC, a Two-Armed Humanoid RobotISAC, a Two-Armed Humanoid RobotVanderbilt UniversityVanderbilt University

EE141

2002 2010 2020 2030

Biomimetics and Bio-inspired SystemsBiomimetics and Bio-inspired SystemsImpact on Space Transportation, Space Science and Earth ScienceImpact on Space Transportation, Space Science and Earth Science

Mis

sio

n C

om

ple

xity

Biological Mimicking

Embryonics

Extremophiles

DNA Computing

Brain-like computing

Self Assembled Array

Artificial nanoporehigh resolution

Mars in situlife detector

Sensor Web

Biological nanoporelow resolution

Skin and Bone

Self healing structureand thermal protection

systems

Biologically inspired aero-space systems

Space Transportation

Memristors

EE141

Sounds like science fictionSounds like science fiction

If you’re trying to look far ahead, and what you see seems like science fiction, it might be wrong.

But if it doesn’t seem like science fiction, it’s definitely wrong.

From presentation by Feresight Institute

EE141

Questions?Questions?