View
220
Download
2
Tags:
Embed Size (px)
Citation preview
Integrated Learning & Integrated Learning & Training for Interactive Training for Interactive
CharactersCharacters
Integrated Learning & Integrated Learning & Training for Interactive Training for Interactive
CharactersCharactersBruce Blumberg & the Bruce Blumberg & the
Synthetic Characters GroupSynthetic Characters Group
www.media.mit.edu/~brucewww.media.mit.edu/~bruce
Bruce Blumberg & the Bruce Blumberg & the Synthetic Characters GroupSynthetic Characters Group
www.media.mit.edu/~brucewww.media.mit.edu/~bruce
Field work…Field work…
Where we have been and are going…Where we have been and are going…
Practical & compelling real-time learningPractical & compelling real-time learning
• Easy for interactive characters to learn what they ought to be able to learn
• Easy for a human trainer to guide learning process
• A compelling user experience
• Provide heuristics and practical design principles
• Easy for interactive characters to learn what they ought to be able to learn
• Easy for a human trainer to guide learning process
• A compelling user experience
• Provide heuristics and practical design principles
Our bias & focusOur bias & focus
• Learning occurs within an innate Learning occurs within an innate structure that biases…structure that biases…• Attention
• Motivation
• Innate frequency, form and organization of behavior
• When certain things are most easily learned
• What are the catalytic components of the What are the catalytic components of the scaffolding that dramatically facilitate the scaffolding that dramatically facilitate the learning & training process?learning & training process?
• Learning occurs within an innate Learning occurs within an innate structure that biases…structure that biases…• Attention
• Motivation
• Innate frequency, form and organization of behavior
• When certain things are most easily learned
• What are the catalytic components of the What are the catalytic components of the scaffolding that dramatically facilitate the scaffolding that dramatically facilitate the learning & training process?learning & training process?
Where we draw from…Where we draw from…
• Reinforcement learningReinforcement learning• Barto & Sutton 98, Mitchell 97, Kaelbling 90, Drescher 91
• Animal training and ethologyAnimal training and ethology• Lindsay 00, Lorenz & Leyhausen 73, Ramirez 99, Pryor 99, Coppinger 01
• Motor learningMotor learning• van de Panne et al 93,94, Grzeszczuk & Terzopoulos 95, Hodgins &
Pollard 97, Gleicher 98, Faloutsos et al 01
• Behavior ArchitecturesBehavior Architectures• Reynolds 87, Tu & Terzopoulos 94, Perlin & Goldberg 96, Funge et al 99,
Burke et al 01
• Computer games & digital petsComputer games & digital pets• Dogz, AIBO, Black & White
• Reinforcement learningReinforcement learning• Barto & Sutton 98, Mitchell 97, Kaelbling 90, Drescher 91
• Animal training and ethologyAnimal training and ethology• Lindsay 00, Lorenz & Leyhausen 73, Ramirez 99, Pryor 99, Coppinger 01
• Motor learningMotor learning• van de Panne et al 93,94, Grzeszczuk & Terzopoulos 95, Hodgins &
Pollard 97, Gleicher 98, Faloutsos et al 01
• Behavior ArchitecturesBehavior Architectures• Reynolds 87, Tu & Terzopoulos 94, Perlin & Goldberg 96, Funge et al 99,
Burke et al 01
• Computer games & digital petsComputer games & digital pets• Dogz, AIBO, Black & White
Dobie T. Coyote Goes to SchoolDobie T. Coyote Goes to School
Short Dobie Video
Reinforcement Learning (R.L.) As Starting Point
Reinforcement Learning (R.L.) As Starting Point
A1 A2 A3
S1 Q(1,1) Q(1,2) Q(1,3)
S2 Q(2,1) Q(2,2) Q(2,3)
S3 Q(3,1) Q(3,2) Q(3,3)
Utility of taking action A3 in state S2
Set of all possible actions
Set of all possible states of world
• Dogs solve a simpler problem in a Dogs solve a simpler problem in a much larger space & one that is more much larger space & one that is more relevant to interactive charactersrelevant to interactive characters
• Dogs solve a simpler problem in a Dogs solve a simpler problem in a much larger space & one that is more much larger space & one that is more relevant to interactive charactersrelevant to interactive characters
The problem facing dogs (real and synthetic)
The problem facing dogs (real and synthetic)
Set of all possible actions
Set of all motivational
goals
Set of all possible stimuli
What do I do, when, in order to best satisfy my motivational goals?
The space of possible stimuli is wicked bigThe space of possible stimuli is wicked big
Set of all possible stimuli
SmellsMotion
Sounds
Dog sounds
SpeechWhistles
Modality of Stimuli
Time of Occurrence
State Space
The space of possible actions is also very bigThe space of possible actions is also very big
Set of all possible actions
Action
Time of Performance
Figure -8
Shake
Low shake
High -5
Beg
Down
Left ear twitch
Action Space
Who gets credit for good things happening?
Who gets credit for good things happening?
Yumm..
Action
Figure -8
Shake
Low shake
High -5
Beg
Down
Left ear twitch
Motion
Sounds
Dog sounds
SpeechWhistles
Modality of Stimuli
Dogs seem to constrain search for causal agents
Dogs seem to constrain search for causal agents
Time
Consequences Window:Trainer “clicks” signaling reward is coming.
When reward is actually received
Attention Window:Cue given immediately before or as dog is moving into desired pose
Sit Approach Eat
Dogs make the problem tractable by constraining search for causal agents to narrow temporal windows
Dogs seem to use implicit feedback to guide perceptual learning
Dogs seem to use implicit feedback to guide perceptual learning
Sit
Time
“sit-utterance” perceived.
Approach Eat
“click” perceived.
Dog decides to sit
Build & update perceptual model of “sit-utterance”
Dogs use rewarded action to identify potentially promising state to explore and to guide formation of perceptual models
Dogs seem to give credit where credit is due
Dogs seem to give credit where credit is due
Sit
Time
“sit-utterance” perceived.
Approach Eat
“click” perceived.
Dog decides to sit
1. Credit sitting in presence of “sit-utterance”2. Build & update perceptual model of “sit-
utterance”
Dogs seem to give credit where credit is due…
Dogs seem to give credit where credit is due…• Trainer repeatedly lures dog Trainer repeatedly lures dog
through a trajectory or into a through a trajectory or into a pose pose
• Eventually, dog performs Eventually, dog performs behavior spontaneouslybehavior spontaneously
• ImplicationImplication• Dog associates reward with
resulting body configuration or trajectory and not just with “follow-your nose”
• Trainer repeatedly lures dog Trainer repeatedly lures dog through a trajectory or into a through a trajectory or into a pose pose
• Eventually, dog performs Eventually, dog performs behavior spontaneouslybehavior spontaneously
• ImplicationImplication• Dog associates reward with
resulting body configuration or trajectory and not just with “follow-your nose”
D.L.: Take Advantage of Predictable Regularities
D.L.: Take Advantage of Predictable Regularities• Constrain search for causal agents by Constrain search for causal agents by
taking advantage of temporal proximity taking advantage of temporal proximity & natural hierarchy of state spaces& natural hierarchy of state spaces• Use consequences to bias choice of action
• But vary performance and attend to differences
• Explore state and action spaces on “as-Explore state and action spaces on “as-needed” basisneeded” basis• Build models on demand
• Constrain search for causal agents by Constrain search for causal agents by taking advantage of temporal proximity taking advantage of temporal proximity & natural hierarchy of state spaces& natural hierarchy of state spaces• Use consequences to bias choice of action
• But vary performance and attend to differences
• Explore state and action spaces on “as-Explore state and action spaces on “as-needed” basisneeded” basis• Build models on demand
D.L.: Make Use of All Feedback: Explicit & Implicit
D.L.: Make Use of All Feedback: Explicit & Implicit• Use rewarded action as context for Use rewarded action as context for
identifying identifying •Promising state space and action space to
explore
•Good examples from which to construct perceptual models, e.g.,
•A good example of a “sit-utterance” is one that occurs within the context of a rewarded Sit.
• Use rewarded action as context for Use rewarded action as context for identifying identifying •Promising state space and action space to
explore
•Good examples from which to construct perceptual models, e.g.,
•A good example of a “sit-utterance” is one that occurs within the context of a rewarded Sit.
D.L.: Make Them Easy to TrainD.L.: Make Them Easy to Train
• Respond quickly to “obvious” Respond quickly to “obvious” contingenciescontingencies
• Support Luring and ShapingSupport Luring and Shaping•Techniques to prompt infrequently expressed
or novel motor actions
• ““Trainer friendly” credit Trainer friendly” credit assignmentassignment•Assign credit to candidate that matches
trainer’s expectation
• Respond quickly to “obvious” Respond quickly to “obvious” contingenciescontingencies
• Support Luring and ShapingSupport Luring and Shaping•Techniques to prompt infrequently expressed
or novel motor actions
• ““Trainer friendly” credit Trainer friendly” credit assignmentassignment•Assign credit to candidate that matches
trainer’s expectation
The SystemThe System
Representation of State: PerceptRepresentation of State: Percept
• Percepts are Percepts are atomic perception atomic perception unitsunits
• Recognize and Recognize and extract features extract features from sensory datafrom sensory data
• Model-basedModel-based
• Organized in Organized in dynamic hierarchydynamic hierarchy
• Percepts are Percepts are atomic perception atomic perception unitsunits
• Recognize and Recognize and extract features extract features from sensory datafrom sensory data
• Model-basedModel-based
• Organized in Organized in dynamic hierarchydynamic hierarchy
Representation of State-Action Pairs: Action Tuples
Representation of State-Action Pairs: Action Tuples
Percept Action
Value
Novelty
Reliability
ValuePercept Activation
Action Tuples are organized Action Tuples are organized in dynamic hierarchy and in dynamic hierarchy and compete probabilistically compete probabilistically based on their learned value based on their learned value and reliability and reliability
Representation of Action: Labeled Path Through Space of Body Configurations
Representation of Action: Labeled Path Through Space of Body Configurations• A motor program generates a path A motor program generates a path
through a graph of annotated through a graph of annotated poses, e.g.,poses, e.g.,•Sit animation
•Follow-your-nose procedure
• Paths can be compared and Paths can be compared and classified just like perceptual classified just like perceptual events using Motor Model Perceptsevents using Motor Model Percepts
• A motor program generates a path A motor program generates a path through a graph of annotated through a graph of annotated poses, e.g.,poses, e.g.,•Sit animation
•Follow-your-nose procedure
• Paths can be compared and Paths can be compared and classified just like perceptual classified just like perceptual events using Motor Model Perceptsevents using Motor Model Percepts
Use Time to Constrain Search for Causal Agents
Use Time to Constrain Search for Causal Agents
Sit
Attention Window:Look here for cues that appear correlated with increased likelihood of action being followed by a good thing
Good Thing
Consequences Window:Assume any good or bad things that happen here are associated with the preceding action and the context in which it was performed
Scratch
Time
Four Important Tasks Are Performed During Credit Assignment
Four Important Tasks Are Performed During Credit Assignment• Choose most worthy Action Tuple Choose most worthy Action Tuple
heuristically based on reliability heuristically based on reliability and novelty statisticsand novelty statistics
• Update valueUpdate value• Create new Action Tuples as Create new Action Tuples as
appropriateappropriate• Guide State and Action Space Guide State and Action Space
DiscoveryDiscovery
• Choose most worthy Action Tuple Choose most worthy Action Tuple heuristically based on reliability heuristically based on reliability and novelty statisticsand novelty statistics
• Update valueUpdate value• Create new Action Tuples as Create new Action Tuples as
appropriateappropriate• Guide State and Action Space Guide State and Action Space
DiscoveryDiscovery
Implicit Feedback Guides State Space Discovery
Implicit Feedback Guides State Space Discovery
Good Thing appears. Create a new Percept with “beg” example as initial model
Time
Utterance occurs within window but not classified by any existing percept
“beg”
This means that Percepts are only created to recognize “promising” utterances
Beg Good ThingScratch
Implicit Feedback Identifies Good Examples
Implicit Feedback Identifies Good Examples
Beg Good Thing
Good Thing appears. Update model of “beg” utterance using “beg” that occurred in attention window
Scratch
Time
Classify utterance as “beg”.
“beg”
This means model is built using good examples
Unrewarded Examples Don’t Get Added to Models
Unrewarded Examples Don’t Get Added to Models
Beg Sit
Beg ends without food appearing. Do not update model since example may have been bad.
Scratch
Time
Utterance classified as “Beg” by mistake. Beg becomes active.
“Leg”
Actually, bad examples can be used to build model of “not-Beg.”
Most Worthy Action Tuple Gets CreditMost Worthy Action Tuple Gets Credit
Sit
Time
“sit-utterance” perceived.
Good Thing
“click” perceived.
<true/Sit> begins
But credit goes to <“sit-utterance”/Sit>
Create New Action Tuples As AppropriateCreate New Action Tuples As Appropriate
Implicit Feedback Guides Action Space Discovery
Implicit Feedback Guides Action Space Discovery
Good Thing appears. Compare accumulated path to known paths
Time
“Follow-your-nose” action accumulates path through pose-space
Down
Down gets the credit for Good Thing appearing, rather than “Follow-your-nose.”
Follow-your-nose Good Thing
If Path Is Novel, Create a New Motor Program and Action
If Path Is Novel, Create a New Motor Program and Action
Good Thing appears. Compare accumulated path to known paths
Time
“Follow-your-nose” action accumulates path through pose-space
Figure-8 is created and subsequent examples of Figure-8 are used to improve model of path
Figure-8
Follow-your-nose Good Thing
Dobie T. Coyote…Dobie T. Coyote…
Long Dobie Video
Limitations and Future WorkLimitations and Future Work
• Important extensions Important extensions •Other kinds of learning (e.g., social or
spatial)
•Generalization
•Sequences
•Expectation-based emotion system
• How will the system scale?How will the system scale?
• Important extensions Important extensions •Other kinds of learning (e.g., social or
spatial)
•Generalization
•Sequences
•Expectation-based emotion system
• How will the system scale?How will the system scale?
Useful InsightsUseful Insights
• UseUse•Temporal proximity to limit search.
•Hierarchical representations of state, action and state-action space & use implicit feedback to guide exploration
• “trainer friendly” credit assignment
• Luring and shaping are essentialLuring and shaping are essential
• UseUse•Temporal proximity to limit search.
•Hierarchical representations of state, action and state-action space & use implicit feedback to guide exploration
• “trainer friendly” credit assignment
• Luring and shaping are essentialLuring and shaping are essential
Future work: the problem of sequences…
Future work: the problem of sequences…
Who gets credit for good things happening?
Who gets credit for good things happening?
stalkgrab-bite
eye
orient
kill-bitechase
Yumm..
Time
Conventional idea: back propagation from goal
Conventional idea: back propagation from goal
stalkgrab-bite
eye
orient
kill-bitechase
Yumm..
Time Credit flows backward
Conventional idea: back propagation from goal
Conventional idea: back propagation from goal
stalkgrab-bite
eye
orient
kill-bitechase
Yumm..
Time Credit flows backward
Conventional idea: back propagation from goal
Conventional idea: back propagation from goal
stalkgrab-bite
eye
orient
kill-bitechase
Yumm..
Time Credit flows backward
Back propagation is a slow way to learn…Back propagation is a slow way to learn…
• Search space is potentially hugeSearch space is potentially huge
• Individual elements of sequence Individual elements of sequence may need to be perfected in order may need to be perfected in order to reach goal at all.to reach goal at all.
• Necessary but rarely successful Necessary but rarely successful behaviors may be very difficult to behaviors may be very difficult to learn.learn.
• Search space is potentially hugeSearch space is potentially huge
• Individual elements of sequence Individual elements of sequence may need to be perfected in order may need to be perfected in order to reach goal at all.to reach goal at all.
• Necessary but rarely successful Necessary but rarely successful behaviors may be very difficult to behaviors may be very difficult to learn.learn.
Leyhausen’s suggestion…Leyhausen’s suggestion…
stalkgrab-bite
eye
orient
kill-bitechase
Time Each element is innately self-motivating and has innate reward metric
motivation & reward
motivation & reward
motivation & reward
motivation & reward
motivation & reward
motivation & reward
Leyhausen’s suggestion…Leyhausen’s suggestion…
stalkgrab-bite
eye
orient
kill-bitechase
Time Each element is innately self-motivating and has innate reward metric
motivation & reward
motivation & reward
motivation & reward
motivation & reward
motivation & reward
motivation & reward
Functional goal plays incidental roleFunctional goal plays incidental role
stalkgrab-bite
eye
orient
kill-bitechase
Time Propagated value from functional goal plays incidental role
Yumm..
Coppinger’s suggestion, part 1…Coppinger’s suggestion, part 1…
grab-bite
eye
orient
kill-bitechase
Time Varying innate tendency to follow behavior with “next” in sequence
Internal motivation
External motivation
stalk
Coppinger suggestion, part 2Coppinger suggestion, part 2
Wolf
Border Collie
Live stock Guarding dog
Coppinger’s suggestion, part 3 Coppinger’s suggestion, part 3
Border Collie
Livestock Guarding dog
Sensitive period for social development
Onset of predatory behaviors
Border Collies incorporate predatory patterns into social play because of early onset of these patterns
After Coppinger
Future work: the problem of sequencesFuture work: the problem of sequences
• What can we learn from how animals What can we learn from how animals “learn” sequences?“learn” sequences?• Sequence may be “learned” apart from function
• Elements may be self-motivating and have local metric of goodness
• Innate bias of varying degree to perform all or part of “sequence”
• Role of developmental timing in determining function
• What can we learn from how animals What can we learn from how animals “learn” sequences?“learn” sequences?• Sequence may be “learned” apart from function
• Elements may be self-motivating and have local metric of goodness
• Innate bias of varying degree to perform all or part of “sequence”
• Role of developmental timing in determining function
Future work: learning from learningFuture work: learning from learning
• Does it pay to explore?Does it pay to explore?•Does exploring lead to good things or bad
things?
• Best environment in which to Best environment in which to explore?explore?
• What can be learned from watching What can be learned from watching others?others?
• How do I know if I am doing it right?How do I know if I am doing it right?
• Does it pay to explore?Does it pay to explore?•Does exploring lead to good things or bad
things?
• Best environment in which to Best environment in which to explore?explore?
• What can be learned from watching What can be learned from watching others?others?
• How do I know if I am doing it right?How do I know if I am doing it right?
Predictability & controlPredictability & control
• Is it a predictable world?Is it a predictable world?• Can I predict the potential arrival of good or bad
things so as to:
• Increase chances of good thing happening
• Decrease chances of bad thing happening
• Is it a controllable world?Is it a controllable world?• Can I control the world so as to satisfy my
motivational goals?
• Is it a predictable world?Is it a predictable world?• Can I predict the potential arrival of good or bad
things so as to:
• Increase chances of good thing happening
• Decrease chances of bad thing happening
• Is it a controllable world?Is it a controllable world?• Can I control the world so as to satisfy my
motivational goals?
P & C are learned, and in turn affect learningP & C are learned, and in turn affect learning
Bad ThingBad Thing Good ThingGood Thing
I got smacked by surpriseI got smacked by surprise Increase Increase attentionattention
I got a cookie by surpriseI got a cookie by surprise
I will get smacked unless I will get smacked unless I do something (maybe)I do something (maybe)
Active Active explorationexploration
I will get a cookie if I do I will get a cookie if I do something (maybe)something (maybe)
I will get smacked unless I will get smacked unless I run awayI run away
ExploitationExploitation I will get a cookie if I sitI will get a cookie if I sit
I will get smacked no I will get smacked no matter what I domatter what I do
Low Low ExplorationExploration
I will get a cookie no matter I will get a cookie no matter what I dowhat I do
Predictability and ControllabilityPredictability and Controllability
-p/c
-p/-c
p/c
p/-c
-p unpredictable
p predictable
c controllable
-c uncontrollable
After Lindsay
Anxiety Confidence
Depression Frustration
Practical & compelling real-time learningPractical & compelling real-time learning
• Easy for interactive characters to learn what they ought to be able to learn
• Easy for a human trainer to guide learning process
• A compelling user experience
• Provide heuristics and practical design principles
• Easy for interactive characters to learn what they ought to be able to learn
• Easy for a human trainer to guide learning process
• A compelling user experience
• Provide heuristics and practical design principles
AcknowledgementsAcknowledgements
• Members of the Synthetic Members of the Synthetic Characters Group, past, present & Characters Group, past, present & futurefuture
• Gary WilkesGary Wilkes
• Funded by the Digital Life Funded by the Digital Life ConsortiumConsortium
• Members of the Synthetic Members of the Synthetic Characters Group, past, present & Characters Group, past, present & futurefuture
• Gary WilkesGary Wilkes
• Funded by the Digital Life Funded by the Digital Life ConsortiumConsortium
The problemThe problem
• If each element in sequence has 3 If each element in sequence has 3 variants, there are 729 possible variants, there are 729 possible combinations of which 1 may work combinations of which 1 may work (ignoring stimuli)(ignoring stimuli)
• If there are 12 possible stimuli, there are If there are 12 possible stimuli, there are 1,586,874,322,944 possible combinations 1,586,874,322,944 possible combinations of stimuli-action pairs to explore.of stimuli-action pairs to explore.
• Don’t know if it is the right sequence until Don’t know if it is the right sequence until goal is reachedgoal is reached
• What happens if “variant” needs to be What happens if “variant” needs to be learned?learned?
• If each element in sequence has 3 If each element in sequence has 3 variants, there are 729 possible variants, there are 729 possible combinations of which 1 may work combinations of which 1 may work (ignoring stimuli)(ignoring stimuli)
• If there are 12 possible stimuli, there are If there are 12 possible stimuli, there are 1,586,874,322,944 possible combinations 1,586,874,322,944 possible combinations of stimuli-action pairs to explore.of stimuli-action pairs to explore.
• Don’t know if it is the right sequence until Don’t know if it is the right sequence until goal is reachedgoal is reached
• What happens if “variant” needs to be What happens if “variant” needs to be learned?learned?
Big idea: innate biases facilitate learning Big idea: innate biases facilitate learning
• Biases include…Biases include…• Temporal Proximity implies causality
• Attend more readily to certain classes of stimuli than to others (motion vs. speech)
• Lazy discovery (pay attention once you have a reason to pay attention)
• Elements may be “innately” self-motivating and have local metric of “goodness”
• Biases include…Biases include…• Temporal Proximity implies causality
• Attend more readily to certain classes of stimuli than to others (motion vs. speech)
• Lazy discovery (pay attention once you have a reason to pay attention)
• Elements may be “innately” self-motivating and have local metric of “goodness”