View
219
Download
0
Tags:
Embed Size (px)
Citation preview
10th Kovacs Colloquium UNESCO
Water Resource Planning and Water Resource Planning and Management using Motivated Management using Motivated Machine LearningMachine Learning
Janusz StarzykSchool of Electrical Engineering and Computer Science, Ohio University, USA
www.ent.ohiou.edu/~starzyk
10th Kovacs Colloquium UNESCO, Hydrocomplexity:
New Tools for Solving Wicked Water
10th Kovacs Colloquium UNESCO
Challenges in Water Management Embodied Intelligence (EI) Embodiment of Mind EI Interaction with Environment How to Motivate a Machine Motivated Learning ML Experiment Abstract Motivations and Goal
Hierarchy Promises of EI
OutlineOutline
Flood in Poland
10th Kovacs Colloquium UNESCO
Water management is challenging since:
Strategies are developed mostly on national level
There is a competition between countries for water
Water policy plans effects environment and society, health and development, and economy
Growing demands for water Need to integrate water management and
policy making There is an acute need for legitimate
scientific data Decision making in water-related health, food
and energy systems are critical to economical development and national security
Challenges in Water Challenges in Water ManagementManagement
South–North Water Transfer Project China
10th Kovacs Colloquium UNESCO
Decision makers must consider important questions:
How do we make water use sustainable? How to protect water resources from overuse
and contamination? Water problems are interconnected and too
complex to be handled by a single institution or a single group of people
It is a challenge to evolve strategies and institutional frameworks for quick policy changes towards an acceptable water use
It is necessary to create assessment and modeling tools to improve policy making resolve conflicting issues and facilitate interaction.
Challenges in Water Challenges in Water ManagementManagement
10th Kovacs Colloquium UNESCO
Why accurate integrated models to support decision making are important ?
Computerized models were used for many years to support water related decisions.
Models often simplify dynamics of economic, social and environmental interactions and lead to inappropriate policy making and management decisions.
This work proposes models that emerge from interaction with real dynamically changing environments with all of their complexities and societal dependencies.
The main objective is to suggest an integrated modeling framework that may assist with the tasks of water related decision making.
Challenges in Water Challenges in Water ManagementManagement
10th Kovacs Colloquium UNESCO
What are deficiencies of machine created models?
Artificial neural networks, fuzzy logic, and genetic algorithms have been used to model resource planning and water management
It is difficult to apply these tools in real-life decisions as the related parameters are not explicitly known
This work presents a machine learning approach that motivates machine to develop into a useful toll.
Motivated machine learning can characterize data and make predictions about their dynamic changes It could support intelligent decision making in
dynamically changing environment It could observe impacts of alternative water
management policies It may consider social, cultural, political, economic and
institutional elements of decision making
Challenges in Water Challenges in Water ManagementManagement
10th Kovacs Colloquium UNESCO
Embodied intelligence may support decision making:
EI mimics biological intelligent systems, extracting general principles of intelligent behaviour
It uses emerging, self-organizing, goal creation (GC) system that motivates EI to learn how to interact with the environment Knowledge is not entered into such systems, but is a
result of useful actions in a given environment. This knowledge is validated through active interaction
with the environment. Motivated intelligent systems adapt to unpredictable
and dynamic situations in the environment by learning, which gives them a high degree of autonomy
Learning in such systems is incremental, with continuous prediction of the input associations based on the emerging models - only new information is registered in the memory
Challenges in Water Challenges in Water ManagementManagement
10th Kovacs Colloquium UNESCO
How to use the motivated learning scheme to integrate modelling and decision making?
It is suggested to apply ML approach to water management in changing environments where the existing methods fail or work with difficulty. For instance, using classical machine learning to
represent physical processes works only under the assumption that the same processes will repeat.
However, if a process changes beyond certain limits, the prediction will not be correct.
ML systems may overcome this difficulty and such systems can be trained to advice humans.
Design concepts, computational mechanisms, and architectural organization of embodied intelligence are presented in this talk
The talk will explain internal motivation mechanism that leads to effective goal oriented learning, abstract goal creation and goal management
Challenges in Water Challenges in Water ManagementManagement
10th Kovacs Colloquium UNESCO
IntelligenceIntelligence
http://www.home-business-smarts.net/
Mainstream Science on Intelligence December 13, 1994: An Editorial With 52 Signatories, by Linda S. Gottfredson, University of Delaware
Intelligence is a very general mental capability that, among other things, involves the ability to reason, plan, solve problems, think abstractly, comprehend complex ideas, learn quickly and learn from experience.
10th Kovacs Colloquium UNESCO
Animals’ Intelligence Animals’ Intelligence Defining intelligence
through humans is not appropriate to design intelligent machines:
– Animals are intelligent too
Dog IQ test: Dogs can learn 165 words (similar to 2 year olds) Average dog has the mental abilities of a 2-year-old child (or better) They would beat a 3- or 4-year-old in basic arithmetic, Dogs show some basic emotions, such as happiness, anger and disgust “The social life of dogs is very complex - more like human teenagers -
interested in who is moving up in the pack, who is sleeping with who etc,“ says professor Stanleay Coren from University of British Columbia
10th Kovacs Colloquium UNESCO
Computational Models of Intelligence Computational Models of Intelligence
Five paradigms of Computational intelligence
http://rtpis.mst.edu/images/paradigms_of_CI.jpg
How to define and compute intelligence?
10th Kovacs Colloquium UNESCO
Traditional AITraditional AI Embodied Intelligence Embodied Intelligence Abstract intelligence
attempt to simulate “highest” human faculties:
– language, discursive reason, mathematics, abstract problem solving
Environment model Condition for problem
solving in abstract way “brain in a vat”
Embodiment knowledge is implicit in the
fact that we have a body– embodiment supports brain
development
Intelligence develops through interaction with environment Situated in environment Environment is its best model
10th Kovacs Colloquium UNESCO
Design principles of intelligent systemsDesign principles of intelligent systemsfrom Rolf Pfeifer “Understanding of Intelligence”, 1999
Interaction with complex environment
cheap design ecological balance redundancy principle parallel, loosely
coupled processes asynchronous sensory-motor
coordination value principle
Agent
Drawing by Ciarán O’Leary- Dublin Institute of Technology
10th Kovacs Colloquium UNESCO
Embodied Intelligence Embodied Intelligence
– Mechanism: biological, mechanical or virtual agent
with embodied sensors and actuators– EI acts on environment and perceives its actions– Environment hostility is persistent and stimulates EI to act– Hostility: direct aggression, pain, scarce resources, etc– EI learns so it must have associative self-organizing memory– Knowledge is acquired by EI
Definition Embodied Intelligence (EI) is a
mechanism that learns how to survive in a hostile environment
10th Kovacs Colloquium UNESCO
Embodied Intelligence Embodied Intelligence
For water resource planning and management hostility of the environment means Insufficient water resources Poor water quality Growing demand of industry for water Conflicts between stakeholders, etc
These hostile signals represent the primitive pains that grow unless they are addressed by proper actions
Surviving in this environment (politically) is to keep these signals below specified level, otherwise economical crises, social unrest, drought or famine will follow
10th Kovacs Colloquium UNESCO
Embodiment
Actuators
Sensors
Intelligence core
channel
channel
Embodiment
Sensors
Intelligence core
Environment
channel
channelActuators
Embodiment
Actuators
Sensors
Intelligence core
channel
channel
Embodiment
Sensors
Intelligence core
Environment
channel
channelActuators
Embodiment of a MindEmbodiment of a Mind
Embodiment is a part of the environment that EI controls to interact with the rest of the environment It contains intelligence core
and sensory motor interfaces under its control
Necessary for development of intelligence
Not necessarily constant or in the form of a physical body
Boundary transforms modifying brain’s self-determination
10th Kovacs Colloquium UNESCO
Brain learns own body’s dynamic Self-awareness is a result of
identification with own embodiment Embodiment can be extended by
using tools and machines Successful operation is a function
of correct perception of environment and own embodiment
Embodiment of a MindEmbodiment of a Mind
10th Kovacs Colloquium UNESCO
INPUT OUTPUT
Simulation or
Real-World System
TaskEnvironment
Agent Architecture
Long-term Memory
Short-term Memory
Reason
ActPerceive
RETRIEVAL LEARNING
EI Interaction with EnvironmentEI Interaction with Environment
From Randolph M. Jones, P : www.soartech.com
10th Kovacs Colloquium UNESCO
How to Motivate a MachineHow to Motivate a Machine ? ?
The fundamental question is how to motivate a machine to do anything, in particular to increase its “brain” complexity?
How to motivate it to explore the environment and learn how to effectively work in this environment?
Can a machine that only implements externally given goals be intelligent?If not how these goals can be created?
10th Kovacs Colloquium UNESCO
I suggest that hostility of environment motivates us. It is the pain that moves us. Our intelligence that tries to minimize this pain motivates our actions,
learning and development
We need both the environment hostility and the mechanism that learns how to reduce inflicted by the environment pain
How to Motivate a MachineHow to Motivate a Machine ? ?
In this work I propose, based on the pain, mechanism that motivates the machine to act, learn and develop.
Without the pain there will be no motivation to develop.
10th Kovacs Colloquium UNESCO
Motivated Learning Motivated Learning I suggest a goal-driven mechanism to motivate
a machine to act, learn, and develop. A simple pain based goal creation system. It uses externally defined pain signals that are
associated with primitive pains. Machine is rewarded for minimizing the primitive
pain signals.
Definition: Motivated learning (ML) is learning based on the self-organizing system of goal creation in embodied agent. Machine creates abstract goals based on the primitive pain signals. It receives internal rewards for satisfying its goals (both primitive and
abstract). ML applies to EI working in a hostile environment.
10th Kovacs Colloquium UNESCO
Pain-center and Goal CreationPain-center and Goal Creation Simple Mechanism Creates hierarchy of
values Motivation is to reduce
the primitive pain level Leads to formulation of
complex goals Reinforcement :
• Pain increase• Pain decrease
Forces exploration
+
-
Environment
Sensor
MotorPain level
Dual pain levelPain increase
Pain decrease
(-)
(+)
Motivation
(-)
(-)
(+)
(+)
Wall-E’s goal is to keep his plants from dying
(+)
(-)
Goal
10th Kovacs Colloquium UNESCO
Primitive Goal CreationPrimitive Goal Creation
- +
Pain
Dry soilPrimitive
level
opentank
sit on garbage
refillfaucet
w. can water
Dual pain
Reinforcing a proper action
10th Kovacs Colloquium UNESCO
Abstract Goal HierarchyAbstract Goal Hierarchy
Abstract goals are created to reduce abstract pains in order to satisfy the primitive goalsA hierarchy of abstract goals is created - they satisfy the lower level goals
ActivationStimulationInhibitionReinforcementEchoNeedExpectation
- +
+
Dry soilPrimitive Level
Level I
Level IIfaucet
-
w. can
open
water
+
Sensory pathway(perception, sense)
Motor pathway(action, reaction)
Level IIItank
-
refill
10th Kovacs Colloquium UNESCO
Motivated Learning ExperimentMotivated Learning Experiment
Sensory-motor pairs and their effect on the environment
Sensory Motor Increases Decreases
Dry soil Water from Can Moisture Water in Can
No Water in Can Water from Tank Water in Can Water in Tank
No Water in Tank Water from Reservoir Water in Tank Water in Reservoir
No Water in Reservoir Water from Lake Water in Reservoir Water in Lake
No Water in Lake Regulate Usage Water in Lake -
Case study: “How can Wall-E water his plants if the water resources are limited and hard to find?”
10th Kovacs Colloquium UNESCO
Action scatters in 5 ML simulations
0 100 200 300 400 500 6000
5
10
15
20
25
30
35
40Goal Scatter Plot
Go
al I
D
Discrete time
Motivated Learning ExperimentMotivated Learning Experiment
10th Kovacs Colloquium UNESCO
The average pain signals in 100 ML simulations
0 100 200 300 400 500 6000
0.5
Primitive pain – dry soil
Pa
in
0 100 200 300 400 500 6000
0.10.2
Lack of water in can
Pa
in
0 100 200 300 400 500 6000
0.10.2
Lack of water in tank
Pa
in
0 100 200 300 400 500 6000
0.10.2
Lack of water in reservoir
Pa
in
0 100 200 300 400 500 6000
0.050.1
Lack of water in lake
Pa
in
Discrete time
Motivated Learning ExperimentMotivated Learning Experiment
10th Kovacs Colloquium UNESCO
Averaged performance over 10 trials:
MLRL
Machine using ML learns to control all abstract pains and maintains the primitive pain signal on a low level.
ML vs. Reinforcement LearningML vs. Reinforcement Learning
10th Kovacs Colloquium UNESCO
Multiple dependencies:- two resources that can provide money
Motivated Learning Experiment IIMotivated Learning Experiment II
10th Kovacs Colloquium UNESCO
When the environment is abundant in both resources
Competition between Work and Sell valuables
The loser and its associated further goals will be ignored by the system
Motivated Learning Experiment IIMotivated Learning Experiment II
10th Kovacs Colloquium UNESCO
ML Abstract Goal HierarchyML Abstract Goal Hierarchy
10th Kovacs Colloquium UNESCO
Compare ML and RLCompare ML and RLMean primitive pain Pp value as a function of the number of iterations.
>10 levels of hierarchy >complex environment
- green line for RL - blue line for ML.
10th Kovacs Colloquium UNESCO
Reinforcement LearningReinforcement Learning Motivated Learning Motivated Learning Single value function Measurable rewards
Can be optimized
Predictable Objectives set by
designer Maximizes the reward
Potentially unstable
Learning effort increases with complexity
Always active
Multiple value functions One for each goal
Internal rewards Cannot be optimized
Unpredictable Sets its own objectives Solves minimax problem
Always stable
Learns better in complex environment than RL
Acts when needed
http://www.bradfordvts.co.uk/images/goal.jpg
10th Kovacs Colloquium UNESCO
Machine Working for Humanity?Machine Working for Humanity? If you’re trying to look far ahead, and
what you see seems like science fiction, it might be wrong.
But if it doesn’t seem like science fiction, it’s definitely wrong.
From presentation by Foresight Institute
10th Kovacs Colloquium UNESCO
Questions?Questions?