Embed Size (px)
Citation preview
Reinforcement Learning Simulations and
Robotics
Models
● Partially observable – noise in sensors
● Policy search methods rather than value function-based approaches
● Isolate key parameters by choosing an appropriate representation for a value function or policy
● Incorporating prior knowledge and transfer knowledge from simulations
Safety
● Key issue of the learning process● Doesn't apply to the rest of the RL community● Perkins and Barto
– RL agents based on Lyapunov functions– Switching between the underlying controllers – Always safe and offers basic performance
guarantees.
Grid World Themed Movements
● Classical RL approaches ● Discrete states and actions ● Projected for navigational tasks● Use actions like “move to the cell to the left”● Use a lower level controller to take care of
accelerating moving and stopping while ensuring precision
Quick Reward Shaping
● Rewards → Quick success – Real-world experience costly
● Specifying good reward functions – Requires domain knowledge
– Difficult in practice
● Intermediate rewards instead of binary
Tracking Solution
● Used to help convergence ● The dynamics of a robot can change
– Temperature– Wear on gears or motors– Other external factors
Building an Accurate Model
● Challenging ● Requires very many data samples ● Under-modeling errors accumulate
– Simulated robot can quickly diverge from the real-world system
● Transfer requires significant modifications if model is not accurate
Approximate models
● Verifying and testing algorithms in simulation● Establishing proximity to theoretical optimal
solution● Calculating approximate gradients for local
policy improvement● Identifying strategies for collecting more data● Performing “Mental rehearsal”
Mental Rehearsal
● Practicing in simulation● The simulated learning step● Used after learning a forward model from real
world● Only the resulting policy is transferred to the robot● Model-based methods
– Sample efficient– Often requires a great deal of memory
Mental Rehearsal Issues
● Simulation Biases● Stochasticity of the real world● Efficient optimization when sampling from a
simulator
Mental Rehearsal Solutions
● Add a stochastic model of distribution to your simulation
● Average results over model uncertainty ● Artificially add noise the the simulation
– Avoids policy over-fitting
– Smooths model errors
● Explicity model uncertainty
Grounded Simulation Learning
Iterative optimization framework for speeding up robot learning using an imperfect simulator
1. Behavior is optimized in simulation
2. Behavior is tested on robot and compared to expected results from the simulation
3. Simulator is modified using machine-learning approach to come closer to reality
GSL: Fitness Sim
● Imperfect simulation of the robot● Evaluates the parametrized behavior of
the robot● Function must be modifiable● Used to make the simulation better match the
real robot’s behavior.
GSL Fitness Robot
● Small number of evaluations ● Evaluates the fitness of the parametrized
behavior on the robot itself
GSL Explore Robot
● A small number of explorations can be
run on the real robot
● Collect states and actions relevant to the current parameterization of the behavior
● While exploring
GSL Learn
● Used to learn a model of the effects of actions
on state of the real robot.
● This model will be used to modify Fitness sim to make it better reflect the behavior on the real robot.
GSL Optimize
● In simulation● Optimization to find better parameters
Ball in Cup Real Robot Example
Ball in Cup Real Robot Results
● 42-45 episodes to get the ball n the cup● 70-80 episodes to be consistent● Always converged tot he maximum after 100
episodes
Simulation in Robot RL
● Simulation matched recorded data very well● Simulated policies usually missed ● First improve a demonstrated policy in
simulation and only perform the fine-tuning on the real robot
● Importance sampler– Considers only the n best previous episodes
SARSA
● Popular base RL algorithm for robotics● Compatible with Q-Value reuse
● The mapping Q-Value Reuse function
Q-Value Reuse
Transfer Methods
● Weak Transfer: Time spent in source task doesn't count against the learner in the target
● Strong Tranfer: Source time does count
Two Step Transfer
● Learned sequentially from multiple source tasks
● The Q-Value Reuse function for two step
