Dynamic Programming and Optimal rrw1/oc/dp2013.pdf 1 Dynamic Programming Dynamic programming and the

  • View
    0

  • Download
    0

Embed Size (px)

Text of Dynamic Programming and Optimal rrw1/oc/dp2013.pdf 1 Dynamic Programming Dynamic programming and the

  • Dynamic Programming

    and Optimal Control

    Richard Weber

    Graduate Course at London Business School

    Winter 2013

    i

  • Contents

    Table of Contents 1

    1 Dynamic Programming 1 1.1 Control as optimization over time . . . . . . . . . . . . . . . . . . . . . . 1 1.2 The principle of optimality . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.3 Example: the shortest path problem . . . . . . . . . . . . . . . . . . . . 1 1.4 The optimality equation . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.5 Markov decision processes . . . . . . . . . . . . . . . . . . . . . . . . . . 4

    2 Examples of Dynamic Programming 5 2.1 Example: optimization of consumption . . . . . . . . . . . . . . . . . . . 5 2.2 Example: exercising a stock option . . . . . . . . . . . . . . . . . . . . . 6 2.3 Example: secretary problem . . . . . . . . . . . . . . . . . . . . . . . . . 7

    3 Dynamic Programming over the Infinite Horizon 9 3.1 Discounted costs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.2 Example: job scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.3 The infinite-horizon case . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3.4 The optimality equation in the infinite-horizon case . . . . . . . . . . . . 11 3.5 Example: selling an asset . . . . . . . . . . . . . . . . . . . . . . . . . . 12

    4 Positive Programming 14 4.1 Example: possible lack of an optimal policy. . . . . . . . . . . . . . . . . 14 4.2 Characterization of the optimal policy . . . . . . . . . . . . . . . . . . . 14 4.3 Example: optimal gambling . . . . . . . . . . . . . . . . . . . . . . . . . 15 4.4 Value iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 4.5 Example: search for a moving object . . . . . . . . . . . . . . . . . . . . 16 4.6 Example: pharmaceutical trials . . . . . . . . . . . . . . . . . . . . . . . 17

    5 Negative Programming 19 5.1 Example: a partially observed MDP . . . . . . . . . . . . . . . . . . . . 19 5.2 Stationary policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 5.3 Characterization of the optimal policy . . . . . . . . . . . . . . . . . . . 20 5.4 Optimal stopping over a finite horizon . . . . . . . . . . . . . . . . . . . 21 5.5 Example: optimal parking . . . . . . . . . . . . . . . . . . . . . . . . . . 22

    6 Optimal Stopping Problems 23 6.1 Bruss’s odds algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 6.2 Example: Stopping a random walk . . . . . . . . . . . . . . . . . . . . . 24 6.3 Optimal stopping over the infinite horizon . . . . . . . . . . . . . . . . . 24 6.4 Sequential Probability Ratio Test . . . . . . . . . . . . . . . . . . . . . . 26 6.5 Bandit processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 6.6 Example: Two-armed bandit . . . . . . . . . . . . . . . . . . . . . . . . 27

    ii

  • 6.7 Example: prospecting . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

    7 Bandit Processes and the Gittins Index 29 7.1 Index policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 7.2 Multi-armed bandit problem . . . . . . . . . . . . . . . . . . . . . . . . 29 7.3 Gittins index theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 7.4 Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 7.5 Proof of the Gittins index theorem . . . . . . . . . . . . . . . . . . . . . 31 7.6 Example: Weitzman’s problem . . . . . . . . . . . . . . . . . . . . . . . 32

    8 Applications of Bandit Processes 33 8.1 Forward induction policies . . . . . . . . . . . . . . . . . . . . . . . . . . 33 8.2 Example: playing golf with more than one ball . . . . . . . . . . . . . . 33 8.3 Target processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 8.4 Bandit superprocesses . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 8.5 Example: single machine stochastic scheduling . . . . . . . . . . . . . . 35 8.6 Calculation of the Gittins index . . . . . . . . . . . . . . . . . . . . . . . 35 8.7 Branching bandits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 8.8 Example: Searching for a single object . . . . . . . . . . . . . . . . . . . 37

    9 Average-cost Programming 38 9.1 Average-cost optimality equation . . . . . . . . . . . . . . . . . . . . . . 38 9.2 Example: admission control at a queue . . . . . . . . . . . . . . . . . . . 39 9.3 Value iteration bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 9.4 Policy improvement algorithm . . . . . . . . . . . . . . . . . . . . . . . . 40

    10 Continuous-time Markov Decision Processes 42 10.1 Stochastic scheduling on parallel machines . . . . . . . . . . . . . . . . . 42 10.2 Controlled Markov jump processes . . . . . . . . . . . . . . . . . . . . . 44 10.3 Example: admission control at a queue . . . . . . . . . . . . . . . . . . . 45

    11 Restless Bandits 47 11.1 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 11.2 Whittle index policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 11.3 Whittle indexability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 11.4 Fluid models of large stochastic systems . . . . . . . . . . . . . . . . . . 49 11.5 Asymptotic optimality . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

    12 Sequential Assignment and Allocation Problems 53 12.1 Sequential stochastic assignment problem . . . . . . . . . . . . . . . . . 53 12.2 Sequential allocation problems . . . . . . . . . . . . . . . . . . . . . . . 54 12.3 SSAP with arrivals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 12.4 SSAP with a postponement option . . . . . . . . . . . . . . . . . . . . . 57 12.5 Stochastic knapsack and bin packing problems . . . . . . . . . . . . . . 58

    iii

  • 13 LQ Regulation 59

    13.1 The LQ regulation problem . . . . . . . . . . . . . . . . . . . . . . . . . 59

    13.2 The Riccati recursion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

    13.3 White noise disturbances . . . . . . . . . . . . . . . . . . . . . . . . . . 61

    13.4 LQ regulation in continuous-time . . . . . . . . . . . . . . . . . . . . . . 62

    13.5 Linearization of nonlinear models . . . . . . . . . . . . . . . . . . . . . . 62

    14 Controllability and Observability 63

    14.1 Controllability and Observability . . . . . . . . . . . . . . . . . . . . . . 63

    14.2 Controllability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

    14.3 Controllability in continuous-time . . . . . . . . . . . . . . . . . . . . . . 65

    14.4 Example: broom balancing . . . . . . . . . . . . . . . . . . . . . . . . . 65

    14.5 Stabilizability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

    14.6 Example: pendulum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

    14.7 Example: satellite in a plane orbit . . . . . . . . . . . . . . . . . . . . . 67

    15 Observability and the LQG Model 68

    15.1 Infinite horizon limits . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

    15.2 Observability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

    15.3 Observability in continuous-time . . . . . . . . . . . . . . . . . . . . . . 70

    15.4 Example: satellite in planar orbit . . . . . . . . . . . . . . . . . . . . . . 70

    15.5 Imperfect state observation with noise . . . . . . . . . . . . . . . . . . . 70

    16 Kalman Filter and Certainty Equivalence 72

    16.1 The Kalman filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

    16.2 Certainty equivalence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

    16.3 The Hamilton-Jacobi-Bellman equation . . . . . . . . . . . . . . . . . . 74

    16.4 Example: LQ regulation . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

    16.5 Example: harvesting fish . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

    17 Pontryagin’s Maximum Principle 78

    17.1 Example: optimization of consumption . . . . . . . . . . . . . . . . . . . 78

    17.2 Heuristic derivation of Pontryagin’s maximum principle . . . . . . . . . 79

    17.3 Example: parking a rocket car . . . . . . . . . . . . . . . . . . . . . . . 80

    17.4 Adjoint variables as Lagrange multipliers . . . . . . . . . . . . . . . . . 82

    18 Using Pontryagin’s Maximum Principle 83

    18.1 Transversality conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

    18.2 Example: use of transversality conditions . . . . . . . . . . . . . . . . . 83

    18.3 Example: insects as optimizers . . . . . . . . . . . . . . . . . . . . . . . 84

    18.4 Problems in which time appears explicitly . . . . . . . . . . . . . . . . . 84

    18.5 Example: monopolist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

    18.6 Example: neoclassical economic growth . . . . . . . . . . . . . . . . . . 86

    iv

  • 19 Controlled Diffusion Processes 88 19.1 The dynamic programming equation . . . . . . . . . . . . . . . . . . . . 88 19.2 Diffusion processes and controlled diffusion processes . . . . . . . . . . . 88 19.3 Example: noisy LQ regulation in continuous time . . . . . . . . . . . . . 89 19.4 Example: passage to a stopping set . . . . . . . . . . . . . . . . . . . . . 90

    20 Risk-sensitive Optimal Control 92 20.1 Whittle risk sensitivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 20.2 The felicity of LEQG assumptions . . . . . . . . . . . . . . . . . . . . . 92 20.3 A risk-sensitive certainty-equivalence principle . . . . . . .