Solving High Dimensional Hamilton-Jacobi-Bellman Equations Using Low

Rank Tensor DecompositionYoke Peng Leong

California Institute of Technology

Joint work with Elis Stefansson, Matanya Horowitz, Joel Burdick

MotivationSynthesize optimal feedback controllers for

nonlinear dynamical systems in high dimensions

1

> 40 degree of freedoms> 12 degree of freedoms

Outline

• Motivation• Problem formulation• Low rank tensor decomposition• Alternating least squares & improvements• Example• Summary & future work

2

Problem Formulation

Stochastic nonlinear affine system:

Synthesize a controller, u(x), that minimize the cost function:

Brownian noise

3

HJB Equation

Dynamic programming gives the HJB equation

where the optimal controller is given by

Value function (cost-to-go):

PDE Nonlinear

4

Log transformation:

Linearly Solvable HJB Equation

Condition:

HJB equation (a nonlinear PDE)

Any system of the following form can satisfy the condition

Linearly solvable HJB equation (a linear PDE)

Desirability function

W. H. Fleming, C. J. Holland, P. Dai Pra, R. Filliger, H. Kappen, E. Todorov, F. Stulp, E. A. Theodorou, K. Dvijotham, S. Schaal

5

Convex Optimization

Suboptimal Stabilizing Controllers for Linearly Solvable Systems, Y. P. Leong, M. B. Horowitz, J. W. Burdick, CDC 2015Linearly Solvable Stochastic Control Lyapunov Functions, Y. P. Leong, M. B. Horowitz, J. W. Burdick, SIAM Journal on Control and Optimization, Accepted

Upper bound solution

Relaxation

Sum of squares program

6

Main ResultsTheorem: The approximate solution is a stochastic control Lyapunov function (SCLF).

Corollary: The suboptimal controller is stabilizing in probability.

Proof: Relaxed HJB and satisfies the definition of SCLF.

The approximate solution gives a suboptimal stabilizing controller.

Theorem: Given the controller , then Expected cost of a system using the given controller

The approximate solution gives an upper bound to the actual cost when using the suboptimal controller.

Proof: Manipulate the relaxed HJB and the error bound of approximate value function.

7

Convex Optimization

Suboptimal Stabilizing Controllers for Linearly Solvable Systems, Y. P. Leong, M. B. Horowitz, J. W. Burdick, CDC 2015Linearly Solvable Stochastic Control Lyapunov Functions, Y. P. Leong, M. B. Horowitz, J. W. Burdick, SIAM Journal on Control and Optimization, Accepted

Upper bound solution

Relaxation

Sum of squares program

Problem:Curse of dimensionality

8

Related Works

• Sparse grid approximation (J. Garcke, A. Kröner)• Taylor polynomial approximation + patchy technique (C.

O. Aguilar, A. J. Krener)• Max-plus expansion (W. M. McEneaney)• Model reduction (K. Kunisch, S. Volkwein, L. Xie, S.

Gombao)• Level-set algorithm (I. M. Mitchell, C. J. Tomlin)etc…

9

GoalSolve linear HJB equations for high

dimensional dynamical systems

10

> 40 degree of freedoms> 12 degree of freedoms

Outline

• Motivation• Problem formulation• Low rank tensor decomposition• Alternating least squares & improvements• Example• Summary & future work

11

Low Rank Tensor Decomposition

Low rank tensor decomposition

Separated representation G. BeylkinM. J. Mohlenkamp

12

Low Rank Tensor DecompositionLow rank tensor decomposition

13

Low Rank Tensor DecompositionLow rank tensor decomposition

CANDECOMP/PARAFAC tensor decomposition

Tensor term

Separation rank Basis function

Normalization constant

R. A. Harshman, J. D. Carroll, J.-J. Chang

14

Low Rank Tensor DecompositionLow rank tensor decomposition

CANDECOMP/PARAFAC tensor decompositionFunction Operator

B. N. Khoromskij. Tensors-structured numerical methods in scientific computing: Survey on recent advances. Chemometrics and Intelligent Laboratory Systems, Jan. 2012.Y. Sun and M. Kumar. A tensor decomposition approach to high dimensional stationary Fokker-Planck equations. ACC, 2014

15

Linearly Solvable HJB Equation16

Stochastic nonlinear affine system:

Cost function:

Linearly Solvable HJB Equation

Rewrite

Tensor form

17

Tensor Decomposition of HJB 18

x =

Boundary

SolutionOperator

1

1

1

Linearly Solvable HJB Equation

Rewrite

Tensor form

Benefit: Memory and operations scale linearly with dimension

But, it is NP-Hard (J. Håstad)

19

Outline

• Motivation• Problem formulation• Low rank tensor decomposition• Alternating least squares & improvements• Example• Summary & future work

20

Alternating Least Squares

1. Fix the separation rank, rF

2. For each dimension k, solve a least squares problem

3. Iterate through all k and repeat until the residual is small enough

4. Increase rF if residual cannot decrease anymore but it is still too large

21

regularization𝛼 ~ 𝜇

approximately rounding error

G. Beylkin, M. J. Mohlenkamp

Alternating Least Squares 22

Normal equation

G. Beylkin, M. J. Mohlenkamp

Complexity

Accuracy

Rounding error

Number of grid

Issues with ALS

Ill-condition

Note:• A includes HJB and

boundary condition• F is the solution

23

Sequential Computation of Solution 24

Residual

Error

Issues with ALSNormal equation

Ill-condition

Note:• A includes HJB and

boundary condition• F is the solution

25

Tensor Decomposition of HJB 26

x =

Boundary

SolutionOperator

1

1

1

Rescaling Boundary Condition 27

Residual

Error

Sequential Alternating Least SquaresModified ALS with:• Sequential computation of solution• Boundary condition rescaling

MATLAB code is available at http://www.cds.caltech.edu/~yleong/

28

Residual

Error

Sequential Alternating Least SquaresModified ALS with:• Sequential computation of solution• Boundary condition rescaling

29

Recent work:M. J. Reynolds, A. Doostan, G. Beylkin. Randomized Alternating Least Squares For Canonical Tensor Decompositions: Application To A PDE With Random Data, SIAM J. Scientific Computing, 2016

Randommatrix

Outline

• Motivation• Problem formulation• Low rank tensor decomposition• Alternating least squares & improvements• Example• Summary & future work

30

Inverted Pendulum 31

Linux machine with 3 GHz i7 processor and 64 GB RAMMATLAB 2014a

Solution rank, rF = 328

H. M. Osinga and J. Hauser, “The geometry of the solution set of nonlinear optimal control problems,” J. Dynamics and Differential Equations, 2006.

Accuracy

Number of grid = 201

Inverted Pendulum 32

Vertical Take-off and Landing Aircraft 33

J. Hauser, S. Sastry, G. Meyer. Nonlinear control design for slightly non-minimum phase systems: application to V/STOL aircraft. Automatica. 1992

Number of grid = 100

Vertical Take-off and Landing Aircraft 34

Number of grid = 100

Quadcopter 35

L. R. Carrillo, A. E. López, R. Lozano, C. Pégard. Modeling the quad-rotor mini-rotorcraft. In Quad Rotorcraft Control 2013. Springer London.

Number of grid = 100

Outline

• Motivation• Problem formulation• Low rank tensor decomposition• Alternating least squares & improvements• Example• Summary & future work

36

Summary

• Low rank tensor decomposition allows for high dimensional HJB representations and computations that scales linearly with dimensions

• SeALS improves ALS by alleviating the ill-condition issue (MATLAB code is available online at http://www.cds.caltech.edu/~yleong/)– Sequentially computing the solution– Rescaling boundary condition

37

Future Work

• Improve the algorithm using different discretization schemes (e.g. Chebyshev spectral differentiation)

• Analyze the algorithm more carefully to quantify convergence and accuracy

• Analyze the properties of the controller given by the solution of SeALS

• Apply to more difficult problems to find out when SeALSbreaks

38