Optimal Control and Hamilton-Jacobi Equations · HJB Equation State Constraints Active Mathematical Domains Motivated by Optimal Control Nonsmooth, Set-Valued Analysis, Variational

Value FunctionHJB Equation

State Constraints

Optimal Control and Hamilton-JacobiEquations

Helene Frankowska

CNRS and UNIVERSITE PIERRE et MARIE CURIE

Control and Optimization, Monastir, Tunisia

May 15-19, 2017

H. Frankowska Optimal Control


State Constraints

Deterministic Control System

{x(t) = f (t, x(t), u(t)), u(t) ∈ U a.e. in [0, 1] (CS)x(t0) = x0

U is a complete separable metric space, t denotes the time

f : [0, 1]× Rn × U → Rn, x0 ∈ Rn

Controls are Lebesgue measurable functions u(·) : [0, 1]→ U

A trajectory of (CS) is any absolutely continuous functionx ∈W 1,1([t0, 1];Rn) satisfying for some control u(·)

x(t) = f (t, x(t), u(t)) a.e. in [0,1]

We assume that f (t, x , ·) is continuous f (·, x , u) is measurable,f (t, x ,U) are closed and ∃ γ : [0, 1]→ R+ integrable such thatsupu∈U |f (t, x , u)| ≤ γ(t)(1 + |x |)



State Constraints




f : [0, 1]× Rn × U → Rn, x0 ∈ Rn



x(t) = f (t, x(t), u(t)) a.e. in [0,1]




State Constraints




f : [0, 1]× Rn × U → Rn, x0 ∈ Rn



x(t) = f (t, x(t), u(t)) a.e. in [0,1]




State Constraints

Semilinear Control SystemX is a separable Banach space. Consider the densely definedunbounded linear operator A - the infinitesimal generator of astrongly continuous semigroup S(t) : X → X ,

f : [0, 1]× X × U → X , x0 ∈ X

and the semilinear control system

x(t) = Ax + f (t, x(t), u(t)), u(t) ∈ U, x(t0) = x0

Its mild trajectory is defined by

x(t) = S(t − t0) x0 +∫ t

t0S(t − s) f (s, x(s), u(s)) ds ∀ t ∈ [t0, 1]

Many of the deterministic results were already adapted to theframework of semilinear control systems (controlled PDEs).Naturally this means some assumptions on semigroups:S(·) has to be compact to prove existence of optimal controls.



State Constraints

Optimal Control

The concept of optimal control can be described as the processof influencing the behavior of a dynamical system so as toachieve the desired goal:to maximize a profit, to minimize the energy, to get from one pointto another one, etc.

“After correctly stating the problem of optimal control and havingat hand some satisfactory existence theorems, augmented bynecessary conditions for optimality, we can consider that we havesufficiently substantial basis to study some special problems, as forinstance Moon Flight Problem”.

From a book on Optimal Control, 1969



State Constraints

Value Function of the Mayer Problemg : Rn → R ∪ {+∞}, ξ0 ∈ Rn. Consider the Mayer’s problem:

min {g(x(1)) | x is a trajectory of (CS), x(0) = ξ0}

The value function associated with this problem is defined by:∀ (t0, x0) ∈ [0, 1]× Rn

V (t0, x0) = inf{g(x(1)) | x is a trajectory of (CS), x(t0) = x0}

V (1, ·) = g(·). In general V is nonsmooth even for smooth data.

If x(·) is a trajectory of (CS), then for any t0 ≤ t1 ≤ t2 ≤ 1,V (t1, x(t1)) ≤ V (t2, x(t2)).x(·) is optimal if and only if V (t, x(t)) ≡ g(x(1)).

Dynamic Programming Principle: ∀ h > 0 such that t0 + h ≤ 1

V (t0, x0) = inf{V (t+h, x(t+h)) | x is a trajectory of (CS), x(t0) = x0}



State Constraints

Hamilton-Jacobi EquationThe Hamiltonian H : [0, 1]× Rn × Rn → R is defined by

H(t, x , p) = maxu∈U〈p, f (t, x , u)〉

If V ∈ C 1 it satisfies the Hamilton-Jacobi equation

−Vt(t, x) + H (t, x , −Vx (t, x)) = 0, V (1, ·) = g(·)

Optimal (feedback) control u(t, x) ∈ U is chosen by :

〈−Vx (t, x), f (t, x , u(t, x))〉 = H (t, x , −Vx (t, x))

If u(t, ·) is Lipschitz, then the solution x(·) of

x(t) = f (t, x(t), u(t, x)) a.e. in [0, 1], x(0) = ξ0

is optimal for the Mayer problem.Even if data are smooth, V is not differentiable.



State Constraints

Active Mathematical Domains Motivated byOptimal Control

Nonsmooth, Set-Valued Analysis, Variational Analysis (since1975)Solutions of HJB equations : viscosity and bilateral (since1983)Control under state constraints (since 2000)First order necessary optimality conditions (since 1957)Second order necessary optimality conditions (since 1965)Sensitivity relations in control (since 1986).....



State Constraints

Outline

1 Value Function of the Mayer Problem

2 Hamilton-Jacobi-Bellman Equation

3 State Constraints



State Constraints

Existence of Optimal Controls

Below we always assume that for a.e. t and ∀ r > 0, f (t, ·, u) iscr (t)-Lipschitz on B(0, r) ∀ u ∈ U, with integrable cr : [0, 1]→ R.

For the Mayer problem let (xi , ui ) be a minimizing sequence oftrajectory-control pairs. By Gronwall’s Lemma, {‖xi‖∞} isbounded and |xi (t)| ≤ γ(t)(‖xi‖∞ + 1).Take a subsequence {xij} converging uniformly to somex : [0, 1]→ Rn with xij converging weakly in L1([0, 1];Rn) to somey : [0, 1]→ Rn. Then ˙x = y , x(0) = ξ0 and

˙x(t) ∈ conv f (t, x(t),U) a.e.

where conv denotes convex hull.If f (t, x ,U) is convex, then, by the measurable selection theorem,there exists a control u such that ˙x(t) = f (t, x(t), u(t)) a.e.



State Constraints

Existence of Optimal ControlsIf g is lower semicontinuous, then

lim infj→∞

g(xij (1)) ≥ g(x(1))

and therefore u is optimal control.TheoremIf f (t, x ,U) are convex and g is lower semicontinuous, then for theMayer problem an optimal solution does exist.

For semilinear control systems only a part of this proof applies andone has either to assume, that S(t) is compact for all t > 0 or that

{(x , f (t, x , u)) | u ∈ U, x ∈ X}is convex. This last assumption is very strong.

For nonlinear stochastic control systems the convexity assumptionsare worse even in the finite dimensional framework.



State Constraints

Relaxation Theorem

x(t) ∈ conv f (t, x(t),U) a.e. in [t0, 1] (RS)has more W 1,1([t0, 1];Rn) solutions (relaxed trajectories) thanthe control system (CS) for the same initial condition x(t0) = x0.

Theorem

Let x : [t0, 1]→ Rn be a relaxed trajectory. Then for every ε > 0there exists a trajectory x of (CS) such that ‖x − x‖∞ ≤ ε andx(t0) = x(t0).

CorollaryIf g is continuous, then the infimum in the Mayer problem isattained by a relaxed trajectory and V = V rel , where

V rel (t0, x0) = min{g(x(1)) | x ∈W 1,1 satisfies (RS), x(t0) = x0}H. Frankowska Optimal Control


State Constraints

Regularity of Value Function

Under our assumptions, sets of solutions of (CS) and (RS) dependon initial state in a locally Lipschitz way. Hence

TheoremIf ∀ x ∈ Rn, f (t, x ,U) is convex and g is lowersemicontinuous, then V is lower semicontinuous.If g is continuous, then V is continuous.If g is locally Lipschitz, then V (t, ·) is locally Lipschitz forevery t ∈ [0, 1]. Furthermore if γ is essentially bounded, thenV is locally Lipschitz.If g ∈ C 1, f (t, ·, u) ∈ C 1, H(t, ·, ·) ∈ C 2 on Rn × (Rn\{0}),∇g never vanishes and for every initial datum the optimalsolution of the relaxed problem is unique, then V ∈ C 1.



State Constraints

Generalized Differentials of Nonsmooth Functions

Let ϕ : Rn → R ∪ {±∞} and x0 ∈ Rn be such that ϕ(x0) 6= ±∞.The Frechet superdifferential of ϕ at x0 is the closed convex set

∂+ϕ(x0) = {p ∈ Rn| lim supx→x0

ϕ(x)− ϕ(x0)− 〈p, x − x0〉|x − x0|

≤ 0}

The Frechet subdifferential of ϕ at x0 is the set:

∂−ϕ(x0) = {p ∈ Rn| lim infx→x0

ϕ(x)− ϕ(x0)− 〈p, x − x0〉|x − x0|

≥ 0}

We always have ∂+ϕ(x0) = −∂−(−ϕ)(x0).



State Constraints

Directional Derivatives of Nonsmooth Functions

The contingent epiderivative of ϕ at x0 in the direction v is

D↑ϕ(x0)(v) = lim infε→0+, v ′→v

ϕ(x0 + εv ′)− ϕ(x0)ε

and the contingent hypoderivative of ϕ at x0 in the direction v

D↓ϕ(x0)(v) = lim supε→0+, v ′→v

ϕ(x0 + εv ′)− ϕ(x0)ε

We always have

∂−ϕ(x0) = {p ∈ Rn | D↑ϕ(x0)(v) ≥ 〈p, v〉 ∀ v ∈ Rn}

∂+ϕ(x0) = {p ∈ Rn | D↓ϕ(x0)(v) ≤ 〈p, v〉 ∀ v ∈ Rn}



State Constraints

Sufficient Conditions for Optimality

Theorem

Assume that g is locally Lipschitz, 0 ≤ t0 < 1. Consider atrajectory z of (CS). If for a.e. t ∈ [t0, 1], ∃ p(t) ∈ Rn such that

(〈p(t), z(t)〉 , −p(t)) ∈ ∂+V (t, z(t))

then z is optimal at (t0, z(t0)).

Proof — ψ(t) := V (t, z(t)) is absolutely continuous. Lett ∈ [t0, 1] be such that ψ(t) and z(t) do exist and our assumptionis verified. Then ψ(t) ≥ 0 and

0 = 〈(〈p(t), z(t)〉,−p(t)), (1, z(t))〉 ≥ D↓V (t, z(t))(1, z(t))

≥ lim supε→0+V (t + ε, z(t + ε))− V (t, z(t))

ε= ψ(t)

Thus ψ(t) = 0 and therefore t 7→ V (t, z(t)) is constant.H. Frankowska Optimal Control


State Constraints

Contingent Inequalities

Theorem

Assume g is lsc, f is continuous in time uniformly in u andf (t, x ,U) are convex. Then for all (t0, x0) ∈ Dom(V ),

(i) t0 < 1 =⇒ infu∈U D↑V (t0, x0)(1, f (t0, x0, u)) ≤ 0(ii) t0 > 0 =⇒ supu∈U D↑V (t0, x0)(−1,−f (t0, x0, u)) ≤ 0(iii) t0 < 1 =⇒ infu∈U D↓V (t0, x0)(1, f (t0, x0, u)) ≥ 0

Proof — Consider a trajectory x of (CS) such that x(t0) = x0V (t, x(t)) ≡ g(x(1)) and εi → 0+ such that x(t0+εi )−x(t0)

εi→ v .

Then, by convexity, v ∈ f (t0, x0,U). ThereforeD↑V (t0, x0)(1, v) ≤ 0. Fix u0 ∈ U and consider the solution of(CS) with constant control u0. Then for ε > 0V (t0−ε, x(t0−ε))−V (t0, x0) ≤ 0 ≤ V (t0+ε, x(t0+ε))−V (t0, x0).Divide by ε and take lower and upper limits.



State Constraints

Notion of Solution to (HJB)

Corollary (Bilateral Solution)LSC value function satisfies V (1, ·) = g, ∀ (t, x) ∈ ]0, 1[×Rn,

∀ (pt , px ) ∈ ∂−V (t, x),−pt + H(t, x ,−px ) = 0∀ x ∈ Rn, V (0, x) = lim inft→0+, x→x V (t, x)∀ x ∈ Rn, V (1, x) = lim inft→1−, x→x V (t, x)

Corollary (Viscosity Solution)Continuous value function satisfies V (1, ·) = g, ∀ t ∈ ]0, 1[, x{

∀ (pt , px ) ∈ ∂−V (t, x),−pt + H(t, x ,−px ) ≥ 0∀ (t, pt , px ) ∈ ∂+V (t, x),−pt + H(x ,−px ) ≤ 0

If a function W satisfies conditions of either corollary, then it isequal to the value function.



State Constraints

Uniqueness of Solution to (HJB)

Steps of the proof of uniquenessDeduce from inequalities involving Frechet super/subdifferentials contingent inequalities. This step is not simplebecause D↑V (t0, x0)(·, ·) and D↓V (t0, x0)(·, ·) are not convexfunctions. It requires lots of nonsmooth analysis results.Show that any function W satisfying contingent inequalities isnondecreasing along trajectories of (CS). Use invariancetheorems.Show that for any function W satisfying contingentinequalities and any (t0, x0) ∈ dom(W ), there exists atrajectory x(·) of (CS) such that W (t, x(t)) is nonincreasing.Use the viability theorem.These two monotonicity properties imply W = V



State Constraints

State ConstraintsFor a given closed set K ⊂ Rn, state constraints are expressed by

x(t) ∈ K for all t ∈ [0, 1]

A trajectory x(·) as above is called a viable (or feasible) trajectoryof the control system. SK (x0) denotes set of all viable trajectoriesdefined on [0, 1] starting at x0 at time 0.

f (a0, u1), f (b0, u2), f (c0, u2) are tangent to K at a0, b0 and c0.At d0 there is no u ∈ U such that f (d0, u) is tangent to K .



State Constraints

Example : Milyutin, 2000 (also for higher order)

min

∫ T0

(x(t) + |u(t)|p

p

)dt, p > 1

x ′′′(t) = u(t), x(0) = ξ0, x ′(0) = ξ1, x ′′(0) = ξ2

u(t) ∈ R, x(t) ≥ 0

If an initial condition (ξ0, ξ1, ξ2) is admissible, then the optimalsolution exists and is unique.

For “most” of the admissible initial conditions (ξ0, ξ1, ξ2) thereexist T = T (ξ0, ξ1, ξ2) > 0 such that the optimal trajectoryreaches the boundary of constraints an infinite (countable) numberof times with T being their accumulation point.



State Constraints

Neighboring Feasible Trajectories (NFT) EstimatesDistance from z ∈ Rn to K , dK (z) := min{|z − y | : y ∈ K}.

Can we “control” trajectories violating state constraints ?What are sufficient conditions for the following property :

For every trajectory x (·) of (CS) with x (0) ∈ K there exists aviable trajectory x (·) ∈ SK (x (0)) satisfying the following NFTestimates in ‖ · ‖W 1,1

‖x − x‖W 1,1 ≤ C maxt∈[0,1]

dK (x (t))

or the NFT estimates in ‖ · ‖∞

‖x − x‖∞ ≤ C maxt∈[0,1]

dK (x (t))

where C depends on the magnitude of |x(0)|, but not on x (·) ?H. Frankowska Optimal Control


State Constraints




‖x − x‖W 1,1 ≤ C maxt∈[0,1]

dK (x (t))


‖x − x‖∞ ≤ C maxt∈[0,1]

dK (x (t))



State Constraints




‖x − x‖W 1,1 ≤ C maxt∈[0,1]

dK (x (t))


‖x − x‖∞ ≤ C maxt∈[0,1]

dK (x (t))



State Constraints




‖x − x‖W 1,1 ≤ C maxt∈[0,1]

dK (x (t))


‖x − x‖∞ ≤ C maxt∈[0,1]

dK (x (t))



State Constraints



State Constraints

K with C 1,1 Boundary

Let U be compact, f be time independent, and ∃ k ≥ 0 such thatf (·, u) is k-Lipschitz (uniformly in u).

Let K be the closure of an open set with C 1,1 boundary ∂K andassume the inward pointing condition (Soner, 1986) : ∀ x ∈ ∂K

minu∈U〈nx , f (x , u)〉 < 0,

where nx is the outward unit normal to K at x

Then NFT estimates in ‖ · ‖∞ hold true.

Can be extended to f measurable in time, by adding an inwardpointing condition on a neighborhood of ∂K .



State Constraints




minu∈U〈nx , f (x , u)〉 < 0,






State Constraints




minu∈U〈nx , f (x , u)〉 < 0,






State Constraints

Tangent and Normal ConesLet (X , d) be a metric space Aτ ⊂ X , τ ∈ T .

Limsupτ→τ0Aτ = {v ∈ X | lim infτ→τ0

dAτ (v) = 0}

Liminfτ→τ0Aτ = {v ∈ X | lim supτ→τ0

dAτ (v) = 0}

(the Peano-Kuratowski upper and lower limits)The contingent cone TK (x) to K ⊂ Rn at x ∈ K is the set ofv ∈ Rn such that ∃ εi → 0+, vi → v satisfying x + εi vi ∈ K

TK (x) = Limsupε→0+K − xε

The limiting normal cone to a closed subset K ⊂ Rn at x ∈ K isNL

K (x) = Limsupy→K x [TK (y)]−

→K stands for the convergence in K andTK (y)− = {p ∈ Rn | 〈p, v〉 ≤ 0 ∀ v ∈ TK (y)}



State Constraints

General Set KThe Clarke normal cone to K at x ∈ K

NK (x) := conv NLK (x)

Normalized normals : N1K (x) := NK (x) ∩ Sn−1

Generalized Inward Pointing Condition (HF, Rampazzo, 2000)

minu∈U

maxp∈N1

K (x)〈p, f (t, x , u)〉 < 0 ∀ x ∈ ∂K

Assume f (·, ·, u) is locally Lipschitz (uniformly in u). Then NFTestimates in ‖ · ‖∞ hold true.

A counterexample to NFT estimates in ‖ · ‖W 1,1 was given in P.Bettiol, A. Bressan & R. Vinter, SICON (2010) for f independentfrom t, x and K being an intersection of two half spaces in R2,



State Constraints





minu∈U

maxp∈N1

K (x)〈p, f (t, x , u)〉 < 0 ∀ x ∈ ∂K





State Constraints





minu∈U

maxp∈N1

K (x)〈p, f (t, x , u)〉 < 0 ∀ x ∈ ∂K





State Constraints





minu∈U

maxp∈N1

K (x)〈p, f (t, x , u)〉 < 0 ∀ x ∈ ∂K





State Constraints

Other Counterexamplesand a counterexample to NFT estimates in ‖ · ‖∞ when K is anintersection of two half spaces in R3 and f is independent from xand is measurable in time. May we expect a weaker logarithmicestimate when maxt∈[0,1] dK (x(t)) > 0 ?

‖x − x‖∞ ≤ C maxt∈[0,1]

dK (x(t))∣∣∣∣∣log( max

t∈[0,1]dK (x(t)))

∣∣∣∣∣or the Holder estimate : for some α ∈ (0, 1)

‖x − x‖∞ ≤ C maxt∈[0,1]

dK (x (t))α

There are a counterexample to the logarithmic estimates for findependent from x and continuous in time and acounterexample to Holder estimates for f independent from x

and measurable in time.H. Frankowska Optimal Control


State Constraints

Other Counterexamplesand a counterexample to NFT estimates in ‖ · ‖∞ when K is anintersection of two half spaces in R3 and f is independent from xand is measurable in time. May we expect a weaker logarithmicestimate when maxt∈[0,1] dK (x(t)) > 0 ?

‖x − x‖∞ ≤ C maxt∈[0,1]

dK (x(t))∣∣∣∣∣log( max

t∈[0,1]dK (x(t)))

∣∣∣∣∣or the Holder estimate : for some α ∈ (0, 1)

‖x − x‖∞ ≤ C maxt∈[0,1]

dK (x (t))α

There are a counterexample to the logarithmic estimates for findependent from x and continuous in time and acounterexample to Holder estimates for f independent from x

and measurable in time.H. Frankowska Optimal Control


State Constraints

Counterexample (Bettiol, Bressan, Vinter)

f (t, x ,U) = G := co {(2, 1), (−2, 1), (0, 0)}

K = {(x1, x2) : x2 ≥ |x1|}



State Constraints

Trajectory Violating State Constraintsx(t) = (x1(t), t)



State Constraints

Getting Rid of this Counterexample

G1 = G ∪ {(2, 3), (−2, 3)}

Then NFT estimates in W 1,1 hold true



State Constraints

Revisiting Classical ConditionLet K be the closure of an open set with C 2 boundary ∂K andassume the inward pointing condition : ∀ x ∈ ∂K , ∃ ux ∈ U with〈nx , f (x , ux )〉 < 0. Then for every u ∈ U with 〈nx , f (x , u)〉 ≥ 0

〈nx , f (x , ux )− f (x , u)〉 < 0

The generalization of this last inward pointing condition is :∀ t ∈ [0, 1] , ∀ x ∈ ∂K ,

(IPC)

∀ v ∈ U with maxn∈N1

K (x)〈n, f (t, x , v)〉 ≥ 0

∃w ∈ Liminf(s,y)→(t,x)f (s, y ,U)

maxn∈N1K (x)〈n,w − f (t, x , v)〉 < 0

The relaxed inward pointing condition : ∀ t ∈ [0, 1] , ∀ x ∈ ∂K ,

(IPCrel ) same as (IPC) but w ∈ Liminf(s,y)→(t,x) conv f (s, y ,U)



State Constraints


〈nx , f (x , ux )− f (x , u)〉 < 0


(IPC)


K (x)〈n, f (t, x , v)〉 ≥ 0


maxn∈N1K (x)〈n,w − f (t, x , v)〉 < 0





State Constraints


〈nx , f (x , ux )− f (x , u)〉 < 0


(IPC)


K (x)〈n, f (t, x , v)〉 ≥ 0


maxn∈N1K (x)〈n,w − f (t, x , v)〉 < 0





State Constraints

For f (·, ·, U) having a Closed Graph

Theorem (NODEA, 2013)Assume that f (·, ·,U) has closed graph and (IPC). Then NFTestimates in ‖ · ‖W 1,1 hold true.

If f is measurable in time, then a similar result is valid but auniform inward pointing condition has to be imposed on aneighborhood of the boundary of K .

Theorem (NODEA, 2013)Assume that f (·, ·,U) has closed graph and (IPCrel ). Then for anyviable relaxed trajectory x rel (·) and every δ > 0 there exists atrajectory x(·) of (CS) such thatx(0) = x rel (0), x(t) ∈ Int K ∀ t ∈ (0, 1] and ‖x(·)− x(·)‖∞ ≤ δ



State Constraints







State Constraints







State Constraints

Relaxed Semilinear Control System

x(t) ∈ Ax + conv f (t, x(t),U), x(t0) = x0

Its mild trajectory is defined by

x(t) = S(t − t0) x0 +∫ t

t0S(t − s) v(s) ds ∀ t ∈ [t0, 1]

where v(s) ∈ conv f (s, x(s),U).Assume X is Hilbert and K = Int K ⊂ X . Denote by Z the set ofpoints z ∈ X\∂K admitting a unique projection P∂K (z) on ∂K .For every z ∈ Z , set

nz = z − P∂K (z)‖z − P∂K (z)‖X

sgn(doriented∂K (z)) .



State Constraints

Relaxation of Semilinear Control Systems underState Constraint, MCRF, 2016

The analogue of relaxation theorem is valid under the followinginward pointing condition:

∀ x ∈ ∂K , ∃ η, ρ, M > 0 such that ∀ t ∈ [0, 1], ∀ x ∈ K ∩ B(x , η),∀ v ∈ U satisfying sup

τ≤η, z∈Z∩B(x ,η)〈nz , S(τ) f (t, x , v)〉 ≥ 0, we have{

v ∈ U : ‖f (t, x , v)− f (t, x , v)‖ ≤ M ,

supτ≤η, z∈Z∩B(S(τ)x ,η)

〈nz ,S(τ) (f (t, x , v)− f (t, x , v))⟩< −ρ

}6= ∅ .

If K is convex, then Z can be replaced by X\K .



State Constraints

Relaxation of Semilinear Control Systems underState Constraint, MCRF, 2016

The analogue of relaxation theorem is valid under the followinginward pointing condition:

∀ x ∈ ∂K , ∃ η, ρ, M > 0 such that ∀ t ∈ [0, 1], ∀ x ∈ K ∩ B(x , η),∀ v ∈ U satisfying sup

τ≤η, z∈Z∩B(x ,η)〈nz , S(τ) f (t, x , v)〉 ≥ 0, we have{

v ∈ U : ‖f (t, x , v)− f (t, x , v)‖ ≤ M ,

supτ≤η, z∈Z∩B(S(τ)x ,η)

〈nz ,S(τ) (f (t, x , v)− f (t, x , v))⟩< −ρ

}6= ∅ .

If K is convex, then Z can be replaced by X\K .



State Constraints

Value Function of the Constrained Mayer Problem

For g : Rn → R⋃{+∞} the Mayer problem under state

constraints is

minimize {g(x(1)) | x(·) ∈ SK (ξ0)}

The value function is defined by

V (t0, x0) = inf{g(x(1)) | x(·) is a viable trajectory of (CS), x(t0) = x0}

The relaxed Mayer problem is :

min {g(x(1)) | x(·) is a viable relaxed trajectory of (RS), x(0) = ξ0}



State Constraints

Value Function of the Constrained Mayer Problem

For g : Rn → R⋃{+∞} the Mayer problem under state

constraints is

minimize {g(x(1)) | x(·) ∈ SK (ξ0)}

The value function is defined by

V (t0, x0) = inf{g(x(1)) | x(·) is a viable trajectory of (CS), x(t0) = x0}

The relaxed Mayer problem is :

min {g(x(1)) | x(·) is a viable relaxed trajectory of (RS), x(0) = ξ0}



State Constraints

Lipschitz Continuity of the Value Function

Corollary (NODEA, 2013)Assume (IPCrel ), f is continuous in t, uniformly in u, γ isessentially bounded and that g is locally Lipschitz.Then V is locally Lipschitz on [0, 1]× K and is equal to the valuefunction of the relaxed problem.Furthermore if x(·) is an optimal solution to the Mayer problem,then it is also optimal for the relaxed problem.

Outward Pointing Condition (OPC)

∀ t ∈ [0, 1] , x ∈ ∂K , v ∈ U with minn∈N1K (x) 〈n, f (t, x , v)〉 ≤ 0

∃ u ∈ U, minn∈N1K (x)∩Sn−1 〈n, f (t, x , u)− f (t, x , v)〉 ≥ 0.



State Constraints

Lipschitz Continuity of the Value Function

Corollary (NODEA, 2013)Assume (IPCrel ), f is continuous in t, uniformly in u, γ isessentially bounded and that g is locally Lipschitz.Then V is locally Lipschitz on [0, 1]× K and is equal to the valuefunction of the relaxed problem.Furthermore if x(·) is an optimal solution to the Mayer problem,then it is also optimal for the relaxed problem.

Outward Pointing Condition (OPC)

∀ t ∈ [0, 1] , x ∈ ∂K , v ∈ U with minn∈N1K (x) 〈n, f (t, x , v)〉 ≤ 0

∃ u ∈ U, minn∈N1K (x)∩Sn−1 〈n, f (t, x , u)− f (t, x , v)〉 ≥ 0.



State Constraints

Constrained HJB Equation under (IPC)

Theorem (Uniqueness of constrained viscosity solutions,Calc. Var.& PDEs, 2013)Let W : [0, 1]× K → R. Assume (IPC) and that g and H arecontinuous and W (1, ·) = g(·). Then the following statements areequivalent:(a) W is the value function of the Mayer problem;(b) W is continuous and

(i) −pt + H(t, x ,−px ) ≤ 0, ∀ (pt , px ) ∈ ∂+W (t, x),∀ (t, x) ∈]0, 1[×Int K ;(ii) −pt + H(t, x ,−px ) ≥ 0, ∀ (pt , px ) ∈ ∂−W (t, x),∀ (t, x) ∈]0, 1[×K .

In other words, V is the unique continuous viscosity solution of theconstrained HJB equation.



State Constraints

Constrained HJB Equation under (OPC)

Theorem (Uniqueness of constrained bilateral solutions, Calc.Var.& PDEs, 2013)Assume (OPC), that H is continuous, g is lsc and letW : [0, 1]× K → R ∪ {+∞}, W (1, ·) = g(·). If f (t, x ,U) areconvex, then the following two statements are equivalent:(a) W is the value function of the Mayer problem;(b) W is lower semicontinuous and

(i) −pt + H(t, x ,−px ) = 0, ∀ (pt , px ) ∈ ∂−W (t, x),∀ (t, x) ∈]0, 1[×Int K;(ii) −pt + H(t, x ,−px ) ≥ 0, ∀ (pt , px ) ∈ ∂−W (t, x),∀ (t, x) ∈ (]0, 1[×∂K ) ∪ ({0} × K );(iii) lim inf (t ′, x ′)→ (1, x)

(t ′, x ′) ∈]0, 1[×Int K

W (t ′, x ′) = g(x), ∀ x ∈ K.



State Constraints

Conclusions and Open Questions

Value function is the unique solution ofHamilton-Jacobi-Bellman equationSame in the presence of state constraintsFor the controlled PDEs that can be reduced to semilinearcontrol systems without state constraints HJB were alreadyinvestigatedMany Open Questions in : HJB associated to semilinearcontrol systems in the presence of state constraintsMany Open Questions in : Stochastic Optimal Control

NFT theorems for semilinear control systems - to appear inESAIM: Control, Optimisation and Calculus of Variations


Documents

Optimal Control and Hamilton-Jacobi Equations · HJB Equation State Constraints Active Mathematical Domains Motivated by Optimal Control Nonsmooth, Set-Valued Analysis, Variational