The Envelope Theorem, Euler and Bellman Equations,

without Differentiability∗

Ramon Marimon † Jan Werner ‡

September 18, 2019

Click Here for the Most Recent Version

Abstract

We extend the envelope theorem, the Euler equation, and the Bellman equation to dy-

namic optimization problems where binding constraints can give rise to non-differentiable

value functions and multiplicity of Lagrange multipliers. The envelope theorem – an ex-

tension of Milgrom and Segal’s (2002) theorem – establishes a relation between the Euler

and the Bellman equation. We show that solutions and multipliers of the Bellman equation

may fail to satisfy the Euler equations of the infinite-horizon problem. In standard dynamic

programming the failure of Euler equations results in inconsistent multipliers, but not in

non-optimal outcomes. However, in dynamic optimization problems with forward-looking

constraints, i.e., recursive contracts of Marcet and Marimon (2019), this failure can result in

inconsistent promises and non-optimal outcomes. We introduce an envelope selection con-

dition which guarantees that solutions generated from the Bellman equation satisfy the Euler

equations with or without forward-looking constraints. A recursive method of solving dy-

namic optimization problems with non-differentiable value function involves expanding the

co-state and imposing the envelope selection condition.∗This is a substantially revised version of previous drafts. We thank the participants in seminars and conferences

where our work has been presented and, in particular Davide Debortoli and Daisuke Oyama, for their comments.†European University Institute, UPF-Barcelona GSE, CEPR and NBER‡University of Minnesota

1

Keywords: limited enforcement, dynamic programming, Envelope Theorem, Euler equation,

Bellman equation, sub-differential calculus.

JEL Classification: C02, C61, D90, E00.

2

1 Introduction

The Euler equation and the Bellman equation are the two basic tools used to analyse dynamic

optimization problems. Euler equations are the first-order inter-temporal necessary conditions for

optimal solutions and, under standard concavity-convexity assumptions, they are also sufficient

conditions, provided that a transversality condition holds. Euler equations are usually second-

order difference equations. The Bellman equation allows the transformation of an infinite-horizon

optimization problem into a recursive problem, resulting in time-independent policy functions

determining the actions as functions of the states. The envelope theorem provides the bridge

between the Bellman equation and the Euler equations, establishing the necessity of the latter for

the former. This bridge allows us to reduce the second-order difference equation system of Euler

equations to a first-order system, fully determined by the policy function of the Bellman equation

with corresponding initial conditions, provided that the value function is differentiable.

Differentiability makes the bridge between the Bellman equation and the Euler equation tight.

If the value function is differentiable, the state provides univocal information about the derivative

and, therefore, the inter-temporal change of values across states (Bellman) is uniquely associated

with the change of marginal values (Euler) via the envelope theorem – as a result, the enve-

lope theorem allows for the passage of properties between the Euler and the Bellman equations.

That is, the necessity and sufficiency properties of the Euler equations on the one hand, and the

properties of Bellman policy functions on the other. However, the value function may not be

differentiable when constraints are binding and, in this case, knowing the state and its value does

not provide univocal information about the derivative.

Recursive methods of dynamic programming have been widely applied in macroeconomics

over the last 30 years since the publication of Stokey et al. (1989). Usual assumptions such as

interiority of optimal paths imply differentiability of the value function and enable application of

the standard envelope theorem. However, the issue of non-differentiability cannot be ignored in

a wide range of currently used models. Models where households, firms, or countries, may face

binding constraints at optimal choices are, nowadays, more the norm than the exception.

The differentiability problem caused by binding constraints is even more pervasive in dy-

namic models with forward-looking constraints.1 Incentive constraints stemming from limited

1Koeppl (2006) provided an early example of non-differentiabilty of the value function in a dynamic model of

3

enforcement or moral hazard often take the form of forward-looking constraints where feasibility

of current actions depends on future actions. Optimization problems with such constraints may

have time-inconsistent solutions rendering the standard dynamic programming methods based on

the Bellman equation inapplicable. Marcet and Marimon (2019) developed an alternative method

- called the theory of recursive contracts - that recovers recursive structure of an optimization

problem with forward-looking constraints and enables characterization and computation of the

solutions. The method relies on a saddle-point Bellman equation with an extended co-state.

This paper extends the standard Bellman’s theory of dynamic programming and the theory

of recursive contracts of Marcet and Marimon (2019) to encompass non-differentiability of the

value function resulting from the presence of binding constraints. To that end, we first generalize

the envelope theorem – a result of significant interest in its own right – using methods of sub-

differential calculus and saddle-point analysis (see Rockafellar (1970, 1981)). The new envelope

theorem allows us to reconstruct the bridge between the Euler and Bellman, or saddle-point

Bellman, equations when the value function is non-differentiable under binding constraints.

We proceed in three steps. First, we derive the envelope theorem for static constrained op-

timization problems without assuming differentiability of the value function or interiority of the

solutions. Our result extends the envelope theorem for directional derivatives of Milgrom and

Segal (2002, Corollary 5) beyond maximization problems with concave objective and constraint

functions. Further, we provide characterizations of the superdifferential of a concave value func-

tion and the subdifferential of a convex value function. From the envelope theorem, we derive

several sufficient conditions for differentiability of the value function. For example, if there is

a unique saddle-point, the value function is differentiable and the standard form of the enve-

lope theorem holds. A sufficient condition for differentiability of concave value function is that

the saddle-point multiplier be unique. For convex value function, a sufficient condition for dif-

ferentiability is that the solution be unique. Our envelope theorem applies to value functions

arising in recursive dynamic programming. We provide a characterization of the superdifferen-

tial of concave value function of the Bellman equation. The well-known result of Benveniste

and Scheinkman (1979) establishing differentiability of the concave value function at an interior

solution follows from our results because the saddle-point multiplier is zero and hence unique at

risk-sharing with enforcement constraints.

4

an interior solution.2

Second, we turn to the Euler equations of the standard dynamic programming. Unless the so-

lution is interior, marginal values of the constraints (i.e. the Lagrange multipliers) are part of the

Euler equations. Applying the envelope theorem of Section 3, we show how the Euler equations

can be derived from the Bellman equation without assuming differentiability of the value func-

tion. It follows that whenever there are multiple Lagrange multipliers of the Bellman equation

(at which points the value function is not differentiable), some sequences of multipliers may fail

to satisfy the Euler equations and therefore not be saddle-point multipliers of the infinite-horizon

optimization. This potential inconsistency of multipliers does not affect the solutions of the Bell-

man equation in the absence of forward-looking constraints. We introduce the envelope selection

condition which guarantees that multipliers generated from the Bellman equation satisfy the Eu-

ler equations. The envelope selection is a consistency condition on multipliers and subgradients

of the value function. The recursive method of solving dynamic programming problems can be

extended to provide solutions with consistent multipliers by expanding the co-state to include a

subgradient of the value function and taking a policy function from the Bellman equation that

satisfies the envelope selection condition. If the value function is everywhere differentiable, this

co-state and the selection condition are redundant since the subgradient is unique.

Third, we characterise the relationship between the Euler equations and the saddle-point Bell-

man equation of Marcet and Marimon (2019) in a partnership problem with forward-looking

participation constraints. The problem of inconsistent multipliers of the Bellman equation in

the standard dynamic programming becomes inconsistency of solutions and multipliers of the

saddle-point Bellman equation in recursive contracts if the value function is not differentiable.

This was pointed out by Messner and Pavoni (2004) who constructed an example of a partnership

problem with non-differentiable value function in which the saddle-point Bellman equation gen-

erates non-optimal outcomes. Applying the envelope theorem without differentiability, we derive

the envelope selection condition which guarantees that solutions and multipliers generated from

the saddle-point Bellman equation satisfy the Euler equations. The recursive method of solving

recursive contracts, i.e., an algorithm, involves expanding the co-state to include a subgradient of

2The result of Rincon-Zapatero and Santos (2009) that the value function in concave dynamic programming

problems is differentiable if the multiplier is unique follows from our results, as well.

5

the value function and imposing the envelope selection condition.3 The envelope selection con-

dition is equivalent to the intertemporal consistency condition of Marcet and Marimon (2019). If

the value function is differentiable, the selection condition is redundant and, accordingly, there is

no need to expand the co-state.4

Dynamic macroeconomics abounds in optimization problems with forward-looking constraints

(see Ljungqvist and Sargent (2018)). Recursive contracts of Marcet and Marimon (2019) are

nowadays the leading method to formulate and solve dynamic incentive and limited-commitment

problems, see Messner, Pavoni and Sleet (2012) and Mele (2014).

The envelope theorem has long history in mathematics and economics, as well. Earlier ver-

sions of it for non-parametric constraints have been known as Danskin’s Theorem (see Oyama

and Takenawa (2018) and references therein). Our approach provides representation of direc-

tional derivatives in optimization problem with parametric constraints using saddle-points of the

Lagrangian. We extend the results of Milgrom and Segal (2002) by discarding the assumptions

of concave objective and constraint functions. A different approach using multipliers of the

Kuhn-Tucker first-order conditions instead of saddle-point multipliers in problems with differ-

entiable objective and constraint functions (see Dubeau and Govin (1982), Rockafellar (1984),

and Morand et al. (2015, 2018)) leads to bounds on directional derivatives because Kuhn-Tucker

multipliers are a superset of saddle-point multipliers. Envelope theorems with saddle-point rep-

resentation of directional derivatives are directly applicable to value functions of the Bellman

equation in dynamic programming and recursive contracts, while those with Kuhn-Tucker multi-

pliers are not since they rely on differentiability of the objective function and, in general, provide

only bounds. Extensions of the envelope theorem to non-smooth optimization problems using

generalized Kuhn-Tucker multipliers can be found in Morand et al. (2015, 2018). If the (general-

ized) Kuhn-Tucker multiplier is unique - for instance, when the constraints are non-binding - or

when the objective and constraint functions are concave, the set of Kuhn-Tucker multipliers co-

incides with saddle-point multipliers and the former provide exact envelope representation, too.

3Since the subgradient is independent of the action, it can be substituted as co-state by a suitable Largange

multiplier.4Cole and Kubler (2012) identify the consistency problem of the saddle-point method with forward-looking

constraints, but offer a way to resolve it for a limited class of models (i.e., two-agent partnership problems). There

is no use of the envelope theorem in their approach.

6

Recent papers by Oyama and Takenawa (2018) and Clausen and Strub (2016) provide further

explorations of the conditions for differentiability of the value function.

The paper is organized as follows: Sections 2 and 3 cover the static envelope theorem and

provide examples of applications of the results. Section 4 contains the analysis of the standard

dynamic programming. It includes an example where the problem of inconsistency of multipli-

ers arises. Section 5 extends the theory of recursive contracts for the partnership problem with

forward-looking participation constraints. It includes an example where the value function is

non-differentiable and the saddle-point Bellman equation may generate non-optimal outcomes.

2 The Envelope Theorem

We consider the following parametric constrained optimization problem:

maxy∈Y

f(x, y) (1)

subject to

h1(x, y) ≥ 0, . . . , hk(x, y) ≥ 0. (2)

Parameter x lies in the set X ⊂ <m. Choice variable y lies in Y ⊂ <n. Objective function f is a

real-valued function on Y ×X. Each constraint function hi is a real-valued function on Y ×X.5

The value function of the problem (1–2) is denoted by V (x).

The Lagrangian function associated with (1–2) is

L(x, y, λ) = f(x, y) + λh(x, y), (3)

where λ ∈ <k+ is a vector of (positive) multipliers6. The saddle-point of L is a pair (y∗, λ∗),

where y∗ ∈ Y is called a saddle-point solution and λ∗ ∈ <k+ a saddle-point multiplier, such that

L(x, y, λ∗) ≤ L(x, y∗, λ∗) ≤ L(x, y∗, λ), (4)

for every y ∈ Y and λ ∈ <k+. We shall use the following notation for saddle-point value

L(x, y∗, λ∗) = SP minλ≥0

maxy∈YL(x, y, λ), (5)

5Note that optimization problems with equality constraints can be represented in form (1–2) by taking hi = −hjfor some i and j.

6We use the product notation: λh(x, y) =∑k

i=1 λihi(x, y).

7

with SP denoting the saddle-point operator defined by (4) with minimization over λ and maxi-

mization over y.7

The set of saddle-points of L at x is a product of two sets and is denoted by Y ∗(x) × Λ∗(x)

where Y ∗(x) ⊂ Y and Λ∗(x) ⊂ <k+, see Lemma 1, Appendix B. Since the slackness condition

λ∗ihi(x, y∗) = 0 holds for every saddle-point (y∗, λ∗), for every i, it follows that the value of

(1–2) equals the saddle-point value, that is,

V (x) = L(x, y∗, λ∗) = SP minλ≥0

maxy∈YL(x, y, λ), (6)

Furthermore, the slackness condition implies that if the ith constraint is not binding, that is,

hi(x, y∗) > 0 for a saddle-point solution y∗, then λ∗i = 0 for every saddle-point multiplier λ∗.

We shall impose the following conditions:

A1. Y is convex; f and hi are continuous functions of (x, y), for every i.

A2. The constraint set Γ(x) = {y ∈ Y : h(x, y) ≥ 0} is compact for every x ∈ X.

A3. The correspondence Γ : X → Y is continuous.

A4. For every x ∈ X and every i, there exists yi ∈ Y such that hi(x, yi) > 0 and hj(x, yi) ≥ 0

for j 6= i.

A5. Y ∗(x)× Λ∗(x) 6= ∅ for every x ∈ X.

Assumption A1 is standard. A2 guarantees the existence of a solution to (1–2) for every x.

A3 guarantees continuity of the value function while A4 is a weak form of the Slater’s condition.

Assumption A5 says that the set of saddle-points of L is nonempty for every x.

Assumption A3 asserts lower and upper hemi-continuity of Γ. A sufficient condition for up-

per hemi-continuity is that Γ is uniformly compact near x for every x ∈ X, that is, there is a

neighborhood Nx of x such that the set⋃x′∈Nx

Γ(x′) is compact. A sufficient condition for lower

hemi-continuity is that Γ(x) = cl{y ∈ Y : h(x, y) >> 0}, where cl denotes the closure.8 As-

7 This SP notation was introduced in Marcet and Marimon (2019). The min and max operators merely indicate

which variables are being minimised and maximised, respectively, in the saddle-point.8The reason is that the correspondence {y ∈ Y : h(x, y) >> 0} is lower hemi-continuous under A1 and so is its

closure.

8

sumption A5 holds if functions f and hi are concave in y and the Slater’s condition A4 holds.9 If

A5 holds, then the saddle-point solutions Y ∗(x) are precisely the solutions to (1–2), see the proof

of Lemma 1. We refer to them simply as solutions. If f and hi are differentiable in y, then the

Kuhn-Tucker first-order conditions hold for every saddle-point of L and the set of saddle-point

multipliers Λ∗(x) is a subset of the set of Kuhn-Tucker multipliers. Those two sets of multi-

pliers are equal if functions f and hi are differentiable and concave in y (see Theorem 28.3 in

Rockafellar (1970)).

The envelope theorem is best stated in terms of directional derivatives. We first consider one-

dimensional parameter set X – a convex subset of the real line. Directional derivatives are then

the left- and right-hand derivatives. The right- and left-hand derivatives of the value function V

at x are

V ′(x+) = limt→0+

V (x+ t)− V (x)

t, (7)

and

V ′(x−) = limt→0−

V (x+ t)− V (x)

t, (8)

if the limits exist.

We have the following result:

Theorem 1: Suppose that X ⊂ <, conditions A1-A5 hold, and partial derivatives ∂f∂x

and ∂hi∂x

are

continuous functions of (x, y). Then the value function V is right- and left-hand differen-

tiable at every x ∈ intX and the directional derivatives are

V ′(x+) = maxy∗∈Y ∗(x)

minλ∗∈Λ∗(x)

[∂f∂x

(x, y∗) + λ∗∂h

∂x(x, y∗)

](9)

and

V ′(x−) = miny∗∈Y ∗(x)

maxλ∗∈Λ∗(x)

[∂f∂x

(x, y∗) + λ∗∂h

∂x(x, y∗)

], (10)

where the order of maximum and minimum does not matter.

Theorem 1 generalizes Corollary 5 in Milgrom and Segal (2002) by discarding assumptions

of concavity of functions f and hi and compactness of set Y. It is worth pointing out that differ-

entiability of functions f and hi with respect to the variable y is not assumed in Theorem 1. This9Lemma 36.2 in Rockafellar (1970) provides necessary and sufficient conditions for existence of a saddle-point

without concavity of f or hi.

9

will be important in applications to dynamic programming in Sections 4 and 5. If f and hi were

continuously differentiable functions of y, then the weak Slater’s condition A4 could be replaced

by a Constrained Qualification condition, in which case the condition A3 could be weakened to

uniform compactness near x for every x.10 A short discussion of CQ conditions will be given in

Section 3.1.

For a multi-dimensional parameter set X in <m, the directional derivative of the value func-

tion V at x ∈ X in the direction x ∈ <m such that x+ x ∈ X is defined as

V ′(x; x) = limt→0+

V (x+ tx)− V (x)

t.

If partial derivatives of V exist, then the directional derivative V ′(x; x) is equal to the scalar

product DV (x)x, where DV (x) is the vector of partial derivatives, i.e. the gradient vector.

Theorem 1 can be applied to the single-variable value function V (t) ≡ V (x + tx) for which

it holds V ′(0+) = V ′(x; x). If Dxf(x, y) and Dxhi(x, y) are continuous functions of (x, y), then

it follows that the directional derivative of V is

V ′(x; x) = maxy∗∈Y ∗(x)

minλ∗∈Λ∗(x)

[Dxf(x, y∗) + λ∗Dxh(x, y∗)

]x, (11)

and the order of maximum and minimum does not matter.

3 Differentiability and Subdifferentials of the Value Function

3.1 Differentiability

The value function V on X ⊂ < is differentiable at x if the one-sided derivatives are equal to

each other. Sufficient conditions for differentiability can be obtained from Theorem 1.

Corollary 1: Under the assumptions of Theorem 1, each of the following conditions is sufficient

for differentiability of value function V at x ∈ intX:

(i) there is a unique saddle-point,10It follows from Dubeau and Govin (1982, Section 3), that the key results of Lemma 1, Appendix B - in particular,

that the solution and the saddle-point correspondences Y ∗ and Λ∗ are compact-valued and upper hemi-continuous -

continue to hold under assumption A1, uniform compactness of Γ and a Constrained Qualification condition.

10

(ii) there is a unique solution and hi does not depend on x for every i.

(iii) there is a solution with non-binding constraints, and ∂f∂x

does not depend on y.11

(iv) there is a unique saddle-point multiplier and ∂f∂x

and ∂hi∂x

do not depend on y, for every

i.

Condition (i) holds if there is a unique saddle-point solution with non-binding constraints so

that zero is the unique saddle-point multiplier. The condition of ∂f∂x

or ∂hi∂x

not depending on y

in part (iv) is essentially the additive separability of f and hi in x and y. Under the separability

condition, uniqueness of multiplier is necessary and sufficient for differentiability of the value

function. A result related to Corollary 1 (iii) can be found in Kim (1993).

A sufficient condition for uniqueness of saddle-point solution to (1–2) is that f be strictly

concave and hi be concave in y. A sufficient condition for uniqueness of saddle-point multiplier

is the following standard Constrained Qualification condition which implies uniqueness of the

Kuhn-Tucker multiplier:

CQ (1) f and hi are continuously differentiable functions of y,

(2) vectors Dyhi(x, y∗) for i ∈ I(x, y∗) are linearly independent, where

I(x, y∗) = {i : hi(x, y∗) = 0} is the set of binding constraints.

A weaker form of Constrained Qualification which is necessary and sufficient for uniqueness of

Kuhn-Tucker multiplier can be found in Kyparisis (1985).12 Note that the CQ condition holds

vacuously for solution y∗ with non-binding constraints.

Under condition (i) or (iv) of Corollary 1, the derivative of the value function is

V ′(x) =∂f

∂x(x, y∗) + λ∗

∂h

∂x(x, y∗). (12)

Under condition (ii) or (iii), it holds

V ′(x) =∂f

∂x(x, y∗). (13)

11It is sufficient that the constraints with hi depending on x are non-binding. Other constraints may bind.12That condition is the strict Mangasarian-Fromovitz CQ condition. The weak MFCQ condition is necessary and

sufficient for compactness of the set of Kuhn-Tucker multipliers, see Dubeau and Govin (1982).

11

For the multi-dimensional parameter set X in <m, the value function is differentiable if

V ′(x; x) = −V ′(x;−x) for every x ∈ <m. This holds under any of the sufficient conditions

of Corollary 1 with Dxf and Dxhi substituted for partial derivatives in (iv) and (v). If V is differ-

entiable, then the gradient DV (x) is well defined, and the multi-dimensional counterpart of (12)

is

DV (x) = Dxf(x, y∗) + λ∗Dxh(x, y∗). (14)

3.2 Concave and Convex Value Functions

If the value function on the parameter set X in <m is concave or convex, the envelope theorem

can be stated using the superdifferential or the subdifferential. We consider the concave case first.

Sufficient conditions for V to be concave are stated in the following well-known result, the proof

of which is omitted.

Proposition 1. If the objective function f and all constraint functions hi are concave functions

of (x, y) on Y ×X, then the value function V is concave.

The superdifferential ∂V (x) of the concave value function V is the set of all vectors φ ∈ <m

such that

V (x′) + φ(x− x′) ≤ V (x) for every x′ ∈ X.

We have the following:

Theorem 2: Suppose that conditions A1-A5 hold, derivatives Dxf and Dxhi are continuous

functions of (x, y) for every i, and V is concave. Then

∂V (x) =⋂

y∗∈Y ∗(x)

⋃λ∗∈Λ∗(x)

{Dxf(x, y∗) + λ∗Dxh(x, y∗)} (15)

for every x ∈ intX .

Sufficient conditions for differentiability of concave value function follow from Theorem 2.

Corollary 2: Under the assumptions of Theorem 2, the following hold for x ∈ intX:

12

(i) If the saddle-point multiplier is unique, then value function V is differentiable at x

and (14) holds for every y∗ ∈ Y ∗(x).

(ii) If hi does not depend on x for every i, then value function V is differentiable at x and

(14) holds for every y∗ ∈ Y ∗(x).

In Corollary 2 (i), it is sufficient that the multiplier is unique for the constraints with hi

depending on x. Corollary 2 (i) implies that the value function is differentiable if there is a

solution with non-binding constraints - those that depend on x - for then the unique saddle-point

multiplier is zero. A saddle-point multiplier may be unique even if some constraints are binding.

Examples are given in Section 3.3. Corollary 2 (ii) extends Corollary 3 in Milgrom and Segal

(2002) to parametrized constraints.

We now provide a similar characterization for convex value functions. Sufficient conditions

for V to be convex are stated without proof in the following:

Proposition 2. If the objective function f(y, ·) is convex in x for every y ∈ Y and all constraint

functions hi are independent of x, then the value function V is convex.

If V is convex, then the subdifferential ∂V (x) is the set of all vectors φ ∈ <m such that13

V (x′) + φ(x− x′) ≥ V (x) for every x′ ∈ X.

We have the following:

Theorem 3: Suppose that conditions A1-A5 hold, derivatives Dxf and Dxhi are continuous

functions of (x, y) for every i, and V is convex. Then

∂V (x) =⋂

λ∗∈Λ∗(x)

co( ⋃y∗∈Y ∗(x)

{Dxf(x, y∗) + λ∗Dxh(x, y∗)}), (16)

for every x ∈ intX , where co( ) denotes the convex hull.

Sufficient conditions for differentiability of convex value function follow from Theorem 3.

13We use the same notation for the superdifferential and the subdifferential as is customary in the literature.

13

Corollary 3: Under the assumptions of Theorem 3, if the saddle-point solution is unique at x ∈intX , then value function V is differentiable and (14) holds for every λ∗ ∈ Λ∗(x).14

Results of this section and Section 2 can be easily extended to minimization problems. An

extension to saddle-point problems is presented in Appendix A.

3.3 Examples

Example 1 (Perturbation of constraints): suppose that the objective function f in (1) is in-

dependent of the parameter x and constraint functions are of the form hi(x, y) = hi(y) − xi.

Conditions A1-A4 are assumed to hold. This optimization problem is a perturbation of the non-

parametric problem in which x is set equal to zero. Rockafellar (1970), Chapter 29, provides an

extensive discussion of the concave perturbed problem.

Corollary 1 (iv) implies that if the saddle-point multiplier λ∗ exists and is unique, then the

value function is differentiable and DV (x) = −λ∗ by (14). If f and hi are concave for every i,

then the set of saddle-points is non-empty and V is concave. By Theorem 2, the superdifferential

of V is ∂V (x) = −Λ∗(x).15

Example 2 (A planner’s problem): consider the resource allocation problem in an economy

with k agents. The planner’s problem is

max{ci}

k∑i=1

µiui(ci) (17)

s.t.n∑i=1

ci ≤ x, (18)

ci ≥ 0, ∀i,

where µ = (µ1, . . . , µk) ∈ <k++ is a vector of welfare weights and x ∈ <L++ represents total

resources. Utility functions ui are continuous so that conditions A1-A4 hold. Let V (x, µ) be the

value of (17) as function of weights µ and total resources x. It follows from Corollary 1 (iv) that14Oyama and Takenawa (2017) provide an example showing that the assumption of continuity of the derivative

Dxf in Corollary 3 cannot be dispensed with.15See Theorem 29.1 and Corollary 29.1.3 in Rockafellar (1970) for the respective results in the concave perturbed

problem. Some extensions can be found in Bonnisseau and Le Van (1996).

14

V is differentiable in x if the saddle-point multiplier of constraint (18) exists and is unique. If

utility functions ui are differentiable, then the CQ condition holds, implying that the multiplier

is unique. The derivative is DxV = λ∗, where λ∗ is the multiplier of the constraint (18). V is

a convex function of µ. The subdifferential ∂µV is (by Theorem 3) the convex hull of the set of

vectors (u1(c∗1), . . . , uk(c∗k)) over all saddle-point solutions c∗ to (17). V is differentiable in µ if

the saddle-point solution exists and is unique.

Consider an example with L = 1, k = 2, and ui(c) = c. Let the welfare weights be

parametrized by a single parameter µ so that µ1 = µ and µ2 = 1 − µ with 0 < µ < 1. The

value function is V (x, µ) = max{µ, 1−µ}x. It is differentiable with respect to µ at every µ 6= 12

and every x. The solution c∗ is unique for every µ 6= 12. V is not differentiable with respect to µ

at µ = 12. The left-hand directional derivative at µ = 1

2is −x while the right-hand derivative is x

in accordance with Theorem 1. V is everywhere differentiable with respect to x.

4 Dynamic Optimization and Euler Equations

In this section we extend the standard results of dynamic programming to non-differentiable

value functions. Using the results of Sections 2 and 3, we show how to derive Euler equations

from the Bellman equation without differentiability of the value function. If the value function is

non-differentiable, then there may be sequences of solutions and multipliers generated from the

Bellman equation for which Euler equations do not hold because multipliers are inconsistent. We

develop a recursive method of selecting solutions with consistent multipliers.

We consider the following dynamic constrained maximization problem studied in Stokey et

al. (1989):

max{xt}∞t=1

∞∑t=0

βtF (xt, xt+1) (19)

s.t. hi(xt, xt+1) ≥ 0, i = 1, ..., k, t ≥ 0,

for given x0 ∈ X , where {xt}∞t=1 is a bounded sequence (i.e., {xt} ∈ `n∞) such that xt ∈ X ⊂ <n

for every t. Functions F and hi are real-valued functions on X × X . We impose the following

conditions:

D1. X is convex, F is bounded, and β ∈ (0, 1).

15

D2. F and hi are concave and differentiable functions of (x, y) on X ×X.

D3. The constraint correspondence Γ(x) = {y ∈ X : h(x, y) ≥ 0} is uniformly compact near x

for every x ∈ X.

D4. For every x ∈ X there exists y ∈ X such that hi(x, y) > 0 for every i.

The saddle-point problem associated with (19) is

SP max{xt}∞t=1

min{λt}∞t=1,λt≥0

∞∑t=0

βt[F (xt, xt+1) + λt+1h(xt, xt+1)

], (20)

for given x0 ∈ X , where λt ∈ <k are Lagrange multipliers and {xt} ∈ `n∞. It follows from

Dechert (1992, Theorem 3) that if a sequence {x∗t} is a solution to (19), then under assumptions

D1, D2 and D4 there exists a summable sequence of multipliers {λ∗t} ∈ `k1 such that {x∗t , λ∗t} is

a saddle-point of (20). Conversely, if {x∗t , λ∗t} is a saddle-point of (20), then {x∗t} is a solution to

(19).

By a standard variational argument, the first-order necessary conditions for saddle-point

{x∗t , λ∗t}∞t=1 of (20) are the following intertemporal Euler equations:

DyF (x∗t , x∗t+1) + λ∗t+1Dyh(x∗t , x

∗t+1) + β

[DxF (x∗t+1, x

∗t+2) + λ∗t+2Dxh(x∗t+1, x

∗t+2)]

= 0 (21)

for every t ≥ 0, with x∗0 = x0. Equations (21) together with complementary slackness conditions

and the given constraints, define a system of second-order difference equations for {x∗t , λ∗t}.Complementary slackness conditions are

λ∗t+1h(x∗t , x∗t+1) = 0, h(x∗t , x

∗t+1) ≥ 0. (22)

The sufficiency of the Euler equation and a transversality condition, see Stokey et al. (1989),

continue to hold when a constraint is binding.

Proposition 3: Suppose that conditions D1-D2 hold. Let {x∗t , λ∗t}, with {x∗t} ∈ `n∞, x∗0 =

x0, λ∗t ≥ 0, and h(x∗t , x

∗t+1) ≥ 0 for every t ≥ 0, satisfy the Euler equations (21) and

the complementary slackness conditions (22). If the transversality condition

limt→∞

βt[DxF (x∗t , x

∗t+1) + λ∗t+1Dxh(x∗t , x

∗t+1)]

= 0, (23)

holds, then {x∗t , λ∗t} is a saddle-point of (20). In particular, {x∗t} is a solution to (19).

16

Proof: see Appendix.

Let V (x0) be the value function of (19). It satisfies the Bellman equation

V (x) = maxy

{F (x, y) + βV (y)} (24)

s.t. hi(x, y) ≥ 0, i = 1, ..., k

for every x ∈ X. The value function V is concave and bounded under assumptions D1-D2. If D3-

D4 are also satisfied, then the envelope Theorem 2 can be applied to (24). This is so because, first,

the correspondence Γ is continuous under D2-D4 (see Section 2), and second, the set of saddle-

points Y ∗(x)×Λ∗(x) of (24) is non-empty under D2 and D4 for every x. Theorem 2 implies that

if DxF and Dxhi are continuous functions of (x, y) for every i, then the superdifferential of V is

∂V (x) =⋂

y∗∈Y ∗(x)

⋃λ∗∈Λ∗(x)

{DxF (x, y∗) + λ∗Dxh(x, y∗)}. (25)

Corollary 2 (i) implies that V is differentiable if saddle-point multiplier λ∗ is unique. If there is a

solution y∗ with non-binding constraints, then the unique multiplier is zero and V is differentiable

at x. The latter is the well-known result of Benveniste and Scheinkman (1979).

For every solution {x∗t} to (19), x∗t+1 is a solution to the Bellman equation (24) at x∗t for every

t ≥ 0. The converse holds as well under assumptions D1, see Stokey et al. (1989; Theorem 4.3).

The latter result not only shows that it is sufficient to sequentially solve the Bellman equation

(24) to obtain a solution to (19) but also that solutions are time-consistent. That is, if {x∗t}∞t=1 is

a solution to (19) at x0 and the Bellman equation is restarted at x∗τ the resulting solution – say,

{x∗t}τt=1, {xt}∞t=τ+1 – is also a solution to (19) at x0.

It is well-known that if constraints are not binding and the value function is differentiable,

then the first-order conditions of the Bellman equation together with the envelope theorem imply

the Euler equations. Indeed, (25) simplifies then to DV (x) = DxF (x, y∗), the Euler equation

simplifies to

DyF (x∗t , x∗t+1) + βDxF (x∗t+1, x

∗t+2) = 0,

and the latter obtains from the first-order conditions of (24). In general, it is convenient to intro-

duce the saddle-point Bellman equation corresponding to (24):

V (x) = SP minλ≥0

maxy{F (x, y) + λh(x, y) + βV (y)} . (26)

17

The saddle-points of (26) are the set Y ∗(x)× Λ∗(x).

If (x∗t , λ∗t ) and (x∗t+1, λ

∗t+1) are consecutive elements of a saddle-point sequence of (20), then

(x∗t+1, λ∗t+1) is a saddle-point of (26) at x∗t , for every t ≥ 0. The converse implication requires a

consistency condition that involves subgradients obtained from the first-order conditions of (26).

The first-order condition for saddle-point (x∗t+1, λ∗t+1) at x∗t , for t ≥ 0, states that there exists a

subgradient vector φ∗t+1 ∈ ∂V (x∗t+1) such that

DyF (x∗t , x∗t+1) + λ∗t+1Dyh(x∗t , x

∗t+1) + βφ∗t+1 = 0, (27)

see Rockafellar (1981, Chapter 5).

Proposition 4: Suppose that conditions D1-D4 hold and DxF and Dxhi are continuous func-

tions for every i. Let {x∗t , λ∗t}∞t=1, with {x∗t} ∈ `n∞, be a sequence of saddle-points gener-

ated by the saddle-point Bellman equation (26), starting at x∗0 = x0, and let {φ∗t}∞t=1 be the

corresponding sequence of subgradients with φ∗t+1 ∈ ∂V (x∗t+1) satisfying (27) for t ≥ 0. If

the following envelope selection condition

φ∗t = DxF (x∗t , x∗t+1) + λ∗t+1Dxh(x∗t , x

∗t+1) (28)

holds for every t ≥ 1, then {x∗t , λ∗t}∞t=1 is a saddle-point of (20).

Proof: The first-order condition (27) together with the envelope selection condition (28) for φ∗t+1

imply that the Euler equation (21) holds at x∗t ; furthermore, the transversality condition (23) can

be written as limt→∞ βtφ∗t = 0. This transversality condition holds because the subgradients

{φ∗t} of a bounded and concave value function are bounded (see Rockafellar (1970), Theorem

24.7). The conclusion follows now from Proposition 3. 2

The envelope selection condition (28) guarantees consistency of multipliers generated by the

saddle-point Bellman equation. It can be dispensed with if the saddle-point multiplier is unique

(which is sufficient for value function V to be differentiable) but not if there are multiple multi-

pliers. Inconsistency of multipliers occurs if, given (x∗t+1, λ∗t+1) and φ∗t+1 ∈ ∂V (x∗t+1) satisfying

the first-order condition (27) at x∗t , a multiplier λ∗t+2 is chosen when solving (26) at x∗t+1 without

satisfying the envelope selection condition (28). Then the Euler equation (21) is not satisfied for

λ∗t+1 and λ∗t+2, and the sequence {x∗t , λ∗t}∞t=1 need not be a saddle-point of (20). This may hap-

pen if V is not differentiable at x∗t+1 and the saddle-point Bellman equation is restarted at x∗t+1

without recalling the selection φ∗t+1 ∈ ∂V (x∗t+1) previously made.

18

Proposition 4 does not provide a recursive method of generating consistent solutions from

the saddle-point Bellman equation, but it clearly suggests what that method should be: extend

the state x with a co-state φ ∈ ∂V (x) and impose the envelope selection condition. To that

end, we define the selective value function V s(x;φ) of state x and co-state φ ∈ ∂V (x) as the

value function of the saddle-point Bellman equation (26) with the additional restriction that the

saddle-point satisfies the envelope selection condition. That is,

V s(x;φ) = SP minλ≥0

maxy{F (x, y) + λh(x, y) + βV (y)} (29)

s.t. DxF (x, y) + λDxh(x, y) = φ.

Clearly, V s(x;φ) = V (x) for φ ∈ ∂V (x), but the the saddle-points of (29) are a subset of those

of (26). If (y∗, λ∗) is a saddle-point of (29) and φ∗′ ∈ ∂V (y∗) satisfies the first-order condition

φ∗′ = −β−1 [DyF (x, y∗) + λ∗Dyh(x, y∗)] , (30)

then the Bellman equation V s(x;φ) = F (x, y∗) + βV s(y∗;φ∗′) holds.

Using the selective value function V s we define policy functions ϕ : X × <n −→ X × <n

and ` : X × <n −→ <k+ as time-invariant selections from solutions to saddle-point Bellman

equation (29) and the first-order condition (30). That is, ϕ(x, φ) = (y∗, λ∗) where (y∗, λ∗) is a

saddle-point of (29), and `(x, φ) = φ∗′ where φ∗′ ∈ ∂V (y∗) satisfies (30). In particular, if V

is differentiable at y∗, then φ∗′ = DV (y∗). These policy functions can be used in a recursive

algorithm to solve the saddle-point problem (20). Given the initial condition x0, one can choose

arbitrary φ0 ∈ ∂V (x0),16 and apply policy functions (ϕ, `) recursively to generate a sequence

{x∗t , λ∗t , φ∗t}. Using policy function ϕ of the selective value function V s guarantees that the

envelope selection condition is satisfied for the resulting sequence. More formally, Corollary 4

summarizes our results and re-establishes the link between the Bellman equation and the Euler

equations known from the differentiable case.

Corollary 4: Suppose that conditions D1-D4 hold. If {x∗t , λ∗t , φ∗t}∞t=1, with {x∗t} ∈ `n∞, is a

sequence generated by policy functions (ϕ, `) of (29), i.e. (x∗t+1, λ∗t+1) = ϕ(x∗t , φ

∗t ) and

φ∗t+1 = `(x∗t , φ∗t ), with x∗0 = x0 and φ∗0 ∈ ∂V (x0), then {x∗t , λ∗t}∞t=1 is a saddle-point of

16Since the value function V is concave, it is differentiable almost everywhere and the choice of φ0 is unique -

φ0 = DV (x0) - for almost every x0.

19

(20) at x0. Furthermore, if at date τ + 1 a new sequence {x∗t , λ∗t}∞t=τ+1 is generated using

possibly different policy functions (ϕ, ) starting from initial state x∗τ and co-state φ∗τ , then

({x∗t , λ∗t}τt=1, {x∗t , λ∗t}∞t=τ+1) is a saddle-point of (20) at x0.

Example 3 illustrates the main results of this section.

Example 3 (A dynastic problem): consider a dynasty of overlapping generations who live for

two periods. A household is formed by a young and an old member. In each period, the young

decides for the household consumption for the next period. The allocation problem of the dynasty

is to maximise the discounted utility of all future generations as follows

max{xt}∞t=1

∞∑t=0

βt [xt + 2xt+1] (31)

s.t. xt + xt+1 ≤ 4, xt+1 ≤ xt, xt ≥ 0, t ≥ 0,

for given x0 ∈ [0, 4]. Note that problem (31) does not contain forward-looking constraints, since

the choice of xt+1 at xt does not involve future variables or values. Let V (x0) be the value

function of (31). It is easy to see that the value function is

V (x) =

3x(1− β)−1 if x ≤ 2

8− x+ 3(4− x)β(1− β)−1 if x ≥ 2.

(32)

The unique solution to (31) for x0 = 2 is the constant sequence x∗t = 2.

The value function V is concave. It is not differentiable at x = 2 where the super-differential

is

∂V (2) = [−(3β(1− β)−1 + 1), 3(1− β)−1]. (33)

The Euler equations for a saddle-point of (31) are

2− λ∗1,t − λ∗2,t + β[1− λ∗1,t+1 + λ∗2,t+1

]= 0 (34)

for t ≥ 1, where λ∗1,t and λ∗2,t are the multipliers of the first and the second constraints in (31),

respectively. For x0 = 2 and the solution x∗t = 2, both constraints are binding. This implies

that the slackness conditions are vacuous, and therefore the saddle-point multipliers are arbitrary

positive solutions to difference equations (34) that satisfy the transversality condition (23).

20

The saddle-point Bellman equation is

V (x) = SP minλ≥0

maxy{x+ 2y + λ1(4− y − x) + λ2(x− y) + βV (y)} . (35)

The unique (saddle-point) solution to (35) at x = 2 is y∗ = 2. Starting from x0 = 2, the saddle-

point Bellman equation generates the sequence of solutions x∗t = 2. The first-order condition at

x∗t is

2− λ∗1,t+1 − λ∗2,t+1 + βφ∗t+1 = 0, (36)

for some φ∗t+1 ∈ ∂V (x∗t+1). Equation (36) is the only restriction on multipliers generated by

the saddle-point Bellman equation starting from x0 = 2. The envelope theorem implies that

φ∗t+1 = 1− λ1,t+2 + λ2,t+2 for a date-(t + 2) multiplier λt+2, but that multiplier can be different

from λ∗t+2. Given φ∗t+1 ∈ ∂V (2), the first-order condition (36) can be written more explicitly as

2− β(3β(1− β)−1 + 1

)≤ λ∗1,t+1 + λ∗2,t+1 ≤ 2 + 3β(1− β)−1. (37)

The set of multipliers (37) is a superset of multipliers satisfying the Euler equation (34).

The envelope selection condition (28) restricts the set of saddle-point multipliers to those

satisfying

φ∗t = 1− λ∗1,t+1 + λ∗2,t+1. (38)

Clearly (38), combined with the first-order condition (36) implies the Euler equation (34). Thus

the set of multipliers generated by the saddle-point Bellman equation and satisfying (38) is the

same as the set of saddle-point multipliers of (31) in accordance with Propositions 3 and 417.

Alternatively, sequences of multipliers satisfying the Euler equation (34) (i.e. with multipliers

satisfying (37)) but not (38) are not saddle-point multipliers of (20).

In sum, the intertemporal Euler equation (21) is a necessary first-order condition for a solution

to the dynamic programming problem (19), and by Proposition 3 it is also sufficient. The same is

true for a sequence of saddle-points of the Bellman equation (24) if the saddle-point multiplier is

unique. If there are multiple multipliers, the Euler equations may fail to hold for some selection

17Note that the envelope selection condition (38) in this example is a one-to-one correspondence between φ∗t and

λ∗t+1 and, therefore, the latter can be used as a co-state variable. This is so because functions F and hi are additively

separable.

21

of multipliers. We introduced the envelope selection condition which guarantees that multipliers

generated from the Bellman equation satisfy the Euler equations. The recursive method of solving

the dynamic programming problem with consistent multipliers involves expanding the co-state

to include a subgradient of the value function and using a policy function of the selective value

function

The problem of time-inconsistency of multipliers discussed in this section can lead to non-

optimal outcomes in the presence of forward-looking constraints.

5 Recursive Contracts

In this section we extend the results of Section 4 to the recursive contracts of Marcet and Mari-

mon (2019). Recursive contracts are dynamic optimization problems with forward-looking con-

straints. The presence of forward-looking constraints makes the standard methods of dynamic

programming inapplicable. Marcet and Marimon (2019) show that recursive contracts can be

solved by transforming optimization problems with forward-looking constraints into saddle-point

problems. Solutions to a saddle-point Bellman equation are solutions to the original contracting

problem provided that an intertemporal consistency condition is satisfied. The necessity of this

consistency condition was motivated by an example given by Messner and Pavoni (2004), who

showed that non-unique solutions to the saddle-point Bellman equation can fail to be solutions to

the contracting problem. We show that imposing the envelope selection condition and extending

the co-state generates time-consistent recursive solutions even if the value function is not differ-

entiable. The envelope selection condition is shown to be equivalent in this context to Marcet and

Marimon’s intertemporal consistency condition.

5.1 The partnership problem with limited enforcement

We consider a canonical deterministic partnership problem18 with limited enforcement, which

takes the form18Marcet and Marimon (2019) develop their theory in a more general dynamic stochastic formulation. The ap-

proach presented in this section can be easily extended to their general setup with infinite-horizon forward-looking

constraints.

22

Vµ(y0) = max{ct}∞0

∞∑t=0

βtm∑i=1

µi u(ci,t) (39)

s.t.m∑i=1

yi,t −m∑i=1

ci,t ≥ 0, (40)

∞∑n=0

βn u(ci,t+n)− vi(yi,t) ≥ 0, (41)

ci,t ≥ 0, for all i, t ≥ 0,

where the sequence {ct} is bounded, i.e. {ct} ∈ `m∞. The sequence of incomes {yt} is bounded

and follows a law of motion yt+1 = g(yt) for some g : <m++ → <m++ and given initial income

vector y0. We impose the following conditions:

P1. The function u : <+ → <+ is bounded, increasing, concave, and differentiable. Further-

more µi > 0 for every i, and β ∈ (0, 1).

P2. Given y0 ∈ <m++, there exists ε > 0 and {ct}∞t=0 ∈ `m∞, with ci,t > 0 for every i and t, such

that constraints (40) and (41) hold with ε instead of 0 on the right-hand side.

A convenient way of analysing problem (39) is to consider the following constrained saddle-

point problem resulting from writing the Lagrangean with all the forward-looking constraints

(41), rearranging terms, and collecting the partial sums of multipliers into weights µi,t (see Marcet

and Marimon (2019) for details):

SP max{ct}∞t=0

min{µt}∞t=1

∞∑t=0

βtm∑i=1

[µi,t+1 (u(ci,t)− vi(yi,t)) + µi,tvi(yi,t)

](42)

s.t.m∑i=1

ci,t ≤m∑i=1

yi,t

µi,t+1 ≥ µi,t

ci,t ≥ 0, for all i, t,

where µ0 = µ, and {ct} ∈ `m∞ and {µt} ∈ `m∞. The unconstrained saddle-point problem for the

23

Lagrangian of (42) is

SP max{ct,λt+1}∞t=0

min{µt,γt}∞t=1

∞∑t=0

βtm∑i=1

{µi,t+1 (u(ci,t)− vi(yi,t)) + µi,tvi(yi,t) (43)

− λi,t+1(µi,t+1 − µi,t) + γt+1 (yi,t − ci,t)}.

If {c∗t} ∈ `m∞ is a solution to (39), then under assumptions P1-P2 there exists a bounded sequence

of weights {µ∗t} ∈ `m∞ and summable sequences of multipliers {λ∗t} ∈ `m1 and {γ∗t} ∈ `1 (see

Dechert (1992, Theorem 2)) such that {c∗t , λ∗t+1, µ∗t+1, γ

∗t+1} is a saddle-point of (43).19 Con-

versely, if {c∗t , λ∗t+1, µ∗t+1, γ

∗t+1} is a saddle-point of (43), then constraints (40) and (41) are satis-

fied and {c∗t} is a solution to (39).

The first-order necessary conditions with respect to µi,t+1 for saddle-point {c∗t , λ∗t+1, µ∗t+1, γ

∗t+1}

of (43) are

u(c∗i,t)−(vi(yi,t) + λ∗i,t+1

)+ β

(vi(yi,t+1) + λ∗i,t+2

)= 0 (44)

for every i and t ≥ 0. The respective condition with respect to ci,t is µ∗i,t+1u′(c∗i,t) = γ∗t+1. Equa-

tion (44) is the intertemporal Euler equation for the partnership problem. The Euler equation

together with first-order conditions for ci,t, the constraints, and complementary slackness condi-

tions for γ∗t+1 and λ∗i,t+1, are a system of first-order difference equations.

As in Section 4 (see Proposition 3), Euler equations and a transversality condition are suffi-

cient conditions for a solution to (43).

Proposition 5: Suppose that condition P1 holds. Let {c∗t , λ∗t+1, µ∗t+1, γ

∗t+1}, with {c∗t}∞t=0 ∈ `m∞,

{µ∗t}∞t=1 ∈ `m∞, µ∗0 = µ, λ∗t+1 ≥ 0 and γ∗t+1 ≥ 0 for every t, satisfy the Euler equations (44),

the constraints and complementary slackness conditions20, and the first-order conditions

with respect to ci,t. If the transversality condition

limt→∞

βt[vi(yi,t) + λ∗i,t+1] = 0 (45)

holds for every i, then {c∗t , λ∗t+1, µ∗t+1, γ

∗t+1} is a saddle-point of (43). In particular, {c∗t} is

a solution to the partnership problem (39).19In the derivation of (42) weights µ∗t are obtained as partial sums of a summable sequence and therefore are a

convergent, hence bounded, sequence.20These are: λ∗i,t+1(µ∗i,t+1 − µ∗i,t) = 0, µ∗i,t+1 − µ∗i,t ≥ 0 for all i, and γ∗t+1

∑mi=1

(yi,t − c∗i,t

)=

0,∑m

i=1

(yi,t − c∗i,t

)≥ 0.

24

Proof: see Appendix.

The constrained saddle-point problem (42) has recursive structure that can be expressed by

the following saddle-point Bellman equation:

W ( y, µ) = SP minµ′

maxc

m∑i=1

[µ′i(u(ci)− vi(yi)) + µivi(yi)] + βW (y′, µ′) (46)

s.t.m∑i=1

ci ≤m∑i=1

yi (47)

µ′i ≥ µi (48)

ci ≥ 0, for all i,

where y′ = g(y). It follows from Theorem 1 in Marcet and Marimon (2019) that the value

function Vµ satisfies equation (46), that is, W (y0, µ) = Vµ(y0) for every µ ∈ <m+ . The value

function W is convex and homogeneous of degree one with respect to µ, and bounded under

assumption P1. If, in addition P2 is assumed, the existence has been established in Marcet and

Marimon (2019; Theorem 3).

A saddle-point of the Bellman equation (46) is denoted by (c∗, λ∗, µ∗′, γ∗) where γ∗ is a

multiplier of constraint (47) and λ∗ is a multiplier of (48). The set of all saddle-points is a

product of two sets M∗(y, µ) and N∗(y, µ) so that (c∗, λ∗) ∈ M∗(y, µ) and (µ∗′, γ∗) ∈ N∗(y, µ)

for every saddle-point (c∗, λ∗, µ∗′, γ∗). Under assumptions P1-P2, the setsM∗(y, µ) andN∗(y, µ)

are non-empty. The envelope theorem for saddle-point problems of Appendix A, see (73), can be

applied to W.21 It implies that the subdifferential of W is

∂µW (y, µ) = v(y) + {λ∗| (c∗, λ∗) ∈M∗(y, µ) for some c∗}. (49)

Function W is differentiable with respect to µ at (y, µ) if and only if there is unique multiplier

λ∗ that is common to all solutions c∗.

For every sequence {c∗t , λ∗t+1, µ∗t+1, γ

∗t+1}∞t=0 which is a saddle-point of (43), (c∗t , λ

∗t+1, µ

∗t+1, γ

∗t+1)

is a saddle-point of the saddle-point Bellman equation (46) at (yt, µ∗t ) for every t. It follows that

for every solution {c∗t} to partnership problem (39), there exist weights {µ∗t} such that (c∗t , µ∗t+1)

is a solution to the saddle-point Bellman equation for every t. As in Proposition 4, the converse

21The constraint set given by (48) is compact when P2 is satisfied, see Marcet and Marimon (2019; Section 4.)

25

result requires an envelope selection condition that involves subgradients from the first-order con-

ditions of (46). The first-order conditions with respect to µ′ for saddle-point (c∗t , λ∗t+1, µ

∗t+1, γ

∗t+1)

at (yt, µ∗t ) state that there exists subgradient vector φ∗t+1 ∈ ∂µW (yt+1, µ

∗t+1) such that

u(c∗i,t)−(vi(yi,t) + λ∗i,t+1

)+ βφ∗i,t+1 = 0 (50)

for every i.

Proposition 6: Suppose that conditions P1-P2 hold. Let {c∗t , λ∗t+1, µ∗t+1, γ

∗t+1} be a sequence

of saddle-points generated by saddle-point Bellman equation (46) starting at (y0, µ), with

{c∗t}∞t=0 ∈ `m∞ and {µ∗t}∞t=1 ∈ `m∞, and let {φ∗t}∞t=1 be the corresponding sequence of sub-

gradients with φ∗t+1 ∈ ∂µW (yt+1, µ∗t+1) satisfying (50) for t ≥ 0. If the following envelope

selection condition

φ∗i,t = vi(yi,t) + λ∗i,t+1 (51)

holds for every i and every t ≥ 1, then {c∗t , λ∗t+1, µ∗t+1, γ

∗t+1} is a saddle-point of (43) and

{c∗t} is a solution to (39).

Proof: The first-order conditions (50) together with the envelope selection condition (51) imply

the Euler equation (44) for every t ≥ 0. The complementary slackness conditions and the first-

order condition with respect to ci,t follow from the respective conditions for (46). Since W is

bounded and convex, the sequence of subgradients {φ∗t} is bounded. The transversality condition

(45), which by (51) can be written as limt→∞ βtφ∗t = 0, holds. The conclusion follows from

Proposition 5. 2

Proposition 6 corresponds to – and provides an alternative proof of – Theorem 2 and its

Corollary in Marcet and Marimon (2019) where the envelope selection condition (51) is replaced

by the intertemporal consistency condition

φ∗i,t = u(c∗i,t) + βφ∗i,t+1. (52)

Because of the first-order condition (50), those two conditions are equivalent in this context.

Note that condition (52) and the transversality condition limt→∞ βtφ∗i,t = 0 imply that

φ∗i,t =∞∑n=0

βnu(c∗i,t+n).

26

Using (51) and λ∗i,t+1 ≥ 0, it follows that

∞∑n=0

βnu(c∗i,t+n) ≥ vi(yi,t). (53)

That is, the intertemporal consistency condition – equivalently, the envelope selection condition

– implies that the participation constraints (41) hold.

As in Section 4 (see Corollary 4), the envelope selection condition (51) guarantees consistency

of multipliers and solutions generated by the saddle-point Bellman equation. It can be dispensed

with if the saddle-point multiplier is unique, which is a sufficient condition for value function W

to be differentiable in µ. Inconsistency of multipliers may occur if, given (c∗t , λ∗t+1, µ

∗t+1, γ

∗t+1)

with the corresponding subgradient φ∗t+1 ∈ ∂µW (yt+1, µ∗t+1) from the first-order condition (50)

at t, at t+1 multiplier λ∗t+2 is chosen without satisfying envelope selection condition (51). This is

likely to happen if the saddle-point Bellman equation is solved only knowing (yt+1, µ∗t+1). Then

the Euler equation (44) is not satisfied for λ∗t+1 and λ∗t+2, and the sequence {c∗t , λ∗t+1, µ∗t+1, γ

∗t+1}

need not be a saddle-point of (43). However, in contrast with Section 4, since the multiplier λ∗t+1

has to be chosen together with consumption c∗t in the set M∗(yt, µ∗t ), an inconsistent choice of

the multiplier may lead to consumption that is either suboptimal, or violates the participation

constraints.

We define the selective value function of state y and co-state (µ, φ) for φ ∈ ∂µW (y, µ) in a

similar way as in Section 4, that is:

W s( y, µ;φ) = SP minµ′

maxc,λ

m∑i=1

[µ′i(u(ci)− vi(yi)) + µivi(yi)− λi (µ′i − µi)] + βW (y′, µ′)

(54)

s.t. v(yi) + λi = φi, (55)m∑i=1

ci ≤m∑i=1

yi,

ci ≥ 0, λi ≥ 0, for all i,

where y′ = g(y).22 It holds that W s( y, µ;φ) = W ( y, µ) for φ ∈ ∂µW (y, µ), but the (saddle-

point) solutions to (54) may be a proper subset of solutions of (46). The first-order condition for22Note that the multiplier λ is uniquely determined by constraint (55).

27

saddle-point (c∗, λ∗, µ∗′, γ∗) of (54) with respect to µ is

u(c∗i )− (vi(yi) + λ∗i ) + βφ∗′i = 0, (56)

where φ∗′ ∈ ∂µW (y′, µ∗′). It holds that

W s( y, µ;φ) =m∑i=1

[µ∗′i (u(c∗i )− vi(yi)) + µivi(yi)] + βW s(y′, µ∗′;φ∗′).

The policy functions ϕ : <3m+ → <2m

+ and ` : <3m+ → <m+1

+ are defined by ϕ(y, µ, φ) =

(c∗, λ∗, µ∗′, γ∗) such that (c∗, λ∗, µ∗′, γ∗) is a saddle-point of (54) and `(y, µ, φ) = φ∗′ where

φ∗′ ∈ ∂µW (y′, µ∗′) satisfies the first-order condition (56).

Policy functions (ϕ, `) can be used to generate sequences of saddle-points {c∗t , λ∗t+1, µ∗t+1, γ

∗t+1}

and subgradients {φ∗t+1} such that (c∗t , λ∗t+1, µ

∗t+1, γ

∗t+1) = ϕ(yt, µ

∗t , φ∗t ) and φ∗t+1 = `(yt, µ

∗t , φ∗t )

for every t ≥ 0, with initial state y0 and co-state (µ∗0, φ∗0) where µ∗0 = µ and φ∗0 ∈ ∂µW (y0, µ). It

follows from Proposition 5 that {c∗t , λ∗t+1, µ∗t+1, γ

∗t+1}∞t=0 is a saddle-point of (43). The sequence

{c∗t , λ∗t+1, µ∗t+1, γ

∗t+1, φ

∗t+1} can be found by solving the system of equations (51) and (50) to-

gether with first-order conditions w.r. ci,t and complementary slackness conditions. All these

equations are first-order difference equations.

Corollary 5 which follows summarizes our results for recursive contracts. It extends the main

sufficiency result of Marcet and Marimon (2019; Theorem 2 and its Corollary) by providing

a recursive algorithm – based on a further extension of their co-state – for solving recursive

contracts with forward-looking constraints, akin to the one introduced in Section 4.

Corollary 5: Suppose that conditions P1-P2 hold. If {c∗t , λ∗t+1, µ∗t+1, γ

∗t+1} and {φ∗t+1} are se-

quences of saddle-points and subgradients generated by policy functions (ϕ, `) of (54),

starting from µ∗0 = µ and φ∗0 ∈ ∂µW (y0, µ), with {c∗t}∞t=0 ∈ `m∞ and {µ∗t}∞t=0 ∈ `m∞, then

{c∗t , λ∗t+1, µ∗t+1, γ

∗t+1} is a saddle-point of (43) at µ and {c∗t} is a solution to (39). Fur-

thermore, if at date τ + 1 a new sequence {c∗t , λ∗t+1, µ

∗t+1, γ

∗t+1}∞t=τ+1 is generated using

possibly different policy functions (ϕ, ) starting from initial state yτ and co-state (µ∗τ , φ∗τ ),

then ({c∗t , λ∗t+1, µ∗t+1, γ

∗t+1}τt=1, {c∗t , λ

∗t+1, µ

∗t+1, γ

∗t+1}∞t=τ+1) is also a saddle-point of (43) at

µ and {c∗t} is a solution to (39).

Note date-t co-state variable φ∗t can be replaced by λ∗t+1. This is so because the envelope

selection condition (49) is a one-to-one relationship between the subgradient φ∗t and the multiplier

λ∗t+1.

28

In sum, the intertemporal Euler equation (44) along with other first-order conditions are nec-

essary and sufficient for a solution to the partnership problem (39). Saddle-point Bellman equa-

tion can be used to find the solution recursively, but the envelope selection condition is required

to guarantee that the sequence of solutions generated by the Bellman equation satisfies the Euler

equation. Such sequence can be generated by a policy function of the selective value function.

As in Section 4, we provide an example that illustrates the results of this section.

Example 4 (Messner and Pavoni (2004) revisited): consider a partnership problem (39) with

two agents and linear utilities,

W (µ) = max{ct}∞0

∞∑t=0

βt2∑i=1

µici,t (57)

s.t. c1,t + c2,t ≤ y,∞∑j=0

βjc1,t+j ≥ 0,∞∑j=0

βjc2,t+j ≥ b(1− β)−1, (58)

ci,t ≥ 0, i = 1, 2, for all t ≥ 0,

where 0 < b < y, µi > 0 for i = 1, 2, and 0 < β < 1. Note that agent 1’s participation constraint

is not binding.

The value function is

W (µ) =

(1− β)−1[µ1(y − b) + µ2b] if µ1 ≥ µ2

(1− β)−1µ2y if µ1 ≤ µ2.

(59)

If µ1 > µ2, the constant sequence c∗1,t = y − b and c∗2,t = b is a solution to (57), but we shall

see that there are many other solutions. The value function W is convex and differentiable if

µ1 6= µ2, but it is not differentiable at µ1 = µ2 where the sub-differential is

∂W (µ) = (1− β)−1co{(y − b, b), (0, y)}. (60)

The saddle-point Bellman equation (46) is

W (µ) = SP minµ′

maxc

{2∑i=1

µ′ici − (µ′2 − µ2)b(1− β)−1 + βW (µ′)

}(61)

s.t. c1 + c2 ≤ y,

µ′i ≥ µi, ci ≥ 0, i = 1, 2.

29

A sequence of saddle-points {c∗t , λ∗t+1, µ∗t+1, γ

∗t+1} of (61) generated by a policy function can be

found by recursively solving a system of equations consisting of the first-order conditions

µ∗i,t+1 − γ∗t+1 = 0, (62)

c∗1,t − λ∗1,t+1 + βφ∗1,t+1 = 0 (63)

c∗2,t − (b(1− β)−1 + λ∗2,t+1) + βφ∗2,t+1 = 0, (64)

with φ∗t+1 ∈ ∂W (µ∗t+1), the complementary slackness conditions for λ∗t+1, and the envelope

selection conditions

φ∗1,t = λ∗1,t+1 (65)

φ∗2,t = b(1− β)−1 + λ∗2,t+1. (66)

Suppose that the initial state is µ∗0 = µ such that µ1 > µ2. Since W is differentiable at µ∗0, the

initial co-state is φ∗0 = DW (µ∗0), that is, φ∗1,0 = (y − b)(1 − β)−1 and φ∗2,0 = b(1 − β)−1. From

equations (65) - (66) we obtain λ∗1,1 = (y − b)(1− β)−1 and λ∗2,1 = 0, and using complementary

slackness µ∗1,1 = µ1 and µ∗2,1 = µ∗1,1. Because W is not differentiable at µ∗1, φ∗1,1 and φ∗2,1 can be

arbitrary, as long as they satisfy: φ∗1,1 + φ∗2,1 = y(1− β)−1, φ∗1,1 ≥ 0 and φ∗2,1 ≥ b(1− β)−1 (see

(60)). The consumption plan c∗0 can be any selection satisfying c∗1,0 = (y−b)(1−β)−1−βφ∗1,1 and

c∗2,0 = b(1 − β)−1 − βφ∗2,1. Selection of c∗0 determines φ∗1. We have thus derived φ∗1 = `(µ∗0, φ∗0),

and (c∗0, µ∗1, λ∗1) = ϕ(µ∗0, φ

∗0) for a policy function (ϕ, `)23.

Next, iteration of equations (62) - (66) with given state µ∗1 and co-state φ∗1 gives µ∗2 = µ∗1,

c∗1,1 = φ∗1,1 − βφ∗1,2, c∗2,1 = φ∗2,1 − βφ∗2,2 where φ∗1,2 + φ∗2,2 = y(1 − β)−1, φ∗1,2 ≥ 0 and φ∗2,2 ≥b(1−β)−1. Again, a selection of c∗1 determines φ∗2.With this step, we have derived φ∗2 = `(µ∗1, φ

∗1),

and (c∗1, µ∗2, λ∗2) = ϕ(µ∗1, φ

∗1). All subsequent iterations follow the same pattern with µ∗t = µ∗1 for

all t > 1, and µ∗t being the point of non-differentiability of value function W. Time-invariant

policy function selects a stationary solution to the equations for t ≥ 2. Corollary 5 implies that

every sequence generated in this way is an optimal solution to (57). For example, consumption

sequence c∗1,t = y − b and c∗2,t = b is optimal and can be generated by a policy function with

λ∗1,t = (y − b)(1− β)−1, λ∗2,t = 0, φ∗1,t = (y − b)(1− β)−1 and φ∗2,t = b(1− β)−1 for t ≥ 1.

We show next that a sequence of saddle-points of the Bellman equation (61) may not be an

optimal solution to (57) if the envelope selection conditions (65) - (66) are not satisfied. Consider23To simplify notation, we eliminated multipliers γ∗t (given by (62)) from consideration.

30

the same sequence of weights {µ∗t} as before and a sequence of consumptions, multipliers, and

subgradients given by c∗1,t = 0, c∗2,t = y for t ≥ 0, λ∗1,t = 0, λ∗2,t = (y − b)(1 − β)−1 for t ≥ 1,

φ∗1,t = 0 and φ∗2,t = y(1 − β)−1 for t ≥ 1, and φ∗0 = DW (µ∗0). Note that φ∗t ∈ ∂W (µ∗t ) for

all t. This sequence satisfies first-order conditions (62) - (64). However, the envelope selection

conditions (65) - (66) are violated for t = 1 implying that the Euler equation is violated at date

0. This consumption sequence is not optimal since there is excessive consumption for agent 2.

Similarly, if βy ≥ b, then the sequence given by c∗1,t = y, c∗2,t = 0 for t ≥ 0, λ∗1,t =

(y − b)(1 − β)−1, λ∗2,t = 0 for t ≥ 1, φ∗1,t = (βy − b)[β(1 − β)]−1, φ∗2,t = b[β(1 − β)]−1 for

t ≥ 1, and φ∗0 = DW (µ∗0) satisfies equations (62 - 63) and also φ∗t ∈ ∂W (µ∗t ). Here, the envelope

selection conditions (65) - (66) are violated at every t ≥ 1. This consumption sequence does not

satisfy the participation constraints (58).

6 Conclusions

This paper makes three contributions to constrained optimization problems when the value func-

tion may be non-differentiable because of binding constraints. First, it extends the envelope theo-

rem to non-concave problems and provides novel characterizations of super- and sub-differentials

of concave and convex value functions. Second, it uncovers a previously unknown time-inconsistency

in standard dynamic programming problems: restarting the Bellman equation at a later state, say

x∗t , may result in inconsistent Lagrange multipliers for which the Euler equations fail to hold and

the continuation saddle-point is not part of the saddle-point of the infinite-horizon optimization

problem at x0. Nevertheless, the solutions to the Bellman equation are time-consistent in the

standard dynamic programming problems. However, in the presence of forward-looking con-

straints, the time-inconsistency of multipliers of the (saddle-point) Bellman equation can turn

into inconsistency of solutions, that is, non-optimal or infeasible solutions. Our third contribution

is a recursive method of restoring time-consistency of multipliers and/or solutions by imposing

the envelope selection condition. This method extends the co-state and introduces the selective

value function to recover the link between the solutions to the Bellman and Euler equations. It is

the proper recursive method for problems with occasionally binding constraints where the value

31

function may not be differentiable 24.

Appendix

A. Saddle-point problems

For the use in Section 5, we extend results of Sections 2 and 3 to saddle-point problems.

Consider the following parametric saddle-point problem

SP maxy∈Y

minz∈Z

f(x, y, z) (67)

subject to

hi(x, y) ≥ 0, gi(x, z) ≤ 0, i = 1, . . . , k (68)

where Y ∈ <n, Z ∈ <l and x is the parameter in X ⊂ <m. Assume that function f, hi and gi are

continuous, the constraint set given by (68) is a compact-valued and continuous correspondence

of x, and the weak Slater condition holds, that is, for every x and every i there exist yi ∈ Y

such that hi(x, yi) > 0 and hj(x, yi) ≥ 0 for j 6= i, and zi ∈ Z such that gi(x, zi) < 0 and

gj(x, yi) ≤ 0 for j 6= i.

Let V (x) denote the value function of the problem (67–68). The Lagrangian function associ-

ated with (67–68) is

L(x, y, z, λ, γ) = f(x, y, z) + λh(x, y) + γg(x, z), (69)

where λ ∈ <k+ and γ ∈ <k+ are vectors of multipliers. A saddle-point ofL is vector (y∗, z∗, λ∗, γ∗)

where L is maximized with respect to y ∈ Y and γ ∈ <k+, and minimized with respect to z ∈ Zand λ ∈ <k+. The set of saddle-points of L at x is a product of two sets M∗(x) and N∗(x) so that

(y∗, γ∗) ∈ M∗(x) and (z∗, λ∗) ∈ N∗(x) for every saddle-point (y∗, z∗, λ∗, γ∗), see Appendix B

or Lemma 36.2 in Rockafellar (1970). As in Section 2, we assume that the set of saddle-points is

non-empty for every x. If (y∗, z∗, λ∗, γ∗) is a saddle-point, then (y∗, z∗) is a solution to (67–68).

24To solve problems with occasionally binding constraints, Guerrieri and Iacovello (2015) consider separate

regimes where constraints are binding or not binding, which can be formulated by adding as co-state the corre-

sponding Lagrange multiplier which is zero or positive depending on the regime. They do not use a policy function

of the Bellman equation in their algorithm, but instead impose the Euler equation directly.

32

If partial derivatives ∂f∂x, ∂h∂x

and ∂g∂x

are continuous functions of (x, y, z), then directional

derivatives of the value function V at x ∈ intX for single-dimensional parameter set X ⊂ < are

V ′(x+) = max(y∗,γ∗)∈M∗(x)

min(z∗,λ∗)∈N∗(x)

[∂f∂x

(x, y∗, z∗) + λ∗∂h

∂x(x, y∗) + γ∗

∂g

∂x(x, z∗)

](70)

and

V ′(x−) = min(y∗,γ∗)∈M∗(x)

max(z∗,λ∗)∈N∗(x)

[∂f∂x

(x, y∗, z∗) + λ∗∂h

∂x(x, y∗) + γ∗

∂g

∂x(x, z∗)

](71)

where the order of maximum and minimum does not matter. Corollary 1 can be extended to

saddle-point problems, as well.

Suppose that multi-dimensional parameter x can be decomposed in x = (x1, x2) so that the

constraints in the saddle-point problem (67–68) can be written as

hi(x1, y) ≥ 0, gi(x

2, z) ≤ 0, i = 1, . . . , k. (72)

If function f is concave in x1 and convex in x2, while the functions hi are concave in x1and y

and the functions gi are convex in x2 and z, then V (x1, x2) is concave in x1 and convex in x2.

The super-sub-differential calculus of Section 3.2 can be extended to this class of saddle-point

problems. For instance, the subdifferential of value function V with respect to x2 is

∂Vx2(x) =⋂

(z∗,λ∗)∈N∗co

⋃(y∗,γ∗)∈M∗

{Dx2f(x, y∗, z∗) + λ∗Dx2h(x, y∗) + γ∗Dx2g(x, z∗)}

.

(73)

B. Proofs

Lemma 1. Under assumptions A1-A5, the following hold:

(i) The set of saddle-points of the Lagrangian (3) is a product of two sets Y ∗(x)×Λ∗(x) where

Y ∗(x) ⊂ Y and Λ∗(x) ⊂ <k+.

(ii) The sets Y ∗(x) and Λ∗(x) are compact for every x ∈ X.

(iii) The correspondences Y ∗ and Λ∗ are upper hemi-continuous on X.

33

Proof: It follows from Lemma 36.2 in Rockafellar (1970) that if the set of saddle-points is non-

empty (assumption A5), then the set of saddle-points solutions is

Y ∗(x) = argmaxy∈Y minλ∈<k

+

[f(x, y) + λh(x, y)], (74)

while the set of saddle-point multipliers is

Λ∗(x) = argminλ∈<k+

maxy∈Y

[f(x, y) + λh(x, y)], (75)

Consequently, the set of saddle-points is the product Y ∗(x)× Λ∗(x). This proves (i).

For parts (ii) and (iii), we first note that the function minλ∈<k+

[f(x, y) + λh(x, y)] which

is maximized over y in (74) equals f(x, y) whenever h(x, y) ≥ 0 and −∞ if h(x, y) � 0.

This implies that Y ∗(x) is the set of solutions to the constrained maximization problem (1–2).

Since the constraint correspondence Γ is compact-valued and continuous, the Maximum Theorem

implies that Y ∗ is compact-valued and upper hemi-continuous on X . Furthermore, the value

function V is continuous on X.

To show the desired properties of Λ∗(x), we proceed as follows: Let yi be of the weak Slater’s

condition A4. The saddle-point property (4) implies that

f(x, yi) + λ∗h(x, yi) ≤ V (x) (76)

for every saddle-point multiplier λ∗ at x. Using A4, we obtain

λ∗i ≤V (x)− f(x, yi)

hi(x, yi). (77)

We denote the RHS of (77) by λi(x) and note that it is a continuous function of x. Consider

a sequence {xn} with xn ∈ X converging to x ∈ X. Let {λ∗n} be a sequence of saddle-point

multipliers λ∗n ∈ Λ∗(xn). Since 0 ≤ λ∗n ≤ λ(xn) and λ(xn) converges to λ(x), it follows that the

sequence {λ∗n} has a convergent subsequence (for which we use the same notation) with the limit

denoted by λ∗. Let {y∗n} be sequence of solutions with y∗n ∈ Y ∗(xn). Since correspondence Y ∗

is upper hemi-continuous, there exists a subsequence (retaining the same notation) converging to

some y∗ ∈ Y ∗(x). It is now easy to show taking the limits as xn converges to x in saddle-point

inequalities (4) that λ∗ is a saddle-point multiplier at x, that is, λ∗ ∈ Λ∗(x). This shows that Λ∗

is upper hemi-continuous and compact-valued on X . 2

34

Proof of Theorem 1:

We shall prove that equations (9) and (10) hold for arbitrary x0 ∈ intX. Let ∆f(s, y) denote

the difference quotient of function f with respect to x at x0, that is

∆f(s, y) =f(x0 + s, y)− f(x0, y)

s

for s 6= 0. For s = 0, we set ∆f(0, y) = ∂f∂x

(x0, y). Assumptions of Theorem 1 imply that

function ∆f(s, y) is continuous in (s, y) on {X − x0} × Y.Similar notation ∆hi(s, y) is used for each function hi, and ∆L(s, y, λ) for the Lagrangian.

Functions ∆hi(s, y) are continuous in (s, y). Note that ∆L(s, y, λ) = ∆f(s, y) + λ∆h(s, y),

where we use the scalar-product notation λ∆h(s, y) =∑

i λi∆hi(s, y).

Let (y∗s , λs) denote a saddle-point of Lagrangian L at x0 + s, that is, λ∗s ∈ Λ∗(x0 + s) and

y∗s ∈ Y ∗(x0 + s), where the two sets are non-empty by A5. Saddle-point property (4) together

with (5) imply that

V (x0 + s) ≥ L(x0 + s, y∗0, λ∗s) (78)

and

V (x0) ≤ L(x0, y∗0, λ

∗s). (79)

Subtracting (79) from (78) and dividing the result on both sides by s > 0, we obtain

V (x0 + s)− V (x0)

s≥ ∆L(s, y∗0, λ

∗s) = ∆f(s, y∗0) + λ∗s∆h(s, y∗0). (80)

Since (80) holds for every y∗0 ∈ Y ∗(x0), we can take the maximum on the right-hand side and

obtainV (x0 + s)− V (x0)

s≥ max

y∗0∈Y ∗(x0)[∆f(s, y∗0) + λ∗s∆h(s, y∗0)]. (81)

Consider function Ψ defined as

Ψ(s, λ) = maxy∗0∈Y ∗(x0)

[∆f(s, y∗0) + λ∆h(s, y∗0)] (82)

so that the expression on the right-hand side of (81) is Ψ(s, λ∗s). Since Y ∗(x0) is compact by

Lemma 1 (ii), it follows from the Maximum Theorem that Ψ is a continuous function of (s, λ).

Further, since λ∗s ∈ Λ∗(x0 + s) and Λ∗ is an upper hemi-continuous correspondence by Lemma 1

(iii), we obtain

lim infs→0+

Ψ(s, λ∗s) ≥ minλ∗0∈Λ∗(x0)

Ψ(0, λ∗0) = minλ∗0∈Λ∗(x0)

maxy∗0∈Y ∗(x0)

[∂f

∂x(x0, y

∗0) + λ∗0

∂h

∂x(x0, y

∗0)] (83)

35

where we used the scalar-product notation λ∗0∂h∂x

=∑

i λ∗i0∂hi∂x

. It follows from (83) and (81) that

lim infs→0+

V (x0 + s)− V (x0)

s≥ min

λ∗0∈Λ∗(x0)max

y∗0∈Y ∗(x0)

[∂f∂x

(x0, y∗0) + λ∗0

∂h

∂x(x0, y

∗0)]. (84)

Similar to (78) and (79), we have

V (x0 + s) ≤ L(x0 + s, y∗s , λ∗0) (85)

and

V (x0) ≥ L(x0, y∗s , λ

∗0), (86)

for every λ∗s ∈ Λ∗(x0 + s) and y∗s ∈ Y ∗(x0 + s). Subtracting side-by-side and dividing by s > 0,

we obtainV (x0 + s)− V (x0)

s≤ ∆f(s, y∗s) + λ∗0∆h(s, y∗s). (87)

Taking the minimum over λ∗0 ∈ Λ∗(x0) on the right-hand side of (87) results in

V (x0 + s)− V (x0)

s≤ min

λ∗0∈Λ∗(x0)[∆f(s, y∗s) + λ∗0∆h(s, y∗s)]. (88)

Consider function Φ defined as

Φ(s, y) = minλ∗0∈Λ∗(x0)

[∆f(s, y) + λ∗0∆h(s, y)]

so that the expression on the right-hand side of (88) is Φ(s, y∗s). It follows from the Maximum

Theorem that Φ is a continuous function of (s, y). Using upper hemi-continuity of correspon-

dence Y ∗ (see Lemma 1 (iii)), we obtain

lim sups→0+

Φ(s, y∗s) ≤ maxy∗0∈Y ∗(x0)

Φ(0, y∗0) = maxy∗0∈Y ∗(x0)

minλ∗0∈Λ∗(x0)

[∂f

∂x(x0, y

∗0) + λ∗0

∂h

∂x(x0, y

∗0)]. (89)

It follows now from (89) and (88) that

lim sups→0+

V (x0 + s)− V (x0)

s≤ max

y∗0∈Y ∗(x0)min

λ∗0∈Λ∗(x0)

[∂f∂x

(x0, y∗0) + λ∗0

∂h

∂x(x0, y

∗0)]. (90)

It holds (see Lemma 36.1 in Rockafellar (1970)) that

maxy∗0∈Y ∗(x0)

minλ∗0∈Λ∗(x0)

[∂f∂x

(x0, y∗0)+λ∗0

∂h

∂x(x0, y

∗0)]≤ min

λ∗0∈Λ∗(x0)max

y∗0∈Y ∗(x0)

[∂f∂x

(x0, y∗0)+λ∗0

∂h

∂x(x0, y

∗0)].

(91)

36

It follows from (84), (90) and (91) that the right-hand side derivative V ′(x0+) exists and is given

by

V ′(x0+) = maxy∗0∈Y ∗(x0)

minλ∗0∈Λ∗(x0)

[∂f∂x

(x0, y∗0) + λ∗0

∂h

∂x(x0, y

∗0)]

where the order of maximum and minimum does not matter. This establishes eq. (9) of Theorem

1. The proof of (10) is similar. 2

Proof of Theorem 2: By Theorem 23.2 in Rockafellar (1970), φ ∈ ∂V (x0) if and only if

V ′(x0; x)φ ≤ xφ for every x such that x0 + x ∈ X. Using (11), we obtain that φ ∈ ∂V (x0)

if and only if

minλ∗0∈Λ∗(x0)

[Dxf(x0, y

∗0) +

k∑i=1

λ∗i0Dxhi(x0, y∗0)]x ≤ φx for every x, (92)

for every y∗0 ∈ Y ∗(x0), where we used the fact that inequality (92) holds for every y∗0 if and only

if it holds for the maximum over y∗0. The left-hand side of (92), as a function of x, is the negative

of the support function of the set

⋃λ∗0∈Λ∗(x0)

{Dxf(x0, y∗0) +

k∑i=1

λ∗i0Dxhi(x0, y∗0)}. (93)

Since Λ∗(x0) is convex and compact, the set (93) is compact and convex. Theorem 13.1 in

Rockafellar (1970) implies that (92) is equivalent to

φ ∈⋃

λ∗0∈Λ∗(x0)

{Dxf(x0, y∗0) +

k∑i=1

λ∗i0Dxhi(x0, y∗0)}

for every y∗0 ∈ Y ∗(x0). Consequently, φ ∈ ∂V (x0) if and only if

φ ∈⋂

y∗0∈Y ∗(x0)

⋃λ∗0∈Λ∗(x0)

{Dxf(x0, y∗0) +

k∑i=1

λ∗i0Dxhi(x0, y∗0)}.

2

Proof of Corollary 2: If Λ∗(x0) is a singleton set or hi does not depend on x for every i, then

⋃λ∗0∈Λ∗(x0)

{Dxf(x0, y∗0) +

k∑i=1

λ∗i0Dxhi(x0, y∗0)}

37

is a singleton set for every y∗0. The intersection of singleton sets in (15) can either be a singleton

set or an empty set. Since V is concave, the subdifferential ∂V (x0) is non-empty, and hence it

must be singleton. This proves that V is differentiable at x0. 2

Proof of Theorem 3: The proof is similar to that of Theorem 2. Using (11) and Theorem 23.2 in

Rockafellar (1970), we obtain that φ ∈ ∂V (x0) if and only if

maxy∗0∈Y ∗(x0)

[Dxf(x0, y

∗0) +

k∑i=1

λ∗i0Dxhi(x0, y∗0)]x ≥ φx for every x, (94)

for every λ∗0 ∈ Λ∗(x0). The left-hand side of (94) is the support function of the compact (but not

necessarily convex) set

⋃y∗0∈Y ∗(x0)

{Dxf(x0, y∗0) +

k∑i=1

λ∗i0Dxhi(x0, y∗0)}.

Theorem 13.1 in Rockafellar (1970) implies that φ ∈ ∂V (x0) if and only if

φ ∈⋂

λ∗0∈Λ∗(x0)

co

( ⋃y∗0∈Y ∗(x0)

{Dxf(x0, y∗0) +

k∑i=1

λ∗i0Dxhi(x0, y∗0)}

).

2

Proof of Corollary 3: The proof is analogous to that of Corollary 2.

Proof of Proposition 3: We prove first that {x∗t} maximizes the Lagrangian in (20) when the

sequence of multipliers is fixed as {λ∗t}. Consider any sequence {xt} ∈ `n∞ such that xt ∈ X and

x∗0 = x0. Let

DT =T∑t=0

βt{F (x∗t , x

∗t+1) + λ∗t+1h(x∗t , x

∗t+1)−

[F (xt, xt+1) + λ∗t+1h(xt, xt+1)

]}. (95)

Using the assumption of concavity of F and hi, we obtain from (95) that

DT ≥T∑t=0

βt{[DxF (x∗t , x

∗t+1) + λ∗t+1Dxh(x∗t , x

∗t+1)]

(x∗t − xt)

+[DyF (x∗t , x

∗t+1) + λ∗t+1Dyh(x∗t , x

∗t+1)] (x∗t+1 − xt+1

)}.

38

Rearranging terms we have

DT ≥∑T−1

t=0 βt{[DyF (x∗t , x

∗t+1) + λ∗t+1Dyh(x∗t , x

∗t+1)

+β[DxF (x∗t+1, x

∗t+2) + λ∗t+2Dxh(x∗t+1, x

∗t++2)

]] (x∗t+1 − xt+1

)}+βT

[DyF (x∗T , x

∗T+1) + λ∗T+1Dyh(x∗T , x

∗T+1)

] (x∗T+1 − xT+1

).

Using Euler equation (21), we obtain

DT ≥ −βT+1[DxF (x∗T+1, x

∗T+2) + λ∗T+2Dxh(x∗T+1, x

∗T+2)

] (x∗T+1 − xT+1

). (96)

Substituting (95) into (96), we obtain

T∑t=0

βt[F (x∗t , x

∗t+1) + λ∗t+1h(x∗t , x

∗t+1)]−

T∑t=0

βt[F (xt, xt+1) + λ∗t+1h(xt, xt+1)

]≥ −βT+1

[DxF (x∗T+1, x

∗T+2) + λ∗T+2Dxh(x∗T+1, x

∗T+2)

] (x∗T+1 − xT+1

)(97)

for every T. Next, we take limits in (97) as T goes to infinity. Since sequences {xt} and {x∗t} are

bounded, the transversality condition (23) implies that the RHS is zero in the limit. Complemen-

tary slackness conditions imply that the first term on the LHS converges to∑∞

t=0 βtF (x∗t , x

∗t+1).

This implies that the second term has a well defined limit and that {x∗t}maximizes the Lagrangian

at {λ∗t}.The proof that {λ∗t} minimizes the Lagrangian in (20) when the sequence of choice variables is

fixed as {x∗t} is straightforward and omitted. 2

Proof of Proposition 5: The proof is similar to that of Proposition 3. We show first that {µ∗t , γ∗t}minimizes the Lagrangian in (43) when {c∗t , λ∗t} are fixed. Consider any sequence {µt, γt} such

that {µt} ∈ `m∞ and µt ≥ 0, γt ≥ 0 and µ0 = µ. Let DT be the difference between date-T partial

sums of the Lagrangians for {µ∗t , γ∗t} and {µt, γt}. We have

DT =T∑t=0

βt

{m∑i=1

[∆µi,t+1

(u(c∗i,t)− vi(yi,t)

)+ ∆µi,tvi(yi,t)

−λ∗i,t+1(∆µi,t+1 −∆µi,t) + (γ∗t+1 − γt+1)(yi,t − c∗i,t)]},

where ∆µt = µ∗t −µt. It follows from the Euler equation (44) and complementary slackness that

DT = βT+1m∑i=1

{−[µ∗i,T+1 − µi,T+1][vi(yi,T+1) + λ∗i,T+2]− γt+1(yi,t − c∗i,t)

}. (98)

39

Since γt+1 ≥ 0 and c∗i,t ≤ yi,t, it follows from (98) that

DT ≤ −βT+1m∑i=1

[µ∗i,T+1 − µi,T+1][vi(yi,T+1) + λ∗i,T+2]. (99)

Since µi,T and µ∗i,T are bounded, the transversality condition (45) implies that the limit on the

RHS of (99) is zero. Therefore limT→∞DT ≤ 0.

Next we prove that {c∗t , λ∗t} maximizes the Lagrangian in (43) when {µ∗t , γ∗t} are fixed. Let

DT =T∑t=0

βt

{m∑i=1

[µ∗i,t+1(u(c∗i,t)− u(ci,t)) + γ∗t+1(ci,t − c∗i,t)

]}.

Using concavity of u, we have

DT ≥T∑t=0

βt

{m∑i=1

[µ∗i,t+1u

′(c∗i,t)(c∗i,t − ci,t) + γ∗t+1(ci,t − c∗i,t)

]}. (100)

Using the first-order condition γ∗t+1 = µ∗i,t+1u′(c∗i,t), it follows from (100) that DT ≥ 0 and

consequently limT→∞ DT ≥ 0. 2

40

References

L. Benveniste and J. Scheinkman (1979), “Differentiability of the Value Function in Dynamic

Models of Economics” Econometrica, 47, 727–732.

J.-M. Bonnisseau and C. Le Van (1996), “On the Subdifferential of the Value Function in Eco-

nomic Optimization Problems,” Journal of Mathematical Economics, 25, 55–75.

A. Clausen and C. Strub (2016), “A General and Intuitive Envelope Theorem, ” Edinburgh School

of Economics Discussion Paper Series, 274.

W. D. Dechert (1992): “Lagrange Multipliers in Infinite Horizon Discrete Time Optimal Control

Models” Journal of Mathematical Economics, 9, 285–302.

J. Gauvin and F. Dubeau (1982), “Differential Properties of the Marginal Function in Mathemat-

ical Programming,” Mathematical Programming Study, 19, 101–119.

L. Guerrieri and M. Iacoviello (2015), “OccBin: A toolkit for solving dynamic models with

occasionally binding constraints easily,” Journal of Monetary Economics, 70, 22–38.

T. Kim (1993), “Differentiability of the Value Function,” Seoul Journal of Economics, 6, 257–

265.

T.V. Koeppl (2006), “Differentiability of the Efficient Frontier when Commitment to Risk Sharing

is Limited,” The B.E. Journal of Macroeconomics, 6(1),1-6.

J. Kyparisis (1985), “On Uniqueness of Kuhn-Tucker Multipliers in Nonlinear Programming,”

Mathematical Programming, 32, 242–246.

L. Ljungqvist and T. Sargent (2018): Recursive Macroeconomic Theory (Fourth Edition). The

MIT Press.

A. Marcet and R. Marimon (2019), ”Recursive Contracts,” ” Econometrica, 87(5), 1589-1631.

A. Mele A. (2014): “Repeated Moral Hazard and Recursive Lagrangians,”Journal of Economic

Dynamics and Control 15(2), 501-525.

P. Milgrom and I. Segal (2002), “Envelope Theorems for Arbitrary Choice Sets” Econometrica,

70, 583–601.

M. Messner and N. Pavoni (2004), “On the Recursive saddle-point Method,” IGIER Working

paper 255, Universita Bocconi.

M. Messner, N. Pavoni and C. Sleet (2012): “Recursive Methods for Incentive Problems,”Review

of Economic Dynamics 42(C), 69-85.

41

O. Morand, K. Reffett and S. Tarafdar (2015), “A Nonsmooth Approach to Envelope Theorems,”

Journal of Mathematical Economics, 61, 157–165.

O. Morand, K. Reffett and S. Tarafdar (2018), “Generalized Envelope Theorems: Applications to

Dynamic Programming,” Journal of Optimization Theory and Applications, 176, 650–687.

D. Oyama and T. Takenawa (2018), “On the (Non-)Differentiability of the Optimal Value Func-

tion when the Optimal Solution is Unique,” Journal of Mathematical Economics, 76, 21–32.

J. P. Rincon-Zapatero and M. Santos (2009), “Differentiability of the Value Function without

Interiority Assumptions.” Journal of Economic Theory, 144, 1948-1964.

R.T. Rockafellar. Convex analysis. Princeton University Press, 1970.

R.T. Rockafellar, The Theory of Subgradients and its Applications to Problems of Optimization.

Convex and Nonconvex Functions. Heldermann Verlag, Berlin, 1981.

R.T. Rockafellar (1984), Directional Differentiability of the Optimal Value Function in a Nonlin-

ear Programming Problem. ” Mathematical Programming Study, 21, 213–226.

N.L. Stokey, R. E. Lucas and E. C. Prescott, Recursive Methods in Economic Dynamics. Cam-

bridge, Ma.: Harvard University Press, 1989.

42