michaudValueFn

ECON 4741H - Quantitative Analysis of the Macroeconomy Amanda M Michaud

Numerical Methods: Value Function Iteration

We have spent many weeks working with local approximation methods to compute statistics

from data generated by real business cycle models. Now we will conclude the course by learn-

ing numerical methods suitable for a larger range of problems. In particular, Value Function

Iteration will allow us to characterize properties of the policy function in problems for which

the policy function does not have a closed-form solution. In order to do this, we will need to

apply our knowledge of recursive dynamic programming.

1 The Value Function

As you recall, the key feature of dynamic programming problems is that we are solving the

same type of maximization problem every period, subject to a different state variable. In our

cake-eating problem, we solve how much cake to eat (ω − ω′) in the current period given how

much to save for future periods (ω′). We don’t care why we have cake size ω, we only care about

how much we have today and how much we will save for the future. We also take as given that,

when faced with the similar problem tomorrow, we will behave optimally. Therefore, we don’t

need to solve tomorrow’s decision problem today. In order to know how much to leave for the

future, we only need to know the present discounted value of all cake sizes we could save. This

is given by the value function V (ω).1

V (ω) = maxω′∈[0,ω]

u(ω − ω′) + βV (ω′)

We have discussed how the above problem may be a little tricky in general because V (·) is

an unknown function. In general we must prove this function exists and that the problem is

well defined. See previous notes for some sufficient conditions.

We will now consider the functional equation as defining an operator T in function space:

Vn = TVn−1. Explicitly2:

Vn(ω) = maxω′∈[0,ω]

u(ω − ω′) + βVn−1(ω′)

Just as a function is defined as an operator that maps real numbers into real number, this

operator maps functions into functions. We will consider problems that satisfy the following:

• Constraint set is compact.

• Current period return function u(·) is strictly concave, strictly monotone, and continuous.

1The following are synonymous: Value Function, Bellman Equation, Recursive Dynamic Program, Functional

Equation2Don’t be confused by the n subscript denoting the iteration number and the time subscript t we used for

finite horizon models. We will only consider infinite-horizon models when dealing with dynamic programming.

Alternatively you could put time in the state space and specify V(T)=0 for your finite horizon

1


• Discounting: β ∈ [0, 1)

• Vn−1 is monotone.

Contraction Mapping Theorem

Under suitable assumptions, the Bellman equation defines an operator Vn =

TnV0 that:

• Will converge to a fixed point V ∗ = TV ∗

• Will converge to a the same fixed point V ∗ given any initial value V0

These assumptions loosely serve as sufficient conditions to apply the Contraction Map-

ping Theorem which states the operator T maps monotone, continuous functions into strictly

monotone, strictly concave, continuous functions. This means if we choose some function Vn−1

that is monotone and continuous and then apply the operator T we will be left with a strictly

monotone, strictly concave, continuous function Vn. Furthermore, the operator defines a con-

traction towards a unique fixed point. This means:

• d(Vn − Vn−1)→ 0 as n→∞.

In other words, for a large number of iterations we will find a unique function such that

when put into our operator, we get the same function back: V = TV .

• Starting guess V0 can be any monotone, continuous function and we will converge to the

same point.

Here is a simple illustration of what is going on. Let’s take the functional equation for the

cake eating problem with utility u(ω) = ln(ω) and as an input function we will use V0(ω′) =

ln(ω′). Substituting this into the functional equation leaves us with V1(ω) as the value of a

well-defined maximization problem:

V1(ω) = maxω′∈[0,ω]

ln(ω − ω′) + βV0(ω′) (1)

V1(ω) = maxω′∈[0,ω]

ln(ω − ω′) + βln(ω′) (2)

Now we solve the maximization problem to figure out the value of V1(ω). It is a concave function,

so we know the maximum is where the first derivative with respect to the choice variable equals

zero. Our first order conditions give a policy function that holds for all ω: ω′ = βωβ+1 . We can

now substitute the policy function into the functional equation to get the value for all ω:

V1(ω) = ln(ω − βω

β + 1) + βln(

βω

β + 1) (3)

(4)

2


Our operator has indeed transformed a monotone, continuous function into a strictly mono-

tone, strictly concave, continuous function. We can continue to repeat this process. Define

A = ln( 11+β ) + βln( β

1+β ):

V2(ω) = maxω′∈[0,ω]

ln(ω − ω′) + β[(1 + β)ln(ω′) +A] (5)

(6)

Which gives new policy: ω′ = β+β2

1+β+β2ω. We can see that this is quickly converging to the

analytic solution: ω′ = (1− β) ∗ ω, but that it would take infinite time to get there exactly.

2 Value Function Iteration

General Procedure for Value Function Iteration

1. Start with an arbitrary strictly increasing and strictly monotone function

V0

2. Compute V1 = TV0

3. Iterate Vn = TVn−1

4. Stop when d(Vn, Vn−1) < ε Where d(·) is some notion of distance between

two functions and ε is a small positive number.

2.1 Numeric Considerations

Matlab is not a symbolic language. It is not going to understand how to define and store new

functions. To see this, consider what it takes to graph any function in Matlab. You must first

define a vector x of n discrete points and then compute another set of n points defined by

yi(xi) = F (xi). Asking Matlab to keep track of a new function G(x) = H(F (x)) isn’t going

to go over so well. Therefore, instead of keeping track of functions, we will keep track of sets

of points and the corresponding values of the functions we are interested in at those points.

Much like when we plot lines, if we use enough points, we will have a pretty good idea of the

properties of the functions we are interested in.

2.1.1 1) Defining Grids

Our value functions are functions of state variables. Therefore, we will need to define a grid

points representing these state variables to be able to characterize the value function. A grid

is a set of n points {xi}ni=1 ∈ [x1, xn] such that xi < xj ∀j > i. For the state variables we will

call this collection of points S ≡ {si}nsi=1. Which points should we select? We must make the

following considerations:

3


Which interval? This is rather obvious. We should define the grid for the state variable(s)

over the range of values we are interested in! For Example:

• Cake Eating Problem: [0, ω̄] where ω̄ is the largest size of cake we think somebody

facing this problem would start out with.

• Neoclassical Growth Model: [(K), K̄] such that Kss ∈ [(K), K̄] and the interval

captures a reasonable range of business cycle fluctuations, say the steady state value

for a low and high shock.

How Many Points? The accuracy of our characterization of the value function is obviously

increasing in the number of points we choose, however you will see our computational

time is increasing in the number of points as well. In fact, for problems with multiple

state variables, computation time increases exponentially with grid size. Economists,

computer scientists, and others in the know refer to this as the Curse of Dimensionality.

Choosing the right n seems like a nice optimization problem in itself. Selecting higher n

has diminishing returns in accuracy and increasing marginal computational burden. What

is the optimal choice? That requires a metric weighting accuracy and patience. You be

the judge. At the minimum you should run your program for a given sized grid, then

increase the grid size by 5% or so and make sure that your results do not change.

Where to put the points? The easiest choice of placement for grid points is to distribute

them evenly across the interval you would like to study. The Matlab command linspace

does just that:

• xgrid = linspace(xlow,xhigh,n)

Where xlow and xhigh are the lower and upper bounds on the interval of choice and n

is the number of grid points. A better idea is to put more points where the function

has more curvature, ie: where the absolute value of the second derivative is very high.

This can be accomplished by defining a new grid where points are defined by a concave

function over linearly spaced grid points. How do we know where the value function is

likely to have curvature if we do not know what the value function is? A good rule of

thumb would be to use your objective function to get a sense. We will want to compute

a few iterations, graph our function, and then possibly change grid points around.

2.1.2 2) Initial Guess V0

Now that we have a grid for our state variables S ≡ {si}nsi=1, we can define another set of

points V0 ≡ {v0i}nsi=1 that correspond to the function we have in mind evaluated at each si:

v0i = V0(si). How should we choose these points? We know by the contraction mapping theorem

that the operator defined by the Bellman equation converges to a unique function regardless of

the initial guess. However, we are no longer working with functions, we are working with sets

of points that approximate functions. We cannot prove a theorem that says these sets of points

4


will converge to a unique set of points because of approximation error. Therefore we will want

to be careful with our initial guess.

Our initial guess should be guided by the problem itself. What do we think are reasonable

values? One good start is the present discounted value of the objective function given the

current state. For instance, for the cake eating problem we could define:

v0i =1

1− βu(si/

2)

where u() is a concave utility function and si ∈ [0, ω̄] is a point on the state grid. By dividing

by 2 I hope to be in a reasonable place on the utility function in terms of curvature, but to

also still have enough difference between the guess for different points on the grid. If you find

your initial guess is bad, take a look after a few iterations and update your beliefs on what your

guess should look like.

2.1.3 3) Given {v0i}nsi=1 solve for the policy

The next step is to compute the updated guess for the value function as defined by the operator.

To do this we need to solve for the policy given our current ”guess” {v0i}nsi=1. This reduces to:

gi = argmaxju(si, sj) + βv0j

In other words, we only know values of v0 for points we have chosen, so our policy must yield

a state variable for tomorrow that lies on those same grid points. 3 Therefore we have to tell

Matlab: given some current state si find another point on the state grid sj that gives the highest

value of the above problem. The most basic way to do this is the following:

3We will do better later

5


Exploiting Concavity

1. Start with a given point on our state grid si

2. Calculate R(i, 1) = u(si, s1) + βv01 for the first possible policy choice

sj = s1


sj = s2

• if R(i, 2) > R(i, 1), choosing s′ = s2 yields higher value than choos-

ing s′ = s1, so store gi = 2.

4. Keep going: for each subsequent j, calculate R(i, j) = u(si, sj) + βv0j .

• if R(i, j) > R(i, gi), choosing s′ = sj yields higher value than choos-

ing s′ = gi, the policy that has yielded the highest value thus far.

• if R(i, j) < R(i, gi), choosing s′ = sj yields lower value than choos-

ing s′ = gi, so we should stick with our current contender for the

max: s′ = gi.

5. Repeat for all j = 1, 2, ...ns

Now Matlab has a find command that will search over R(i, j) and find i∗, j∗. However, you

should note that this requires making ns× ns calculations and then searching one-by-one over

the grid. For a grid of size 20, that is already 400 calculations and points to sift through. How

can we do better? It can be shown that the operator defined by the Bellman equation maps

increasing and strictly concave functions into strictly increasing and strictly concave functions.

Let’s use these two properties.

6


Exploiting Concavity

1. Start with a given point on our state grid si


sj = s1


sj = s2

• if R(i, 2) > R(i, 1), keep going (calc R(i, 3), compare with R(i, 2)

and so on.

• if R(i, n+ 1) < R(i, n) STOP!!! Because of concavity, we know the

marginal return to increasing the state variable tomorrow will be

smaller and smaller and so every point beyond sn will yield lower

value.

Exploiting Monotonicity

1. Start with the first point on our state grid si

2. Do the above (”Exploiting Concavity”) to find the policy g1 for this

point.

• Because the value function and objective functions are strictly in-

creasing in the state variable, we know that the policy function

should also be strictly increasing. ie: in the cake eating problem,

when we have a larger cake, we eat more today and have more left

over for tomorrow.

• Given s2 > s1 start the above procedure (”Exploiting Concavity”)

with g1 as the first point tested.

• Continue testing subsequent points until one yields a lower value.

Stop here and store g2.

3. Repeat for s3 by starting with g2

4. Repeat for si, i = 4, 5....n

7


2.1.4 4) Calculate updated guess V1

Here we update given our policy and define a new set of points using our policy rule gi for each

i:

v1i = u(si, sgi) + βv0gi ∀i = 1, 2, ...ns

2.1.5 5) Check for convergence

Now we see if we have come ”close enough” to a fixed point. Matlab loves matrices, so lets

define our notion of distance as the following:

err = abs(V1−V2)I

Where I is a vector of size ns filled completely with 1’s. This gives is matrix notation for the

sum of the distance between v0i and v1ifor each point on our grid.

If err¡ε. We can go home. If not, we start the process over with V1 as our new guess for the set

of points that characterize the true value function evaluated at the grid points we have chosen.

define V0 = V1 and repeat this process until we get to an iteration where err¡ε.

3 Closing Remarks

That is all I have for you for now. Value function iteration is a powerful tool for problems

with no analytic solution. It is widely used and there are many refinements on it. In general

these operate by reducing computation time or better approximating the actual function. One

problem you may have noticed in the above is that our choice of policy was restricted to lie on

the state grid. If we have time we will discuss how to do better here.

8

Documents

michaudValueFn