© 2009 Warren B. Powell© 2008 Warren B. Powell Slide 1
SMART:A Stochastic Multiscale Energy Policy Modelusing Approximate Dynamic Programming
Power Systems Modeling ConferenceUniversity of Florida
March, 2009
Warren PowellAbraham George
Alan LamontJeffrey Stewart
© 2009 Warren B. Powell, Princeton University
© 2009 Warren B. Powell
Goals for an energy policy model Potential policy questions
» How do we design policies to achieve energy goals (e.g. 20% renewables by 2015) with a given probability?
» How does the imposition of a carbon tax change the likelihood of meeting this goal?
» What might happen if ethanol subsidies are reduced or eliminated?» How would a long-term rise or fall in oil prices affect
investments? Some challenges:
» The marginal value of wind and solar farms depends on the ability to work with intermittent supply.
» The impact of intermittent supply will be affected by our use of storage.
» The need for storage (and the value of wind and solar) depends on the entire portfolio of energy producing technologies.
© 2009 Warren B. Powell
Demand modeling
Commercial electric demand
7 days
© 2009 Warren B. Powell
Intermittent energy sources
Wind speed
Solar energy
© 2009 Warren B. Powell
Wind
30 days
1 year
© 2009 Warren B. Powell
Storage
Batteries Ultracapacitors
FlywheelsHydroelectric
© 2009 Warren B. Powell
Long term uncertainties….
2010 2015 2020 2025 2030
Tax policy Batteries
Solar panels Carbon capture and sequestration
Price of oil
Climate change
© 2009 Warren B. Powell
Goals for an energy policy model
Model capabilities we are looking for» Multi-scale
• Multiple time scales (hourly, daily, seasonal, annual, decade)• Multiple spatial scales• Multiple technologies (different coal-burning technologies,
new wind turbines, …)• Multiple markets
– Transportation (commercial, commuter, home activities)– Electricity use (heavy industrial, light industrial, business,
residential)– ….
» Stochastic (handles uncertainty)• Hourly fluctuations in wind, solar and demands• Daily variations in prices and rainfall• Seasonal variations in weather• Yearly variations in supplies, technologies and policies
© 2009 Warren B. Powell
Prior work
Existing deterministic models» MARKAL, NEMS, EFOM, META*Net» Use linear programming with nonlinear market equilibration» Typically focus on long horizons (decades) with long time steps
(1-5 years)» Not set up to handle uncertainty or fine-grained spatial and
temporal features
Techniques incorporating uncertainty» LP-averaging – Stochastic MARKAL » Decision trees – SOCRATES» Stochastic programming – SETSTOCH » Markov decision processes (you have to be kidding)
© 2009 Warren B. Powell
Decision making under uncertainty
Mixing optimization and uncertainty
TimeEnergysources
?
??
?
? “large optimization model (e.g. NEMS, MARKAL, …)”
© 2009 Warren B. Powell
Decision making under uncertainty
Mixing optimization and uncertainty
TimeEnergysources
6.2
12.334
No
Solar
142
Scenario 1
© 2009 Warren B. Powell
Decision making under uncertainty
Mixing optimization and uncertainty
TimeEnergysources
3.6
18.917
No
Wind
89.1Scenario 2
© 2009 Warren B. Powell
Decision making under uncertainty
Mixing optimization and uncertainty
TimeEnergysources
5.9
22.314
Yes
Solar
117Scenario 3
© 2009 Warren B. Powell
5.9
22.314
Yes
Solar
117
3.6
18.917
No
Wind
89.1
Decision making under uncertainty
6.2
12.334
No
Solar
142Scenario 1
Scenario 2
Scenario 3
Now we have to combine the results of these three optimizations to make decisions.
© 2009 Warren B. Powell
Making decisions
Stochastic programming using scenario trees
Jan Feb Mar Apr May… Sep… Jan
Now
Known Unknown
0.3
0.4 probability (weight)
0.3
Wetter
DrierAverage over 30Flow forecast scenarios
15% exceedence
50% exceedence
85% exceedence
15% exceedence
50% exceedence
85% exceedence
Stochastic programming does not “cheat”, but it does not scale. It is best designed for coarse-grained sources of uncertainty, but will not handle fine-grained temporal resolution.
© 2009 Warren B. Powell
Stochastic MARKAL
Received Wednesday, March 18 2009
© 2009 Warren B. Powell
Energy resource modeling
SMART: A Stochastic, Multiscale Allocation model for energy Resources, Technology and policy» Stochastic – able to handle different types of uncertainty:
• Fine-grained – Daily fluctuations in wind, solar, demand, prices, …• Coarse-grained – Major climate variations, new government policies,
technology breakthroughs
» Multiscale – able to handle different levels of detail:• Time scales – Hourly to yearly• Spatial scales – Aggregate to fine-grained disaggregate• Activities – Different types of demand patterns
» Decisions• Hourly dispatch decisions• Yearly investment decisions• Takes as input parameters characterizing government policies,
performance of technologies, assumptions about climate
© 2009 Warren B. Powell
Energy resource modeling
tttt DRS ,,System state:
Resource state
Market demands
“System parameters”:State of technology (costs, performance)Climate, weather (temperature, rainfall, wind)Government policies (tax rebates on solar panels)Market prices (oil, coal)
© 2009 Warren B. Powell Slide 19
Energy resource modeling
The decision variable:
New capacity
Retired capacity
Storage
:
Type
Location
Technology
tx for each
© 2009 Warren B. Powell
Energy resource modeling
Exogenous information:
Slide 20
ˆ ˆ ˆNew information = , ,t t t tW R D
ˆ Exogenous changes in capacity, reserves
ˆ New demands for energy from each source
ˆ Exogenous changes in parameters.
t
t
t
R
D
Information can be: Fine-grained:
Wind, solar, demand, prices, …
Coarse-grained:Changes in technologies, policies, climate
© 2009 Warren B. Powell Slide 21
Energy resource modeling
The transition function
1 1( , , )Mt t t tS S S x W
Also known as the “system model,” “plant model” or just “model.”
All the physics of the problem.
© 2009 Warren B. Powell Slide 22
Introduction to ADP The challenge of dynamic programming:
Problem: Three curses of dimensionality» State space
» Outcome space (the expectation)
» Action space (x is a vector)
1 1( ) max ( , ) ( ) |t t t t t t t tx
V S C S x E V S S
X
© 2009 Warren B. Powell Slide 23
Introduction to ADP
The computational challenge:
How do we find ? 1 1( )t tV S
How do we compute the expectation?
How do we find the optimal solution?
1 1( ) max ( , ) ( ) |t t t t t t t tx
V S C S x E V S S
X
© 2009 Warren B. Powell Slide 24
Introduction to ADP
We first use the value function around the post-decision state variable, removing the expectation:
We then replace the value function with an approximation that we estimate using machine learning techniques:
( ) max ( , ) ( , )x xt t t t t t t t t
xV S C S x V S S x
X
( ) max ( , ) ( , )xt t t t t t t t t
xV S C S x V S S x
X
© 2009 Warren B. Powell© 2008 Abraham George 25
Making decisions
Following an ADP policy
© 2009 Warren B. Powell© 2008 Abraham George 26
Making decisions
Following an ADP policy
© 2009 Warren B. Powell© 2008 Abraham George 27
Making decisions
Following an ADP policy
© 2009 Warren B. Powell 28
Approximate dynamic programming
With luck, the objective function will improve steadily
Objec tive func tion
97
97.5
98
98.5
99
99.5
100
1 11 21 31 41 51 61 71 81
Itera tion
© 2009 Warren B. Powell© 2008 Warren B. Powell Slide 29
Approximate dynamic programming
Step 1: Start with a pre-decision state
Step 2: Solve the deterministic optimization using
an approximate value function:
to obtain .
Step 3: Update the value function approximation
Step 4: Obtain Monte Carlo sample of and
compute the next pre-decision state:
Step 5: Return to step 1.
, 1 ,1 1 1 1 1 1 ˆ( ) (1 ) ( )n x n n x n n
t t n t t n tV S V S v
1 ,ˆ max ( , ) ( ( , ) )n n n M x nt x t t t t t tv C S x V S S x
ntx
ntS
( )ntW
1 1( , , ( ))n M n n nt t t tS S S x W
Simulation
Deterministicoptimization
Recursivestatistics
© 2009 Warren B. Powell© 2008 Warren B. Powell Slide 30
Approximate dynamic programming
Step 1: Start with a pre-decision state
Step 2: Solve the deterministic optimization using
an approximate value function:
to obtain .
Step 3: Update the value function approximation
Step 4: Obtain Monte Carlo sample of and
compute the next pre-decision state:
Step 5: Return to step 1.
, 1 ,1 1 1 1 1 1 ˆ( ) (1 ) ( )n x n n x n n
t t n t t n tV S V S v
1 ,ˆ max ( , ) ( ( , ) )n n n M x nt x t t t t t tv C S x V S S x
ntx
ntS
( )ntW
1 1( , , ( ))n M n n nt t t tS S S x W
Simulation
Deterministicoptimization
Recursivestatistics
© 2009 Warren B. Powell
Approximate dynamic programming
Optimization Simulation
Cplex
Approximate dynamic programming combines simulation and optimization in a rigorous yet flexible framework.
OptimizationStrengths
Produces optimal decisions.Mature technology
WeaknessesCannot handle uncertainty.Cannot handle high levels of
complexity
SimulationStrengths
Extremely flexibleHigh level of detailEasily handles uncertainty
WeaknessesModels decisions using
user-specified rules.Low solution quality.
© 2009 Warren B. Powell Slide 32
The investment problem
oiltx
2008
oiltR ˆ oil
tD ˆ oiltˆ oil
tR
New information 2009
1oiltR 1
oiltx 1
ˆ oiltD 1
ˆ oilt 1
ˆ oiltR
New information
windtxwind
tR ˆ windtD ˆ wind
tˆ windtR 1
windtR 1
windtx 1
ˆ windtD 1
ˆ windt 1
ˆ windtR
coaltxcoal
tR ˆ coaltD ˆ coal
tˆ coaltR 1
coaltR 1
coaltx 1
ˆ coaltD 1
ˆ coalt 1
ˆ coaltR
nat gastxnat gas
tR ˆ nat gastD ˆ nat gas
tˆ nat gastR 1
nat gastx 1
nat gastR 1
ˆ nat gastD 1
ˆ nat gast
ˆ nat gastR
© 2009 Warren B. Powell Slide 33
The dispatch problem
Hourly electricity “dispatch” problem
© 2009 Warren B. Powell Slide 34
Energy resource modeling
Hourly model» Decisions at time t impact t+1 through the amount of water held in
the reservoir.Hour t Hour t+1
© 2009 Warren B. Powell Slide 35
Energy resource modeling
Hourly model» Decisions at time t impact t+1 through the amount of water held in
the reservoir.
Value of holding water in the reservoir for future time periods.
Hour t
© 2009 Warren B. Powell Slide 36
Energy resource modeling
© 2009 Warren B. Powell Slide 37
Energy resource modeling
2008
Hour 1 2 3 4 87602009
1 2
© 2009 Warren B. Powell Slide 38
Energy resource modeling
2008
Hour 1 2 3 4 87602009
1 2
© 2009 Warren B. Powell Slide 39
Energy resource modeling
2008 2009
© 2009 Warren B. Powell Slide 40
Energy resource modeling
2008 2009
2009oil
2009wind
2009nat gas
2009coal
© 2009 Warren B. Powell Slide 41
Energy resource modeling
2008 2009 2010 2011 2038
© 2009 Warren B. Powell Slide 42
Energy resource modeling
2008 2009 2010 2011 2038
© 2009 Warren B. Powell Slide 43
Energy resource modeling
2008 2009 2010 2011 2038
~5 seconds ~5 seconds ~5 seconds ~5 seconds ~5 seconds
© 2009 Warren B. Powell Slide 44
Energy resource modeling
Use statistical methods to learn the value of resources in the future.
Resources may be:» Stored energy
• Hydro• Flywheel energy• …
» Storage capacity• Batteries• Flywheels• Compressed air
» Energy transmission capacity• Transmission lines• Gas lines• Shipping capacity
» Energy production sources• Wind mills• Solar panels• Nuclear power plants
Amount of resourceV
alue
( )t tV R
© 2009 Warren B. Powell
Energy resource modeling
Following sample paths» Demands, prices, weather, technology, policies, …
Slide 45
2030
Achievedgoal w/
Prob. 0.70
Met
ric
(e.g
. % r
enew
able
)
ˆ ˆ ˆ, ,t t t tW R D
Need to consider:Little noise (wind, rain, demand, prices, …)Big noise (technology, policy, climate, …)
© 2009 Warren B. Powell
Convergence analysis
» For scalar, piecewise linear approximations:• Nascimento, J. and W. B. Powell, “An Optimal Approximate
Dynamic Programming Algorithm for the Lagged Asset Acquisition Problem,” Mathematics of Operations Research, Vol. 34, No. 1, pp. 210-237 (2009).
• Nascimento, J. and W. B. Powell, “Optimal approximate dynamic programming algorithms for a general class of storage problems,” under review at SIAM J. on Control and Optimization.
» For general continuous states and actions:• J. Ma and W. B. Powell, “Convergence Proofs for Least Squares
Policy Iteration Algorithm of High-Dimensional Infinite Horizon Markov Decision Process Problems,” under review at Machine Learning.
» Stepsizes:• George, A. and W. B. Powell, “Adaptive Stepsizes for Recursive
Estimation with Applications in Approximate Dynamic Programming,” Machine Learning, Vol. 65, No. 1, pp. 167-198, (2006).
© 2009 Warren B. Powell
Convergence analysis
Research on approximations:» George, A., W.B. Powell and S. Kulkarni, “Value Function
Approximation Using Hierarchical Aggregation for Multiattribute Resource Management,” Journal of Machine Learning Research, Vol. 9, pp. 2079-2111 (2008).
» L. Hannah, D. Blei and W. B. Powell, “Density Estimation and Regression with Dirichlet Process-Generalized Linear Mixture Models,” in preparation for submission to Machine Learning.
» J. Ma and W. B. Powell, “A Convergent Algorithm for Continuous State Value Function Approximations Using Kernel Regression,” in preparation.
© 2009 Warren B. Powell
Energy resource modeling
Benchmarking» Compare ADP to optimal LP for a deterministic
problem• Annual model
– 8,760 hours over a single year– Focus on ability to match hydro storage decisions
• 20 year model– 24 hour time increments over 20 years– Focus on investment decisions
» Comparisons on stochastic model• Stochastic rainfall analysis
– How does ADP solution compare to LP?• Carbon tax policy analysis
– Demonstrate nonanticipativity
Slide 48
© 2009 Warren B. Powell 49
0.00%
0.50%
1.00%
1.50%
2.00%
2.50%
0 50 100 150 200 250 300 350 400 450 500
Iterations
Per
cent
age
erro
r fr
om o
ptim
al
0.06% over optimal
Energy resource modeling ADP objective function relative to optimal LP
2.50
2.00
1.50
1.00
0.50
0.00
© 2009 Warren B. Powell
Optimal from linear program
Energy resource modeling
Optimal from linear program
Reservoir level
Demand
Rainfall
© 2009 Warren B. Powell 51
Approximate dynamic programming
Energy resource modeling
ADP solution
Reservoir level
Demand
Rainfall
© 2009 Warren B. Powell
Optimal from linear program
Energy resource modeling
Optimal from linear program
Reservoir level
Demand
Rainfall
© 2009 Warren B. Powell
Approximate dynamic programming
Energy resource modeling
ADP solution
Reservoir level
Demand
Rainfall
© 2009 Warren B. Powell 54
Annual energy model
0
100
200
300
400
500
600
700
0 100 200 300 400 500 600 700 800
Time period
Pre
cipi
tatio
n
Sample paths
© 2009 Warren B. Powell 55
Res
ervo
ir le
vel Optimal for individual
scenarios
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
0 100 200 300 400 500 600 700 800
Time period
ADP
Energy resource modeling
© 2009 Warren B. Powell Slide 56
Multidecade energy model
Optimal vs. ADP – daily model over 20 years
0.00%
5.00%
10.00%
15.00%
20.00%
25.00%
30.00%
35.00%
40.00%
0 100 200 300 400 500 600Iterations
Perc
en
t o
ver
op
tim
al
0.24% over optimal
© 2009 Warren B. Powell
Energy policy modeling
Traditional optimization models tend to produce all-or-nothing solutions
Cost differential: IGCC - Pulverized coal
Pulverized coal is cheaperIGCC is cheaper
Investment in IGCC
Traditionaloptimization
Approximate dynamicprogramming
© 2009 Warren B. Powell
Energy policy modeling
Policy study: » What is the effect of a potential (but uncertain) carbon
tax in year 8?
1 2 3 4 5 6 7 8 9
Year
Car
bon
tax
0
© 2009 Warren B. Powell
Energy policy modeling
10000
20000
30000
40000
50000
60000
70000
80000
0 2 4 6 8 10 12 14 16 18 20
Year
Inst
alle
d C
apac
ity
Renewable technologies
Carbon-based technologies
No carbon tax
© 2009 Warren B. Powell
Energy policy modeling
10000
20000
30000
40000
50000
60000
70000
80000
0 2 4 6 8 10 12 14 16 18 20
Year
Inst
alle
d C
apac
ity
With carbon tax
Carbon-based technologies
Renewable technologies
Carbon tax policy unknown
Carbon tax policy determined
© 2009 Warren B. Powell
Energy policy modeling
10000
20000
30000
40000
50000
60000
70000
80000
0 2 4 6 8 10 12 14 16 18 20
Year
Inst
alle
d C
apac
ity
With carbon tax
Carbon-based technologies
Renewable technologies
© 2009 Warren B. Powell
Conclusions
Capabilities» SMART can handle problems with over 300,000 time
periods so that it can model hourly variations in a long-term energy investment model.
» It can simulate virtually any form of uncertainty, either provided through an exogenous scenario file or sampled from a probability distribution.
» Accurate modeling of climate, technology and markets requires access to exogenously provided scenarios.
» It properly models storage processes over time.» Current tests are on an aggregate model, but the
modeling framework (and library) is set up for spatially disaggregate problems.
© 2009 Warren B. Powell
Conclusions
Limitations» Value function approximations capture the resource
state vector, but are limited to very simple exogenous state variations.
» More research is needed to test the ability of the model to use multiple storage technologies.
» Extension to spatially disaggregate model will require significant engineering and data.
» Run times will start to become an issue for a spatially disaggregate model.
© 2009 Warren B. Powell© 2008 Warren B. Powell Slide 64