Optimal Decision Making for Electric Vehicles Providing Electric Grid Frequency Regulation:
A Stochastic Dynamic Programming Approach
Jonathan Donadee Ph.D. Student, ECE, Carnegie Mellon University
Marija Ilic IEEE Fellow, Professor of ECE and EPP, Carnegie Mellon University
[email protected] 9th International Conference on Computational Management Science
Imperial College London, UK April 20th, 2012
Support for this research was provided by Fundação para a CiΓͺncia e a Tecnologia (Portuguese Foundation for Science and Technology)
through the Carnegie Mellon Portugal Program
Outline
Problem Introduction
Deterministic Equivalent Problem Model
Stochastic Dynamic Programming Algorithm
Simulation Results
Conclusions
Questions
2
Electrical Grid Frequency Regulation
Electrical Grid AC frequency must be maintained within Β± 0.05 Hz
60Hz (USA)
50Hz (Europe)
Demand < Supply, f
Demand > Supply, f
Some generators follow βAGCβ or βRegulationβ control signal on second to minute basis
Generators bid capacity (MW) into hourly markets
Graphic Source: PJM ISO Training
Power Supply and Demand Balance
Regulation is on seconds to minutes timescale
3
One Dayβs AGC Signal
0 5 10 15 20 25-1.5
-1
-0.5
0
0.5
1
Hours
No
rma
lize
d S
ign
al
1 Day of PJM AGC Signal
4
Motivation
In the future, greater need for fast responding electrical grid resources Compensate renewable resource forecast errors
Mitigate trend of decreasing system inertia smaller imbalances causing larger frequency deviations
Integration of Electric Vehicles Minimize EV charging cost
Deterministic models and methods are not well suited for managing uncertain resources
5
Electric Vehicle Participation in Frequency Regulation EVs can adjust charge rate to follow AGC signal
Weβll focus on charging only strategies, no discharging
Specify preferred charge rate, Pavg
Specify capacity for regulating, B Adjust according to negative of AGC signal scaled by B
6
0 β€ π΅ β€ πππ£π
0 β€ πππ£π β€ ππππ₯ 0 β€ π΅ β€ ππππ₯ β πππ£π
Smart Charging Scenario
Driver arrives at home and plugs in vehicle Inputs time of departure
Inputs inconvenience cost for not finishing on time ($/hr)
Smart charger optimizes Pavg, B decisions for each hour
Smart charger can participate in markets directly, without delay
Picture Source: Ford.com
7
A Stochastic Model is Needed
State of charge can take any non-decreasing path
t1 t2 t3 tf
t
e0
Emax
Average charge rate to hit Emax at tf
Bound of Possible State of Charge
Ebatt
8
A Stochastic Model is Needed
t1 t2 t3 tf
t
e0
Emax
Average charge rate to hit Emax at tf
Bound of Possible State of Charge
Ebatt
State of charge can take any non-decreasing path
Regulation contract is violated if charging finishes early
Regulation Bid Violated
9
A Stochastic Model is Needed
t1 t2 t3 tf
t
e0
Emax
Average charge rate to hit Emax at tf
Bound of Possible State of Charge
Ebatt
State of charge can take any non-decreasing path
Regulation contract is violated if charging finishes early
Driver inconvenienced if vehicle is not charged on time
Regulation Bid Violated
Driver
Inconvenienced
10
A Stochastic Model is Needed
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.80
2
4
6
8
10
12
14
16
18
Integrated Hourly Energy
Co
un
ts
Histogram of July 2011 PJM AGC Signal Energy
t1 t2 t3 tf
t
e0
Emax
Average charge rate to hit Emax at tf
Bound of Possible State of Charge
Regulation Bid Violated Ebatt
Driver
Inconvenienced
11
A Stochastic Model is Needed
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.80
2
4
6
8
10
12
14
16
18
Integrated Hourly Energy
Co
un
ts
Histogram of July 2011 PJM AGC Signal Energy
t1 t2 t3 tf
t
e0
Emax
Average charge rate to hit Emax at tf
Bound of Possible State of Charge
Regulation Bid Violated Ebatt
Driver
Inconvenienced
Providing regulation makes future battery state of charge uncertain
Literature ignores effect of regulation or optimizes considering expected value
Stochastic Model needed to: value risk of regulation contract violation (pro-rated by time)
value risk of inconveniencing EV driver
Optimize choice of average charge rates and regulation contracts size under uncertainty 12
Dynamic Programming Solution
Solve many hour long optimization problems in a backwards recursion
Minimize Expected Future Cost given current time and state of charge Find a single decision to minimize average cost over all future
outcomes ππ,β β Ξ© β
πβ πΈπ,β is a Stochastic Deterministic Equivalent Problem
πβ πΈπ,β = minππ,β ,π΅π,β
πΌπ π½β πΈπ,β, ππ,β , π΅π,β, π βπ + πβ+1 ππ‘π
π
13
DEP and Optimal Value Function
1618
2022
24
Hour 5
Hour 6
0
0.05
0.1
0.15
0.2
0.25
Energy (kWh)
Time
Op
tima
l Va
lue
Fu
nct
ion
Co
st (
$)
Path Bounds
(from Regulation Bid)
Energy in sample Ο
π 6
Energy at Pavg
Solving for V5(16.8) Using 30 Sample Regulation Signals
14
Sample Path Generation
Each DEP uses 30, hour long, AGC signals, π βπ
Sample historical data using crude monte carlo
ππ,β β Ξ© β
Integrate signal over 5 minutes
Becomes normalized energy for discrete state equations
π π»π is a correlated 12 dimensional vector
Assume AGC independent across hours
15
State Equations
πππ = ππβπ
π + βπ β π· β π© β βπ β πΉπβππ β ππβπ
π , π β₯ π, βπ
ππ‘π = πΈπ,β , π‘ = 1, βπ
ππ‘π β₯ ππ‘β1
π , βπ‘, βπ
ππ‘π β€ πΈπππ₯, βπ‘, βπ
π π‘π β₯ 0, βπ‘, βπ
Energy Actually
Consumed
β
π β βπ‘
T0 1Hr16
17
18
19
20
21
22
23
24
En
erg
y (
kW
h)
Time
Example State Dynamics
16
Regulation Contract Risk
If the battery reaches full state of charge, cannot provide regulation
Payment is pro-rated, time-based Regulation contract violation indicator, π’
Allows penalty to be a function of time, not energy π’ is a binary variable
π’π‘π β₯ π’π‘β1
π , π‘ β π‘π, βπ
π’π‘π β ππππ₯ β βπ‘ β₯ π π‘
π, π‘ β π‘π, βπ
ππ‘+1π β₯ π’π‘
π β πΈπππ₯, π‘ β π‘π, , βπ
t
e0
Emax
Average charge rate to hit Emax at tf
Bound of Possible State of Charge
Ebatt
π’1 = 0 π’2 = 0 π’3 = 1
17
Driver Inconvenience Risk When not fully charged by
unplug time Charge at ππππ₯ until battery is
full
ππ =πΈπππ₯βππ‘π
π
ππππ₯ time late on
sample path Ο
πΏ Driverβs inconvenience cost ($/hr)
πΏ β ππ + ππ»+1(πΈπππ₯ β ππ‘ππ)
t1 t2 t3 tf
t
e0
Emax
Average charge rate to hit Emax at tf
Bound of Possible State of Charge
Ebatt
Driver
Inconvenienced
ππ
18
Objective Function (if final decision)
π =1
π ππ» βπ΅π,π» β π π‘
π β βπ‘ β π π‘π
π‘πβ1
π‘=1
+ π β βπ‘ β π΅π,π» β π’π‘π
π‘
+ πΏ β ππ + ππ»+1(πΈπππ₯ β ππ‘ππ)
πβΞ©π
ππ» πΈπ,π» = minππ,π» ,π΅π,π» ππ»ππ,π» β ππ»π΅π,π» + π
19
Objective Function (if final decision)
π =1
π ππ» βπ΅π,π» β π π‘
π β βπ‘ β π π‘π
π‘πβ1
π‘=1
+ π β βπ‘ β π΅π,π» β π’π‘π
π‘
+ πΏ β ππ + ππ»+1(πΈπππ₯ β ππ‘ππ)
πβΞ©π
ππ» πΈπ,π» = minππ,π» ,π΅π,π» ππ»ππ,π» β ππ»π΅π,π» + π
Baseline
Energy Cost
20
Objective Function (if final decision)
π =1
π ππ» βπ΅π,π» β π π‘
π β βπ‘ β π π‘π
π‘πβ1
π‘=1
+ π β βπ‘ β π΅π,π» β π’π‘π
π‘
+ πΏ β ππ + ππ»+1(πΈπππ₯ β ππ‘ππ)
πβΞ©π
ππ» πΈπ,π» = minππ,π» ,π΅π,π» ππ»ππ,π» β ππ»π΅π,π» + π
Baseline
Energy Cost
Regulation Contract
Revenue
21
Objective Function (if final decision)
π =1
π ππ» βπ΅π,π» β π π‘
π β βπ‘ β π π‘π
π‘πβ1
π‘=1
+ π β βπ‘ β π΅π,π» β π’π‘π
π‘
+ πΏ β ππ + ππ»+1(πΈπππ₯ β ππ‘ππ)
πβΞ©π
ππ» πΈπ,π» = minππ,π» ,π΅π,π» ππ»ππ,π» β ππ»π΅π,π» + π
Energy Cost
Adjustement
Baseline
Energy Cost
Regulation Contract
Revenue
22
Objective Function (if final decision)
π =1
π ππ» βπ΅π,π» β π π‘
π β βπ‘ β π π‘π
π‘πβ1
π‘=1
+ π β βπ‘ β π΅π,π» β π’π‘π
π‘
+ πΏ β ππ + ππ»+1(πΈπππ₯ β ππ‘ππ)
πβΞ©π
ππ» πΈπ,π» = minππ,π» ,π΅π,π» ππ»ππ,π» β ππ»π΅π,π» + π
Energy Cost
Adjustement
Contract
Violation Cost
Baseline
Energy Cost
Regulation Contract
Revenue
23
Objective Function (if final decision)
π =1
π ππ» βπ΅π,π» β π π‘
π β βπ‘ β π π‘π
π‘πβ1
π‘=1
+ π β βπ‘ β π΅π,π» β π’π‘π
π‘
+ πΏ β ππ + ππ»+1(πΈπππ₯ β ππ‘ππ)
πβΞ©π
ππ» πΈπ,π» = minππ,π» ,π΅π,π» ππ»ππ,π» β ππ»π΅π,π» + π
Energy Cost
Adjustement
Driver
Inconvenience Cost
Contract
Violation Cost
Baseline
Energy Cost
Regulation Contract
Revenue
24
Objective Function (if final decision)
π =1
π ππ» βπ΅π,π» β π π‘
π β βπ‘ β π π‘π
π‘πβ1
π‘=1
+ π β βπ‘ β π΅π,π» β π’π‘π
π‘
+ πΏ β ππ + ππ»+1(πΈπππ₯ β ππ‘ππ)
πβΞ©π
πβ πΈπ,π» = minππ,π» ,π΅π,π» ππ»ππ,π» β ππ»π΅π,π» + π
Energy Cost
Adjustement
Driver
Inconvenience Cost
Contract
Violation Cost
Baseline
Energy Cost
Regulation Contract
Revenue
Future Energy
Purchases
25
Objective Function (not final decision)
π =1
π πβ βπ΅π,β β π π‘
π β βπ‘ β π π‘π
π‘πβ1
π‘=1
+ π β βπ‘ β π΅π,β β π’π‘π
π‘
+ πΏ β ππ + πβ+1(πΈπππ₯ β ππ‘ππ)
πβΞ©π
πβ πΈπ,β = minππ,β ,π΅π,β πβππ,β β πβπ΅π,β + π
Energy Cost
Adjustement
Contract
Violation Cost
Baseline
Energy Cost
Regulation Contract
Revenue
+ πβ+1 ππ‘ππ
Future Optimal Value Function
26
Exact Linearization of Contract Violation Cost π β βπ‘ β π΅ β π’π‘
ππ‘ is nonlinear
Replace with π β βπ‘ β π₯π‘
ππ‘
Add constraints When π’ = 0, π₯=0 When π’ = 1, π₯= π΅
π₯π‘π β€ π΅
π₯π‘π β€
ππππ₯
2 π’π‘
π
π₯π‘π β₯ π΅ β
ππππ₯
2 1 β π’π‘
π
27
Stochastic Dynamic Programming
Ebatt
E5
Time (Hours)
Unplug Time
E1
Emax
π5 πΈ5
h1 h2 h3 h4 h5
28
Stochastic Dynamic Programming
Emax
Ebatt
Time (Hours)
Unplug Time
h1 h2 h3 h4 h5
E1
E5
π5 πΈ5
29
Stochastic Dynamic Programming
Emax
Ebatt
Time (Hours)
Unplug Time
h1 h2 h3 h4 h5
E1
E5
π5 πΈ5
30
Stochastic Dynamic Programming
Emax
Ebatt
Time (Hours)
Unplug Time
h1 h2 h3 h4 h5
E1
E5
π5 πΈ5
31
Stochastic Dynamic Programming
Emax
Ebatt
Time (Hours)
Unplug Time
h1 h2 h3 h4 h5
E1
E5
π5 πΈ5
32
Stochastic Dynamic Programming
Emax
Ebatt
Time (Hours)
Unplug Time
h1 h2 h3 h4 h5
E1
E5
π5 πΈ5
33
Stochastic Dynamic Programming
Emax
Ebatt
Time (Hours)
Unplug Time
h1 h2 h3 h4 h5
E1
E5
π5 πΈ5
34
Stochastic Dynamic Programming
Emax
Ebatt
Time (Hours)
Unplug Time
h1 h2 h3 h4 h5
E1
E5
π5 πΈ5
35
Find state, cost points on the convex hull of all points Andrews Monotone Chain Algorithm
Basically compares slopes
Future Cost - ππ ππ‘ππ
π5 πΈ5
E5 36
Find state, cost points on the convex hull of all points Andrews Monotone Chain Algorithm
Basically compares slopes
Future Cost - ππ ππ‘ππ
On the hull
Not on the hull π5 πΈ5
E5 37
Future Cost - ππ ππ‘ππ
Create inequalities from points on the convex hull
Add new inequality constraints to DEP π4 πΈπ,4
π5 πΈ5
Cut j
πβ+1 ππ‘ππ β₯ πΌππ‘ππππππ‘π β ππππππ β ππ‘π
π , βπ
E5 38
Stochastic Dynamic Programming
1618
2022
24
Hour 5
Hour 6
0
0.05
0.1
0.15
0.2
0.25
Energy (kWh)
Time
Op
tim
al V
alu
e F
un
ctio
n C
ost ($
)
4
5
39
πβ+1 ππ‘ππ
ππ‘π1
Stochastic Dynamic Programming
Repeat backwards recursion until the current state is reached
Ebatt
Time (Hours)
Unplug Time
Emax
h1 h2 h3 h4 h5
E1
40
Stochastic Dynamic Programming
Emax
Ebatt
Time (Hours)
Unplug Time
Repeat backwards recursion until the current state is reached
h1 h2 h3 h4 h5
E1
41
Stochastic Dynamic Programming
Emax
Ebatt
Time (Hours)
Unplug Time
Repeat backwards recursion until the current state is reached
h1 h2 h3 h4 h5
E1
42
Stochastic Dynamic Programming
Emax
Ebatt
Time (Hours)
Unplug Time
Repeat backwards recursion until the current state is reached
h1 h2 h3 h4 h5
E1
43
Implementation
Calculate Optimal Value Functions, Vh
At initial state, time Solve one DEP for P1, B1
Implement decision and wait 1hr, see what happens
Given new state, Optimize decision,
implement, wait
At unplug time If not full, charge at Pmax
Else, Done!
1 2 3 4 5 6 7 8
12
14
16
18
20
22
24
26
Ba
tte
ry S
tate
of C
ha
rge
(kW
h)
Simulation Timestep (hr)
Simulation Results
Energy Bounds
(from Regulation Bid)
Actual Energy
State
Energy at Pavg
44
Forward Simulation for Comparison
Simulate 150 different, 7 hour long realizations of AGC Signal
Each trial uses the same Optimal Value Functions
Initial state
Deterministic prices
Set of samples in DEPs
Compare with an expected value formulation 1 sample, using expected value of AGC signal
45
Results- 150 forward Simulations
0 0.5 1 1.5 2 2.5 3 3.5 4 4.50
20
40
60
80
100
120Histogram of Expected Value Formulation Costs
Cost($)
Cou
nts
0.16 0.18 0.2 0.22 0.24 0.26 0.28 0.30
5
10
15
20
25
30
35
Cost($)
Counts
Histogram of Stochastic Formulation Costs
$20/hr Inconvenience cost
Stochastic
Model
Expected Value
Model
ΞΌ $ 0.23 $ 0.44
Ξ£2 5.0 E-4 0.35
Late trials 0% 29%
$200/hr Inconvenience cost
Stochastic
Model
Expected Value
Model
Ξ $ 0.23 $ 2.29
Ξ£2 5.0 E-4 35.58
Late trials 0% 29%
46
Observations
Expected Value formulation often inconveniences driver, while Stochastic formulation is robust
For final decision, P,B are chosen such that driver is not inconvenienced on any sample path
Cost of uncharged energy Γ· 30 > All hourly Energy Prices
Vast majority of DEP solutions are on the CH
good approximation of πβ
If Charging, regulation contract size, B is on upper
bound , but decisions are dependent
47
Future Work
Method Improvement/Evaluation Bias Estimation and Correction Number of AGC Samples Number of Discretizations Parallelize
Model Expansion Investigate AGC signal properties Uncertain Prices- ARIMA or GARCH Form CH in 4 dimensions with QuickHull
Fleet Aggregation Apply method to other technologies (flywheels) Integrate into broader Smart Distribution Network model
48
Conclusions
Stochastic models are necessary for demand side frequency regulation
We have accurately modeled risks of providing frequency regulation
Our method is tractable and parallelizable
49
Thank you!
1 2 3 4 5 6 7 8
12
14
16
18
20
22
24
26
Ba
tte
ry S
tate
of C
ha
rge
(kW
h)
Simulation Timestep (hr)
Simulation Results
0.16 0.18 0.2 0.22 0.24 0.26 0.28 0.30
5
10
15
20
25
30
35
Cost($)
Counts
Histogram of Stochastic Formulation Costs
50
Support for this research was provided by Fundação para a CiΓͺncia e a Tecnologia (Portuguese Foundation for Science and Technology)
through the Carnegie Mellon Portugal Program