2

Optimal Dispatch in a Balancing Market withIntermittent Renewable Generation

Priyanka Shinde, Student Member, IEEE, Mohammad Reza Hesamzadeh, Senior Member, IEEE,Paresh Date, and Derek W. Bunn

Abstract—In this paper, three single-stage stochastic programsare proposed and compared for optimal dispatch by a SystemOperator (SO) into balancing markets (BM). The motivation forthe models is to represent a possible requirement to undertakesystem balancing with increasing amounts of intermittent re-newable generation. The proposed optimization models are re-formulated as tractable Mixed Integer Linear Programs (MILPs)and these consider both fuel cost and intermittency cost of thegenerators, when the SO activates the up- or down-regulationbids. These three models are based on the main approachesseen in practice: dual-imbalance pricing, single imbalance pricingand single imbalance pricing with spot reversion. A scenario-generation algorithm based on predictive conditional dynamicdensity distributions is also proposed. We perform a comparativeanalysis of these three proposed models in terms of how they helpthe SO to optimize their balancing market actions consideringintermittent-renewable generators. The single imbalance pricingis found to be the most market efficient.

Index Terms—Balancing market, Imbalance settlement cost(ISC), Stochastic program, GAMLSS

NOMENCLATUREThe main notation is presented below for a quick reference.Additional symbols are introduced throughout the text.Indicesi generating unitω scenarios of power generation

Parametersgmi,ω metered output of generator i in scenario wgsi scheduled output of generator igupi up-regulation quantity bid of generator igdni down-regulation quantity bid of generator iλupi up-regulation price bid of generator iλdni down-regulation price bid of generator iSIL System Imbalance Levelβ CVaR parameter (specified probability level)πω Probability weight of scenario ωG Total number of generatorsλspot Spot priceµ Mean of metered output of generators

Manuscript received June 5, 2019; revised October 23, 2019 andFebruary 24, 2020; accepted July 26, 2020.

P. Shinde is with the School of Electrical Engineering and ComputerScience, KTH Royal Institute of Technology, Sweden.

M.R. Hesamzadeh is with the School of Electrical Engineering andComputer Science, KTH Royal Institute of Technology, Sweden; Programon Energy and Sustainable Development (PESD), Stanford University andGerman Institute for Economic Research (DIW Berlin).

P. Date is with the College of Engineering, Design and PhysicalSciences, Brunel University, United Kingdom.

D. Bunn is with the Decision Sciences, Management Science andOperations, London Business School, United Kingdom.

σ Standard deviation of metered output of generatorswindt Wind generation in MWh at time tdemt Demand in MWh at time thydt Hydropower in MWh at time tΞ Big-M parameterε Constant used in linearizing ’If’ condition

Variablesφupi Binary variable for up-regulation bid of generator iφdni Binary variable for down-regulation bid of generator iφ Binary variable for system up-regulation statusφ Binary variable for system down-regulation statusλdnreg Down-regulation priceλupreg Up-regulation priceδmi,ω Imbalance quantity of generator i in scenario wαβ Surplus generation for a given probability value βαβ Unmet level of demand for a given probability value β∆ω Net deviation in scenario ω∆−ω Net negative deviation in scenario ω∆+ω Net positive deviation in scenario ω

b−ω Binary variable to linearize ∆−ωb+ω Binary variable to linearize ∆+

ω

Variables used in DIP model for Big-M linearizationI+i,w Bilinear variable for positive deviation and

down-regulationI−i,w Bilinear variable for negative deviation and

up-regulationsω Non-linear term for E(ISC) due to negative deviationcω Non-linear term for E(ISC) due to positive deviationIupi,w Non-linear variable equal to product of sω and φupiIdni,w Non-linear variable equal to product of cω and φdnizdnω Variable to linearize the product of φdni and cωzupω Variable to linearize the product of φupi and sωu−ω Non-linear variable for linearizing CVaR constraint for

unmet demandu+ω Non-linear variable for linearizing CVaR constraint for

surplus generationb1ω Binary variable to linearize Max operator for surplus

generationb2ω Binary variable to linearize Max operator for unmet

demand

Variables used in SIP-SR model for Big-M linearizationNupi,ω Bilinear variable equal to product of φupi and ∆−ω

Ndni,ω Bilinear variable equal to product of φdni and ∆+

ω

cupω Bilinear variable related to up-regulation costcdnω Bilinear variable related to down-regulation cost

3

qiω Binary variable to identify marginal down-regulatinggenerator

riω Binary variable to identify marginal up-regulatinggenerator

φ Binary variable to indicate equal activated volumes forup- and down-regulation

kupi Non-linear variable equal to product of φupi and φkdni Non-linear variable equal to product of φdni and φh+ Bilinear variable equal to product of ∆+

ω and φh− Bilinear variable equal to product of ∆−ω and φtupω Bilinear variable equal to product of ∆+

ω and φtdnω Bilinear variable equal to product of ∆−ω and φ

SetΩ Set of decision variables

OperatorsI(condition) “If” operatorE(expression) “Expectation” operator.

I. INTRODUCTION

IN electricity markets, renewable generation has beenincreasing substantially following decreasing technology

costs and supportive government policies [1]. In the EuropeanUnion, the share of renewable generation has increased by64% from 2007 to 2017 [2]. Reference [3] reports 25 TWhproduction from renewable generation in the third quarterof 2018 in UK (a 10% increase year-on-year). A similarlyincreasing pattern is reported for the US electricity system [4].This increasing share of renewable generation brings majorenvironmental benefits but it also introduces new operationalchallenges. Although renewable generation has low marginalcosts and is CO2 free, it is highly variable and intermittent[5]. This feature implies a need for more real-time balancingservices by the system operator (SO). Authors in [6] and [7]discuss the balancing task of the SO in an electricity systemwith high penetration of intermittent-renewable generation.They suggest that at some point the intermittent-renewablegenerators will have to actively participate in the balancingmarket (BM) and be part of the solution as well as thecause. On the legislation side, the European Union direc-tive 2003/54/EC enforces the non-discriminatory and market-based provision of balancing services [8]. This encouragesintermittent-renewable generators to provide balancing ser-vices in the BM (see for example [9] for Britain and [10]for Nordic countries).Accordingly, authors in [11] propose different models for theSO to facilitate the active participation of wind generators inthe balancing markets. In [12], a Lyapunov optimization basedreal-time strategy has been proposed in order to minimize theimbalance costs for a virtual power plant. This formulationdoes not need the specification of the probability distributionsof the renewable power generation. In [13], a combined energyand reserve market model has been proposed to increase thewind producers’ revenue by allowing them to optimally bidin both markets and help them reduce their imbalance costs.A mathematical model to maximize the overall profit of aportfolio of a wind farm and a combined heat and power byminimizing their imbalances in a two-price balancing market

has been proposed in [14]. A bidding strategy to minimizethe imbalance costs for wind power producers is describedin [15] where forecast errors in wind power are representedas stochastic processes. But the risk associated with windpower forecast errors is not accounted for in this model.Reference [16] discusses the lack of economic incentivesfor the wind power producers in Portugal to minimize theirimbalances as the wind imbalance costs are socialised. In[17], a method to choose the generation adjustments and thevolumes of elastic demand that match the imbalances in the netinelastic demand (inelastic demand minus RES production),with respect to the day-ahead scheduling, has been proposed.In [18], the link between the power system cost due to windpower imbalances and the imbalance settlement payments towind power producers is investigated. They draw a comparisonbetween the one and two-price settlement systems.

TABLE IOVERVIEW OF LITERATURE IN RELATED AREAS

Ref. BMC Imbal. pricing Role of Prob. Stoch. Prob.+ISC SO constr. gen. scen.

Single Dual in BM gen.[6] 7 7 7 7 7 7 7[11] 7 X X 7 7 7 7[15] 7 X 7 7 7 X 7[19] 7 X 7 7 X X 7[20] 7 7 7 X 7 7 7[21] 7 7 7 X 7 7 7[22] 7 7 7 X 7 7 7[23] 7 X X 7 7 X 7[24] 7 X 7 7 7 7 7[25] 7 7 X 7 7 7 X[26] 7 X 7 7 7 7 7

Our X X X X X X Xpaper

Table I summarizes an overview of the published researchon the various aspects of specifications. The first column inTable I, ‘BMC+ISC’ refers to the approach of consideringboth balancing market cost and imbalance settlement costto take the balancing market decisions. The second column,‘Imbal. pricing (Single and Dual)’, mentions the imbalancepricing schemes. While some of the literature is limitedto the producers’/consumers’ perspectives of minimizing theimbalance settlement costs, the third column, ‘Role of SO inBM’, aims to highlight which papers have discussed aboutthe involvement of the SO in the balancing market. ‘Prob.const.’ (Probabilistic constraint) emphasizes the approach forconsidering the uncertainty in the system, which in our caseis a CVaR (Conditional Value at Risk) constraint. The fifthcolumn, ‘Stoch. gen.’ (Stochastic generation), highlights thosepapers that have considered stochastic generation in theirwork. ‘Prob. scen. gen.’ (Probabilistic scenario generation)tries to show if the method chosen for generating scenariosis probabilistic or not.The above research generally, but implicitly, recognizes therole of the imbalance settlement cost (ISC) influencing theactive participation of the intermittent-renewable generatorsin the BM. This point is further explained in [6] and[20]. Accordingly, proper modeling of the ISC in the SO

4

optimal-dispatch can ensure the efficient participation of theintermittent-renewable generators in the balancing market andmaybe incentivize the intermittent-renewable generators toadapt technologies to reduce their ISC exposure (eg. hybridfacilities with battery-storage devices) or to hedge more effec-tively (eg. with weather derivatives). Furthermore, in principle,the SO as a regulated entity would be expected to perform awelfare maximizing approach to system balancing, and thusinclude all relevant costs to participants, including ISCs, but inpractice this is not yet widespread. Hence this is the motivationfor this paper.

II. BACKGROUND

In this section, we present a brief description of a fewelectricity markets to explain their basic functioning:• Nordic electricity market: At the day-ahead (DA) stage,

all the producers and consumers willing to participatein the DA market submit their bidding curves to theelectricity market operator. Then the market operatorclears the DA market based on uniform price auction anddefines the DA price, all the producers and consumersbelow that price are cleared in the DA auction. Basedon the updated forecasts, these producers and consumerscan adjust their position in the market by trading inthe intraday market. At the gate closure (60 minutesbefore actual delivery of electricity), the position becomesbinding and cannot be changed further. However, at thetime of actual delivery of electricity, these producersand consumers deviate from their scheduled value. Itis then the responsibility of the SO to activate bids inthe balancing market to match the supply and demandof electricity. According to the existing method, thereis a merit-order dispatch (marginal cost pricing (MCP)auction) performed by the SO, in the balancing market1, to decide which producer or consumer needs to becalled upon, based on their marginal cost of productionor marginal utility of consumption. In the imbalance set-tlement stage, the SO calculates the imbalance settlementcost for each producer or consumer who participated inthe market based on their deviation from the committedvalue and the direction of the system imbalance [29].

• Great Britain (GB): In GB electricity market, there iscontinuous trading up until the gate closure, which hap-pens one hour before the actual delivery of electricity, andthat is when all generators and retailers must nominateto the SO what they expect to generate or consume inthe delivery period. But the participants are allowed totrade even after the gate closure for a further hour untilthe beginning of the actual delivery period. The deliveryperiod length (also known as Settlement Period (SP)) is30 minutes. It is the responsibility of the SO to ensure 50Hz frequency, by activating bids and offers continuously

1Here we refer to the m-FRR (manual Frequency Restoration Reserve)of the balancing market. This is the main balancing resource in the Nordicmarket apart from this the other reserves are FCR-N (Frequency ContainmentReserve-Normal), FCR-D (Frequency Containment Reserves-Disturbance)and a-FRR (automatic Frequency Restoration Reserve). For more informationcheck [27], [28].

in every SP2. For each SP, a single imbalance price (SIP)is calculated. Based on whether the offer volumes exceedthe bid volumes or vice versa, the NIV (Net ImbalanceVolume) can be either positive or negative. The marginalprice for 1 MW of the NIV gives the SIP [19], [31].

• PJM 3: PJM compiles the bids in ascending price or-der and auction is performed to clear the bids untilthe point of intersection of the supply-demand curve.This sets the DA price for a particular hour for thedelivery of electricity on the next day. Real-time marketis another market that serves electricity for immediatedelivery. Supply and demand curves are paired and basedon the intersection point (MCP auction), the prices arecalculated every five minutes. Based on the situation offluctuations in the system, PJM informs the producerswhat their output should be. But if they deviate fromPJM’s instructions, they are penalized. If they followthe instructions of PJM, they are compensated for that.Another important market in PJM is the ancillary ser-vices market, which comprises of regulation and reservesmarket. The regulation market is in place to correctthe small fluctuations in the electricity supply/demandthat can affect the stability of power system. It allowsto compensate those producers/consumers that have theability to adjust their production/consumption as per therequirement in the system [33].

III. CONTRIBUTIONS

This paper contributes to the relevant research in this contextin several ways:• Our paper proposes a mechanism for the SO to dispatch

generators to activate their up- and down-regulation bidson the basis of having lowest expected costs, taking intoaccount both their production costs and potential devi-ations from nominated outputs. We propose a stochasticframework for clearing the real time balancing market bythe SO.

• In our formulation, we introduce a CVaR constraint,which allows surplus generation or unmet demand, withan appropriate small probability. This produces usefuland computationally tractable solutions under supply sideuncertainty, where hard constraints with no surplus gen-eration or unmet demand might lead to infeasibility.

• The proposed method has been mathematically appliedto compare three imbalance pricing models namely DualImbalance Pricing (DIP), Single Imbalance Pricing withSpot Reversion (SIP-SR) and Single Imbalance Pricing(SIP) models.

• The three proposed models have been tested on a stylisedfive-generator system and a more realistic eight-generator

2Short term operating reserve (STOR) provides the grid in GB withadditional power when actual demand in the network is greater than theforecast and/or there is unforeseen generation unavailability [30].

3PJM: Pennsylvania-Jersey-Maryland, which is the regional transmissionorganiziation that is responsible for coordinating the wholesale electricitymarket in some or all parts of Indiana, Delaware, Virginia, West Virginia,Maryland, Illinois, Pennsylvania, Michigan, Kentucky, North Carolina, Ten-nessee, New Jersey, Ohio, and the District of Columbia [32].

5

gate closure actual delivery

timegim

gis

Balancing

market

imbalance

settlement

gir

gir

gis

post delivery

Fig. 1. The proposed balancing market operation (Dotted line representsuncertainty)

system. In the case of eight-generator system, the un-certain parameters in the stochastic MILP models wereestimated and predicted using a state-of-the art algorithm,the Generalized Additive Models for Location, Scaleand Shape (GAMLSS). We demonstrate the performanceof our proposed MILP models and the GAMLSS-baseduncertainty-modeling algorithm with realistic numericalstudies.

The rest of the paper is structured as follows. Section IVdevelops the optimal-dispatch model for the SO. Section Vpresents the GAMLSS-based forecasting model for generationimbalances. An illustrative example is provided in Section VI.Numerical results based on variations of a Nordic case studyis provided in Section VII. Finally, Section VIII concludes thepaper.

IV. MATHEMATICAL FORMULATION

We assume three stages in our analysis. At gate closure thescheduled generations (gsi ) are reported to the SO. Then theSO receives up-regulation offers from generators for increasingoutput and down-regulation bids for reducing output, (gri ).If called, generators receive payments for their generation,or pay for their reduced generation. The SO also forecastssystem demand in real time and estimates the imbalancesettlement charges. Since the metered output of generators(gmi ) are available only at the imbalance settlement stage, theyare uncertain parameters in balancing-market optimal dispatchmodel. These stages are illustrated in Fig. 1. We now considerthree variations in the formation of imbalance charges.

A. Dual Imbalance Pricing (DIP)

The DIP mechanism uses two imbalance prices, often calledsystem buy price (SBP) and system sell price (SSP) [34].The SBP is the average price at which the system has to buyelectricity during a balancing period and the SSP reflects theaverage price at which the system has to sell electricity in orderto dispose of surplus energy. Participants will then pay SBP orreceive SSP depending upon whether their imbalance volumeswere short or long. This mechanism has been favoured by reg-ulators who want to deter speculation. It was the mechanismintroduced in Great Britain in 2001 (although it has since beenchanged), also in France, Spain, Italy and elsewhere. At gateclosure, let λupi , λdni , gupi and gdni represent bid prices andvolumes for generator i participating in the balancing market.For notational simplicity, we assume that each generatorsubmits only one bid pair. Let φupi (respectively, φdni ) bebinary variables which take values 1 or 0 depending whetherthe bid (respectively, offer) from generator i is accepted ornot.

Time

Gen

era

tio

n

Uncertainty band

metered generationMean of total

Total scheduled generation

Total imbalance

Regulation volume

(a)

Pro

ba

bil

ity

de

nsi

ty

i im

mean -VaR

1-

-CVaR

(b)

Fig. 2. (a) Time series for scheduled generation and depiction of imbalanceat a specific time (b) Illustration of δmi

1) Objective function of proposed optimal-dispatch model:The proposed objective function for the SO optimal-dispatchmodel is given by (1), where BSC stands for Balance Settle-ment Cost:

BSC =∑i

(λupi gupi φupi − λ

dni gdni φdni ) + (1a)

max(∑i

δmi , 0)

∑i λ

dni gdni φdni∑i gdni φdni

I(∑i

φdni ≥ 1) + (1b)

min(∑i

δmi , 0)

∑i λ

upi g

upi φupi∑

i gupi φupi

I(∑i

φupi ≥ 1) (1c)

In (1), both gupi and gdni are assumed to be positive. gmi is themetered output of generator i and is considered to be a randomparameter with known means and known continuous jointdistribution. gsi is the scheduled power dispatch of generator i.The δmi = gmi − (gsi +gupi φupi −gdni φdni ) models the deviationfor generator i. The SO objective function in (1) can be splitinto two parts, (1a) is the Balancing Market Cost (BMC) while(1b) and (1c) represent the positive and negative ImbalanceSettlement Cost (ISC), respectively. Observe that whilst ISCsare the charges incurred by the generators, they are also therecourse costs incurred by the SO in balancing, because of thenon-firm regulating dispatch.2) Constraints of proposed optimal-dispatch model: Letf(φ, gm) =

∑i δmi =

∑i(g

mi − (gsi + gupi φupi − gdni φdni ))

be the deviation function, where φ and gm are vector valuedvariables corresponding to φi and gmi . Fig. 2a depicts thetime series for total scheduled generation and the mean ofthe total metered generation. The blue color band highlightsthe uncertainty in the total metered generation. At a giventime instant, the red colored line represents the total scheduledgeneration at that time, orange colored line gives the regulationvolume, and green colored line shows the total imbalance(deviation) at that time. Our paper discusses one such specifictime instant. In Fig. 2b, the probability density of the totalimbalance (deviation) is demonstrated, the details about whichare discussed further in this section. In our formulation, weallow surplus generation (f(φ, gm) > 0) or unmet demand(f(φ, gm) < 0) with an appropriate small probability. Theprobability of f(φ, gm) not exceeding a given α is given by:

Ψ(φ, α) =

∫f(φ,gm)≤α

p(gm)dgm (2)

6

where p(gm) is the joint distribution function of gm. Similarly,the probability of −f not exceeding a given α (f exceeding−α) is given by:

Ψ(φ, α) =

∫−f(φ,gm)≤α

p(gm)dgm (3)

For a specified probability level β, let the surplus generationbe defined by:

αβ(φ) := minα ∈ R+, Ψ(φ, α) ≥ β (4)

and the unmet level of demand is defined by:

αβ(φ) := minα ∈ R+, Ψ(φ, α) ≥ β (5)

Thus, given a probability level β, the surplus generation doesnot exceed α with probability β. In other words, α is oneendpoint of the non-empty interval of values α such thatΨ(φ, α) = β. Similarly, unmet demand does not exceed α withprobability β. We then define the average surplus generationif it already exceeds α as:

Lβ(φ) = (1− β)−1

∫f(φ,gm)≥αβ(φ)

f(φ, gm)p(gm)dgm (6)

Similarly, we define the average unmet demand if it alreadyexceeds α as:

Lβ(φ) = (1− β)−1

∫−f(φ,gm)≥αβ(φ)

f(φ, gm)p(gm)dgm

(7)Note that the probability of f(φ, gm) exceeding α is 1 − β,so that Lβ(φ) gives conditional expectation, i.e., the averagesurplus generation, when it exceeds α. Similar comments holdfor Lβ(φ).Lβ(φ) and Lβ(φ) can be incorporated into our optimizationmodel through appropriately relaxed versions of these func-tions. These are defined as follows:

F β(φ, α) = α+ (1− β)−1

∫gm

(f(φ, gm)− α)+dgm

F β(φ, α) = α+ (1− β)−1

∫gm

(−f(φ, gm)− α)+dgm

where (x)+ = max(x, 0). Then F β(φ, α) (respectivelyF β(φ, α)) is convex and continuously differentiable in α

(respectively α). Further, Lβ(φ) = minα∈R F β(φ, α) anda similar result holds for Lβ(φ) [35]. Now, if we generatesamples of gm, we can write an approximation to F β(φ, α)as:

Fβ(φ, α) = α+1

N(1− β)

∑j

(f(φ, vec(gmi,ω))− α)+ (9)

where vec(xi) represents a vector with xi as its ith element,with its dimension determined by the context. Fβ(φ, α) isdefined similarly. For a given System Imbalance Level (SIL),Fβ(φ, α) ≤ SIL and Fβ(φ, α) ≤ SIL can be written as in(10).

α+1

N(1− β)(f(φ, vec(gmi,ω))− α)+ ≤ SIL (10a)

α+1

N(1− β)(−f(φ, vec(gmi,ω))− α)+ ≤ SIL (10b)

where in (10), SIL in MWh is a behavioural parameter toreflect SO aversion to large real-time imbalances. It representsthe total imbalance in the system tolerated by the SO and riskconstraints (2) represent the probabilistic tolerance of this largeimbalance.3) The proposed optimal-dispatch model: As we consider themeasured generation gmi,ω to be an uncertain parameter, it canbe modeled using a set of scenarios. We assume that scenariosof gi,ω (ω = 1, 2, · · · ,N ) are available, along with the associ-ated probability weights πω . The discussion of how to generatescenarios is explained in Section V. Explicit modeling of theISC in the optimization model (12) exposes the intermittent-renewable generators participating in the balancing market totheir imbalance penalties (because of the deviation from theirnominated outputs). The deviation δmi,ω depends on the stochas-tic measured generation gmi,ω and the expression of deviationcan be given as: δmi,ω = gmi,ω − (gsi + gupi φupi − gdni φdni ). Thisterm of δmi,ω when introduced in (1b) and (1c) motivates us totake the expected value of the expression in (1b) and (1c) dueto the stochasticity. Therefore, the expression of BSC becomes(11):

BSC =∑i

(λupi gupi φupi − λ

dni gdni φdni ) + (11a)

E(max(∑i

δmi,ω, 0)

∑i λ

dni gdni φdni∑i gdni φdni

I(∑i

φdni ≥ 1)) + (11b)

E(min(∑i

δmi,ω, 0)

∑i λ

upi g

upi φupi∑

i gupi φupi

I(∑i

φupi ≥ 1)) (11c)

Here it can be observed that (11b) and (11c) together comprisethe expected value of ISC.Hence, the proposed optimization model for SO consideringboth BMC and ISC can be presented as in (12).

MinimizeΩ1=φupi ,φdni ,α,α

BSC = BMC + E(ISC)

subject to: (10), φupi , φdni ∈ 0, 1, α, α ∈ R,∀ i (12)

4) The MILP formulation of the proposed optimal-dispatchmodel: We now present a MILP model of the deterministicequivalent problem for (12). The total deviation in a scenariow can be given as ∆ω =

∑i δmi,ω where δmi,ω = gmi,ω − (gsi +

gupi φupi − gdni φdni ). The ∆+ω = Max(∆ω ,0) can be linearized

as shown in (13).

0 ≤ ∆+ω ≤ b+ωΞ(1) (13a)

∆ω ≤ ∆+ω ≤ ∆ω + (1− b+ω )Ξ(1) (13b)

where Ξ(1) is a suitable large constant. The ∆−ω = Max(-∆ω ,0)can be linearized similarly as in (13). Referring to the positivedeviation part of (11), we denote cω =

∑i λdni gdni φdni ∆+

ω∑i gdni φdni

whichcan be written as (14):

cω∑i

gdni φdni =∑i

λdni gdni φdni ∆+ω (14)

Using change of variable Idni,w = cωφdni in (14), the non-linear

term can be linearized by implementing (15):

0 ≤ Idni,w ≤ φdni Ξ(2) (15a)

cω − (1− φdni )Ξ(2) ≤ Idni,w ≤ cω (15b)

7

where Ξ(2) is a suitably large constant. Also, substitutingI+i,w = φdni ∆+

ω in (14), it can be linearized as in (15). Thelinearized form of (14), after implementing the change ofvariable, can be written as in (16):∑

i

gdni Idni,w =∑i

λdni gdni I+i,w (16)

Consider the negative deviation part of (1), we define sω =∑i λupi gupi φupi ∆−

ω∑i gupi φupi

which means:

sω∑i

gupi φupi =∑i

λupi gupi φupi ∆−ω (17)

By implementing the substitutions, Iupi,w = sωφupi and

I−i,w = φupi ∆−ω , the linearized form of (17) can be writtenas

∑i gupi Iupi,w =

∑i λ

upi g

upi I−i,w by following the similar

equations as in (15). The If condition in (1b) is written aslinear constraints

∑i φ

dni ≥ 1 − Ξ(3)(1 − φ) and

∑i φ

dni ≤

1 + Ξ(3)(φ). A similar linearization can be adopted for (1c)and replaced with φ. Then the objective function becomes:

MinimizeΩ2

∑i

(λupi gupi φupi − λ

dni gdni φdni ) +∑

ω

πω(φcω − φsω)(18)

where Ω2 = Ω1 ∪ φ, φ, b+ω , b−ω , cω, sω,∆+ω ,∆

−ω , I

dni,w, I

upi,w,

I−i,w, I+i,w. If we use zdnω = φcω and zupω = φsω , then the

nonlinear expressions (φcω and φsω) can be linearized as in(15). The probabilistic constraint can be written as:

α+1

(1− β)

N∑ω

πωu+ω ≤ SIL (19)

where u+ω = Max(∆+

ω − α, 0). Also,

α+1

(1− β)

N∑ω

πωu−ω ≤ SIL (20)

where u−ω = Max(∆−ω − α, 0).Linearization of Max(·) operator is possible in similar wayas in (13). Accordingly, the optimization problem (12) can bere-formulated as the stochastic mixed-integer linear program(MILP) in (21):Minimize

ΩA

∑i

(λupi gupi φupi − λ

dni gdni φdni ) +

∑ω

πω(zdnω − zupω )

(21a)subject to : (13), (15), (16), (19), (20) (21b)

φ, φ, φupi , φdni , b+ω , b

−ω ∈ 0, 1 ∀ i, ω (21c)

where ΩA = Ω2 ∪ zupω , zdnω , u+ω , u

−ω is the set of decision

variables in (21).

B. Benchmark model for the DIP model

An illustration of the benchmark model for the DIP model isgiven in Fig. 3. In the benchmark model, the decision of the SOto activate up- or down-regulation bids is taken solely basedon the marginal cost of the generators. This is achieved byminimizing expression (11a). The output of the optimizationproblem would then give the values for φupi and φdni . These

Add BMC and ISC to get

BSC

Minimize (11a)

Calculate (11b) and (11c)ISC

Regulation

bid status

BMC

ISC

BSC

Fig. 3. Illustration of Benchmark model for the DIP model

values serve as inputs to the calculation of (11b) and (11c),which collectively give the value for ISC. The optimized valuefor BMC and the calculated value of ISC can be added togetherto find the BSC.

C. Single Imbalance Pricing with Spot Reversion (SIP-SR)Model

The SIP-SR mechanism differs in two respects from above.It is based on marginal rather than average pricing, i.e. theimbalance price is set according to the most expensive gener-ator that is cleared during up-regulation hour, or the cheapestgenerator cleared during down-regulation hour. Further, asshown in Table II, if the deviations contribute to the systemimbalance then the balance provider is charged accordingto the regulation price. However, if the deviations help tomaintain the system in balance then it should not be penalizedand accordingly the spot market price is applied [29]. This isthe current Nordic model.

TABLE IIPRICES IN BALANCING MARKET FOR SIP-SR MODEL

Up-regulation hour Down-regulation hour∆−

ω p=λupreg p=λspot

∆+ω p=λspot p=λdnreg

The objective function of SIP-SR optimal dispatch model canbe written as:

MinimizeΩ3

∑i

(λupi gupi φupi − λ

dni gdni φdni ) + (22a)∑

ω

πω[∆+ωλ

spotI(∑i

φupi gupi >

∑i

φdni gdni )+ (22b)

∆+ωλ

spotI(∑i

φupi gupi =

∑i

φdni gdni )+ (22c)

∆+ωλ

dnregI(

∑i

φupi gupi <

∑i

φdni gdni )− (22d)

∆−ωλupregI(

∑i

φupi gupi >

∑i

φdni gdni )− (22e)

∆−ωλspotI(

∑i

φupi gupi <

∑i

φdni gdni )− (22f)

∆−ωλspotI(

∑i

φupi gupi =

∑i

φdni gdni )] (22g)

Ω3 = φupi , φdni , λdnreg, λupreg,∆

+ω ,∆

−ω where λdnreg =

Min(λdni φdni ) and λupreg = Max(λupi φupi ). Hence, the objective

function (22) becomes:

8

MinimizeΩ3

(∑i

(λupi gupi φupi − λ

dni gdni φdni ) +∑

ω

πω[∆+ωλ

spotI(∑i

φupi gupi >

∑i

φdni gdni )+

∆+ωλ

spotI(∑i

φupi gupi =

∑i

φdni gdni )+

Min(λdni φdni ∆+ω )−Max(λupi φ

upi ∆−ω )

−∆−ωλspotI(

∑i

φupi gupi <

∑i

φdni gdni )−

∆−ωλspotI(

∑i

φupi gupi =

∑i

φdni gdni )])

(23)

If Ndni,ω = φdni ∆+

ω and Nupi,ω = φupi ∆−ω , the non-linearity in-

troduced (bilinear term φdni ∆+ω and φupi ∆−ω ) can be linearized

in the same way as (15). The expression cdnω = Min(λdni Ndni,ω)

is linearized in (24) (the term cupω = Max(λupi Nupi,ω) can be

linearized in the same way as shown in the Appendix):

− qiωΞ(5) ≤ cdnω − λdni Ndni,ω ≤ qiωΞ(5) (24a)∑

i

qiω = G − 1 (24b)

where G is the total number of generators in the electric-ity market and qiω is a binary variable. The If condition,I(∑i φ

upi g

upi >

∑i φ

dni gdni ) in (23) is replaced by binary

variable φ where:

∑i

φupi gupi ≥

∑i

φdni gdni − Ξ(6)(1− φ) + ε (25a)∑i

φupi gupi ≤

∑i

φdni gdni + Ξ(6)φ− ε (25b)

where ε is a non-zero number. The If condition,I(∑i φ

upi g

upi <

∑i φ

dni gdni ) in (23) can be linearized the

same way as in (38) using binary variable φ. The If condition,I(∑i φ

upi g

upi =

∑i φ

dni gdni ) in (23) is linearized by replacing

it by binary variable φ and rewriting the If condition as∑i kupi gupi =

∑i kdni gdni where kupi (= φφupi ) is linearized

using φ+φupi −1 ≤ kupi ≤ φ and kupi ≤ φupi . The kdni = φφdni

can be linearized in the same way as kupi . The objectivefunction can now be written as:

MinimizeΩ4

∑i

(λupi gupi φupi − λ

dni gdni φdni ) +

∑ω

πω[∆+ωλ

spotφ

+∆+ωλ

spotφ+ cdnω − cupω −∆−ωλspotφ−∆−ωλ

spotφ].(26)

Ω4 = Ω3 ∪ φ, φ, φ, b+ω , b−ω , Ndni,ω, N

upi,ω, c

dnω , cupω , k

dni , kupi If

we use tupω = ∆+ωφ, h+ = ∆+

ωφ, tdnω = ∆−ωφ, and h− = ∆−ωφthen the non-linear expressions can be linearized similar to(15). Accordingly, the final stochastic MILP model is given in(27).

MinimizeΩB

∑i

(λupi gupi φupi − λ

dni gdni φdni ) + (27a)∑

ω

πω[λspot(tupω + h+) + cdnω − cupω − λspot(tdnω + h−)]

subject to : (13), (19), (20), (24) (27b)

φ, φ, φ, φupi , φdni , b+ω , b

−ω ∈ 0, 1 ∀ i, ω (27c)

Where ΩB = Ω4 ∪ tupω , tdnω , h+, h−, u+ω , α, u

−ω , α is the set

of decision variables in (27).

D. Benchmark model for the SIP-SR model

As explained in Subsection IV-B, the benchmark model for theSIP-SR model works on the similar principle. Fig. 4 illustratesthe functioning of the benchmark model. The SO ignores thedeviations of the generators while choosing them for up- ordown-regulation. The decision of φupi and φdni is taken byminimizing (22a), which gives the BMC. Based on the outputof the optimization problem, (22b), (22c), (22d), (22e), (22f),and (22g) are calculated to find the ISC. Thus, BMC and ISCare added together to get the BSC value for the benchmarkmodel.

Add BMC and ISC to get

BSC

Minimize (22a)

Calculate (22b), (22c), (22d),

(22e), (22f), and (22g)ISC

Regulation

bid status

BMC

ISC

BSC

Fig. 4. Illustration of Benchmark model for the SIP-SR model

E. Single Imbalance Pricing (SIP) Model

The SIP model here, based on the imbalance pricing mecha-nism introduced in Great Britain (GB) in 2015, is summarizedin Table III.

TABLE IIIPRICES IN BALANCING MARKET FOR SIP MODEL

Up-regulation hour Down-regulation hour∆−

ω p=λupreg p=λdnreg∆+

ω p=λupreg p=λdnreg

The model is quite similar to SIP-SR in that the SO computesthe Net Imbalance Volume (NIV), i.e. the net amount of up-and down- regulation called in the trading period, then, forthat net amount, it computes the marginal offer price or bidprice, but, in this case, without any reversions to the spotprice. The motivation is to make it easier to hedge imbalanceprices by market participants, especially vertically integratedcompanies that may be out of balance in opposite directionsin their generation and retail accounts, and to go further thanSIP-SR by actually encouraging participants to help to balancethe system. Apart from GB, single imbalance pricing exists inBelgium, Austria and Germany subject to various conditions.In this section, we have presented the mathematical formu-lation of our proposed concept applied to three imbalancepricing models. We explain and discuss the dual imbalancepricing (DIP) model, a benchmark model corresponding to theDIP model, single imbalance pricing with spot reversion (SIP-SR) model, a benchmark model for the SIP-SR model, andthe single imbalance pricing (SIP) model. Our concept of SOclearing the balancing market by considering the imbalances

9

TABLE IVINPUT PARAMETER DATA FOR FIVE-GENERATOR SYSTEM IN ALL MODELS

gen λup λdn gup gdn gs

no. ($/MWh) ($/MWh) (MWh) (MWh) (MWh)G1 10 8 30 20 400G2 11 7 40 40 500G3 12 6 25 15 350G4 13 5 35 25 450G5 14 4 45 35 550

of the generators has been applied to different imbalancepricing models. A CVaR constraint has been introduced to theoptimization model in order to account for surplus generationor unmet demand, with an appropriate small probability. Theresulting non-linear optimization models have been linearizedthrough Big-M method.

V. PROPOSED DYNAMIC DENSITY ESTIMATION ANDFORECASTING USING GAMLSS

To be implemented in practice, the above formulations re-quire probabilistic forecasting of imbalances and we adopt aGeneralized Additive Model for Location, Scale and Shape(GAMLSS) as effectively used in this context by [36]. InGAMLSS, one of a variety of density functions can beselected in which the mean and higher moments can bespecified dynamically as conditional upon exogenous drivingvariables, [37]. In our proposed stochastic MILP models (21)and (27), the metered generation is the uncertain variableand the estimation flowchart is shown in Fig. 5. First, wefit the best distribution to the uncertain metered generationusing the Akaike Information Criterion (AIC). The momentsof the selected distribution are linearly regressed on selectedexplanatory variables (see Section VII for more details). Withforecasts for the explanatory variables, the moments of theselected distribution are estimated. This distribution is thenapproximated using a set of scenarios [38].

VI. ILLUSTRATIVE EXAMPLES

We assume five generators G1 to G5 participate in ourproposed balancing market to meet the system demand. TableIV shows the values of input parameter for the five-generatorsystem in all the three proposed models. The simulations wereperformed by setting the values of Ξ(1),Ξ(2),Ξ(3),Ξ(4),Ξ(5),and Ξ(6) equal to 2000 in optimization models (21) and (27).The generation of G1, G2, G3, G4, and G5 are distributedaccording to normal distribution4 N(400, 0), N(500, 4), N(350,8), N(450, 12) and N(550, 16), respectively. Using these nor-mal distributions, 1000 scenarios are sampled and then reducedto 10 scenarios presented in Fig. 6. The MILPs for all thethree proposed models are coded in GAMS and solved usingCPLEX solver [39]. Simulations are carried out on a computerwith 16 GB of RAM and 1.2GHz CPU. The computationaltimes for the DIP, SIP-SR and SIP models were 0.25 seconds,1.69 seconds and 67.89 seconds, respectively. As observed,

4The notation N(µ, σ) represents the normal probability distributionfunction (PDF) with mean µ and standard deviation σ.

Input data

Select explanatory and response variables

Check AIC

If minimum

AIC?No

Yes

Perform linear regression of the statistical moments

of selected PDF over explanatory variables

Approximate forecasted distribution [15]

Output scenarios

PDF

estimation

using

GAMLSS

Select distribution type j

j=j+1

Select generator i

If i=G (last

generator)?

Yes

i=i+1

No

Predict values of explanatory variables for generator i

Predict the moments for generator i

Predicted distribution of metered output of generator i

PDF

prediction

Fig. 5. Estimation and forecasting the dynamic distribution of meteredgeneration gmi using GAMLSS

the computational time for the SIP model is the highest dueto the several binary variables introduced for the linearizationprocess 5. We can reduce the computational time in our MILPmodels in two ways: first, we can employ decompositionalgorithms to break the MILP model into smaller and easier-to-solve optimization problems [40], [41]. Second, we can usescenario generation and reduction algorithm to reduce the sizeof our MILP models [42], [43]. Using the reduced scenarioswith updated probabilities that can represent the entire dataset enables us to reduce the computational time significantly,which is desirable for real-time dispatch decisions. We employthe second approach in our paper and through a proposedscenario generation and reduction algorithm, the number ofscenarios in our MILP models is reduced to 10 scenarioswith updated probabilities. We select generator G1 (the firstgenerator in the merit-order dispatch) as the (renewable-)intermittent generator participating in the balancing market.The standard deviation of output generation for G1 is changedfrom σG1 = 0 MWh (firm output) to σG1 = 70 MWh(intermittent output). As seen in Fig. 7, the BSC increaseswith increasing output intermittency of G1 in SIP and SIP-SR

5It is not the primary focus of the paper to reduce the computation time.However, it could be an interesting future work to look at possible ways toreduce the computation time for the SIP model.

10

=1

=2

=3

=4

=5

=6

=7

=8

=9

=10

gm=[400, 499.49, 350.76, 449.24, 547.92]; =0.118

gm=[400, 498.17, 350.45, 449.39, 566.56]; =0.094

gm=[400, 502.20, 350.96, 451.62, 532.03]; =0.134

gm=[400, 501.61, 350.76, 463.69, 551.32]; =0.117

gm=[400, 502.70, 352.01, 433.79, 552.65]; =0.089

gm=[400, 499.48, 345.06, 441.07, 539.84]; =0.111

gm=[400, 498.33, 357.13, 459.89, 574.46]; =0.074

gm=[400, 500.20, 346.21, 459.69, 539.39]; =0.102

gm=[400, 499.36, 350.87, 436.60, 570.01]; =0.072

gm=[400, 502.04, 341.53, 449.34, 555.52]; =0.089

Regulation

bid status

Imbalance in

each scenario

Stage 1

Fig. 6. Sample scenario tree for one-stage stochastic MILPs for all the threemodels

0 10 20 30 40 50 60 70

Standard deviation of G1 (MWh)

-200

-150

-100

-50

0

50

100

150

200

BS

C (

$/h

) a

t S

IL=

12

5 M

Wh

20

40

60

80

100

120

140

min

imu

m S

IL (

MW

h)

DIP SO cost

SIP SO cost

SIP-SR SO cost

DIP min SIL

SIP min SIL

SIP-SR min SIL

Fig. 7. Effect of varying standard deviation of G1 metered generation onbalancing and settlement cost (BSC) and minimum SIL for five-generatorsystem (BSC axis is scaled down by 10 for SIP model)

models. This indicates that these models do not prefer to staywith the cheaper generator G1 as it becomes more intermittent.Both SIP and SIP-SR use marginal cost pricing, and in thiscase they provide strong signals on the cost of intermittency.The DIP model however, being based on average pricing, isnot able to give this signal as the BSC reduces in its case withthe increase in intermittency of G1. This is one of the reasonswhy the GB market moved from DIP to SIP model.The effect of probabilistic constraint (10) is also shown in Fig.7. The SO can satisfy the energy-balance constraint in 90%of scenarios (β = 0.9) with lower SIL level when the σG1 islow. However, when generator G1 becomes more intermittent(higher σG1), the SO needs to tolerate higher SIL levels inorder to satisfy the probabilistic constraint (10) in 90% ofscenarios.

A. Comparison of the proposed optimal-dispatch models withthe benchmark models

In the benchmark model, the recourse costs incurred by the SOdecision in the BMC (term E(ISC) in optimization model (12))are ignored. This benchmark optimal-dispatch model is solvedand the BMC is calculated. The ISC for each scenario is alsocalculated and weighted average of the imbalance costs E(ISC)is obtained. Results in Table V show the predictive nature ofour optimal-dispatch model. Under σG1 = 30 MWh, theBench-DIP model leads to BSC=226.6 $/h. This is while

TABLE VCOMPARISON OF BMC, ISC AND BSC (ALL VALUES ARE IN $/h) FOR

THE BENCHMARK (BENCH.) AND PROPOSED (PROP.) MODELS FORβ = 0.9 AND SIL=125MWh AT DIFFERENT STANDARD DEVIATION (STD.

DEV.) OF G1

std. DIP SIP-SR SIP

dev. Bench. Prop. Bench. Prop. Bench. Prop.0 BMC -55 210 -55 230 -55 1365

ISC 216.2 -151.7 274.0 -1390.5 274.0 -2185BSC 161.1 58.2 219.0 -160.5 219.0 -820

30 BMC -75 210 -75 930 -75 1035ISC 301.6 -164 445.2 -1040.8 445.2 -1666BSC 226.6 46 370.2 -110.8 370.2 -631

50 BMC -195 210 -195 600 -195 735ISC 295.8 -206.7 338.8 -680.8 342.4 -1233BSC 100.8 3.2 143.8 -80.8 147.4 -498

under our Prop-DIP model, the BSC cost is reduced to46 $/h. Under benchmark model, the SO has a revenue ofBMC=75 $/h in the balancing market, however, the recoursecost of its decision amounts to ISC=301.6 $/h. This iswhile in our proposed optimal-dispatch model, the SO takesinto account the ISC in the BM decision making process.The BMC is increased to 140 $/MWh at the benefit ofreduced ISC to 164 $/MWh revenue. Once we increase theintermittency level of G1 (σG1 = 50 MWh), the same patternis observed. The benchmark optimal-dispatch model ignoresthe ISC leading to 100.8 $/h for total system cost (BSC). Butin our proposed model, the BSC is reduced in order of five toBSC=3.2 $/h at SIL=125 MWh. The comparative analysisof prop-SIP-SR and prop-SIP models with their correspondingbenchmark models show similar pattern for the costs. Forour case studies, the Prop-SIP model has the lowest BSC ascompared to the Prop-DIP and Prop-SIP-SR models.

VII. NUMERICAL RESULTS

In this section, we assume eight generators6 participating in theBM, where each generator represents one area in the Nordicelectricity market7. The metered generation of these generatorsare taken from Nord Pool website [46].

A. Proposed GAMLSS-based scenario generation algorithm inFig. 5To estimate and forecast, the dynamic distribution of meteredgenerations, the GAMLSS packages in R is used [47] andthis linearly regresses the parameters of the assumed distri-bution on lagged values, wind generation, hydropower anddemand. Table VI gives the value of Akaike InformationCriterion (AIC) for different distributions for the data of G1

metered generation. The skew t distribution was found tohave minimum AIC. Similarly, the analysis was performed forother generators in the system. Once the skew t distribution

6With eight generators it was easier to interpret the choice of generatorsby the system operator (SO) for balancing market considering their intermit-tency, which is reflected by the ISC. Therefore, the choice of five- and eight-generator systems was for the ease of demonstration. As all our proposedmodels have been linearized, decomposition methods can be implemented tostudy a real system [38], [40], [41], [44], [45].

7Out of the eight generators, three of them are located in Sweden, onein Denmark and four in Norway.

11

TABLE VIAIC VALUES FOR DIFFERENT DISTRIBUTIONS TESTED ON G1 METERED

GENERATION

Distribution Skew t Log Negative Normal Poissonnormal binomial

AIC values 4000 4894.05 4901.03 4925.47 23262.09

TABLE VIIINTERCEPTS AND COEFFICIENTS OBTAINED FROM GAMLSS FOR THE

GENERATORS G1 TO G4

α1 γ1 γ2 γ3 β1 β2 β3G1 35.6 0.072 0.091 0.424 0.451 0.194 5.81G2 661.1 0.057 0.061 0.338 0.551 0.468 11.58G3 716.6 0.045 0.009 0.431 1.976 0.450 -33.05G4 -774.8 0.115 0.052 0.380 0.273 0.828

is selected for G1 metered generation, different parametersof this distribution (mean, standard deviation, skewness andkurtosis) are linearly regressed on those explanatory variableswhich were significant at 5%. The linear expression (28)shows the regression of the mean value (µG1

t ) for G1 meteredgeneration over its explanatory variables. In (28), α1 indicatesthe intercept of the response variable µG1 . The γ1, γ2 and γ3

are the coefficients of the lag values of µG1t . The coefficients

corresponding to wind generation (windt), demand (demt)and hydropower (hydt) are β1, β2 and β3, respectively.

µG1t =α1 + γ1µ

G1t−1 + γ2µ

G1t−2 + γ3µ

G1t−3 + β1wind

G1t

+ β2demG1t + β3hyd

G1t (28)

Tables VII and VIII give the values of intercepts and coeffi-cients for the mean value of metered generation as obtainedfrom GAMLSS. Table IX shows the values of mean (µ),standard deviation (σ), skewness (ν) and kurtosis (τ ) of themetered generation calculated by GAMLSS. These values areused for scenario generation from the skew t distribution. Each

TABLE VIIIINTERCEPTS AND COEFFICIENTS OBTAINED FROM GAMLSS FOR THE

GENERATORS G5 TO G8

α1 γ1 γ2 γ3 γ4 β2 β3G5 -21.5 0.003 0 -0.006 1.086 -0.034 -1.38G6 687.7 0 0 -0.004 0.816 0.199 -7.83G7 6287 -0.003 -0.015 -0.004 -0.981 0.5346 -39.13G8 11090 0.003 -0.722 -0.254 -0.564 -2.341 41.35

TABLE IXOBTAINED VALUES OF MEAN, STANDARD DEVIATION, SKEWNESS AND

KURTOSIS FOR THE GENERATORS FROM GAMLSS

µ(MWh) σ(MWh) ν τG1 1269.2 266.24 0.6082 5.6916G2 2560.7 494.664 0.8782 144.459G3 10988 236.78 -0.1696 3.309G4 1488.8 209.22 0.6236 12.114G5 1523.8 0 1.8538 8.6119G6 6749.8 0.0433 -0.2543 0.072G7 3214.1 0 0.8149 3.6506G8 3562.2 0 1.3201 3.4879

TABLE XINPUT PARAMETER DATA FOR EIGHT-GENERATOR SYSTEM IN DIP,

SIP-SR AND SIP MODELS

Gen λup λdn gup gdn gs

($/MWh) ($/MWh) (MWh) (MWh) (MWh)G1 15 4 300 300 1404.9G2 16 5 200 200 2700.2G3 17 6 1100 1100 11405G4 18 7 100 100 1604.7G5 19 8 100 100 1523.8G6 20 9 700 700 7122G7 21 10 200 200 3214.1G8 22 11 200 200 3562.2

TABLE XICOMPARISON OF STATUS OF UP- AND DOWN-REGULATION BIDS FOR THE

EIGHT-GENERATOR SYSTEM

Gen G1 G2 G3 G4 G5 G6 G7 G8

DIP φup 1(0) 0(0) 0(0) 0(1) 0(0) 0(0) 0(0) 0(0)Prop(Ben) φdn 0(1) 0(0) 0(0) 0(0) 0(1) 1(0) 0(1) 0(1)SIP-SR φup 0(0) 1(0) 0(0) 0(1) 0(0) 0(0) 0(0) 0(0)Prop(Ben) φdn 0(1) 0(0) 0(0) 1(0) 1(1) 0(0) 1(1) 1(1)SIP φup 0(0) 0(0) 0(0) 0(1) 0(0) 0(0) 0(0) 0(0)Prop(Ben) φdn 0(1) 0(0) 0(0) 0(0) 1(1) 0(0) 1(1) 1(1)

skew t distribution is sampled by 1000 scenarios which aregiven as input to the fast forward selection reduction algorithm[38]. As a result of the scenario reduction algorithm, tenscenarios are obtained along with their updated probabilities,which are then used in the proposed and benchmark models8.The computational time for eight-generator system turnedout to be 0.11 seconds, 1.5 seconds and 840.41 seconds forthe DIP, SIP-SR and SIP models respectively. Again, thecomputational time for the SIP model was the highest even forthis case study due to the several binary variables introducedfor the linearization process.

B. Proposed economic-dispatch models

Table XI shows the status of up- and down-regulation offersand bids for the eight generators when the same input is givento all the proposed and benchmark models. The minimumSIL for DIP and SIP-SR was 603 MWh while for SIP itwas 435 MWh to satisfy the probabilistic energy-balanceconstraint (10) at β = 0.9. It can be seen that some of thegenerators up-regulate while none of them down-regulate for

TABLE XIICOMPARISON OF BSC, BMC AND ISC (ALL VALUES ARE IN $/h) FOR

THE BENCHMARK (BENCH.) AND PROPOSED (PROP.) MODELS (POSITIVEVALUE INDICATES COST AND NEGATIVE VALUE IS REVENUE)

DIP model SIP-SR model SIP modelBench. Prop. Bench. Prop. Bench. Prop.

BMC -4400 -1800 -4400 -2500 -4400 -4200ISC 69.4 -3671.3 656.7 -3809.4 745.7 -3018.6BSC -4330.5 -5471.3 -3743.2 -6309.4 -3654.2 -7218.6

8We use the same data for all the three dispatch models, since thegenerator characteristics can be taken as representative across both the regions(Nordic countries and Great Britain).

12

the given set of inputs. As shown in Table X, G1 is themost expensive for down-regulation and has a high standarddeviation (see Table IX), hence it was not chosen in anyof the proposed models due to its high intermittency level.However, it was chosen by all benchmark models which ignorethe intermittency cost of G1. Table XII clearly shows thepredictive nature of our optimal-dispatch model. The Bench-DIP model results in BSC=−4330.5 $/h while our Prop-DIP model gives BSC=−5471.3 $/h. This indicates a higherrevenue in our Prop-DIP model as compared to the Bench-DIP model. The results followed the trend as observed in theIllustrative examples (see Table V). Even though the SO earnsa higher revenue (BMC=4400 $/h) in Bench-DIP model, therecourse cost of its decision leads to ISC of 69.4 $/h. In ourProp-DIP model however, the ISC=−3671.3 $/h. This resultsin a net reduction in BSC by 1140.8 $/h in our Prop-DIPmodel as compared to Bench-DIP model. This reduction inBSC can be accounted to the fact that in our Prop-DIP model,the objective function of the optimization problem comprisesof both the BMC and ISC (as given in equation (11)) thatare minimized. However, in the benchmark model, only theBMC is minimized and the output for binary variables φdniand φupi from minimizing BMC serves as inputs to (11b) and(11c) respectively for finding out the ISC. This value of ISCfor the benchmark model turns out to be higher than that inour proposed model as the intermittency of the generators donot play any role in selecting the generators to be operated inthe balancing market. Another interesting point to explain thelarge differences in the ISC values of Table XII as compared tothose in Table V would be that it has to do with the differencesin the input values of λup, λdn, gup, gdn and gs in the five-and eight-generator systems. Table IV and Table X give thevalues of the inputs considered for the λup, λdn, gup, gdn andgs in our simulations for five- and eight-generator systems,respectively. In order to obtain values for the scheduled outputof the generation, gs, we take average of the 10 values ofgmi,ω that were obtained from the proposed scenario generationalgorithm based on predictive conditional dynamic densitydistributions. The study has been performed on real marketdata acquired from the Nord Pool website to find the valuesfor gmi,ω . As the magnitudes of all these input values for theeight-generator system in the Table X are larger as comparedto those assumed for the five-generator system in Table IV,the results of the ISCs have larger differences in Table XII ascompared to those in Table V.The reduction in BSC for the Prop-SIP-SR and Prop-SIPmodels were 2566.2 $/h and 3564.4 $/h respectively as com-pared to the corresponding benchmark models. A similar trendof reduction in BSC in the proposed models of Numericalexamples has reaffirmed our observations from the Illustrativeexamples. The less reduction in BSC for the Prop-DIP-modelcan be accounted for in Table X. It can be seen that G1 waschosen by the Prop-DIP model for up-regulation as it was thecheapest for up-regulation but also more variable than G4,which was chosen by the corresponding Bench-DIP model.

VIII. CONCLUSION

This paper proposes three stochastic optimization modelsin order to enable the SO to dispatch into the balancingmarket not just based on the marginal cost but also on theintermittency of the generators. This capability will becomeincreasingly important as more intermittent and stochastic gen-erators come on to the system and indeed wish to take part inbalancing actions. The three one-stage stochastic optimizationmodels are based on different imbalance pricing methods.Out of the three proposed models, the single imbalance pricemodels are able to give a clear signal to the stochastic (cheaperbut more uncertain) generators to invest in technologies orhedge efficiently if they want to be chosen in the balancingmarket by the SO. For the same inputs to the MILP models,SIP model gives more saving for the SO as compared to theDIP model. Therefore, we observe why it was beneficial forthe GB electricity market to move from DIP model to SIPmodel in 2015.

IX. FURTHER DISCUSSIONS AND FUTURE WORK

In Nordic market, the balancing market is a two-stage market.In the first stage, the transmission constraints are ignored andthe dispatching is done only based on the energy imbalances,then if the re-dispatch in the first stage is violating sometransmission constraints, in the second stage, the countertrading will be performed [28]. The balancing market in GBis explicitly designed to deal with energy imbalances andany actions, which the system operator takes for locationalor voltage reasons are recorded separately and included inthe recovery of overall system costs by the SO. They arenot part of the balancing market [30]. Thus, from a practicalview-point while activating the bids in the balancing market,transmission constraints are not considered in the first stageof Nordic balancing market and GB electricity market. As ourpaper primarily focuses on these markets, we do not modeltransmission constraints in our paper. However, the explicitmodeling of transmission constraints in our proposed modelis a good future extension of our work, which adjusts ourmodel for other electricity markets too.Reference [48] explains how the SO in Sweden procures lossesbefore the DA market clearing. In the balancing market, theSO activates the bids based on the merit order list independentof how large losses might occur due to activating a particularbid [29]. Later, the SO pays for the transmission grid losses,which it has to buy before the DA market clearing. Losses inGB are included in the settlement system, not the balancingmarket. All contracts are settled in comparison with meteredvolumes and imbalance volumes after the delivery periods andzonal loss factors are applied at that stage [49]. But in someother electricity markets, if it is important to model losses thenthe modeling of ohmic losses in our proposed formulation canbe considered as a good future extension of our work.The tests performed in Sections VI and VII serve the purposeof demonstrating the functionality of the proposed models.While the existing work tests the proposed models with five-and eight-generator systems, it is desirable to apply thesemodels to a full system [38], [40], [41], [44], [45]. This will

13

be a further extension of the existing work.The issue of fairness and openness in scenario generation andreduction, as employed in our method, is not addressed in theliterature. This is one promising direction of future research.

X. ACKNOWLEDGEMENT

Dr. Date acknowledges useful discussions about the GB elec-tricity market with Dr A. H. Alikhanzadeh. The authors alsoacknowledge useful comments regarding the role of windenergy in the UK balancing market from Jean Hamman andMartin Bradley, both from National Grid UK. We are alsograteful to Robert Eriksson from Svenska kraftnat (SwedishTransmission System Operator) for answering our questionsabout the Nordic balancing market.

REFERENCES

[1] IEA. (2018) World Energy Outlook 2018. [Online]. Available:https://www.iea.org/weo2018/

[2] eurostat. (2019) Renewable energy statistics. [Online]. Available:https://ec.europa.eu/eurostat/statistics-explained/index.php/Renewableenergy statistics#Renewable energy produced in the EU increasedby two thirds in 2007-2017

[3] E. Department for Business and I. Strategy. (2018)UK Energy Statistics, Q3 2018. [Online]. Available:https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment data/file/766776/Press Notice December 2018.pdf

[4] EIA. (2019) U.S. renewable electricity generation has doubled since2008. [Online]. Available: https://www.eia.gov/todayinenergy/detail.php?id=38752

[5] IEA. (2014) The Power of Transformation. [Online]. Available:https://webstore.iea.org/the-power-of-transformation

[6] P. Sorknæs, A. N. Andersen, J. Tang, and S. Strøm, “Market integrationof wind power in electricity system balancing,” Energy Strategy Reviews,vol. 1, no. 3, pp. 174–180, 2013.

[7] H. Lund, “Electric grid stability and the design of sustainable energysystems,” International Journal of Sustainable Energy, vol. 24, no. 1,pp. 45–54, 2005.

[8] “European parliament, council, directive 2003/54/ ec of the europeanparliament and of the council of 26 june 2003 concerning common rulesfor the internal market in electricity and repealing directive 96/92/ec estatements made with regard to decommissioning and waste managementactivities,” pp. 37–56, 2003.

[9] “National grid, managing intermittent and inflexible generation in thebalancing mechanism, national grid,” Sept. 2011.

[10] EWEA. (2015) Balancing responsibility and costsof wind power plants. [Online]. Available: http://www.ewea.org/fileadmin/files/library/publications/position-papers/EWEA-position-paper-balancing-responsibility-and-costs.pdf

[11] L. Vandezande, L. Meeus, R. Belmans, M. Saguan, and J.-M. Glachant,“Well-functioning balancing markets: A prerequisite for wind powerintegration,” Energy Policy, vol. 38, no. 7, pp. 3146–3154, 2010.

[12] Y. Guo, Y. Gong, Y. Fang, and P. P. Khargonekar, “Stochastic minimiza-tion of imbalance cost for a virtual power plant in electricity markets,”in ISGT 2014, Feb 2014, pp. 1–5.

[13] J. Liang, S. Grijalva, and R. G. Harley, “Increased wind revenue andsystem security by trading wind power in energy and regulation reservemarkets,” IEEE Transactions on Sustainable Energy, vol. 2, no. 3, pp.340–347, July 2011.

[14] A. Hellmers, M. Zugno, A. Skajaa, and J. M. Morales, “Operationalstrategies for a portfolio of wind farms and chp plants in a two-pricebalancing market,” IEEE Transactions on Power Systems, vol. 31, no. 3,pp. 2182–2191, May 2016.

[15] J. Matevosyan and L. Soder, “Minimization of imbalance cost tradingwind power on the short-term power market,” IEEE Transactions onPower Systems, vol. 21, no. 3, pp. 1396–1404, 2006.

[16] P. M. S. Frade, J. P. S. Catalao, J. P. Pereira, and J. J. E. Santana,“Balancing reserves in a power system with high wind penetration- evidence from portugal,” in 15th International Conference on theEuropean Energy Market (EEM), June 2018, pp. 1–4.

[17] A. G. Vlachos and P. N. Biskas, “Demand response in a real-timebalancing market clearing with pay-as-bid pricing,” IEEE Transactionson Smart Grid, vol. 4, no. 4, pp. 1966–1975, Dec 2013.

[18] A. Helander, H. Holttinen, and J. Paatero, “Impact of wind power on thepower system imbalances in Finland,” IET renewable power generation,vol. 4, no. 1, pp. 75–84, 2010.

[19] J. Browell, “Risk constrained trading strategies for stochastic generationwith a single-price balancing market,” Energies, vol. 11, no. 6, p. 1345,2018.

[20] R. A. van der Veen and R. A. Hakvoort, “The electricity balancingmarket: Exploring the design challenge,” Utilities Policy, vol. 43, pp.186–194, 2016.

[21] T. Brijs, K. De Vos, C. De Jonghe, and R. Belmans, “Statistical analysisof negative prices in european balancing markets,” Renewable Energy,vol. 80, pp. 53–60, 2015.

[22] L. Hirth and I. Ziegenhagen, “Balancing power and variable renewables:Three links,” Renewable and Sustainable Energy Reviews, vol. 50, pp.1035–1051, 2015.

[23] J. P. Chaves-Avila, R. A. Hakvoort, and A. Ramos, “The impact ofeuropean balancing rules on wind power economics and on short-termbidding strategies,” Energy Policy, vol. 68, pp. 383–393, 2014.

[24] C. Fernandes, P. Frıas, and J. Reneses, “Participation of intermittentrenewable generators in balancing mechanisms: A closer look into thespanish market design,” Renewable Energy, vol. 89, pp. 305–316, 2016.

[25] P. Pinson, C. Chevallier, and G. N. Kariniotakis, “Trading wind gen-eration from short-term probabilistic forecasts of wind power,” IEEETransactions on Power Systems, vol. 22, no. 3, pp. 1148–1156, 2007.

[26] A. Skajaa, K. Edlund, and J. M. Morales, “Intraday trading of windenergy,” IEEE Transactions on Power Systems, vol. 30, no. 6, pp. 3181–3189, 2015.

[27] S. Kraftnat. (2019) Overview of the Requirements on Reserves. [On-line]. Available: https://www.svk.se/siteassets/aktorsportalen/elmarknad/information-om-reserver/reserve-markets.pdf

[28] ENTSOE. (2016) Nordic Balancing Philosophy. [Online]. Avail-able: https://docstore.entsoe.eu/Documents/Publications/SOC/Nordic/Nordic Balancing Philosophy 160616 Final external.pdf

[29] S. Kraftnat, “The swedish electricity market and the role of svenskakraftnat,” Published Jan, 2007. [Online]. Available: https://inis.iaea.org/collection/NCLCollectionStore/ Public/42/022/42022239.pdf

[30] E. M. A. Team. (2019) System Prices Analysis Report.[Online]. Available: https://www.elexon.co.uk/about/key-data-reports/system-prices-analysis-report/

[31] nationalgridESO. (2020) Short-term operating reserve (STOR). [On-line]. Available: https://www.nationalgrideso.com/balancing-services/reserve-services/short-term-operating-reserve-stor

[32] PJM. (2019) About PJM. [Online]. Available: https://www.pjm.com/about-pjm/who-we-are.aspx

[33] ——. (2019) Understanding the DifferencesBetween PJM’s Markets. [Online]. Available:https://learn.pjm.com/-/media/about-pjm/newsroom/fact-sheets/understanding-the-difference-between-pjms-markets-fact-sheet.ashx?la=en

[34] A. L. S. Mendes, N. de Castro, R. Brandao, L. Camara, andM. Moszkowicz, “The role of imbalance settlement mechanisms inelectricity markets: A comparative analysis between uk and brazil,” in13th International Conference on the European Energy Market (EEM),June 2016, pp. 1–6.

[35] R. T. Rockafellar, S. Uryasev et al., “Optimization of conditional value-at-risk,” Journal of risk, vol. 2, pp. 21–42, 2000.

[36] A. Bello, D. W. Bunn, J. Reneses, and A. Munoz, “Medium-termprobabilistic forecasting of electricity prices: A hybrid approach,” IEEETransactions on Power Systems, vol. 32, no. 1, pp. 334–343, Jan 2017.

[37] D. M. Stasinopoulos, R. A. Rigby et al., “Generalized additive modelsfor location scale and shape (gamlss) in r,” Journal of StatisticalSoftware, vol. 23, no. 7, pp. 1–46, 2007.

[38] A. J. Conejo, M. Carrion, J. M. Morales et al., Decision making underuncertainty in electricity markets. Springer, 2010, vol. 1.

[39] IBM, “V12. 1: User’s manual for cplex,” International Business Ma-chines Corporation, vol. 46, no. 53, p. 157, 2009.

[40] E. Moiseeva and M. R. Hesamzadeh, “Strategic bidding of a hydropowerproducer under uncertainty: Modified benders approach,” IEEE Trans-actions on Power Systems, vol. 33, no. 1, pp. 861–873, Jan 2018.

[41] Y. Tohidi, M. R. Hesamzadeh, and F. Regairaz, “Modified bendersdecomposition for solving transmission investment game with riskmeasure,” IEEE Transactions on Power Systems, vol. 33, no. 2, pp.1936–1947, March 2018.

14

[42] J. C. do Prado and W. Qiao, “A stochastic decision-making model foran electricity retailer with intermittent renewable energy and short-termdemand response,” IEEE Transactions on Smart Grid, vol. 10, no. 3,pp. 2581–2592, May 2019.

[43] J. Shen, C. Jiang, Y. Liu, and X. Wang, “A microgrid energy man-agement system and risk management under an electricity marketenvironment,” IEEE Access, vol. 4, pp. 2349–2356, 2016.

[44] A. J. Conejo, E. Castillo, R. Minguez, and R. Garcia-Bertrand, De-composition techniques in mathematical programming: engineering andscience applications. Springer Science & Business Media, 2006.

[45] C. Sagastizabal, “Divide to conquer: decomposition methods for energyoptimization,” Mathematical programming, vol. 134, no. 1, pp. 187–222,2012.

[46] NordPool. (2017) Historical market data. [Online]. Available: https://www.nordpoolgroup.com/historical-market-data/

[47] R. Rigby, D. Stasinopoulos, and C. Akantziliotou, “Instructions on howto use the gamlss package in r (2008),” Available at: http://www. gamlss.org/wp-content/uploads/2013/01/gamlss-manual. pdf.

[48] J. Sahlin, R. Eriksson, M. T. Ali, and M. Ghandhari, “Transmission lineloss prediction based on linear regression and exchange flow modelling,”in 2017 IEEE Manchester PowerTech, June 2017, pp. 1–6.

[49] ELEXON. Line Loss Factors. [Online]. Available: https://www.elexon.co.uk/operations-settlement/losses/

XI. APPENDIX

A. Linearizations related to DIP model

The non-liear expression, ∆−ω = Max(-∆ω ,0), can be linearizedby replacing it with the following two equations:

0 ≤ ∆−ω ≤ b−ωΞ(1) (29a)

−∆ω ≤ ∆−ω ≤ −∆ω + (1− b−ω )Ξ(1) (29b)

A similar linearization has been performed for I+i,w = φdni ∆+

ω

by using the following equations:

0 ≤ I+i,w ≤ φdni Ξ(1) (30a)

∆+ω − (1− φdni )Ξ(1) ≤ I+

i,w ≤ ∆+ω (30b)

Again, for the non-linear expression, Iupi,w = sωφupi , it can be

replaced by the equations given below:

0 ≤ Iupi,w ≤ φupi Ξ(1) (31a)

sω − (1− φupi )Ξ(1) ≤ Iupi,w ≤ sω (31b)

Similarly, for I−i,w = φupi ∆−ω , the following equations can beused for linearizaion:

0 ≤ I−i,w ≤ φupi Ξ(1) (32a)

∆−ω − (1− φupi )Ξ(1) ≤ I−i,w ≤ ∆−ω (32b)

The ‘If’ condition in (1c) is written as linear constraints∑i φ

upi ≥ 1− Ξ(1)(1− φ) and

∑i φ

upi ≤ 1 + Ξ(1)(φ).

The bi-linear term, zdnω = φcω , corresponding to (18), can belinearized as:

0 ≤ zdnω ≤ φΞ(1) (33a)

cω − (1− φ)Ξ(2) ≤ zdnω ≤ cω (33b)

Similarly, another bi-linear expression, zupω = φsω , in (18) canbe replaced by the following equations to make them linear:

0 ≤ zupω ≤ φΞ(1) (34a)

sω − (1− φ)Ξ(1) ≤ zupω ≤ sω (34b)

The Max operator, u+ω = Max(∆+

ω −α, 0), in the probabilis-tic constraint can be linearized as given by:

0 ≤ u+ω ≤ b1ωΞ(1) (35a)

∆+ω − α ≤ u+

ω ≤ ∆+ω − α+ (1− b1ω)Ξ(1) (35b)

Again, the other Max operator, u−ω = Max(∆−ω − α, 0), canbe replaced by:

0 ≤ u−ω ≤ b2ωΞ(1) (36a)

∆−ω − α ≤ u−ω ≤ ∆−ω − α+ (1− b2ω)Ξ(1) (36b)

B. Linearizations related to SIP-SR model

The non-linearity introduced because of up-regulation priceand total negative deviation in the SIP-SR model is given bycdnω =Max(λupi N

upi,ω), in order to linearize it, it can be replaced

with the following equations:

− riωΞ(1) ≤ cupω − λupi N

upi,ω ≤ riωΞ(1) (37a)∑

i

riω = G − 1 (37b)

where riω is a binary variable.

For the ‘If’ condition, I(∑i φ

upi g

upi <

∑i φ

dni gdni ), the below

equations can be used:∑i

φupi gupi ≥

∑i

φdni gdni − Ξ(1)(1− φ) + ε (38a)∑i

φupi gupi ≤

∑i

φdni gdni + Ξ(1)φ− ε (38b)

The non-linear expression, kdni = φφdni , can be linearized asφ+ φdni − 1 ≤ kdni ≤ φ and kdni ≤ φdni

From the objective function (26), tupω = ∆+ωφ can be replaced

by the following equations:

0 ≤ tupω ≤ φΞ(1) (39a)

∆+ω − (1− φ)Ξ(1) ≤ tupω ≤ ∆+

ω (39b)

Another non-linear term, h+ = ∆+ωφ, from (26) can be written

as:

0 ≤ h+ ≤ φΞ(1) (40a)

∆+ω − (1− φ)Ξ(1) ≤ h+ ≤ ∆+

ω (40b)

On similar lines, the term ∆−ωφ, in (26), can be denoted bytdnω and replaced with:

0 ≤ tdnω ≤ φΞ(1) (41a)

∆−ω − (1− φ)Ξ(1) ≤ tdnω ≤ ∆−ω (41b)

Finally, the non-linearity introduced in (26), by h− = ∆−ωφcan be tackled by replacing it with the following equations:

0 ≤ h− ≤ φΞ(1) (42a)

∆−ω − (1− φ)Ξ(1) ≤ h− ≤ ∆−ω (42b)

Please note that, in our case, the same value of Ξ workedfor all the linearization equations. But it can also be differentcorresponding to each specific linearization, even though wehave written Ξ(1) throughout the Appendix section.