11th February 2013

DESC: NDM Algorithm Performance(Gas year 2011/12)

Strand 2: Reconciliation Variance Analysis &NDM Sample Analysis

2

Algorithm Performance 2011/12: Strand 2 Analysis

• Strand 1 (SF and WCF analysis) presented at Nov 2012 DESC• SF performance during 11/12 was mixed compared with 10/11 (generally

closer to the ideal of 1 over summer and further from 1 over winter)

• WCF deviation generally improved over Winter 11/12 and slightly worsened during the Summer 11/12 (compared with 10/11).

• Strand 2: Reconciliation Variance Analysis• Compare allocated demand (derived from algorithms) with

• Actual demand obtained from available reconciliation data

• Strand 2: NDM Sample Consumption Analysis• Compare the actual demand from the NDM sample data with

• Allocated demand for the sample

• Supporting document: detailed explanation with

full examples

3

Reconciliation Variance (RV) Analysis

• Compare actual demand (rec.) to allocated demand (algorithms)

• Use available Meter Point rec. data for band ‘B’ EUCs • Data available at time of analysis (non-monthly, smaller EUC may not

have been received)• No analysis for EUC Band 1 (no rec.)• Uses Standard & Suppressed rec.

• Rejection criteria applied prior to analysis to remove inappropriate or erroneous rec. data • Negative and zero consumptions, actual to allocated ratio

• Profile comparisons are then compared and categorised as:• ‘Peaky’ - ‘Flat’ - ‘Ok’

4

Assessment of Standard and Suppressed Reconciliation

Assessment of Standard and Suppressed Reconciliation(based on reconciliations during April to September 2012)

0%

2%

4%

6%

8%

10%

12%

14%

16%

-100+

-90-1

00

-80-9

0

-70-8

0

-60-7

0

-50-6

0

-40-5

0

-30-4

0

-20-3

0

-10-2

0

-0-1

0

0-1

0

10-2

0

20-3

0

30-4

0

40-5

0

50-6

0

60-7

0

70-8

0

80-9

0

90-1

00

100+

Range - Drift %

% o

f R

ec

on

cil

iati

on

s

Standard Rec Suppressed Rec

Estimated > 2 * Actual Estimated < 0.5* Actual

Range - Drift% ((Actual - Allocated) / Allocated)

5

RV Analysis - Data Envelope

Estimated

Actual

Estimated = 0.5 * Actual

Estimated = Actual

Estimated = 2 * Actual

6

RV Analysis: Levels of Validation Fall Out

• Rejection Criteria: AQ <=3 kWh ; Actual <=0 ; Actual >0 and Allocated > 2 * Actual ;

Actual >0 and Allocated <0.5 * Actual

• Rejection rates higher in summer due to smaller consumptions thereby

resulting in greater % differences

• Profiles consistent with previous years and post-validation numbers good

0

50,000

100,000

150,000

200,000

250,000

300,000

350,000

400,000

Oct1

0

Nov10

Dec10

Jan11

Feb11

Mar1

1

Apr1

1

May11

Jun11

Jul1

1

Aug11

Sep11

Oct1

1

Nov11

Dec11

Jan12

Feb12

Mar1

2

Apr1

2

May12

Jun12

Jul1

2

Aug12

Sep12

Num

ber of Record

s

0

10

20

30

40

50

60

Perc

enta

ge (%

)

Raw Data Validated % Rejected

7

RV Analysis: Rejections – approx. breakdown

12.8%3.2%Actual > 0 and

Allocated < 0.5 x Actual

22.6%7.3%Actual > 0 and

Allocated > 2 x Actual

9.2%3.1%Actual = 0

1.5%1.0%Actual < 0

1.2%1.4%AQ <= 3 kWh pa

Maximum 47.3%

(September 2012)

Minimum 15.9%

(May 2012)Rejection category

• Table shows the rejection category breakdown for:

• May 2012 - which had the smallest rejection %

• September 2012 - which had the largest rejection %

8

RV Analysis: Unreconciled Energy Profile

Unreconciled Energy Profile

(upto and incl Oct 2012 Rec Invoice)

0%

20%

40%

60%

80%

100%

2011_11

2011_12

2012_01

2012_02

2012_03

2012_04

2012_05

2012_06

2012_07

2012_08

2012_09

2012_10

Year_Month

% U

nre

co

nciled

% Reconciled % Unreconciled

9

RV Analysis: Methodology

• Following removal of rejected reconciliations, for each meter point:

• Reconciled energy is identified

• Allocated Energy calculated

• Values are then applied evenly to each day of the reconciliation period

• Average for each of the meter points in the specific EUC is calculated

• Profile is ‘scaled’

• Level of allocated demand (based on AQ) = actual demand (actual)

• Scaling allows profile comparisons and analysis of algorithm performance

• Without scaling analysis would primarily highlight differences in demand levels (affected by other factors)

• Example charts for cross section of EUC Bands (B) and LDZs provided in supporting document

10SC : Consumption Band 04 (Pre-Scaling)RV Analysis – Allocated to Actual

• 1st chart highlights where scaling has not occurred and profile of demand

through the year.

• Following scaling…..

0

500

1000

1500

2000

2500

3000

3500

4000

Oct-11 Nov-11 Dec-11 Jan-12 Feb-12 Mar-12 Apr-12 May-12 Jun-12 Jul-12 Aug-12 Sep-12

Avera

ge D

aily D

em

and p

er

mete

r (k

Wh)

Smooth Actual Smooth Allocated

11SC : Consumption Band 04 (After Scaling) RV Analysis – Allocated to Actual

• Analysis allows comparison of the profiles rather than demand levels

• Indicates an over allocation in the Winter & under allocation in the summer

• ‘Peaky’ allocated profile: Winter over, Summer under (predominant profile)

0

500

1000

1500

2000

2500

3000

3500

4000

Oct-11 Nov-11 Dec-11 Jan-12 Feb-12 Mar-12 Apr-12 May-12 Jun-12 Jul-12 Aug-12 Sep-12

Avera

ge D

aily D

em

and p

er

mete

r (k

Wh)

Smooth Actual Smooth Allocated

12

RV Categorisation: LDZ / EUC Profile & Error LevelsGas Year 2011/12

• ‘% level’ = average difference of allocated to actual over the winter and summer differences (measures ‘peakiness’)

• 2011/12: ‘Peaky’ profile 37%, ‘Ok’ profile 34%, ‘Flat’ 10%, No data for analysis 19%

• 2010/11: ‘Peaky’ profile 50%, ‘Ok’ profile 26%, ‘Flat’ 5%, No data for analysis 19%

• 2009/10: ‘Peaky’ profile 53%, ‘Ok’ Profile 28%, ‘Flat’ 5%, No data for analysis 14%

• Profiles overall for 2011/12 tend to be ‘OK’ or ‘Peaky’

EUC BAND SC NO NW NE EM WM WN WS EA NT SE SO SW

02 B ↑ ∼ ↑ ↑ ↑ ∼ ↑ ↑ ∼ ↑ ↑ ↑ ↑

03 B ∼ ↑ ↑ ↑ ↑ ↑ ⇑ ⇑ ↑ ↑ ↑ ↑ ⇑

04 B ↑ ∼ ∼ ∼ ∼ ∼ ∼ ↑ ∼ ↑ ∼ ∼ ↑

05 B ∼ ∼ ↑ ∼ ∼ ∼ ∼ ∼ ∼ ↑ ∼ ∼ ∼

06 B ⇑ ↓ ↑ ∼ ∼ ↑ ⇑ ↓ ∼ ∼ ∼ ↑ ↓

07 B ↓ ↑ ↑ ∼ ↓ ⇓ ∼ ↑ ↓ ↓ ↑

08 B ⇑ ↓ ∼ ∼ ∼ ∼ ⇑

09 B ⇓

OK / Good ∼ 5% Level ↑ Too Peaky 10% Level ⇑ Too Peaky

No Data (<3) ↓ Too Flat ⇓ Too Flat

13

RV Analysis: Conclusions

• RV analysis highlights a ‘peaky’ trend of:• Over Allocation – Winter

• Under Allocation – Summer

• 2011/12 saw 37% of profiles defined as ‘peaky’ (50% in 10/11):• Levels of rec. rejected similar to previous years

• Available rec. for analysis incomplete, particularly Bands 2/3 (non-monthly read meters)

• BUT – analysis not necessarily representative of population• Consider with SF and WCF analysis and NDM Sample data…

14

NDM Sample Consumption Analysis

• Using the actual NDM Sample consumption for 11/12

• Compare the % error of sample consumption against three models :

• Allocated using 11/12 ALPs & DAFs, real system WCF and SF - (“As Used”)

• Allocated using 11/12 ALPs & DAFs, EWCF and SF = 1 – (Best Estimate ’11)

• Allocated using 12/13 ALPs & DAFs, 11/12 EWCF and SF = 1 – (Best Estimate ’12)

• This is completed by EUC for all LDZs and also by month by LDZ

• Supporting document - detailed explanation with full

examples

15Allocated Error As % of Actual Demand

Weighted average across LDZs. ‘As Used’System WCF and SF – ALPs and DAFs 11/12 Algorithms - NDM Sample derived AQs (not system AQs)

• Positive errors = Under allocation; Negative errors = Over allocation.

• Over year: Positive errors across all consumption bands (indicate population AQs too high)• ‘As Used’ model uses real system SFs which have taken population AQs into account.

• AQs used based on sample consumption which is also expected to be lower than equivalent system AQs

• ‘As Used’ model does not assess EUC profiles, however can provide indicator of system AQ excess…..

Error as a Percentage of Demand - Weighted average across LDZs:

'As Used'

-1%

0%

1%

2%

3%

4%

5%

6%

7%

01B 02B 03B 04B 05B 06B 07B 08B

Pe

rce

nta

ge

Err

or

Oct 11 - Mar 12 Oct 11 - Sep 12 Apr 12 - Sep 12

Figure 3.1

Actual WCF and SF

Actual ALPs and DAFs

16

As Used Model – AQ Assessment

-5.8%2.2%Overall

-7.3%3.3%SW

-5.6%2.3%SO

-6.6%3.5%SE

-6.0%2.7%NT

-5.9%2.3%EA

-6.5%2.6%WS

-8.0%-WN

-5.0%1.3%WM

-5.4%1.9%EM

-4.5%1.0%NE

-6.4%2.0%NW

-5.6%1.8%NO

-4.7%1.2%SC

Observed AQ Reductions in

Gemini at start of gas year 2012/13

Estimated AQ Excess (+) or Deficit (-)

(‘as used’ analysis full year errors)LDZ

17Allocated Error As % of Actual DemandWeighted average across LDZs. ‘Best Estimate 11’

EWCF and SF =1 – ALPs and DAFs 11/12 Algorithms - NDM Sample derived AQs (not system AQs)

• Remove SF impact and use EWCF which avoids potential bias in WCF• Positive errors = Under allocation ; Negative errors = Over allocation• Winter/Summer analysis indicates bands 03 & 06 little too flat and bands 01,02,04,05,07

& 08 little too peaky• Over year: Little overall error in each band (Range +0.06% and +0.53% for all bands)

Error as a Percentage of Demand - Weighted average across LDZs:

'Best Estimate 11'

-4%

-3%

-2%

-1%

0%

1%

2%

3%

4%

01B 02B 03B 04B 05B 06B 07B 08B

Pe

rce

nta

ge

Err

or

Oct 11 - Mar 12 Oct 11 - Sep 12 Apr 12 - Sep 12

Figure 3.2

EWCF and SF = 1

ALPs and DAFs : 2011/12

18Allocated Error As % of Actual DemandWeighted average across LDZs. ‘Best Estimate 12’

EWCF and SF =1 – ALPs and DAFs 12/13 Algorithms - NDM Sample derived AQs (not system AQs)

• ALPs and DAFs for 2012/13 applied to 2011/12 consumption data

• Should provide less error as ALPs and DAFs were derived from this consumption data

• Winter / Summer errors are slightly improved for bands 03,04,05 and 06. Slightly worse for 01,02,07 and 08

• Over whole year, on average, extent of error across all EUCs is slightly reduced using 12/13 algorithms

• Monthly analysis also completed…

Error as a Percentage of Demand - Weighted average across LDZs:

'Best Estimate 12'

-4%

-3%

-2%

-1%

0%

1%

2%

3%

4%

01B 02B 03B 04B 05B 06B 07B 08B

Pe

rce

nta

ge

Err

or

Oct 11 - Mar 12 Oct 11 - Sep 12 Apr 12 - Sep 12

Figure 3.3

EWCF and SF = 1

ALPs and DAFs : 2012/13

19Monthly Actual & Deemed Demand01B (All LDZs)

As previous but by EUC and By Month

• Results also provided for previous models but by EUC Band and Month - Equivalent charts for all

consumption bands included in supporting document

• Band 01B profile – indicates winter over allocation (except Dec & Jan) and summer under allocation

• Relevant to recall weather conditions in 11/12 when interpreting results

• During Winter months October , November and March were warmer than seasonal normal (warmest

March in last 50 years) with Dec and Jan experiencing notable periods of colder than normal weather

• Summer months generally colder than seasonal normal (April was unusually colder than March)

Monthly Actual & Deemed Demands for 01B (across all LDZs)

0

1

2

3

4

5

6

7

8

9

Oct 11 Nov 11 Dec 11 Jan 12 Feb 12 Mar 12 Apr 12 May 12 Jun 12 Jul 12 Aug 12 Sep 12

Dem

and G

Wh

As Used Actual Best Estimate 11 Best Estimate 12

Figure 3.4

20Monthly Actual & Deemed Demand04B (All LDZs)

As previous but by EUC and By Month

• Band 04B profile – indicates winter over allocation (except for colder months of December, January and

February) and summer under allocation (with the exception of August and September)

Monthly Actual & Deemed Demands for 04B (across all LDZs)

0

100

200

300

400

500

600

Oct 11 Nov 11 Dec 11 Jan 12 Feb 12 Mar 12 Apr 12 May 12 Jun 12 Jul 12 Aug 12 Sep 12

Dem

and G

Wh

As Used Actual Best Estimate 11 Best Estimate 12

Figure 3.7

21

Figure 3.19: Daily Actual and Deemed Demands for 01B (across all LDZs)

0

50

100

150

200

250

300

350

400

01/10/2011 01/12/2011 01/02/2012 01/04/2012 01/06/2012 01/08/2012

Dem

and M

Wh

Actual Best Estimate 11 Best Estimate 12

Daily Actual & Deemed Demand01B (All LDZs)

• The daily chart for Band 01 shows that allocated demand was generally close to actual demand. The most notable exception to this occurred during the warmest March in last 50 years and the unseasonably colder weather during April, early May and June.

Under allocation due to colder

weather in April, early May and June

Over allocation due to notably warmer

weather in late February and March

22Daily Actual & Deemed Demand

04B (All LDZs)

• The daily chart for Band 04 shows that allocated demand was generally close to actual demand. The most notable exception to this occurred during the last week of the Christmas holiday period and the cold weather in late April and early May 2012.

Figure 3.22: Daily Actual and Deemed Demands for 04B (across all LDZs)

0

5000

10000

15000

20000

25000

30000

35000

01/10/2011 01/12/2011 01/02/2012 01/04/2012 01/06/2012 01/08/2012

Dem

and M

Wh

Actual Best Estimate 11 Best Estimate 12

Under allocation during last week

of Christmas holiday period

Under allocation due to colder

weather in late April, early May

23

RV Analysis & NDM Sample Analysis

• The “best estimate 11” & “best estimate 12” analyses suggest:

• For bands 01, 02, 04, 05, 07 & 08: over allocation (-ve errors) in the winter and under allocation (+ve errors) in the summer. � profile too peaky.

• For bands 03 & 06: under allocation (+ve errors) in the winter and over allocation (-ve errors) in the summer. � profile too flat.

• The RV analysis indicated profiles that were:

• too peaky in most LDZs in bands 02 & 03 (overall too peaky in bands 02 & 03, at 5% level)

• good in most LDZs in bands 04 & 05 (overall slightly too peaky in bands 04 & 05, below 5% level)

• mixture of good, too peaky and too flat profiles in bands 06, 07, 08(overall slightly too peaky, below 5% level in bands 06 & 07 andoverall too peaky in band 08, at 5% level)

24

RV Analysis & NDM Sample Analysis Conclusions

• Limitations - different, restricted data sets

• Analyses based on different data sets - neither are necessarily representative of population as a whole

• RV analysis excludes band 01B & based on a sub-set of rec data

• NDM sample analysis is based on validated NDM SAMPLE data

• Both analyses suffer from small numbers of contributing meter/supply points at the higher consumption bands

• Important Point: Both approaches, subject to their limitations, suggest only small inaccuracies over the year as a whole

• Full explanatory document on Joint Office website: • ‘Algorithm Performance Strand 2 Evaluation 2011-12.pdf’