1

Evaluation of Energy Efficiency Programs: Adjustments to Energy Savings

Edward Vine

California Institute for Energy and Environment

Center for the Study of Energy Markets (CSEM) Policy Conference

Sacramento, CA

December 9, 2008

2

Outline of Talk

History of Energy Efficiency in California Overview of Evaluation Adjustments to Energy Savings: Net Energy

Savings Cost-Effectiveness Metrics Why do We Care about NTG & TRC? AEA’s Guiding Principles for Evaluators

3

1. History of Energy Efficiency in California

Energy efficiency is California’s highest priority resource for meeting its energy needs in a clean, reliable and low-cost manner

For more than three decades, California has adopted energy efficiency policies and made investments that are among the most aggressive in the nation

These efforts have saved more than 40,000 GWh of electricity and 12,000 MW of peak demand - avoiding the need to build 24 large (500 MW) power plants, and equal to the energy required to power 3.8 million homes

4

Californians use less electricity per person than those in all other states

5

California vs. US Energy Efficiency

Per Capita Electricity ConsumptionkWh/person

-

2,000

4,000

6,000

8,000

10,000

12,000

14,000

196019621964196619681970 1972197419761978198019821984 19861988199019921994199619982000 20022004

California

United States

Source: California Energy Commission

6

California IOU’s Investment in Energy Efficiency

Source: California Energy Commission, modified by Vine (2008)

$0

$100

$200

$300

$400

$500

$600

$700

$800

$900

$1,000

1976197819801982198419861988199019921994199619982000200220042006200820102012

Millions of $2002 per Year

Forecast

Profits decoupled from sales

Performance Incentives

Market Restructuring

Crisis

IRP2% of 2004

IOU Electric Revenues

Public Goods Charges

RRIM

EESPEnergyAction Plan

7

Annual Energy Savings from EE Programs and Standards

Source: Rosenfeld, California Energy Commission

0

5,000

10,000

15,000

20,000

25,000

30,000

35,000

40,000

45,000

19

75

19

76

19

77

19

78

19

79

19

80

19

81

19

82

19

83

19

84

19

85

19

86

19

87

19

88

19

89

19

90

19

91

19

92

19

93

19

94

19

95

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

GW

h/y

ear

Appliance Standards

Building Standards

Utility Efficiency Programs at a cost of

~1% of electric bill

~15% of Annual Electricity Use in California in 2003

8

Energy Action Plan

Since its enactment in 2003, the Loading Order has been integrated into the major CPUC decisions governing energy policy and procurement.

In the EAP, the Loading Order continued:

“Pursue all cost-effective energy efficiency, first.”

Energy resources are prioritized as follows:

1. Energy efficiency/demand response

2. Renewable generation, including renewable DG

3. Increased development of affordable and reliable conventional generation

4. Transmission expansion to support all of California’s energy goals

9

California: The Most Aggressive Energy Efficiency Program in the Nation

Energy Efficiency goals (2004-2013) 26,506 GWh/year 5,000 MW/year 444 Million therms/year

Eliminates need for 10 new power plants

Eliminates 9 million tons of CO2 emissions

(equal to 1.8 million cars)

$10 billion in net savings to consumers

10

Current Program Cycle

2006-08 ~$2 billion in funding for 3 years

$581M in 2006, $646M in 2007, and $742 M in 2008 (excludes EM&V budget)

Annual funding from utility procurement dollars and from the Public Goods Charge

Levelized cost of 3 cents/kWh and 21 cents/therm $2.7 billion in net savings to consumers over 3 years

$163 million EM&V budget for 3 years 8% of total portfolio funding

11

Shareholder Risk/ Reward Incentive Mechanism (RRIM)

Decision 07-09-043 (Sept. 20, 2007) Creates incentives to ensure that utility investors and

managers view energy efficiency as a core part of the utility’s regulated operations that can generate meaningful earnings for its shareholders

Protects ratepayers’ financing investment Ensure that program savings are real and verified

All calculations of the net benefits and kW, kWh and therm achievements are independently verified by the CPUC and their EM&V contractors, based on adopted EM&V protocols

Imposes penalties for substandard performance

12

Adopted Incentive Mechanism Earnings/ Penalty Curve

Reward(% of PEB)

(per unit below CPUC goal)Penalty

65%

85% 100% % of CPUC goals

ER = 9%

ER = 12%

Earnings capped at $450 million

0%

5¢/kWh, $25/kW, 45 ¢/therm below goals, or payback of negative net benefits (cost-effectiveness guarantee), whichever is greater.

Penalty capped at $450 million.

Earnings = ER x PEB

PEB= Performance Earnings BasisER= Earnings Rate (or Shared- Savings Rate)

13

Earnings Claim & Recovery Process

Two interim claims during each 3-year program cycle

Progress payments towards total expected earnings

One final true-up claim after the program cycle is completed

Hold back: 30% of the expected earnings in each interim claim to provide a margin for error in expected earnings

14

15

Scope of Strategic Plan

Includes everything the two state energy agencies (CPUC and CEC) are currently working on, and more

Incorporates: Market transformation Voluntary market actions – to become “Business as

Usual in California Collaboration -- Roles for Local Governments, other

State Agencies and Private Sector Players

4 Big Bold EE strategies as cornerstones of initial bold energy-savings/outcomes

16

• All new commercial construction in California will be zero net energy by 2030.

Commercial New

Construction

• All new residential construction in California will be zero net energy by 2020.

Residential New Construction

• Heating, Ventilation, and Air Conditioning (HVAC) industry will be reshaped

Residential / Small Commercial HVAC

BIG BOLD

Energy Efficiency Strategies

Low- Income Energy Efficiency • All eligible

low-income homes will be energy-efficient by 2020

4

17

Proposed Next Program Cycle

2009-11 ~$4 billion in funding for 3 years IOU program applications submitted and are

being reviewed by the CPUC Decision likely next year Bridge funding adopted by CPUC to fund some of the

2006-8 programs, but no new programs

Many key questions, including: How do (can?) the program applications support the

RRIM & EESP? Do cost-effectiveness indicators need to be adjusted to

meet RRIM & EESP needs: e.g. TRC and net-to-gross?

18

California’s GHG Emissions Goals

Reduce GHG emissions to 2000 levels by 2010

Reduce GHG emissions to 1990 levels by 2020

Reduce GHG emissions to 80% below 1990 levels by 2050

19 19

Magnitude of the ChallengeMagnitude of the Challenge

ARB Emissions Inventory

0

100

200

300

400

500

600

700

1990 2000 2004 2020 2050

Year

Million Metric Tons(CO

2 Equivalent)

1990 Emission Baseline

~173 MMT CO2e Reduction

80% Reduction ~341 MMT CO2e

20

CARB - Climate Change Scoping Plan

Climate Change Scoping Plan (Oct. 2008) Energy efficiency strategies will provide 16% of

the estimated total emission reduction (169 MMTCO2E) by 2020

Focus: increased incentives and more stringent building codes and appliance efficiency standards

Focus is on technological improvements, reflecting national and international studies

Bottom line: Incredible challenge & we need to do much more and urgently

21

2. Overview of Evaluation

Budgets for energy efficiency program implementation and evaluation have increased over time

California: $2 billion (2006-2008); 8% ($163M) on EM&V Shareholders incentives and risks: $450M $3.7 billion (2006-2008) proposed

Increased interest in evaluation results!!! But how does evaluation relate to the program planning

cycle? And how are evaluation findings used by program managers? How to evaluate energy efficiency programs?

22

Energy Efficiency Program Planning Cycle

Planning

Program Design

Program Implementation

Program Evaluation

23

Evaluation Uses by Program & Portfolio Managers

Evaluation results can be used for: Evaluating performance (“report card”)

Accountability and demonstration of success

Improving program design and implementation Improving engineering & behavioral

assumptions Prioritizing program and portfolio budgets Finalizing incentive payments Resource (strategic) planning

24

Six Types of Evaluation

Evaluation Category

Phase at which Implemented Assessment Level

Market and/or Program

Market and/or Program

Implementation phase (post-hoc)

Program

Program & Market

Program or Portfolio

5. Market effects evaluation

Pre-program planning phase (a priori)

Evaluation Type

6. Cost-Effectiveness evaluation

Program

Formative

3. Process evaluation

Outcomes

1. Market assessment (includes market characterization, baseline studies)

2. Potential or feasibility studies

Implementation phase (post-hoc) and/or post-

implementation (ex-post)

4. Impact evaluation

25

Impact Evaluation

Time

EnergyUse

In the AbsenceOf the Program

ActualImpact

Installation

25

26

Impact Data Needed to Collect

Monthly energy consumption

Metered or monitored energy usage

Load shape data (day, season, year)

Hours of operation for building or measures

Physical characteristics of the building and equipment (size and location)

Other physical variables: temperature, flow, weather

Building occupancy schedules and occupant data

27

Data Collection and Analysis Methods for Impact Evaluation

Engineering methods

Basic statistical billing analysis

Multivariate statistical analysis

End-use metering

Short-term monitoring

Integrative methods

28

3. Adjustments to Energy Savings: Net Energy Savings

Gross energy savings

Net energy savings [Additionality] Baselines (naturally occurring … energy use/ energy

savings/ market adoption, etc.) Free riders Program spillover

29

Net Energy Savings & Additionality

International Perspective: Joint Implementation and Clean Development Mechanism Emission reductions must be “additional to any that would

otherwise occur” [“additionality criteria”] [“net emissions”] Determining additionality requires a baseline Focus is on environmental additionality (GHG emissions)

US/Utility Regulatory Perspective: Similar to net energy savings [versus gross energy savings] Determining net savings requires a baseline Accounts for naturally occurring conservation and free riders

30

Baselines [Reference Case]

What would have happened in the absence of a particular project or

program? [counter-factual question] Need to account, if possible, for economic

growth, technological changes, prices, policy or regulatory shifts, population changes, market barriers and trends, etc.

Baselines must be credible and realistic to prevent/mitigate gaming

31

Free Riders #1

Activities are undertaken by participant(s) who would have conducted the same activities if there had been no project

Savings from free riders are not “additional” to what would otherwise have occurred

Therefore, subtract the free riders’ energy savings from gross savings to get net energy savings

32

Free Riders #2

Example in California: If NTG = .8, then Free Riders =.2 (i.e., 20% of energy

savings was due to energy savings from free riders)

Free rider methodology Self reports - measurement limitations: social desirability

bias, cognitive dissonance, and attribution Comparisons with non-program areas - measurement

limitations: data availability, self-selection bias, non-equivalent comparison groups

Net Energy Savings:

Gross Energy Savings

Net to Gross =(NTG)

33

UNFCCC/ CDMIndicators of Project Additionality

Regulatory Additionality Has the project been directly or indirectly mandated by law or

regulation?

Investment Additionality Would the project have occurred under the investing party’s

normal investment decision rules?

Technological Additionality Does the project involve technology or practices that go beyond

conventional practice in the corresponding industry or sector?

Barrier Removal Additionality Does the project remove or work to overcome any information,

institutional or other barriers that would persist in the reference case?

34

Program Spillover Participant spillover

Not normally looked at What program participants did “extra” as a result of participating in

program

Nonparticipant spillover Not normally looked at What program nonparticipants did as a result of hearing about the

program

Spillover methodologies Self reports - measurement limitations: social desirability bias,

cognitive dissonance, and attribution Comparisons with non-program areas - measurement limitations:

data availability, self-selection bias, non-equivalent comparison groups

Market effects (e.g., sales) - measurement limitations: key is data availability and quality

35

4. Cost-Effectiveness Metrics

If TRC >1, program is cost effective

If TRC < 1, program is not cost effective (but may still want to include as part of a portfolio of programs)

Savings are adjusted by NTG Free riders has been the sole adjustment factor so far

BENEFITS [Savings]: Utility Avoided Power Cost

(and small emissions adder)

Costs: Total Project Costs

(All program and participant costs)

Total Resource Cost =(TRC)

36

TRC Concerns

Is the right avoided cost being used? Should avoided cost be based on natural gas combined

cycle gas turbine (CCGT) or renewable energy (wind, solar) power plant?

Are NTG methodological issues too overwhelming? Should some costs be subtracted because people

invest in energy efficiency for non-energy reasons? Comfort, quiet, home value, improved air quality, health,

productivity, jobs, etc. Non-energy benefits (NEBs) are not included

37

Revise TRC

BENEFITS [Savings]: Utility Avoided Power Cost(New Avoided Cost Plus

NEBs)

Costs: Total Project Costs

(All program and 20% participant costs)

Total Resource Cost =(TRC)

38

5. Why Do We Care about TRC & NTG? #1

Regulatory objectives & concerns Are we giving away money wisely? (Efficiency argument) Are we giving away money to people who do not need it?

(Equity argument)

In CA: focus has been on resource acquisition (RA) programs - due to RRIM

Not on market transformation (MT) programs - EESP focus

Not on market (program-centric) - EESP focus

In CA: focus is on free riders - due to RRIM Not on spillover (participant and non-participant) - EESP

focus Not on market effects - EESP focus

39

Why Do We Care about TRC & NTG? #2

Focus varies by: Regional objectives (RA [RRIM] vs MT [EESP]) Program maturity (new versus old programs) Concern over reducing greenhouse gases

Gross reductions Net reductions (cap and trade; emissions trading) -

Additionality

Policy issue: Does current use of TRC and NTG inhibit new,

market transforming energy efficiency interventions? Time to look at market effects? Non-Energy Impacts?

40

IEPEC Conference

To Learn More about Evaluation and These Exciting Topics, Go to:

International Energy Program Evaluation Conference (IEPEC)

August 12-14, 2009 Portland, OR http://www.iepec.org

41

Before Carl Speaks

42

American Evaluation Association:Guiding Principles for Evaluators

Systematic Inquiry Competence Integrity / Honesty Respect for People Responsibilities for General and Public

Welfare

http://www.eval.org/Publications/GuidingPrinciples.asp