34
Presentation by Steve Schiller Senior Advisor - Lawrence Berkeley National Laboratory Electricity Markets and Policy Group Principal - Schiller Consulting Inc. Topics: Guide Background and Objectives Topics Covered in Guide EM&V Objectives and Basic Issues Discussed in Guide Determining Avoided Emissions SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 0 Energy-Efficiency Program Impact Evaluation Guide An introduction to and summary of the practices, planning, and associated issues of documenting energy savings, demand savings, avoided emissions, and other non-energy benefits resulting from end-use energy- efficiency programs. A RESOURCE OF THE STATE AND LOCAL ENERGY EFFICIENCY ACTION NETWORK

Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

  • Upload
    others

  • View
    2

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

Presentation by Steve Schiller Senior Advisor - Lawrence Berkeley National Laboratory

Electricity Markets and Policy Group

Principal - Schiller Consulting Inc

Topics

Guide Background and Objectives

Topics Covered in Guide

EMampV Objectives and Basic Issues Discussed

in Guide

Determining Avoided Emissions

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 0 i

Energy-Efficiency Program Impact Evaluation Guide An introduction to and summary of

the practices planning and associated

issues of documenting energy savings

demand savings avoided emissions

and other non-energy benefits

resulting from end-use energy-

efficiency programs

A RESOURCE OF THE STATE AND LOCAL ENERGY EFFICIENCY ACTION NETWORK

Why Evaluate bull Document impacts Document the energy savings of

projects and programs in order to determine how well they have met their goals eg has there been a good use of the invested money and time Provide PROOF of the effectiveness of energy management

bull Resource Planning To support energy resource planning by understanding the historical and future resource contributions of energy efficiency as compared to other energy resources Provide data to support efficiency as a reliable resource

bull Understand why the effects occurred Identify ways to improve current and future projects and programs as well as select future projects ldquoYou canrsquot manage what you donrsquot measurerdquo and ldquoThings that are measured tend to improverdquo

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 1

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 2

Update to the 2007 NAPEE Impact Evaluation Guide

ldquoNAPEE Guiderdquo bull Widely used and cited both

domestically and internationally

bull Used for training and as a resource by many federal agencies and several state agencies and utilities

bull Cited in state and utility EMampV rules and frameworks

bull Buthellipneeded updating

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

3

The New Guide

bull Describes common terminology structures and approaches used for determining (evaluating)

ndash energy and demand savings

ndash avoided emissions

ndash other non-energy benefits

bull Does not recommend specific approaches - it provides

ndash context

ndash planning guidance

ndash discussion of issues

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

4 i

Energy-Efficiency Program Impact Evaluation Guide An introduction to and summary of

the practices planning and associated

issues of documenting energy savings

demand savings avoided emissions

and other non-energy benefits

resulting from end-use energy-

efficiency programs

A RESOURCE OF THE STATE AND LOCAL ENERGY EFFICIENCY ACTION NETWORK

Intended Audience

bull Primary audience ndash energy agencies and regulators

ndash public and private energy efficiency portfolio administrators such as utilities nonprofit organizations and government agencies

ndash program implementers bull And

ndash policymakers seeking general information

ndash evaluation practitioners - introductions to and summaries of evaluation topics that can be useful for explaining concepts and standard practices

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

5

Outcomes

bull Standard evaluation terminologystructures and best practices - support the adoption continuation and expansion of effective efficiency actions

bull For those just starting or ramping up their efficiency and evaluation activities ndash reduce costs for starting up an evaluation effort and moving ldquoup the learning curverdquo

bull Use of common approaches and terminology can support comparisons across different jurisdictions and facilitate ldquocross-borderrdquo energy efficiency andor greenhouse gas and other air emissions mitigation programs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

6

Expectations From Using the Guide ndash State Energy Officials

After reading this guide the reader from a State Energy Office will be able to

bull Define the basic objectives structure and evaluation approaches that can be used to plan and conduct impact evaluations of programs

bull Support and provide input for an energy efficiency evaluation framework (general guidance documents) and review impact evaluation plans and reports

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

7

The Guide supports lsquocapacity buildingrsquo and also provides input for some

forthcoming EMampV Topics

bull What is the future of EMampV for EE ndash more rigor or less rigor more consistency more deemed savings

bull How will energy savings and emissions reductions be documented for SIPsclimate programs

bull What matters net savings or gross savings just energy savings or NEBs

bull Is there a future for top-down evaluation macro data analyses

bull What is the best role of EMampV and evaluators

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

8

But First A Commercial Interruption

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 9

Jointly managed by the DOE Office of Electricity Delivery and Energy Reliability and the Office of Energy Efficiency and Renewable Energy

UNIFORM METHODS PROJECT

bull Addresses seven common residential and commercial efficiency measures

bull Step-by-step calculations for determining gross first-year savings using MampV approach

bull Also some cross cutting chapters on some EMampV topics (sampling metering persistence peak savings)

bull Adoption is voluntary bull httpumppnnlgov DOE Goals

1113089 Strengthen credibility EE savings calculations 1113089 Provide clear accessible step-by-step protocols 1113089 Support consistency and transparency 1113089 Reduce costs of EMampV 1113089 Allow for comparison of savings

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 11

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

12

Part Chapters Intended Audience

Part 1

Executive Summary

Readers interested in a brief summary and

introduction to impact evaluation

Part 2

Chapter 1 Introduction

Chapter 2 Energy Efficiency Program Evaluation

Overview

Chapter 3 Impact Evaluation Basics

Readers who want an overview of evaluation and

the key aspects of impact evaluation

Part 3

Chapter 4 Calculating Energy Savings

Chapter 5 Determining Net Energy Savings

Chapter 6 Calculating Avoided Air Emissions

Readers who want additional detail on impact

evaluation approaches ndash deemed savings MampV

large-scale consumption data analysis

Part 4

Chapter 7 Impact Evaluation Considerations

Chapter 8 Impact Evaluation Planning

Program implementers evaluators and

managersregulators of evaluations looking for

guidance on key evaluation issues and planning of

evaluations as well as readers with a background in

evaluation may want to go directly to these chapters

Part 5

Appendix A Glossary

Appendix B Other Evaluation Categories and

Approaches ndash Market Process Cost-Effectiveness

Appendix C Resources

References

Readers interested in standard energy efficiency

evaluation definitions and reference materials as

well as summaries of process market evaluations

cost-effectiveness analyses and top-down

evaluation

EXAMPLE Chapter 7 Impact Evaluation Considerations

bull First portion covers issues that often arise in impact evaluations ndash determining baselines

ndash determining demand savings

ndash calculating persistence of savings

ndash addressing uncertainty of savings estimates

ndash setting evaluation budgets

ndash establishing evaluation principles and ethics

bull Second portion provides brief overviews of ndash program feedback through evaluation

ndash resource planning through evaluation

ndash calculating non-energy benefits

ndash evaluating some relatively unique program types (residential behaviors education and training market transformation codes and standards demand response and GHG mitigation)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

13

Glossary ndash a combined effort with several groups to

create more consistency in how we describe EMampV

bull Accuracy A concept that refers to the relationship between the true value of a variable and an estimate of

the value The term can also be used in reference to a model or a set of measured data or to describe a measuring instrumentrsquos capability

bull helliphelliphelliphelliphellip

bull Incremental Annual Savings The difference between the amount of energy savings acquired or planned

to be acquired as a result of energy efficiency activities in one year and the amount of energy savings acquired or planned to be acquired as a result of the energy efficiency activities in the prior year

bull Independent Variables The explanatory factors (eg weather or occupancy) in a regression model that

are assumed to affect the variable under study (eg energy use)

bull Indirect Emissions Emissions that are a consequence of activities that take place within the organizational

boundaries of an entity but occur at sources owned or controlled by another entity For example emissions of electricity used by a manufacturing entity that occur at a power plant represent the manufacturerrsquos indirect emissions Indirect emissions are typically the source of avoided emissions for electric energy efficiency measures

bull helliphelliphelliphellip

bull Workforce Education and Training Programs Programs primarily intended for building maintenance engineers

HVAC contractors engineers architects maintenance personnel and others These programs provide information about energy efficiency concepts recommended energy-efficient technologies or behaviors andor programs that offer energy efficiency or load-reduction

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

14

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16

RDampD

bull Research

bull Development

bull Demonstration

Deployment

Voluntary implementation of projects and programs ndash outreach education and subsidies For example

bull Incenting early actors

bull Incenting consumers distributers and manufacturers

bull Mass market and individual market strategies

Transformed Markets

Standard practice

or

Codes and standards

There is a continuum of efficiency actions

Our goal is to greatly accelerate and

ldquofeedrdquo (widen) this continuum of actions

EMampV is a tool to support these actions

Evaluation Types

Evaluation Type Description Example Uses

Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)

Determines the amount of energy and demand saved

Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives

Identifies how program designs and processes can be improved

Market Effects Evaluation

Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program

Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program

Cost-Effectiveness Evaluation

Quantifies the costs of program implementation and compares them with program benefits

Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

17

Savings Cannot Be Measured - They Are Estimated

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18

Graph of Energy Consumption Before During And After Project Is Installed

Determining Energy Savings

bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as

used by the Northwest RTF

bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group

bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program

bull All or some of the above

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

19

Planning Implementing and Evaluating Efficiency Programs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

20

Evaluation is Integral to Planning-Implementation-Evaluation Process

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21

When to Evaluate

bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle

bull In timely manner with feedback for bull Ongoing program improvement

bull Supporting portfolio assessments

bull Support the planning of future portfolio cycles load forecastsenergy resource plans

bull Can also be used to inform future evaluations in particular through updating deemed savings values

Structure for Defining Evaluation Activities

bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input

bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle

bull Evaluation Research Plans ndash Created for the major EMampV activities

bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22

The Big Issues of EMampV

bull Fundamental issue of EMampV

bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty

bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

23

bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual

bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings

How good is good enough As compared to what

EMampV is About Risk Management

How Good is Good Enough amp As Compared to What

bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources

bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example

ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or

ndash Expending too many resources on ineffective actions or the opposite or

ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24

Side Note Energy Savings vs Energy Avoidance ndash

Consumer vs Regulator Perspectives

bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments

bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments

bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26

Non-Energy Benefits (NEBs)

NEB categories

bull NEBs can be categorized as those accruing to

ndash utilities (energy providers)

ndash society as a whole (such as avoided emissions)

ndash individual participants

bull Can be positive or negative (NEIs)

bull Might exceed energy benefits

Documenting NEBs

bull Measurement of benefits

bull Modeling

bull Surveys

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

27

Why Document Avoided Emissions

bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit

bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by

monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test

ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

28

Calculating Avoided Emissions

Conceptually avoided emissions are calculated using net energy savings and one of two different approaches

bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)

ndash Several sources of emission factors as well as approaches for calculating the factors

ndash Avoided emissions = (net energy savings) x (emission factor)

bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a

grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs

ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

29

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 2: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

Why Evaluate bull Document impacts Document the energy savings of

projects and programs in order to determine how well they have met their goals eg has there been a good use of the invested money and time Provide PROOF of the effectiveness of energy management

bull Resource Planning To support energy resource planning by understanding the historical and future resource contributions of energy efficiency as compared to other energy resources Provide data to support efficiency as a reliable resource

bull Understand why the effects occurred Identify ways to improve current and future projects and programs as well as select future projects ldquoYou canrsquot manage what you donrsquot measurerdquo and ldquoThings that are measured tend to improverdquo

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 1

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 2

Update to the 2007 NAPEE Impact Evaluation Guide

ldquoNAPEE Guiderdquo bull Widely used and cited both

domestically and internationally

bull Used for training and as a resource by many federal agencies and several state agencies and utilities

bull Cited in state and utility EMampV rules and frameworks

bull Buthellipneeded updating

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

3

The New Guide

bull Describes common terminology structures and approaches used for determining (evaluating)

ndash energy and demand savings

ndash avoided emissions

ndash other non-energy benefits

bull Does not recommend specific approaches - it provides

ndash context

ndash planning guidance

ndash discussion of issues

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

4 i

Energy-Efficiency Program Impact Evaluation Guide An introduction to and summary of

the practices planning and associated

issues of documenting energy savings

demand savings avoided emissions

and other non-energy benefits

resulting from end-use energy-

efficiency programs

A RESOURCE OF THE STATE AND LOCAL ENERGY EFFICIENCY ACTION NETWORK

Intended Audience

bull Primary audience ndash energy agencies and regulators

ndash public and private energy efficiency portfolio administrators such as utilities nonprofit organizations and government agencies

ndash program implementers bull And

ndash policymakers seeking general information

ndash evaluation practitioners - introductions to and summaries of evaluation topics that can be useful for explaining concepts and standard practices

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

5

Outcomes

bull Standard evaluation terminologystructures and best practices - support the adoption continuation and expansion of effective efficiency actions

bull For those just starting or ramping up their efficiency and evaluation activities ndash reduce costs for starting up an evaluation effort and moving ldquoup the learning curverdquo

bull Use of common approaches and terminology can support comparisons across different jurisdictions and facilitate ldquocross-borderrdquo energy efficiency andor greenhouse gas and other air emissions mitigation programs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

6

Expectations From Using the Guide ndash State Energy Officials

After reading this guide the reader from a State Energy Office will be able to

bull Define the basic objectives structure and evaluation approaches that can be used to plan and conduct impact evaluations of programs

bull Support and provide input for an energy efficiency evaluation framework (general guidance documents) and review impact evaluation plans and reports

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

7

The Guide supports lsquocapacity buildingrsquo and also provides input for some

forthcoming EMampV Topics

bull What is the future of EMampV for EE ndash more rigor or less rigor more consistency more deemed savings

bull How will energy savings and emissions reductions be documented for SIPsclimate programs

bull What matters net savings or gross savings just energy savings or NEBs

bull Is there a future for top-down evaluation macro data analyses

bull What is the best role of EMampV and evaluators

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

8

But First A Commercial Interruption

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 9

Jointly managed by the DOE Office of Electricity Delivery and Energy Reliability and the Office of Energy Efficiency and Renewable Energy

UNIFORM METHODS PROJECT

bull Addresses seven common residential and commercial efficiency measures

bull Step-by-step calculations for determining gross first-year savings using MampV approach

bull Also some cross cutting chapters on some EMampV topics (sampling metering persistence peak savings)

bull Adoption is voluntary bull httpumppnnlgov DOE Goals

1113089 Strengthen credibility EE savings calculations 1113089 Provide clear accessible step-by-step protocols 1113089 Support consistency and transparency 1113089 Reduce costs of EMampV 1113089 Allow for comparison of savings

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 11

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

12

Part Chapters Intended Audience

Part 1

Executive Summary

Readers interested in a brief summary and

introduction to impact evaluation

Part 2

Chapter 1 Introduction

Chapter 2 Energy Efficiency Program Evaluation

Overview

Chapter 3 Impact Evaluation Basics

Readers who want an overview of evaluation and

the key aspects of impact evaluation

Part 3

Chapter 4 Calculating Energy Savings

Chapter 5 Determining Net Energy Savings

Chapter 6 Calculating Avoided Air Emissions

Readers who want additional detail on impact

evaluation approaches ndash deemed savings MampV

large-scale consumption data analysis

Part 4

Chapter 7 Impact Evaluation Considerations

Chapter 8 Impact Evaluation Planning

Program implementers evaluators and

managersregulators of evaluations looking for

guidance on key evaluation issues and planning of

evaluations as well as readers with a background in

evaluation may want to go directly to these chapters

Part 5

Appendix A Glossary

Appendix B Other Evaluation Categories and

Approaches ndash Market Process Cost-Effectiveness

Appendix C Resources

References

Readers interested in standard energy efficiency

evaluation definitions and reference materials as

well as summaries of process market evaluations

cost-effectiveness analyses and top-down

evaluation

EXAMPLE Chapter 7 Impact Evaluation Considerations

bull First portion covers issues that often arise in impact evaluations ndash determining baselines

ndash determining demand savings

ndash calculating persistence of savings

ndash addressing uncertainty of savings estimates

ndash setting evaluation budgets

ndash establishing evaluation principles and ethics

bull Second portion provides brief overviews of ndash program feedback through evaluation

ndash resource planning through evaluation

ndash calculating non-energy benefits

ndash evaluating some relatively unique program types (residential behaviors education and training market transformation codes and standards demand response and GHG mitigation)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

13

Glossary ndash a combined effort with several groups to

create more consistency in how we describe EMampV

bull Accuracy A concept that refers to the relationship between the true value of a variable and an estimate of

the value The term can also be used in reference to a model or a set of measured data or to describe a measuring instrumentrsquos capability

bull helliphelliphelliphelliphellip

bull Incremental Annual Savings The difference between the amount of energy savings acquired or planned

to be acquired as a result of energy efficiency activities in one year and the amount of energy savings acquired or planned to be acquired as a result of the energy efficiency activities in the prior year

bull Independent Variables The explanatory factors (eg weather or occupancy) in a regression model that

are assumed to affect the variable under study (eg energy use)

bull Indirect Emissions Emissions that are a consequence of activities that take place within the organizational

boundaries of an entity but occur at sources owned or controlled by another entity For example emissions of electricity used by a manufacturing entity that occur at a power plant represent the manufacturerrsquos indirect emissions Indirect emissions are typically the source of avoided emissions for electric energy efficiency measures

bull helliphelliphelliphellip

bull Workforce Education and Training Programs Programs primarily intended for building maintenance engineers

HVAC contractors engineers architects maintenance personnel and others These programs provide information about energy efficiency concepts recommended energy-efficient technologies or behaviors andor programs that offer energy efficiency or load-reduction

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

14

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16

RDampD

bull Research

bull Development

bull Demonstration

Deployment

Voluntary implementation of projects and programs ndash outreach education and subsidies For example

bull Incenting early actors

bull Incenting consumers distributers and manufacturers

bull Mass market and individual market strategies

Transformed Markets

Standard practice

or

Codes and standards

There is a continuum of efficiency actions

Our goal is to greatly accelerate and

ldquofeedrdquo (widen) this continuum of actions

EMampV is a tool to support these actions

Evaluation Types

Evaluation Type Description Example Uses

Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)

Determines the amount of energy and demand saved

Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives

Identifies how program designs and processes can be improved

Market Effects Evaluation

Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program

Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program

Cost-Effectiveness Evaluation

Quantifies the costs of program implementation and compares them with program benefits

Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

17

Savings Cannot Be Measured - They Are Estimated

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18

Graph of Energy Consumption Before During And After Project Is Installed

Determining Energy Savings

bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as

used by the Northwest RTF

bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group

bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program

bull All or some of the above

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

19

Planning Implementing and Evaluating Efficiency Programs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

20

Evaluation is Integral to Planning-Implementation-Evaluation Process

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21

When to Evaluate

bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle

bull In timely manner with feedback for bull Ongoing program improvement

bull Supporting portfolio assessments

bull Support the planning of future portfolio cycles load forecastsenergy resource plans

bull Can also be used to inform future evaluations in particular through updating deemed savings values

Structure for Defining Evaluation Activities

bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input

bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle

bull Evaluation Research Plans ndash Created for the major EMampV activities

bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22

The Big Issues of EMampV

bull Fundamental issue of EMampV

bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty

bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

23

bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual

bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings

How good is good enough As compared to what

EMampV is About Risk Management

How Good is Good Enough amp As Compared to What

bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources

bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example

ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or

ndash Expending too many resources on ineffective actions or the opposite or

ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24

Side Note Energy Savings vs Energy Avoidance ndash

Consumer vs Regulator Perspectives

bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments

bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments

bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26

Non-Energy Benefits (NEBs)

NEB categories

bull NEBs can be categorized as those accruing to

ndash utilities (energy providers)

ndash society as a whole (such as avoided emissions)

ndash individual participants

bull Can be positive or negative (NEIs)

bull Might exceed energy benefits

Documenting NEBs

bull Measurement of benefits

bull Modeling

bull Surveys

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

27

Why Document Avoided Emissions

bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit

bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by

monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test

ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

28

Calculating Avoided Emissions

Conceptually avoided emissions are calculated using net energy savings and one of two different approaches

bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)

ndash Several sources of emission factors as well as approaches for calculating the factors

ndash Avoided emissions = (net energy savings) x (emission factor)

bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a

grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs

ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

29

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 3: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 2

Update to the 2007 NAPEE Impact Evaluation Guide

ldquoNAPEE Guiderdquo bull Widely used and cited both

domestically and internationally

bull Used for training and as a resource by many federal agencies and several state agencies and utilities

bull Cited in state and utility EMampV rules and frameworks

bull Buthellipneeded updating

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

3

The New Guide

bull Describes common terminology structures and approaches used for determining (evaluating)

ndash energy and demand savings

ndash avoided emissions

ndash other non-energy benefits

bull Does not recommend specific approaches - it provides

ndash context

ndash planning guidance

ndash discussion of issues

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

4 i

Energy-Efficiency Program Impact Evaluation Guide An introduction to and summary of

the practices planning and associated

issues of documenting energy savings

demand savings avoided emissions

and other non-energy benefits

resulting from end-use energy-

efficiency programs

A RESOURCE OF THE STATE AND LOCAL ENERGY EFFICIENCY ACTION NETWORK

Intended Audience

bull Primary audience ndash energy agencies and regulators

ndash public and private energy efficiency portfolio administrators such as utilities nonprofit organizations and government agencies

ndash program implementers bull And

ndash policymakers seeking general information

ndash evaluation practitioners - introductions to and summaries of evaluation topics that can be useful for explaining concepts and standard practices

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

5

Outcomes

bull Standard evaluation terminologystructures and best practices - support the adoption continuation and expansion of effective efficiency actions

bull For those just starting or ramping up their efficiency and evaluation activities ndash reduce costs for starting up an evaluation effort and moving ldquoup the learning curverdquo

bull Use of common approaches and terminology can support comparisons across different jurisdictions and facilitate ldquocross-borderrdquo energy efficiency andor greenhouse gas and other air emissions mitigation programs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

6

Expectations From Using the Guide ndash State Energy Officials

After reading this guide the reader from a State Energy Office will be able to

bull Define the basic objectives structure and evaluation approaches that can be used to plan and conduct impact evaluations of programs

bull Support and provide input for an energy efficiency evaluation framework (general guidance documents) and review impact evaluation plans and reports

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

7

The Guide supports lsquocapacity buildingrsquo and also provides input for some

forthcoming EMampV Topics

bull What is the future of EMampV for EE ndash more rigor or less rigor more consistency more deemed savings

bull How will energy savings and emissions reductions be documented for SIPsclimate programs

bull What matters net savings or gross savings just energy savings or NEBs

bull Is there a future for top-down evaluation macro data analyses

bull What is the best role of EMampV and evaluators

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

8

But First A Commercial Interruption

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 9

Jointly managed by the DOE Office of Electricity Delivery and Energy Reliability and the Office of Energy Efficiency and Renewable Energy

UNIFORM METHODS PROJECT

bull Addresses seven common residential and commercial efficiency measures

bull Step-by-step calculations for determining gross first-year savings using MampV approach

bull Also some cross cutting chapters on some EMampV topics (sampling metering persistence peak savings)

bull Adoption is voluntary bull httpumppnnlgov DOE Goals

1113089 Strengthen credibility EE savings calculations 1113089 Provide clear accessible step-by-step protocols 1113089 Support consistency and transparency 1113089 Reduce costs of EMampV 1113089 Allow for comparison of savings

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 11

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

12

Part Chapters Intended Audience

Part 1

Executive Summary

Readers interested in a brief summary and

introduction to impact evaluation

Part 2

Chapter 1 Introduction

Chapter 2 Energy Efficiency Program Evaluation

Overview

Chapter 3 Impact Evaluation Basics

Readers who want an overview of evaluation and

the key aspects of impact evaluation

Part 3

Chapter 4 Calculating Energy Savings

Chapter 5 Determining Net Energy Savings

Chapter 6 Calculating Avoided Air Emissions

Readers who want additional detail on impact

evaluation approaches ndash deemed savings MampV

large-scale consumption data analysis

Part 4

Chapter 7 Impact Evaluation Considerations

Chapter 8 Impact Evaluation Planning

Program implementers evaluators and

managersregulators of evaluations looking for

guidance on key evaluation issues and planning of

evaluations as well as readers with a background in

evaluation may want to go directly to these chapters

Part 5

Appendix A Glossary

Appendix B Other Evaluation Categories and

Approaches ndash Market Process Cost-Effectiveness

Appendix C Resources

References

Readers interested in standard energy efficiency

evaluation definitions and reference materials as

well as summaries of process market evaluations

cost-effectiveness analyses and top-down

evaluation

EXAMPLE Chapter 7 Impact Evaluation Considerations

bull First portion covers issues that often arise in impact evaluations ndash determining baselines

ndash determining demand savings

ndash calculating persistence of savings

ndash addressing uncertainty of savings estimates

ndash setting evaluation budgets

ndash establishing evaluation principles and ethics

bull Second portion provides brief overviews of ndash program feedback through evaluation

ndash resource planning through evaluation

ndash calculating non-energy benefits

ndash evaluating some relatively unique program types (residential behaviors education and training market transformation codes and standards demand response and GHG mitigation)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

13

Glossary ndash a combined effort with several groups to

create more consistency in how we describe EMampV

bull Accuracy A concept that refers to the relationship between the true value of a variable and an estimate of

the value The term can also be used in reference to a model or a set of measured data or to describe a measuring instrumentrsquos capability

bull helliphelliphelliphelliphellip

bull Incremental Annual Savings The difference between the amount of energy savings acquired or planned

to be acquired as a result of energy efficiency activities in one year and the amount of energy savings acquired or planned to be acquired as a result of the energy efficiency activities in the prior year

bull Independent Variables The explanatory factors (eg weather or occupancy) in a regression model that

are assumed to affect the variable under study (eg energy use)

bull Indirect Emissions Emissions that are a consequence of activities that take place within the organizational

boundaries of an entity but occur at sources owned or controlled by another entity For example emissions of electricity used by a manufacturing entity that occur at a power plant represent the manufacturerrsquos indirect emissions Indirect emissions are typically the source of avoided emissions for electric energy efficiency measures

bull helliphelliphelliphellip

bull Workforce Education and Training Programs Programs primarily intended for building maintenance engineers

HVAC contractors engineers architects maintenance personnel and others These programs provide information about energy efficiency concepts recommended energy-efficient technologies or behaviors andor programs that offer energy efficiency or load-reduction

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

14

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16

RDampD

bull Research

bull Development

bull Demonstration

Deployment

Voluntary implementation of projects and programs ndash outreach education and subsidies For example

bull Incenting early actors

bull Incenting consumers distributers and manufacturers

bull Mass market and individual market strategies

Transformed Markets

Standard practice

or

Codes and standards

There is a continuum of efficiency actions

Our goal is to greatly accelerate and

ldquofeedrdquo (widen) this continuum of actions

EMampV is a tool to support these actions

Evaluation Types

Evaluation Type Description Example Uses

Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)

Determines the amount of energy and demand saved

Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives

Identifies how program designs and processes can be improved

Market Effects Evaluation

Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program

Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program

Cost-Effectiveness Evaluation

Quantifies the costs of program implementation and compares them with program benefits

Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

17

Savings Cannot Be Measured - They Are Estimated

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18

Graph of Energy Consumption Before During And After Project Is Installed

Determining Energy Savings

bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as

used by the Northwest RTF

bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group

bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program

bull All or some of the above

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

19

Planning Implementing and Evaluating Efficiency Programs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

20

Evaluation is Integral to Planning-Implementation-Evaluation Process

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21

When to Evaluate

bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle

bull In timely manner with feedback for bull Ongoing program improvement

bull Supporting portfolio assessments

bull Support the planning of future portfolio cycles load forecastsenergy resource plans

bull Can also be used to inform future evaluations in particular through updating deemed savings values

Structure for Defining Evaluation Activities

bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input

bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle

bull Evaluation Research Plans ndash Created for the major EMampV activities

bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22

The Big Issues of EMampV

bull Fundamental issue of EMampV

bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty

bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

23

bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual

bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings

How good is good enough As compared to what

EMampV is About Risk Management

How Good is Good Enough amp As Compared to What

bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources

bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example

ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or

ndash Expending too many resources on ineffective actions or the opposite or

ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24

Side Note Energy Savings vs Energy Avoidance ndash

Consumer vs Regulator Perspectives

bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments

bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments

bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26

Non-Energy Benefits (NEBs)

NEB categories

bull NEBs can be categorized as those accruing to

ndash utilities (energy providers)

ndash society as a whole (such as avoided emissions)

ndash individual participants

bull Can be positive or negative (NEIs)

bull Might exceed energy benefits

Documenting NEBs

bull Measurement of benefits

bull Modeling

bull Surveys

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

27

Why Document Avoided Emissions

bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit

bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by

monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test

ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

28

Calculating Avoided Emissions

Conceptually avoided emissions are calculated using net energy savings and one of two different approaches

bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)

ndash Several sources of emission factors as well as approaches for calculating the factors

ndash Avoided emissions = (net energy savings) x (emission factor)

bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a

grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs

ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

29

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 4: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

Update to the 2007 NAPEE Impact Evaluation Guide

ldquoNAPEE Guiderdquo bull Widely used and cited both

domestically and internationally

bull Used for training and as a resource by many federal agencies and several state agencies and utilities

bull Cited in state and utility EMampV rules and frameworks

bull Buthellipneeded updating

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

3

The New Guide

bull Describes common terminology structures and approaches used for determining (evaluating)

ndash energy and demand savings

ndash avoided emissions

ndash other non-energy benefits

bull Does not recommend specific approaches - it provides

ndash context

ndash planning guidance

ndash discussion of issues

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

4 i

Energy-Efficiency Program Impact Evaluation Guide An introduction to and summary of

the practices planning and associated

issues of documenting energy savings

demand savings avoided emissions

and other non-energy benefits

resulting from end-use energy-

efficiency programs

A RESOURCE OF THE STATE AND LOCAL ENERGY EFFICIENCY ACTION NETWORK

Intended Audience

bull Primary audience ndash energy agencies and regulators

ndash public and private energy efficiency portfolio administrators such as utilities nonprofit organizations and government agencies

ndash program implementers bull And

ndash policymakers seeking general information

ndash evaluation practitioners - introductions to and summaries of evaluation topics that can be useful for explaining concepts and standard practices

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

5

Outcomes

bull Standard evaluation terminologystructures and best practices - support the adoption continuation and expansion of effective efficiency actions

bull For those just starting or ramping up their efficiency and evaluation activities ndash reduce costs for starting up an evaluation effort and moving ldquoup the learning curverdquo

bull Use of common approaches and terminology can support comparisons across different jurisdictions and facilitate ldquocross-borderrdquo energy efficiency andor greenhouse gas and other air emissions mitigation programs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

6

Expectations From Using the Guide ndash State Energy Officials

After reading this guide the reader from a State Energy Office will be able to

bull Define the basic objectives structure and evaluation approaches that can be used to plan and conduct impact evaluations of programs

bull Support and provide input for an energy efficiency evaluation framework (general guidance documents) and review impact evaluation plans and reports

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

7

The Guide supports lsquocapacity buildingrsquo and also provides input for some

forthcoming EMampV Topics

bull What is the future of EMampV for EE ndash more rigor or less rigor more consistency more deemed savings

bull How will energy savings and emissions reductions be documented for SIPsclimate programs

bull What matters net savings or gross savings just energy savings or NEBs

bull Is there a future for top-down evaluation macro data analyses

bull What is the best role of EMampV and evaluators

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

8

But First A Commercial Interruption

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 9

Jointly managed by the DOE Office of Electricity Delivery and Energy Reliability and the Office of Energy Efficiency and Renewable Energy

UNIFORM METHODS PROJECT

bull Addresses seven common residential and commercial efficiency measures

bull Step-by-step calculations for determining gross first-year savings using MampV approach

bull Also some cross cutting chapters on some EMampV topics (sampling metering persistence peak savings)

bull Adoption is voluntary bull httpumppnnlgov DOE Goals

1113089 Strengthen credibility EE savings calculations 1113089 Provide clear accessible step-by-step protocols 1113089 Support consistency and transparency 1113089 Reduce costs of EMampV 1113089 Allow for comparison of savings

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 11

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

12

Part Chapters Intended Audience

Part 1

Executive Summary

Readers interested in a brief summary and

introduction to impact evaluation

Part 2

Chapter 1 Introduction

Chapter 2 Energy Efficiency Program Evaluation

Overview

Chapter 3 Impact Evaluation Basics

Readers who want an overview of evaluation and

the key aspects of impact evaluation

Part 3

Chapter 4 Calculating Energy Savings

Chapter 5 Determining Net Energy Savings

Chapter 6 Calculating Avoided Air Emissions

Readers who want additional detail on impact

evaluation approaches ndash deemed savings MampV

large-scale consumption data analysis

Part 4

Chapter 7 Impact Evaluation Considerations

Chapter 8 Impact Evaluation Planning

Program implementers evaluators and

managersregulators of evaluations looking for

guidance on key evaluation issues and planning of

evaluations as well as readers with a background in

evaluation may want to go directly to these chapters

Part 5

Appendix A Glossary

Appendix B Other Evaluation Categories and

Approaches ndash Market Process Cost-Effectiveness

Appendix C Resources

References

Readers interested in standard energy efficiency

evaluation definitions and reference materials as

well as summaries of process market evaluations

cost-effectiveness analyses and top-down

evaluation

EXAMPLE Chapter 7 Impact Evaluation Considerations

bull First portion covers issues that often arise in impact evaluations ndash determining baselines

ndash determining demand savings

ndash calculating persistence of savings

ndash addressing uncertainty of savings estimates

ndash setting evaluation budgets

ndash establishing evaluation principles and ethics

bull Second portion provides brief overviews of ndash program feedback through evaluation

ndash resource planning through evaluation

ndash calculating non-energy benefits

ndash evaluating some relatively unique program types (residential behaviors education and training market transformation codes and standards demand response and GHG mitigation)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

13

Glossary ndash a combined effort with several groups to

create more consistency in how we describe EMampV

bull Accuracy A concept that refers to the relationship between the true value of a variable and an estimate of

the value The term can also be used in reference to a model or a set of measured data or to describe a measuring instrumentrsquos capability

bull helliphelliphelliphelliphellip

bull Incremental Annual Savings The difference between the amount of energy savings acquired or planned

to be acquired as a result of energy efficiency activities in one year and the amount of energy savings acquired or planned to be acquired as a result of the energy efficiency activities in the prior year

bull Independent Variables The explanatory factors (eg weather or occupancy) in a regression model that

are assumed to affect the variable under study (eg energy use)

bull Indirect Emissions Emissions that are a consequence of activities that take place within the organizational

boundaries of an entity but occur at sources owned or controlled by another entity For example emissions of electricity used by a manufacturing entity that occur at a power plant represent the manufacturerrsquos indirect emissions Indirect emissions are typically the source of avoided emissions for electric energy efficiency measures

bull helliphelliphelliphellip

bull Workforce Education and Training Programs Programs primarily intended for building maintenance engineers

HVAC contractors engineers architects maintenance personnel and others These programs provide information about energy efficiency concepts recommended energy-efficient technologies or behaviors andor programs that offer energy efficiency or load-reduction

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

14

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16

RDampD

bull Research

bull Development

bull Demonstration

Deployment

Voluntary implementation of projects and programs ndash outreach education and subsidies For example

bull Incenting early actors

bull Incenting consumers distributers and manufacturers

bull Mass market and individual market strategies

Transformed Markets

Standard practice

or

Codes and standards

There is a continuum of efficiency actions

Our goal is to greatly accelerate and

ldquofeedrdquo (widen) this continuum of actions

EMampV is a tool to support these actions

Evaluation Types

Evaluation Type Description Example Uses

Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)

Determines the amount of energy and demand saved

Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives

Identifies how program designs and processes can be improved

Market Effects Evaluation

Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program

Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program

Cost-Effectiveness Evaluation

Quantifies the costs of program implementation and compares them with program benefits

Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

17

Savings Cannot Be Measured - They Are Estimated

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18

Graph of Energy Consumption Before During And After Project Is Installed

Determining Energy Savings

bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as

used by the Northwest RTF

bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group

bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program

bull All or some of the above

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

19

Planning Implementing and Evaluating Efficiency Programs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

20

Evaluation is Integral to Planning-Implementation-Evaluation Process

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21

When to Evaluate

bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle

bull In timely manner with feedback for bull Ongoing program improvement

bull Supporting portfolio assessments

bull Support the planning of future portfolio cycles load forecastsenergy resource plans

bull Can also be used to inform future evaluations in particular through updating deemed savings values

Structure for Defining Evaluation Activities

bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input

bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle

bull Evaluation Research Plans ndash Created for the major EMampV activities

bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22

The Big Issues of EMampV

bull Fundamental issue of EMampV

bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty

bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

23

bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual

bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings

How good is good enough As compared to what

EMampV is About Risk Management

How Good is Good Enough amp As Compared to What

bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources

bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example

ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or

ndash Expending too many resources on ineffective actions or the opposite or

ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24

Side Note Energy Savings vs Energy Avoidance ndash

Consumer vs Regulator Perspectives

bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments

bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments

bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26

Non-Energy Benefits (NEBs)

NEB categories

bull NEBs can be categorized as those accruing to

ndash utilities (energy providers)

ndash society as a whole (such as avoided emissions)

ndash individual participants

bull Can be positive or negative (NEIs)

bull Might exceed energy benefits

Documenting NEBs

bull Measurement of benefits

bull Modeling

bull Surveys

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

27

Why Document Avoided Emissions

bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit

bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by

monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test

ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

28

Calculating Avoided Emissions

Conceptually avoided emissions are calculated using net energy savings and one of two different approaches

bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)

ndash Several sources of emission factors as well as approaches for calculating the factors

ndash Avoided emissions = (net energy savings) x (emission factor)

bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a

grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs

ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

29

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 5: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

The New Guide

bull Describes common terminology structures and approaches used for determining (evaluating)

ndash energy and demand savings

ndash avoided emissions

ndash other non-energy benefits

bull Does not recommend specific approaches - it provides

ndash context

ndash planning guidance

ndash discussion of issues

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

4 i

Energy-Efficiency Program Impact Evaluation Guide An introduction to and summary of

the practices planning and associated

issues of documenting energy savings

demand savings avoided emissions

and other non-energy benefits

resulting from end-use energy-

efficiency programs

A RESOURCE OF THE STATE AND LOCAL ENERGY EFFICIENCY ACTION NETWORK

Intended Audience

bull Primary audience ndash energy agencies and regulators

ndash public and private energy efficiency portfolio administrators such as utilities nonprofit organizations and government agencies

ndash program implementers bull And

ndash policymakers seeking general information

ndash evaluation practitioners - introductions to and summaries of evaluation topics that can be useful for explaining concepts and standard practices

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

5

Outcomes

bull Standard evaluation terminologystructures and best practices - support the adoption continuation and expansion of effective efficiency actions

bull For those just starting or ramping up their efficiency and evaluation activities ndash reduce costs for starting up an evaluation effort and moving ldquoup the learning curverdquo

bull Use of common approaches and terminology can support comparisons across different jurisdictions and facilitate ldquocross-borderrdquo energy efficiency andor greenhouse gas and other air emissions mitigation programs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

6

Expectations From Using the Guide ndash State Energy Officials

After reading this guide the reader from a State Energy Office will be able to

bull Define the basic objectives structure and evaluation approaches that can be used to plan and conduct impact evaluations of programs

bull Support and provide input for an energy efficiency evaluation framework (general guidance documents) and review impact evaluation plans and reports

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

7

The Guide supports lsquocapacity buildingrsquo and also provides input for some

forthcoming EMampV Topics

bull What is the future of EMampV for EE ndash more rigor or less rigor more consistency more deemed savings

bull How will energy savings and emissions reductions be documented for SIPsclimate programs

bull What matters net savings or gross savings just energy savings or NEBs

bull Is there a future for top-down evaluation macro data analyses

bull What is the best role of EMampV and evaluators

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

8

But First A Commercial Interruption

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 9

Jointly managed by the DOE Office of Electricity Delivery and Energy Reliability and the Office of Energy Efficiency and Renewable Energy

UNIFORM METHODS PROJECT

bull Addresses seven common residential and commercial efficiency measures

bull Step-by-step calculations for determining gross first-year savings using MampV approach

bull Also some cross cutting chapters on some EMampV topics (sampling metering persistence peak savings)

bull Adoption is voluntary bull httpumppnnlgov DOE Goals

1113089 Strengthen credibility EE savings calculations 1113089 Provide clear accessible step-by-step protocols 1113089 Support consistency and transparency 1113089 Reduce costs of EMampV 1113089 Allow for comparison of savings

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 11

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

12

Part Chapters Intended Audience

Part 1

Executive Summary

Readers interested in a brief summary and

introduction to impact evaluation

Part 2

Chapter 1 Introduction

Chapter 2 Energy Efficiency Program Evaluation

Overview

Chapter 3 Impact Evaluation Basics

Readers who want an overview of evaluation and

the key aspects of impact evaluation

Part 3

Chapter 4 Calculating Energy Savings

Chapter 5 Determining Net Energy Savings

Chapter 6 Calculating Avoided Air Emissions

Readers who want additional detail on impact

evaluation approaches ndash deemed savings MampV

large-scale consumption data analysis

Part 4

Chapter 7 Impact Evaluation Considerations

Chapter 8 Impact Evaluation Planning

Program implementers evaluators and

managersregulators of evaluations looking for

guidance on key evaluation issues and planning of

evaluations as well as readers with a background in

evaluation may want to go directly to these chapters

Part 5

Appendix A Glossary

Appendix B Other Evaluation Categories and

Approaches ndash Market Process Cost-Effectiveness

Appendix C Resources

References

Readers interested in standard energy efficiency

evaluation definitions and reference materials as

well as summaries of process market evaluations

cost-effectiveness analyses and top-down

evaluation

EXAMPLE Chapter 7 Impact Evaluation Considerations

bull First portion covers issues that often arise in impact evaluations ndash determining baselines

ndash determining demand savings

ndash calculating persistence of savings

ndash addressing uncertainty of savings estimates

ndash setting evaluation budgets

ndash establishing evaluation principles and ethics

bull Second portion provides brief overviews of ndash program feedback through evaluation

ndash resource planning through evaluation

ndash calculating non-energy benefits

ndash evaluating some relatively unique program types (residential behaviors education and training market transformation codes and standards demand response and GHG mitigation)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

13

Glossary ndash a combined effort with several groups to

create more consistency in how we describe EMampV

bull Accuracy A concept that refers to the relationship between the true value of a variable and an estimate of

the value The term can also be used in reference to a model or a set of measured data or to describe a measuring instrumentrsquos capability

bull helliphelliphelliphelliphellip

bull Incremental Annual Savings The difference between the amount of energy savings acquired or planned

to be acquired as a result of energy efficiency activities in one year and the amount of energy savings acquired or planned to be acquired as a result of the energy efficiency activities in the prior year

bull Independent Variables The explanatory factors (eg weather or occupancy) in a regression model that

are assumed to affect the variable under study (eg energy use)

bull Indirect Emissions Emissions that are a consequence of activities that take place within the organizational

boundaries of an entity but occur at sources owned or controlled by another entity For example emissions of electricity used by a manufacturing entity that occur at a power plant represent the manufacturerrsquos indirect emissions Indirect emissions are typically the source of avoided emissions for electric energy efficiency measures

bull helliphelliphelliphellip

bull Workforce Education and Training Programs Programs primarily intended for building maintenance engineers

HVAC contractors engineers architects maintenance personnel and others These programs provide information about energy efficiency concepts recommended energy-efficient technologies or behaviors andor programs that offer energy efficiency or load-reduction

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

14

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16

RDampD

bull Research

bull Development

bull Demonstration

Deployment

Voluntary implementation of projects and programs ndash outreach education and subsidies For example

bull Incenting early actors

bull Incenting consumers distributers and manufacturers

bull Mass market and individual market strategies

Transformed Markets

Standard practice

or

Codes and standards

There is a continuum of efficiency actions

Our goal is to greatly accelerate and

ldquofeedrdquo (widen) this continuum of actions

EMampV is a tool to support these actions

Evaluation Types

Evaluation Type Description Example Uses

Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)

Determines the amount of energy and demand saved

Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives

Identifies how program designs and processes can be improved

Market Effects Evaluation

Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program

Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program

Cost-Effectiveness Evaluation

Quantifies the costs of program implementation and compares them with program benefits

Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

17

Savings Cannot Be Measured - They Are Estimated

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18

Graph of Energy Consumption Before During And After Project Is Installed

Determining Energy Savings

bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as

used by the Northwest RTF

bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group

bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program

bull All or some of the above

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

19

Planning Implementing and Evaluating Efficiency Programs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

20

Evaluation is Integral to Planning-Implementation-Evaluation Process

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21

When to Evaluate

bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle

bull In timely manner with feedback for bull Ongoing program improvement

bull Supporting portfolio assessments

bull Support the planning of future portfolio cycles load forecastsenergy resource plans

bull Can also be used to inform future evaluations in particular through updating deemed savings values

Structure for Defining Evaluation Activities

bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input

bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle

bull Evaluation Research Plans ndash Created for the major EMampV activities

bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22

The Big Issues of EMampV

bull Fundamental issue of EMampV

bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty

bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

23

bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual

bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings

How good is good enough As compared to what

EMampV is About Risk Management

How Good is Good Enough amp As Compared to What

bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources

bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example

ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or

ndash Expending too many resources on ineffective actions or the opposite or

ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24

Side Note Energy Savings vs Energy Avoidance ndash

Consumer vs Regulator Perspectives

bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments

bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments

bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26

Non-Energy Benefits (NEBs)

NEB categories

bull NEBs can be categorized as those accruing to

ndash utilities (energy providers)

ndash society as a whole (such as avoided emissions)

ndash individual participants

bull Can be positive or negative (NEIs)

bull Might exceed energy benefits

Documenting NEBs

bull Measurement of benefits

bull Modeling

bull Surveys

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

27

Why Document Avoided Emissions

bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit

bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by

monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test

ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

28

Calculating Avoided Emissions

Conceptually avoided emissions are calculated using net energy savings and one of two different approaches

bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)

ndash Several sources of emission factors as well as approaches for calculating the factors

ndash Avoided emissions = (net energy savings) x (emission factor)

bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a

grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs

ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

29

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 6: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

Intended Audience

bull Primary audience ndash energy agencies and regulators

ndash public and private energy efficiency portfolio administrators such as utilities nonprofit organizations and government agencies

ndash program implementers bull And

ndash policymakers seeking general information

ndash evaluation practitioners - introductions to and summaries of evaluation topics that can be useful for explaining concepts and standard practices

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

5

Outcomes

bull Standard evaluation terminologystructures and best practices - support the adoption continuation and expansion of effective efficiency actions

bull For those just starting or ramping up their efficiency and evaluation activities ndash reduce costs for starting up an evaluation effort and moving ldquoup the learning curverdquo

bull Use of common approaches and terminology can support comparisons across different jurisdictions and facilitate ldquocross-borderrdquo energy efficiency andor greenhouse gas and other air emissions mitigation programs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

6

Expectations From Using the Guide ndash State Energy Officials

After reading this guide the reader from a State Energy Office will be able to

bull Define the basic objectives structure and evaluation approaches that can be used to plan and conduct impact evaluations of programs

bull Support and provide input for an energy efficiency evaluation framework (general guidance documents) and review impact evaluation plans and reports

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

7

The Guide supports lsquocapacity buildingrsquo and also provides input for some

forthcoming EMampV Topics

bull What is the future of EMampV for EE ndash more rigor or less rigor more consistency more deemed savings

bull How will energy savings and emissions reductions be documented for SIPsclimate programs

bull What matters net savings or gross savings just energy savings or NEBs

bull Is there a future for top-down evaluation macro data analyses

bull What is the best role of EMampV and evaluators

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

8

But First A Commercial Interruption

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 9

Jointly managed by the DOE Office of Electricity Delivery and Energy Reliability and the Office of Energy Efficiency and Renewable Energy

UNIFORM METHODS PROJECT

bull Addresses seven common residential and commercial efficiency measures

bull Step-by-step calculations for determining gross first-year savings using MampV approach

bull Also some cross cutting chapters on some EMampV topics (sampling metering persistence peak savings)

bull Adoption is voluntary bull httpumppnnlgov DOE Goals

1113089 Strengthen credibility EE savings calculations 1113089 Provide clear accessible step-by-step protocols 1113089 Support consistency and transparency 1113089 Reduce costs of EMampV 1113089 Allow for comparison of savings

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 11

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

12

Part Chapters Intended Audience

Part 1

Executive Summary

Readers interested in a brief summary and

introduction to impact evaluation

Part 2

Chapter 1 Introduction

Chapter 2 Energy Efficiency Program Evaluation

Overview

Chapter 3 Impact Evaluation Basics

Readers who want an overview of evaluation and

the key aspects of impact evaluation

Part 3

Chapter 4 Calculating Energy Savings

Chapter 5 Determining Net Energy Savings

Chapter 6 Calculating Avoided Air Emissions

Readers who want additional detail on impact

evaluation approaches ndash deemed savings MampV

large-scale consumption data analysis

Part 4

Chapter 7 Impact Evaluation Considerations

Chapter 8 Impact Evaluation Planning

Program implementers evaluators and

managersregulators of evaluations looking for

guidance on key evaluation issues and planning of

evaluations as well as readers with a background in

evaluation may want to go directly to these chapters

Part 5

Appendix A Glossary

Appendix B Other Evaluation Categories and

Approaches ndash Market Process Cost-Effectiveness

Appendix C Resources

References

Readers interested in standard energy efficiency

evaluation definitions and reference materials as

well as summaries of process market evaluations

cost-effectiveness analyses and top-down

evaluation

EXAMPLE Chapter 7 Impact Evaluation Considerations

bull First portion covers issues that often arise in impact evaluations ndash determining baselines

ndash determining demand savings

ndash calculating persistence of savings

ndash addressing uncertainty of savings estimates

ndash setting evaluation budgets

ndash establishing evaluation principles and ethics

bull Second portion provides brief overviews of ndash program feedback through evaluation

ndash resource planning through evaluation

ndash calculating non-energy benefits

ndash evaluating some relatively unique program types (residential behaviors education and training market transformation codes and standards demand response and GHG mitigation)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

13

Glossary ndash a combined effort with several groups to

create more consistency in how we describe EMampV

bull Accuracy A concept that refers to the relationship between the true value of a variable and an estimate of

the value The term can also be used in reference to a model or a set of measured data or to describe a measuring instrumentrsquos capability

bull helliphelliphelliphelliphellip

bull Incremental Annual Savings The difference between the amount of energy savings acquired or planned

to be acquired as a result of energy efficiency activities in one year and the amount of energy savings acquired or planned to be acquired as a result of the energy efficiency activities in the prior year

bull Independent Variables The explanatory factors (eg weather or occupancy) in a regression model that

are assumed to affect the variable under study (eg energy use)

bull Indirect Emissions Emissions that are a consequence of activities that take place within the organizational

boundaries of an entity but occur at sources owned or controlled by another entity For example emissions of electricity used by a manufacturing entity that occur at a power plant represent the manufacturerrsquos indirect emissions Indirect emissions are typically the source of avoided emissions for electric energy efficiency measures

bull helliphelliphelliphellip

bull Workforce Education and Training Programs Programs primarily intended for building maintenance engineers

HVAC contractors engineers architects maintenance personnel and others These programs provide information about energy efficiency concepts recommended energy-efficient technologies or behaviors andor programs that offer energy efficiency or load-reduction

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

14

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16

RDampD

bull Research

bull Development

bull Demonstration

Deployment

Voluntary implementation of projects and programs ndash outreach education and subsidies For example

bull Incenting early actors

bull Incenting consumers distributers and manufacturers

bull Mass market and individual market strategies

Transformed Markets

Standard practice

or

Codes and standards

There is a continuum of efficiency actions

Our goal is to greatly accelerate and

ldquofeedrdquo (widen) this continuum of actions

EMampV is a tool to support these actions

Evaluation Types

Evaluation Type Description Example Uses

Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)

Determines the amount of energy and demand saved

Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives

Identifies how program designs and processes can be improved

Market Effects Evaluation

Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program

Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program

Cost-Effectiveness Evaluation

Quantifies the costs of program implementation and compares them with program benefits

Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

17

Savings Cannot Be Measured - They Are Estimated

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18

Graph of Energy Consumption Before During And After Project Is Installed

Determining Energy Savings

bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as

used by the Northwest RTF

bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group

bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program

bull All or some of the above

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

19

Planning Implementing and Evaluating Efficiency Programs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

20

Evaluation is Integral to Planning-Implementation-Evaluation Process

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21

When to Evaluate

bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle

bull In timely manner with feedback for bull Ongoing program improvement

bull Supporting portfolio assessments

bull Support the planning of future portfolio cycles load forecastsenergy resource plans

bull Can also be used to inform future evaluations in particular through updating deemed savings values

Structure for Defining Evaluation Activities

bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input

bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle

bull Evaluation Research Plans ndash Created for the major EMampV activities

bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22

The Big Issues of EMampV

bull Fundamental issue of EMampV

bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty

bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

23

bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual

bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings

How good is good enough As compared to what

EMampV is About Risk Management

How Good is Good Enough amp As Compared to What

bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources

bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example

ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or

ndash Expending too many resources on ineffective actions or the opposite or

ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24

Side Note Energy Savings vs Energy Avoidance ndash

Consumer vs Regulator Perspectives

bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments

bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments

bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26

Non-Energy Benefits (NEBs)

NEB categories

bull NEBs can be categorized as those accruing to

ndash utilities (energy providers)

ndash society as a whole (such as avoided emissions)

ndash individual participants

bull Can be positive or negative (NEIs)

bull Might exceed energy benefits

Documenting NEBs

bull Measurement of benefits

bull Modeling

bull Surveys

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

27

Why Document Avoided Emissions

bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit

bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by

monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test

ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

28

Calculating Avoided Emissions

Conceptually avoided emissions are calculated using net energy savings and one of two different approaches

bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)

ndash Several sources of emission factors as well as approaches for calculating the factors

ndash Avoided emissions = (net energy savings) x (emission factor)

bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a

grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs

ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

29

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 7: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

Outcomes

bull Standard evaluation terminologystructures and best practices - support the adoption continuation and expansion of effective efficiency actions

bull For those just starting or ramping up their efficiency and evaluation activities ndash reduce costs for starting up an evaluation effort and moving ldquoup the learning curverdquo

bull Use of common approaches and terminology can support comparisons across different jurisdictions and facilitate ldquocross-borderrdquo energy efficiency andor greenhouse gas and other air emissions mitigation programs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

6

Expectations From Using the Guide ndash State Energy Officials

After reading this guide the reader from a State Energy Office will be able to

bull Define the basic objectives structure and evaluation approaches that can be used to plan and conduct impact evaluations of programs

bull Support and provide input for an energy efficiency evaluation framework (general guidance documents) and review impact evaluation plans and reports

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

7

The Guide supports lsquocapacity buildingrsquo and also provides input for some

forthcoming EMampV Topics

bull What is the future of EMampV for EE ndash more rigor or less rigor more consistency more deemed savings

bull How will energy savings and emissions reductions be documented for SIPsclimate programs

bull What matters net savings or gross savings just energy savings or NEBs

bull Is there a future for top-down evaluation macro data analyses

bull What is the best role of EMampV and evaluators

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

8

But First A Commercial Interruption

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 9

Jointly managed by the DOE Office of Electricity Delivery and Energy Reliability and the Office of Energy Efficiency and Renewable Energy

UNIFORM METHODS PROJECT

bull Addresses seven common residential and commercial efficiency measures

bull Step-by-step calculations for determining gross first-year savings using MampV approach

bull Also some cross cutting chapters on some EMampV topics (sampling metering persistence peak savings)

bull Adoption is voluntary bull httpumppnnlgov DOE Goals

1113089 Strengthen credibility EE savings calculations 1113089 Provide clear accessible step-by-step protocols 1113089 Support consistency and transparency 1113089 Reduce costs of EMampV 1113089 Allow for comparison of savings

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 11

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

12

Part Chapters Intended Audience

Part 1

Executive Summary

Readers interested in a brief summary and

introduction to impact evaluation

Part 2

Chapter 1 Introduction

Chapter 2 Energy Efficiency Program Evaluation

Overview

Chapter 3 Impact Evaluation Basics

Readers who want an overview of evaluation and

the key aspects of impact evaluation

Part 3

Chapter 4 Calculating Energy Savings

Chapter 5 Determining Net Energy Savings

Chapter 6 Calculating Avoided Air Emissions

Readers who want additional detail on impact

evaluation approaches ndash deemed savings MampV

large-scale consumption data analysis

Part 4

Chapter 7 Impact Evaluation Considerations

Chapter 8 Impact Evaluation Planning

Program implementers evaluators and

managersregulators of evaluations looking for

guidance on key evaluation issues and planning of

evaluations as well as readers with a background in

evaluation may want to go directly to these chapters

Part 5

Appendix A Glossary

Appendix B Other Evaluation Categories and

Approaches ndash Market Process Cost-Effectiveness

Appendix C Resources

References

Readers interested in standard energy efficiency

evaluation definitions and reference materials as

well as summaries of process market evaluations

cost-effectiveness analyses and top-down

evaluation

EXAMPLE Chapter 7 Impact Evaluation Considerations

bull First portion covers issues that often arise in impact evaluations ndash determining baselines

ndash determining demand savings

ndash calculating persistence of savings

ndash addressing uncertainty of savings estimates

ndash setting evaluation budgets

ndash establishing evaluation principles and ethics

bull Second portion provides brief overviews of ndash program feedback through evaluation

ndash resource planning through evaluation

ndash calculating non-energy benefits

ndash evaluating some relatively unique program types (residential behaviors education and training market transformation codes and standards demand response and GHG mitigation)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

13

Glossary ndash a combined effort with several groups to

create more consistency in how we describe EMampV

bull Accuracy A concept that refers to the relationship between the true value of a variable and an estimate of

the value The term can also be used in reference to a model or a set of measured data or to describe a measuring instrumentrsquos capability

bull helliphelliphelliphelliphellip

bull Incremental Annual Savings The difference between the amount of energy savings acquired or planned

to be acquired as a result of energy efficiency activities in one year and the amount of energy savings acquired or planned to be acquired as a result of the energy efficiency activities in the prior year

bull Independent Variables The explanatory factors (eg weather or occupancy) in a regression model that

are assumed to affect the variable under study (eg energy use)

bull Indirect Emissions Emissions that are a consequence of activities that take place within the organizational

boundaries of an entity but occur at sources owned or controlled by another entity For example emissions of electricity used by a manufacturing entity that occur at a power plant represent the manufacturerrsquos indirect emissions Indirect emissions are typically the source of avoided emissions for electric energy efficiency measures

bull helliphelliphelliphellip

bull Workforce Education and Training Programs Programs primarily intended for building maintenance engineers

HVAC contractors engineers architects maintenance personnel and others These programs provide information about energy efficiency concepts recommended energy-efficient technologies or behaviors andor programs that offer energy efficiency or load-reduction

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

14

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16

RDampD

bull Research

bull Development

bull Demonstration

Deployment

Voluntary implementation of projects and programs ndash outreach education and subsidies For example

bull Incenting early actors

bull Incenting consumers distributers and manufacturers

bull Mass market and individual market strategies

Transformed Markets

Standard practice

or

Codes and standards

There is a continuum of efficiency actions

Our goal is to greatly accelerate and

ldquofeedrdquo (widen) this continuum of actions

EMampV is a tool to support these actions

Evaluation Types

Evaluation Type Description Example Uses

Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)

Determines the amount of energy and demand saved

Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives

Identifies how program designs and processes can be improved

Market Effects Evaluation

Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program

Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program

Cost-Effectiveness Evaluation

Quantifies the costs of program implementation and compares them with program benefits

Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

17

Savings Cannot Be Measured - They Are Estimated

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18

Graph of Energy Consumption Before During And After Project Is Installed

Determining Energy Savings

bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as

used by the Northwest RTF

bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group

bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program

bull All or some of the above

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

19

Planning Implementing and Evaluating Efficiency Programs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

20

Evaluation is Integral to Planning-Implementation-Evaluation Process

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21

When to Evaluate

bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle

bull In timely manner with feedback for bull Ongoing program improvement

bull Supporting portfolio assessments

bull Support the planning of future portfolio cycles load forecastsenergy resource plans

bull Can also be used to inform future evaluations in particular through updating deemed savings values

Structure for Defining Evaluation Activities

bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input

bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle

bull Evaluation Research Plans ndash Created for the major EMampV activities

bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22

The Big Issues of EMampV

bull Fundamental issue of EMampV

bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty

bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

23

bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual

bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings

How good is good enough As compared to what

EMampV is About Risk Management

How Good is Good Enough amp As Compared to What

bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources

bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example

ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or

ndash Expending too many resources on ineffective actions or the opposite or

ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24

Side Note Energy Savings vs Energy Avoidance ndash

Consumer vs Regulator Perspectives

bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments

bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments

bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26

Non-Energy Benefits (NEBs)

NEB categories

bull NEBs can be categorized as those accruing to

ndash utilities (energy providers)

ndash society as a whole (such as avoided emissions)

ndash individual participants

bull Can be positive or negative (NEIs)

bull Might exceed energy benefits

Documenting NEBs

bull Measurement of benefits

bull Modeling

bull Surveys

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

27

Why Document Avoided Emissions

bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit

bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by

monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test

ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

28

Calculating Avoided Emissions

Conceptually avoided emissions are calculated using net energy savings and one of two different approaches

bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)

ndash Several sources of emission factors as well as approaches for calculating the factors

ndash Avoided emissions = (net energy savings) x (emission factor)

bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a

grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs

ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

29

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 8: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

Expectations From Using the Guide ndash State Energy Officials

After reading this guide the reader from a State Energy Office will be able to

bull Define the basic objectives structure and evaluation approaches that can be used to plan and conduct impact evaluations of programs

bull Support and provide input for an energy efficiency evaluation framework (general guidance documents) and review impact evaluation plans and reports

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

7

The Guide supports lsquocapacity buildingrsquo and also provides input for some

forthcoming EMampV Topics

bull What is the future of EMampV for EE ndash more rigor or less rigor more consistency more deemed savings

bull How will energy savings and emissions reductions be documented for SIPsclimate programs

bull What matters net savings or gross savings just energy savings or NEBs

bull Is there a future for top-down evaluation macro data analyses

bull What is the best role of EMampV and evaluators

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

8

But First A Commercial Interruption

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 9

Jointly managed by the DOE Office of Electricity Delivery and Energy Reliability and the Office of Energy Efficiency and Renewable Energy

UNIFORM METHODS PROJECT

bull Addresses seven common residential and commercial efficiency measures

bull Step-by-step calculations for determining gross first-year savings using MampV approach

bull Also some cross cutting chapters on some EMampV topics (sampling metering persistence peak savings)

bull Adoption is voluntary bull httpumppnnlgov DOE Goals

1113089 Strengthen credibility EE savings calculations 1113089 Provide clear accessible step-by-step protocols 1113089 Support consistency and transparency 1113089 Reduce costs of EMampV 1113089 Allow for comparison of savings

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 11

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

12

Part Chapters Intended Audience

Part 1

Executive Summary

Readers interested in a brief summary and

introduction to impact evaluation

Part 2

Chapter 1 Introduction

Chapter 2 Energy Efficiency Program Evaluation

Overview

Chapter 3 Impact Evaluation Basics

Readers who want an overview of evaluation and

the key aspects of impact evaluation

Part 3

Chapter 4 Calculating Energy Savings

Chapter 5 Determining Net Energy Savings

Chapter 6 Calculating Avoided Air Emissions

Readers who want additional detail on impact

evaluation approaches ndash deemed savings MampV

large-scale consumption data analysis

Part 4

Chapter 7 Impact Evaluation Considerations

Chapter 8 Impact Evaluation Planning

Program implementers evaluators and

managersregulators of evaluations looking for

guidance on key evaluation issues and planning of

evaluations as well as readers with a background in

evaluation may want to go directly to these chapters

Part 5

Appendix A Glossary

Appendix B Other Evaluation Categories and

Approaches ndash Market Process Cost-Effectiveness

Appendix C Resources

References

Readers interested in standard energy efficiency

evaluation definitions and reference materials as

well as summaries of process market evaluations

cost-effectiveness analyses and top-down

evaluation

EXAMPLE Chapter 7 Impact Evaluation Considerations

bull First portion covers issues that often arise in impact evaluations ndash determining baselines

ndash determining demand savings

ndash calculating persistence of savings

ndash addressing uncertainty of savings estimates

ndash setting evaluation budgets

ndash establishing evaluation principles and ethics

bull Second portion provides brief overviews of ndash program feedback through evaluation

ndash resource planning through evaluation

ndash calculating non-energy benefits

ndash evaluating some relatively unique program types (residential behaviors education and training market transformation codes and standards demand response and GHG mitigation)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

13

Glossary ndash a combined effort with several groups to

create more consistency in how we describe EMampV

bull Accuracy A concept that refers to the relationship between the true value of a variable and an estimate of

the value The term can also be used in reference to a model or a set of measured data or to describe a measuring instrumentrsquos capability

bull helliphelliphelliphelliphellip

bull Incremental Annual Savings The difference between the amount of energy savings acquired or planned

to be acquired as a result of energy efficiency activities in one year and the amount of energy savings acquired or planned to be acquired as a result of the energy efficiency activities in the prior year

bull Independent Variables The explanatory factors (eg weather or occupancy) in a regression model that

are assumed to affect the variable under study (eg energy use)

bull Indirect Emissions Emissions that are a consequence of activities that take place within the organizational

boundaries of an entity but occur at sources owned or controlled by another entity For example emissions of electricity used by a manufacturing entity that occur at a power plant represent the manufacturerrsquos indirect emissions Indirect emissions are typically the source of avoided emissions for electric energy efficiency measures

bull helliphelliphelliphellip

bull Workforce Education and Training Programs Programs primarily intended for building maintenance engineers

HVAC contractors engineers architects maintenance personnel and others These programs provide information about energy efficiency concepts recommended energy-efficient technologies or behaviors andor programs that offer energy efficiency or load-reduction

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

14

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16

RDampD

bull Research

bull Development

bull Demonstration

Deployment

Voluntary implementation of projects and programs ndash outreach education and subsidies For example

bull Incenting early actors

bull Incenting consumers distributers and manufacturers

bull Mass market and individual market strategies

Transformed Markets

Standard practice

or

Codes and standards

There is a continuum of efficiency actions

Our goal is to greatly accelerate and

ldquofeedrdquo (widen) this continuum of actions

EMampV is a tool to support these actions

Evaluation Types

Evaluation Type Description Example Uses

Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)

Determines the amount of energy and demand saved

Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives

Identifies how program designs and processes can be improved

Market Effects Evaluation

Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program

Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program

Cost-Effectiveness Evaluation

Quantifies the costs of program implementation and compares them with program benefits

Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

17

Savings Cannot Be Measured - They Are Estimated

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18

Graph of Energy Consumption Before During And After Project Is Installed

Determining Energy Savings

bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as

used by the Northwest RTF

bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group

bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program

bull All or some of the above

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

19

Planning Implementing and Evaluating Efficiency Programs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

20

Evaluation is Integral to Planning-Implementation-Evaluation Process

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21

When to Evaluate

bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle

bull In timely manner with feedback for bull Ongoing program improvement

bull Supporting portfolio assessments

bull Support the planning of future portfolio cycles load forecastsenergy resource plans

bull Can also be used to inform future evaluations in particular through updating deemed savings values

Structure for Defining Evaluation Activities

bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input

bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle

bull Evaluation Research Plans ndash Created for the major EMampV activities

bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22

The Big Issues of EMampV

bull Fundamental issue of EMampV

bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty

bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

23

bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual

bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings

How good is good enough As compared to what

EMampV is About Risk Management

How Good is Good Enough amp As Compared to What

bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources

bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example

ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or

ndash Expending too many resources on ineffective actions or the opposite or

ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24

Side Note Energy Savings vs Energy Avoidance ndash

Consumer vs Regulator Perspectives

bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments

bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments

bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26

Non-Energy Benefits (NEBs)

NEB categories

bull NEBs can be categorized as those accruing to

ndash utilities (energy providers)

ndash society as a whole (such as avoided emissions)

ndash individual participants

bull Can be positive or negative (NEIs)

bull Might exceed energy benefits

Documenting NEBs

bull Measurement of benefits

bull Modeling

bull Surveys

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

27

Why Document Avoided Emissions

bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit

bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by

monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test

ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

28

Calculating Avoided Emissions

Conceptually avoided emissions are calculated using net energy savings and one of two different approaches

bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)

ndash Several sources of emission factors as well as approaches for calculating the factors

ndash Avoided emissions = (net energy savings) x (emission factor)

bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a

grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs

ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

29

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 9: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

The Guide supports lsquocapacity buildingrsquo and also provides input for some

forthcoming EMampV Topics

bull What is the future of EMampV for EE ndash more rigor or less rigor more consistency more deemed savings

bull How will energy savings and emissions reductions be documented for SIPsclimate programs

bull What matters net savings or gross savings just energy savings or NEBs

bull Is there a future for top-down evaluation macro data analyses

bull What is the best role of EMampV and evaluators

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

8

But First A Commercial Interruption

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 9

Jointly managed by the DOE Office of Electricity Delivery and Energy Reliability and the Office of Energy Efficiency and Renewable Energy

UNIFORM METHODS PROJECT

bull Addresses seven common residential and commercial efficiency measures

bull Step-by-step calculations for determining gross first-year savings using MampV approach

bull Also some cross cutting chapters on some EMampV topics (sampling metering persistence peak savings)

bull Adoption is voluntary bull httpumppnnlgov DOE Goals

1113089 Strengthen credibility EE savings calculations 1113089 Provide clear accessible step-by-step protocols 1113089 Support consistency and transparency 1113089 Reduce costs of EMampV 1113089 Allow for comparison of savings

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 11

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

12

Part Chapters Intended Audience

Part 1

Executive Summary

Readers interested in a brief summary and

introduction to impact evaluation

Part 2

Chapter 1 Introduction

Chapter 2 Energy Efficiency Program Evaluation

Overview

Chapter 3 Impact Evaluation Basics

Readers who want an overview of evaluation and

the key aspects of impact evaluation

Part 3

Chapter 4 Calculating Energy Savings

Chapter 5 Determining Net Energy Savings

Chapter 6 Calculating Avoided Air Emissions

Readers who want additional detail on impact

evaluation approaches ndash deemed savings MampV

large-scale consumption data analysis

Part 4

Chapter 7 Impact Evaluation Considerations

Chapter 8 Impact Evaluation Planning

Program implementers evaluators and

managersregulators of evaluations looking for

guidance on key evaluation issues and planning of

evaluations as well as readers with a background in

evaluation may want to go directly to these chapters

Part 5

Appendix A Glossary

Appendix B Other Evaluation Categories and

Approaches ndash Market Process Cost-Effectiveness

Appendix C Resources

References

Readers interested in standard energy efficiency

evaluation definitions and reference materials as

well as summaries of process market evaluations

cost-effectiveness analyses and top-down

evaluation

EXAMPLE Chapter 7 Impact Evaluation Considerations

bull First portion covers issues that often arise in impact evaluations ndash determining baselines

ndash determining demand savings

ndash calculating persistence of savings

ndash addressing uncertainty of savings estimates

ndash setting evaluation budgets

ndash establishing evaluation principles and ethics

bull Second portion provides brief overviews of ndash program feedback through evaluation

ndash resource planning through evaluation

ndash calculating non-energy benefits

ndash evaluating some relatively unique program types (residential behaviors education and training market transformation codes and standards demand response and GHG mitigation)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

13

Glossary ndash a combined effort with several groups to

create more consistency in how we describe EMampV

bull Accuracy A concept that refers to the relationship between the true value of a variable and an estimate of

the value The term can also be used in reference to a model or a set of measured data or to describe a measuring instrumentrsquos capability

bull helliphelliphelliphelliphellip

bull Incremental Annual Savings The difference between the amount of energy savings acquired or planned

to be acquired as a result of energy efficiency activities in one year and the amount of energy savings acquired or planned to be acquired as a result of the energy efficiency activities in the prior year

bull Independent Variables The explanatory factors (eg weather or occupancy) in a regression model that

are assumed to affect the variable under study (eg energy use)

bull Indirect Emissions Emissions that are a consequence of activities that take place within the organizational

boundaries of an entity but occur at sources owned or controlled by another entity For example emissions of electricity used by a manufacturing entity that occur at a power plant represent the manufacturerrsquos indirect emissions Indirect emissions are typically the source of avoided emissions for electric energy efficiency measures

bull helliphelliphelliphellip

bull Workforce Education and Training Programs Programs primarily intended for building maintenance engineers

HVAC contractors engineers architects maintenance personnel and others These programs provide information about energy efficiency concepts recommended energy-efficient technologies or behaviors andor programs that offer energy efficiency or load-reduction

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

14

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16

RDampD

bull Research

bull Development

bull Demonstration

Deployment

Voluntary implementation of projects and programs ndash outreach education and subsidies For example

bull Incenting early actors

bull Incenting consumers distributers and manufacturers

bull Mass market and individual market strategies

Transformed Markets

Standard practice

or

Codes and standards

There is a continuum of efficiency actions

Our goal is to greatly accelerate and

ldquofeedrdquo (widen) this continuum of actions

EMampV is a tool to support these actions

Evaluation Types

Evaluation Type Description Example Uses

Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)

Determines the amount of energy and demand saved

Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives

Identifies how program designs and processes can be improved

Market Effects Evaluation

Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program

Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program

Cost-Effectiveness Evaluation

Quantifies the costs of program implementation and compares them with program benefits

Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

17

Savings Cannot Be Measured - They Are Estimated

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18

Graph of Energy Consumption Before During And After Project Is Installed

Determining Energy Savings

bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as

used by the Northwest RTF

bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group

bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program

bull All or some of the above

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

19

Planning Implementing and Evaluating Efficiency Programs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

20

Evaluation is Integral to Planning-Implementation-Evaluation Process

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21

When to Evaluate

bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle

bull In timely manner with feedback for bull Ongoing program improvement

bull Supporting portfolio assessments

bull Support the planning of future portfolio cycles load forecastsenergy resource plans

bull Can also be used to inform future evaluations in particular through updating deemed savings values

Structure for Defining Evaluation Activities

bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input

bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle

bull Evaluation Research Plans ndash Created for the major EMampV activities

bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22

The Big Issues of EMampV

bull Fundamental issue of EMampV

bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty

bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

23

bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual

bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings

How good is good enough As compared to what

EMampV is About Risk Management

How Good is Good Enough amp As Compared to What

bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources

bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example

ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or

ndash Expending too many resources on ineffective actions or the opposite or

ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24

Side Note Energy Savings vs Energy Avoidance ndash

Consumer vs Regulator Perspectives

bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments

bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments

bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26

Non-Energy Benefits (NEBs)

NEB categories

bull NEBs can be categorized as those accruing to

ndash utilities (energy providers)

ndash society as a whole (such as avoided emissions)

ndash individual participants

bull Can be positive or negative (NEIs)

bull Might exceed energy benefits

Documenting NEBs

bull Measurement of benefits

bull Modeling

bull Surveys

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

27

Why Document Avoided Emissions

bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit

bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by

monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test

ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

28

Calculating Avoided Emissions

Conceptually avoided emissions are calculated using net energy savings and one of two different approaches

bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)

ndash Several sources of emission factors as well as approaches for calculating the factors

ndash Avoided emissions = (net energy savings) x (emission factor)

bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a

grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs

ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

29

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 10: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

But First A Commercial Interruption

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 9

Jointly managed by the DOE Office of Electricity Delivery and Energy Reliability and the Office of Energy Efficiency and Renewable Energy

UNIFORM METHODS PROJECT

bull Addresses seven common residential and commercial efficiency measures

bull Step-by-step calculations for determining gross first-year savings using MampV approach

bull Also some cross cutting chapters on some EMampV topics (sampling metering persistence peak savings)

bull Adoption is voluntary bull httpumppnnlgov DOE Goals

1113089 Strengthen credibility EE savings calculations 1113089 Provide clear accessible step-by-step protocols 1113089 Support consistency and transparency 1113089 Reduce costs of EMampV 1113089 Allow for comparison of savings

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 11

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

12

Part Chapters Intended Audience

Part 1

Executive Summary

Readers interested in a brief summary and

introduction to impact evaluation

Part 2

Chapter 1 Introduction

Chapter 2 Energy Efficiency Program Evaluation

Overview

Chapter 3 Impact Evaluation Basics

Readers who want an overview of evaluation and

the key aspects of impact evaluation

Part 3

Chapter 4 Calculating Energy Savings

Chapter 5 Determining Net Energy Savings

Chapter 6 Calculating Avoided Air Emissions

Readers who want additional detail on impact

evaluation approaches ndash deemed savings MampV

large-scale consumption data analysis

Part 4

Chapter 7 Impact Evaluation Considerations

Chapter 8 Impact Evaluation Planning

Program implementers evaluators and

managersregulators of evaluations looking for

guidance on key evaluation issues and planning of

evaluations as well as readers with a background in

evaluation may want to go directly to these chapters

Part 5

Appendix A Glossary

Appendix B Other Evaluation Categories and

Approaches ndash Market Process Cost-Effectiveness

Appendix C Resources

References

Readers interested in standard energy efficiency

evaluation definitions and reference materials as

well as summaries of process market evaluations

cost-effectiveness analyses and top-down

evaluation

EXAMPLE Chapter 7 Impact Evaluation Considerations

bull First portion covers issues that often arise in impact evaluations ndash determining baselines

ndash determining demand savings

ndash calculating persistence of savings

ndash addressing uncertainty of savings estimates

ndash setting evaluation budgets

ndash establishing evaluation principles and ethics

bull Second portion provides brief overviews of ndash program feedback through evaluation

ndash resource planning through evaluation

ndash calculating non-energy benefits

ndash evaluating some relatively unique program types (residential behaviors education and training market transformation codes and standards demand response and GHG mitigation)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

13

Glossary ndash a combined effort with several groups to

create more consistency in how we describe EMampV

bull Accuracy A concept that refers to the relationship between the true value of a variable and an estimate of

the value The term can also be used in reference to a model or a set of measured data or to describe a measuring instrumentrsquos capability

bull helliphelliphelliphelliphellip

bull Incremental Annual Savings The difference between the amount of energy savings acquired or planned

to be acquired as a result of energy efficiency activities in one year and the amount of energy savings acquired or planned to be acquired as a result of the energy efficiency activities in the prior year

bull Independent Variables The explanatory factors (eg weather or occupancy) in a regression model that

are assumed to affect the variable under study (eg energy use)

bull Indirect Emissions Emissions that are a consequence of activities that take place within the organizational

boundaries of an entity but occur at sources owned or controlled by another entity For example emissions of electricity used by a manufacturing entity that occur at a power plant represent the manufacturerrsquos indirect emissions Indirect emissions are typically the source of avoided emissions for electric energy efficiency measures

bull helliphelliphelliphellip

bull Workforce Education and Training Programs Programs primarily intended for building maintenance engineers

HVAC contractors engineers architects maintenance personnel and others These programs provide information about energy efficiency concepts recommended energy-efficient technologies or behaviors andor programs that offer energy efficiency or load-reduction

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

14

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16

RDampD

bull Research

bull Development

bull Demonstration

Deployment

Voluntary implementation of projects and programs ndash outreach education and subsidies For example

bull Incenting early actors

bull Incenting consumers distributers and manufacturers

bull Mass market and individual market strategies

Transformed Markets

Standard practice

or

Codes and standards

There is a continuum of efficiency actions

Our goal is to greatly accelerate and

ldquofeedrdquo (widen) this continuum of actions

EMampV is a tool to support these actions

Evaluation Types

Evaluation Type Description Example Uses

Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)

Determines the amount of energy and demand saved

Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives

Identifies how program designs and processes can be improved

Market Effects Evaluation

Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program

Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program

Cost-Effectiveness Evaluation

Quantifies the costs of program implementation and compares them with program benefits

Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

17

Savings Cannot Be Measured - They Are Estimated

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18

Graph of Energy Consumption Before During And After Project Is Installed

Determining Energy Savings

bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as

used by the Northwest RTF

bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group

bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program

bull All or some of the above

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

19

Planning Implementing and Evaluating Efficiency Programs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

20

Evaluation is Integral to Planning-Implementation-Evaluation Process

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21

When to Evaluate

bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle

bull In timely manner with feedback for bull Ongoing program improvement

bull Supporting portfolio assessments

bull Support the planning of future portfolio cycles load forecastsenergy resource plans

bull Can also be used to inform future evaluations in particular through updating deemed savings values

Structure for Defining Evaluation Activities

bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input

bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle

bull Evaluation Research Plans ndash Created for the major EMampV activities

bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22

The Big Issues of EMampV

bull Fundamental issue of EMampV

bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty

bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

23

bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual

bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings

How good is good enough As compared to what

EMampV is About Risk Management

How Good is Good Enough amp As Compared to What

bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources

bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example

ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or

ndash Expending too many resources on ineffective actions or the opposite or

ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24

Side Note Energy Savings vs Energy Avoidance ndash

Consumer vs Regulator Perspectives

bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments

bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments

bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26

Non-Energy Benefits (NEBs)

NEB categories

bull NEBs can be categorized as those accruing to

ndash utilities (energy providers)

ndash society as a whole (such as avoided emissions)

ndash individual participants

bull Can be positive or negative (NEIs)

bull Might exceed energy benefits

Documenting NEBs

bull Measurement of benefits

bull Modeling

bull Surveys

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

27

Why Document Avoided Emissions

bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit

bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by

monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test

ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

28

Calculating Avoided Emissions

Conceptually avoided emissions are calculated using net energy savings and one of two different approaches

bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)

ndash Several sources of emission factors as well as approaches for calculating the factors

ndash Avoided emissions = (net energy savings) x (emission factor)

bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a

grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs

ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

29

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 11: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

Jointly managed by the DOE Office of Electricity Delivery and Energy Reliability and the Office of Energy Efficiency and Renewable Energy

UNIFORM METHODS PROJECT

bull Addresses seven common residential and commercial efficiency measures

bull Step-by-step calculations for determining gross first-year savings using MampV approach

bull Also some cross cutting chapters on some EMampV topics (sampling metering persistence peak savings)

bull Adoption is voluntary bull httpumppnnlgov DOE Goals

1113089 Strengthen credibility EE savings calculations 1113089 Provide clear accessible step-by-step protocols 1113089 Support consistency and transparency 1113089 Reduce costs of EMampV 1113089 Allow for comparison of savings

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 11

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

12

Part Chapters Intended Audience

Part 1

Executive Summary

Readers interested in a brief summary and

introduction to impact evaluation

Part 2

Chapter 1 Introduction

Chapter 2 Energy Efficiency Program Evaluation

Overview

Chapter 3 Impact Evaluation Basics

Readers who want an overview of evaluation and

the key aspects of impact evaluation

Part 3

Chapter 4 Calculating Energy Savings

Chapter 5 Determining Net Energy Savings

Chapter 6 Calculating Avoided Air Emissions

Readers who want additional detail on impact

evaluation approaches ndash deemed savings MampV

large-scale consumption data analysis

Part 4

Chapter 7 Impact Evaluation Considerations

Chapter 8 Impact Evaluation Planning

Program implementers evaluators and

managersregulators of evaluations looking for

guidance on key evaluation issues and planning of

evaluations as well as readers with a background in

evaluation may want to go directly to these chapters

Part 5

Appendix A Glossary

Appendix B Other Evaluation Categories and

Approaches ndash Market Process Cost-Effectiveness

Appendix C Resources

References

Readers interested in standard energy efficiency

evaluation definitions and reference materials as

well as summaries of process market evaluations

cost-effectiveness analyses and top-down

evaluation

EXAMPLE Chapter 7 Impact Evaluation Considerations

bull First portion covers issues that often arise in impact evaluations ndash determining baselines

ndash determining demand savings

ndash calculating persistence of savings

ndash addressing uncertainty of savings estimates

ndash setting evaluation budgets

ndash establishing evaluation principles and ethics

bull Second portion provides brief overviews of ndash program feedback through evaluation

ndash resource planning through evaluation

ndash calculating non-energy benefits

ndash evaluating some relatively unique program types (residential behaviors education and training market transformation codes and standards demand response and GHG mitigation)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

13

Glossary ndash a combined effort with several groups to

create more consistency in how we describe EMampV

bull Accuracy A concept that refers to the relationship between the true value of a variable and an estimate of

the value The term can also be used in reference to a model or a set of measured data or to describe a measuring instrumentrsquos capability

bull helliphelliphelliphelliphellip

bull Incremental Annual Savings The difference between the amount of energy savings acquired or planned

to be acquired as a result of energy efficiency activities in one year and the amount of energy savings acquired or planned to be acquired as a result of the energy efficiency activities in the prior year

bull Independent Variables The explanatory factors (eg weather or occupancy) in a regression model that

are assumed to affect the variable under study (eg energy use)

bull Indirect Emissions Emissions that are a consequence of activities that take place within the organizational

boundaries of an entity but occur at sources owned or controlled by another entity For example emissions of electricity used by a manufacturing entity that occur at a power plant represent the manufacturerrsquos indirect emissions Indirect emissions are typically the source of avoided emissions for electric energy efficiency measures

bull helliphelliphelliphellip

bull Workforce Education and Training Programs Programs primarily intended for building maintenance engineers

HVAC contractors engineers architects maintenance personnel and others These programs provide information about energy efficiency concepts recommended energy-efficient technologies or behaviors andor programs that offer energy efficiency or load-reduction

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

14

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16

RDampD

bull Research

bull Development

bull Demonstration

Deployment

Voluntary implementation of projects and programs ndash outreach education and subsidies For example

bull Incenting early actors

bull Incenting consumers distributers and manufacturers

bull Mass market and individual market strategies

Transformed Markets

Standard practice

or

Codes and standards

There is a continuum of efficiency actions

Our goal is to greatly accelerate and

ldquofeedrdquo (widen) this continuum of actions

EMampV is a tool to support these actions

Evaluation Types

Evaluation Type Description Example Uses

Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)

Determines the amount of energy and demand saved

Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives

Identifies how program designs and processes can be improved

Market Effects Evaluation

Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program

Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program

Cost-Effectiveness Evaluation

Quantifies the costs of program implementation and compares them with program benefits

Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

17

Savings Cannot Be Measured - They Are Estimated

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18

Graph of Energy Consumption Before During And After Project Is Installed

Determining Energy Savings

bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as

used by the Northwest RTF

bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group

bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program

bull All or some of the above

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

19

Planning Implementing and Evaluating Efficiency Programs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

20

Evaluation is Integral to Planning-Implementation-Evaluation Process

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21

When to Evaluate

bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle

bull In timely manner with feedback for bull Ongoing program improvement

bull Supporting portfolio assessments

bull Support the planning of future portfolio cycles load forecastsenergy resource plans

bull Can also be used to inform future evaluations in particular through updating deemed savings values

Structure for Defining Evaluation Activities

bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input

bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle

bull Evaluation Research Plans ndash Created for the major EMampV activities

bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22

The Big Issues of EMampV

bull Fundamental issue of EMampV

bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty

bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

23

bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual

bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings

How good is good enough As compared to what

EMampV is About Risk Management

How Good is Good Enough amp As Compared to What

bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources

bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example

ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or

ndash Expending too many resources on ineffective actions or the opposite or

ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24

Side Note Energy Savings vs Energy Avoidance ndash

Consumer vs Regulator Perspectives

bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments

bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments

bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26

Non-Energy Benefits (NEBs)

NEB categories

bull NEBs can be categorized as those accruing to

ndash utilities (energy providers)

ndash society as a whole (such as avoided emissions)

ndash individual participants

bull Can be positive or negative (NEIs)

bull Might exceed energy benefits

Documenting NEBs

bull Measurement of benefits

bull Modeling

bull Surveys

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

27

Why Document Avoided Emissions

bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit

bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by

monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test

ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

28

Calculating Avoided Emissions

Conceptually avoided emissions are calculated using net energy savings and one of two different approaches

bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)

ndash Several sources of emission factors as well as approaches for calculating the factors

ndash Avoided emissions = (net energy savings) x (emission factor)

bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a

grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs

ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

29

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 12: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 11

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

12

Part Chapters Intended Audience

Part 1

Executive Summary

Readers interested in a brief summary and

introduction to impact evaluation

Part 2

Chapter 1 Introduction

Chapter 2 Energy Efficiency Program Evaluation

Overview

Chapter 3 Impact Evaluation Basics

Readers who want an overview of evaluation and

the key aspects of impact evaluation

Part 3

Chapter 4 Calculating Energy Savings

Chapter 5 Determining Net Energy Savings

Chapter 6 Calculating Avoided Air Emissions

Readers who want additional detail on impact

evaluation approaches ndash deemed savings MampV

large-scale consumption data analysis

Part 4

Chapter 7 Impact Evaluation Considerations

Chapter 8 Impact Evaluation Planning

Program implementers evaluators and

managersregulators of evaluations looking for

guidance on key evaluation issues and planning of

evaluations as well as readers with a background in

evaluation may want to go directly to these chapters

Part 5

Appendix A Glossary

Appendix B Other Evaluation Categories and

Approaches ndash Market Process Cost-Effectiveness

Appendix C Resources

References

Readers interested in standard energy efficiency

evaluation definitions and reference materials as

well as summaries of process market evaluations

cost-effectiveness analyses and top-down

evaluation

EXAMPLE Chapter 7 Impact Evaluation Considerations

bull First portion covers issues that often arise in impact evaluations ndash determining baselines

ndash determining demand savings

ndash calculating persistence of savings

ndash addressing uncertainty of savings estimates

ndash setting evaluation budgets

ndash establishing evaluation principles and ethics

bull Second portion provides brief overviews of ndash program feedback through evaluation

ndash resource planning through evaluation

ndash calculating non-energy benefits

ndash evaluating some relatively unique program types (residential behaviors education and training market transformation codes and standards demand response and GHG mitigation)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

13

Glossary ndash a combined effort with several groups to

create more consistency in how we describe EMampV

bull Accuracy A concept that refers to the relationship between the true value of a variable and an estimate of

the value The term can also be used in reference to a model or a set of measured data or to describe a measuring instrumentrsquos capability

bull helliphelliphelliphelliphellip

bull Incremental Annual Savings The difference between the amount of energy savings acquired or planned

to be acquired as a result of energy efficiency activities in one year and the amount of energy savings acquired or planned to be acquired as a result of the energy efficiency activities in the prior year

bull Independent Variables The explanatory factors (eg weather or occupancy) in a regression model that

are assumed to affect the variable under study (eg energy use)

bull Indirect Emissions Emissions that are a consequence of activities that take place within the organizational

boundaries of an entity but occur at sources owned or controlled by another entity For example emissions of electricity used by a manufacturing entity that occur at a power plant represent the manufacturerrsquos indirect emissions Indirect emissions are typically the source of avoided emissions for electric energy efficiency measures

bull helliphelliphelliphellip

bull Workforce Education and Training Programs Programs primarily intended for building maintenance engineers

HVAC contractors engineers architects maintenance personnel and others These programs provide information about energy efficiency concepts recommended energy-efficient technologies or behaviors andor programs that offer energy efficiency or load-reduction

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

14

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16

RDampD

bull Research

bull Development

bull Demonstration

Deployment

Voluntary implementation of projects and programs ndash outreach education and subsidies For example

bull Incenting early actors

bull Incenting consumers distributers and manufacturers

bull Mass market and individual market strategies

Transformed Markets

Standard practice

or

Codes and standards

There is a continuum of efficiency actions

Our goal is to greatly accelerate and

ldquofeedrdquo (widen) this continuum of actions

EMampV is a tool to support these actions

Evaluation Types

Evaluation Type Description Example Uses

Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)

Determines the amount of energy and demand saved

Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives

Identifies how program designs and processes can be improved

Market Effects Evaluation

Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program

Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program

Cost-Effectiveness Evaluation

Quantifies the costs of program implementation and compares them with program benefits

Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

17

Savings Cannot Be Measured - They Are Estimated

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18

Graph of Energy Consumption Before During And After Project Is Installed

Determining Energy Savings

bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as

used by the Northwest RTF

bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group

bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program

bull All or some of the above

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

19

Planning Implementing and Evaluating Efficiency Programs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

20

Evaluation is Integral to Planning-Implementation-Evaluation Process

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21

When to Evaluate

bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle

bull In timely manner with feedback for bull Ongoing program improvement

bull Supporting portfolio assessments

bull Support the planning of future portfolio cycles load forecastsenergy resource plans

bull Can also be used to inform future evaluations in particular through updating deemed savings values

Structure for Defining Evaluation Activities

bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input

bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle

bull Evaluation Research Plans ndash Created for the major EMampV activities

bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22

The Big Issues of EMampV

bull Fundamental issue of EMampV

bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty

bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

23

bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual

bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings

How good is good enough As compared to what

EMampV is About Risk Management

How Good is Good Enough amp As Compared to What

bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources

bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example

ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or

ndash Expending too many resources on ineffective actions or the opposite or

ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24

Side Note Energy Savings vs Energy Avoidance ndash

Consumer vs Regulator Perspectives

bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments

bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments

bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26

Non-Energy Benefits (NEBs)

NEB categories

bull NEBs can be categorized as those accruing to

ndash utilities (energy providers)

ndash society as a whole (such as avoided emissions)

ndash individual participants

bull Can be positive or negative (NEIs)

bull Might exceed energy benefits

Documenting NEBs

bull Measurement of benefits

bull Modeling

bull Surveys

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

27

Why Document Avoided Emissions

bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit

bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by

monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test

ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

28

Calculating Avoided Emissions

Conceptually avoided emissions are calculated using net energy savings and one of two different approaches

bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)

ndash Several sources of emission factors as well as approaches for calculating the factors

ndash Avoided emissions = (net energy savings) x (emission factor)

bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a

grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs

ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

29

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 13: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

12

Part Chapters Intended Audience

Part 1

Executive Summary

Readers interested in a brief summary and

introduction to impact evaluation

Part 2

Chapter 1 Introduction

Chapter 2 Energy Efficiency Program Evaluation

Overview

Chapter 3 Impact Evaluation Basics

Readers who want an overview of evaluation and

the key aspects of impact evaluation

Part 3

Chapter 4 Calculating Energy Savings

Chapter 5 Determining Net Energy Savings

Chapter 6 Calculating Avoided Air Emissions

Readers who want additional detail on impact

evaluation approaches ndash deemed savings MampV

large-scale consumption data analysis

Part 4

Chapter 7 Impact Evaluation Considerations

Chapter 8 Impact Evaluation Planning

Program implementers evaluators and

managersregulators of evaluations looking for

guidance on key evaluation issues and planning of

evaluations as well as readers with a background in

evaluation may want to go directly to these chapters

Part 5

Appendix A Glossary

Appendix B Other Evaluation Categories and

Approaches ndash Market Process Cost-Effectiveness

Appendix C Resources

References

Readers interested in standard energy efficiency

evaluation definitions and reference materials as

well as summaries of process market evaluations

cost-effectiveness analyses and top-down

evaluation

EXAMPLE Chapter 7 Impact Evaluation Considerations

bull First portion covers issues that often arise in impact evaluations ndash determining baselines

ndash determining demand savings

ndash calculating persistence of savings

ndash addressing uncertainty of savings estimates

ndash setting evaluation budgets

ndash establishing evaluation principles and ethics

bull Second portion provides brief overviews of ndash program feedback through evaluation

ndash resource planning through evaluation

ndash calculating non-energy benefits

ndash evaluating some relatively unique program types (residential behaviors education and training market transformation codes and standards demand response and GHG mitigation)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

13

Glossary ndash a combined effort with several groups to

create more consistency in how we describe EMampV

bull Accuracy A concept that refers to the relationship between the true value of a variable and an estimate of

the value The term can also be used in reference to a model or a set of measured data or to describe a measuring instrumentrsquos capability

bull helliphelliphelliphelliphellip

bull Incremental Annual Savings The difference between the amount of energy savings acquired or planned

to be acquired as a result of energy efficiency activities in one year and the amount of energy savings acquired or planned to be acquired as a result of the energy efficiency activities in the prior year

bull Independent Variables The explanatory factors (eg weather or occupancy) in a regression model that

are assumed to affect the variable under study (eg energy use)

bull Indirect Emissions Emissions that are a consequence of activities that take place within the organizational

boundaries of an entity but occur at sources owned or controlled by another entity For example emissions of electricity used by a manufacturing entity that occur at a power plant represent the manufacturerrsquos indirect emissions Indirect emissions are typically the source of avoided emissions for electric energy efficiency measures

bull helliphelliphelliphellip

bull Workforce Education and Training Programs Programs primarily intended for building maintenance engineers

HVAC contractors engineers architects maintenance personnel and others These programs provide information about energy efficiency concepts recommended energy-efficient technologies or behaviors andor programs that offer energy efficiency or load-reduction

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

14

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16

RDampD

bull Research

bull Development

bull Demonstration

Deployment

Voluntary implementation of projects and programs ndash outreach education and subsidies For example

bull Incenting early actors

bull Incenting consumers distributers and manufacturers

bull Mass market and individual market strategies

Transformed Markets

Standard practice

or

Codes and standards

There is a continuum of efficiency actions

Our goal is to greatly accelerate and

ldquofeedrdquo (widen) this continuum of actions

EMampV is a tool to support these actions

Evaluation Types

Evaluation Type Description Example Uses

Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)

Determines the amount of energy and demand saved

Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives

Identifies how program designs and processes can be improved

Market Effects Evaluation

Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program

Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program

Cost-Effectiveness Evaluation

Quantifies the costs of program implementation and compares them with program benefits

Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

17

Savings Cannot Be Measured - They Are Estimated

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18

Graph of Energy Consumption Before During And After Project Is Installed

Determining Energy Savings

bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as

used by the Northwest RTF

bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group

bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program

bull All or some of the above

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

19

Planning Implementing and Evaluating Efficiency Programs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

20

Evaluation is Integral to Planning-Implementation-Evaluation Process

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21

When to Evaluate

bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle

bull In timely manner with feedback for bull Ongoing program improvement

bull Supporting portfolio assessments

bull Support the planning of future portfolio cycles load forecastsenergy resource plans

bull Can also be used to inform future evaluations in particular through updating deemed savings values

Structure for Defining Evaluation Activities

bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input

bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle

bull Evaluation Research Plans ndash Created for the major EMampV activities

bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22

The Big Issues of EMampV

bull Fundamental issue of EMampV

bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty

bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

23

bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual

bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings

How good is good enough As compared to what

EMampV is About Risk Management

How Good is Good Enough amp As Compared to What

bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources

bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example

ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or

ndash Expending too many resources on ineffective actions or the opposite or

ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24

Side Note Energy Savings vs Energy Avoidance ndash

Consumer vs Regulator Perspectives

bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments

bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments

bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26

Non-Energy Benefits (NEBs)

NEB categories

bull NEBs can be categorized as those accruing to

ndash utilities (energy providers)

ndash society as a whole (such as avoided emissions)

ndash individual participants

bull Can be positive or negative (NEIs)

bull Might exceed energy benefits

Documenting NEBs

bull Measurement of benefits

bull Modeling

bull Surveys

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

27

Why Document Avoided Emissions

bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit

bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by

monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test

ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

28

Calculating Avoided Emissions

Conceptually avoided emissions are calculated using net energy savings and one of two different approaches

bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)

ndash Several sources of emission factors as well as approaches for calculating the factors

ndash Avoided emissions = (net energy savings) x (emission factor)

bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a

grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs

ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

29

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 14: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

EXAMPLE Chapter 7 Impact Evaluation Considerations

bull First portion covers issues that often arise in impact evaluations ndash determining baselines

ndash determining demand savings

ndash calculating persistence of savings

ndash addressing uncertainty of savings estimates

ndash setting evaluation budgets

ndash establishing evaluation principles and ethics

bull Second portion provides brief overviews of ndash program feedback through evaluation

ndash resource planning through evaluation

ndash calculating non-energy benefits

ndash evaluating some relatively unique program types (residential behaviors education and training market transformation codes and standards demand response and GHG mitigation)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

13

Glossary ndash a combined effort with several groups to

create more consistency in how we describe EMampV

bull Accuracy A concept that refers to the relationship between the true value of a variable and an estimate of

the value The term can also be used in reference to a model or a set of measured data or to describe a measuring instrumentrsquos capability

bull helliphelliphelliphelliphellip

bull Incremental Annual Savings The difference between the amount of energy savings acquired or planned

to be acquired as a result of energy efficiency activities in one year and the amount of energy savings acquired or planned to be acquired as a result of the energy efficiency activities in the prior year

bull Independent Variables The explanatory factors (eg weather or occupancy) in a regression model that

are assumed to affect the variable under study (eg energy use)

bull Indirect Emissions Emissions that are a consequence of activities that take place within the organizational

boundaries of an entity but occur at sources owned or controlled by another entity For example emissions of electricity used by a manufacturing entity that occur at a power plant represent the manufacturerrsquos indirect emissions Indirect emissions are typically the source of avoided emissions for electric energy efficiency measures

bull helliphelliphelliphellip

bull Workforce Education and Training Programs Programs primarily intended for building maintenance engineers

HVAC contractors engineers architects maintenance personnel and others These programs provide information about energy efficiency concepts recommended energy-efficient technologies or behaviors andor programs that offer energy efficiency or load-reduction

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

14

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16

RDampD

bull Research

bull Development

bull Demonstration

Deployment

Voluntary implementation of projects and programs ndash outreach education and subsidies For example

bull Incenting early actors

bull Incenting consumers distributers and manufacturers

bull Mass market and individual market strategies

Transformed Markets

Standard practice

or

Codes and standards

There is a continuum of efficiency actions

Our goal is to greatly accelerate and

ldquofeedrdquo (widen) this continuum of actions

EMampV is a tool to support these actions

Evaluation Types

Evaluation Type Description Example Uses

Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)

Determines the amount of energy and demand saved

Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives

Identifies how program designs and processes can be improved

Market Effects Evaluation

Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program

Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program

Cost-Effectiveness Evaluation

Quantifies the costs of program implementation and compares them with program benefits

Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

17

Savings Cannot Be Measured - They Are Estimated

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18

Graph of Energy Consumption Before During And After Project Is Installed

Determining Energy Savings

bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as

used by the Northwest RTF

bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group

bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program

bull All or some of the above

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

19

Planning Implementing and Evaluating Efficiency Programs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

20

Evaluation is Integral to Planning-Implementation-Evaluation Process

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21

When to Evaluate

bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle

bull In timely manner with feedback for bull Ongoing program improvement

bull Supporting portfolio assessments

bull Support the planning of future portfolio cycles load forecastsenergy resource plans

bull Can also be used to inform future evaluations in particular through updating deemed savings values

Structure for Defining Evaluation Activities

bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input

bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle

bull Evaluation Research Plans ndash Created for the major EMampV activities

bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22

The Big Issues of EMampV

bull Fundamental issue of EMampV

bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty

bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

23

bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual

bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings

How good is good enough As compared to what

EMampV is About Risk Management

How Good is Good Enough amp As Compared to What

bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources

bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example

ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or

ndash Expending too many resources on ineffective actions or the opposite or

ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24

Side Note Energy Savings vs Energy Avoidance ndash

Consumer vs Regulator Perspectives

bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments

bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments

bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26

Non-Energy Benefits (NEBs)

NEB categories

bull NEBs can be categorized as those accruing to

ndash utilities (energy providers)

ndash society as a whole (such as avoided emissions)

ndash individual participants

bull Can be positive or negative (NEIs)

bull Might exceed energy benefits

Documenting NEBs

bull Measurement of benefits

bull Modeling

bull Surveys

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

27

Why Document Avoided Emissions

bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit

bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by

monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test

ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

28

Calculating Avoided Emissions

Conceptually avoided emissions are calculated using net energy savings and one of two different approaches

bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)

ndash Several sources of emission factors as well as approaches for calculating the factors

ndash Avoided emissions = (net energy savings) x (emission factor)

bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a

grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs

ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

29

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 15: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

Glossary ndash a combined effort with several groups to

create more consistency in how we describe EMampV

bull Accuracy A concept that refers to the relationship between the true value of a variable and an estimate of

the value The term can also be used in reference to a model or a set of measured data or to describe a measuring instrumentrsquos capability

bull helliphelliphelliphelliphellip

bull Incremental Annual Savings The difference between the amount of energy savings acquired or planned

to be acquired as a result of energy efficiency activities in one year and the amount of energy savings acquired or planned to be acquired as a result of the energy efficiency activities in the prior year

bull Independent Variables The explanatory factors (eg weather or occupancy) in a regression model that

are assumed to affect the variable under study (eg energy use)

bull Indirect Emissions Emissions that are a consequence of activities that take place within the organizational

boundaries of an entity but occur at sources owned or controlled by another entity For example emissions of electricity used by a manufacturing entity that occur at a power plant represent the manufacturerrsquos indirect emissions Indirect emissions are typically the source of avoided emissions for electric energy efficiency measures

bull helliphelliphelliphellip

bull Workforce Education and Training Programs Programs primarily intended for building maintenance engineers

HVAC contractors engineers architects maintenance personnel and others These programs provide information about energy efficiency concepts recommended energy-efficient technologies or behaviors andor programs that offer energy efficiency or load-reduction

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

14

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16

RDampD

bull Research

bull Development

bull Demonstration

Deployment

Voluntary implementation of projects and programs ndash outreach education and subsidies For example

bull Incenting early actors

bull Incenting consumers distributers and manufacturers

bull Mass market and individual market strategies

Transformed Markets

Standard practice

or

Codes and standards

There is a continuum of efficiency actions

Our goal is to greatly accelerate and

ldquofeedrdquo (widen) this continuum of actions

EMampV is a tool to support these actions

Evaluation Types

Evaluation Type Description Example Uses

Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)

Determines the amount of energy and demand saved

Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives

Identifies how program designs and processes can be improved

Market Effects Evaluation

Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program

Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program

Cost-Effectiveness Evaluation

Quantifies the costs of program implementation and compares them with program benefits

Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

17

Savings Cannot Be Measured - They Are Estimated

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18

Graph of Energy Consumption Before During And After Project Is Installed

Determining Energy Savings

bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as

used by the Northwest RTF

bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group

bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program

bull All or some of the above

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

19

Planning Implementing and Evaluating Efficiency Programs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

20

Evaluation is Integral to Planning-Implementation-Evaluation Process

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21

When to Evaluate

bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle

bull In timely manner with feedback for bull Ongoing program improvement

bull Supporting portfolio assessments

bull Support the planning of future portfolio cycles load forecastsenergy resource plans

bull Can also be used to inform future evaluations in particular through updating deemed savings values

Structure for Defining Evaluation Activities

bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input

bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle

bull Evaluation Research Plans ndash Created for the major EMampV activities

bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22

The Big Issues of EMampV

bull Fundamental issue of EMampV

bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty

bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

23

bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual

bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings

How good is good enough As compared to what

EMampV is About Risk Management

How Good is Good Enough amp As Compared to What

bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources

bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example

ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or

ndash Expending too many resources on ineffective actions or the opposite or

ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24

Side Note Energy Savings vs Energy Avoidance ndash

Consumer vs Regulator Perspectives

bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments

bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments

bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26

Non-Energy Benefits (NEBs)

NEB categories

bull NEBs can be categorized as those accruing to

ndash utilities (energy providers)

ndash society as a whole (such as avoided emissions)

ndash individual participants

bull Can be positive or negative (NEIs)

bull Might exceed energy benefits

Documenting NEBs

bull Measurement of benefits

bull Modeling

bull Surveys

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

27

Why Document Avoided Emissions

bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit

bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by

monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test

ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

28

Calculating Avoided Emissions

Conceptually avoided emissions are calculated using net energy savings and one of two different approaches

bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)

ndash Several sources of emission factors as well as approaches for calculating the factors

ndash Avoided emissions = (net energy savings) x (emission factor)

bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a

grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs

ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

29

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 16: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16

RDampD

bull Research

bull Development

bull Demonstration

Deployment

Voluntary implementation of projects and programs ndash outreach education and subsidies For example

bull Incenting early actors

bull Incenting consumers distributers and manufacturers

bull Mass market and individual market strategies

Transformed Markets

Standard practice

or

Codes and standards

There is a continuum of efficiency actions

Our goal is to greatly accelerate and

ldquofeedrdquo (widen) this continuum of actions

EMampV is a tool to support these actions

Evaluation Types

Evaluation Type Description Example Uses

Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)

Determines the amount of energy and demand saved

Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives

Identifies how program designs and processes can be improved

Market Effects Evaluation

Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program

Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program

Cost-Effectiveness Evaluation

Quantifies the costs of program implementation and compares them with program benefits

Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

17

Savings Cannot Be Measured - They Are Estimated

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18

Graph of Energy Consumption Before During And After Project Is Installed

Determining Energy Savings

bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as

used by the Northwest RTF

bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group

bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program

bull All or some of the above

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

19

Planning Implementing and Evaluating Efficiency Programs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

20

Evaluation is Integral to Planning-Implementation-Evaluation Process

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21

When to Evaluate

bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle

bull In timely manner with feedback for bull Ongoing program improvement

bull Supporting portfolio assessments

bull Support the planning of future portfolio cycles load forecastsenergy resource plans

bull Can also be used to inform future evaluations in particular through updating deemed savings values

Structure for Defining Evaluation Activities

bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input

bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle

bull Evaluation Research Plans ndash Created for the major EMampV activities

bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22

The Big Issues of EMampV

bull Fundamental issue of EMampV

bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty

bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

23

bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual

bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings

How good is good enough As compared to what

EMampV is About Risk Management

How Good is Good Enough amp As Compared to What

bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources

bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example

ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or

ndash Expending too many resources on ineffective actions or the opposite or

ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24

Side Note Energy Savings vs Energy Avoidance ndash

Consumer vs Regulator Perspectives

bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments

bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments

bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26

Non-Energy Benefits (NEBs)

NEB categories

bull NEBs can be categorized as those accruing to

ndash utilities (energy providers)

ndash society as a whole (such as avoided emissions)

ndash individual participants

bull Can be positive or negative (NEIs)

bull Might exceed energy benefits

Documenting NEBs

bull Measurement of benefits

bull Modeling

bull Surveys

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

27

Why Document Avoided Emissions

bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit

bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by

monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test

ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

28

Calculating Avoided Emissions

Conceptually avoided emissions are calculated using net energy savings and one of two different approaches

bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)

ndash Several sources of emission factors as well as approaches for calculating the factors

ndash Avoided emissions = (net energy savings) x (emission factor)

bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a

grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs

ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

29

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 17: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16

RDampD

bull Research

bull Development

bull Demonstration

Deployment

Voluntary implementation of projects and programs ndash outreach education and subsidies For example

bull Incenting early actors

bull Incenting consumers distributers and manufacturers

bull Mass market and individual market strategies

Transformed Markets

Standard practice

or

Codes and standards

There is a continuum of efficiency actions

Our goal is to greatly accelerate and

ldquofeedrdquo (widen) this continuum of actions

EMampV is a tool to support these actions

Evaluation Types

Evaluation Type Description Example Uses

Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)

Determines the amount of energy and demand saved

Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives

Identifies how program designs and processes can be improved

Market Effects Evaluation

Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program

Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program

Cost-Effectiveness Evaluation

Quantifies the costs of program implementation and compares them with program benefits

Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

17

Savings Cannot Be Measured - They Are Estimated

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18

Graph of Energy Consumption Before During And After Project Is Installed

Determining Energy Savings

bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as

used by the Northwest RTF

bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group

bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program

bull All or some of the above

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

19

Planning Implementing and Evaluating Efficiency Programs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

20

Evaluation is Integral to Planning-Implementation-Evaluation Process

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21

When to Evaluate

bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle

bull In timely manner with feedback for bull Ongoing program improvement

bull Supporting portfolio assessments

bull Support the planning of future portfolio cycles load forecastsenergy resource plans

bull Can also be used to inform future evaluations in particular through updating deemed savings values

Structure for Defining Evaluation Activities

bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input

bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle

bull Evaluation Research Plans ndash Created for the major EMampV activities

bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22

The Big Issues of EMampV

bull Fundamental issue of EMampV

bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty

bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

23

bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual

bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings

How good is good enough As compared to what

EMampV is About Risk Management

How Good is Good Enough amp As Compared to What

bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources

bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example

ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or

ndash Expending too many resources on ineffective actions or the opposite or

ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24

Side Note Energy Savings vs Energy Avoidance ndash

Consumer vs Regulator Perspectives

bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments

bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments

bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26

Non-Energy Benefits (NEBs)

NEB categories

bull NEBs can be categorized as those accruing to

ndash utilities (energy providers)

ndash society as a whole (such as avoided emissions)

ndash individual participants

bull Can be positive or negative (NEIs)

bull Might exceed energy benefits

Documenting NEBs

bull Measurement of benefits

bull Modeling

bull Surveys

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

27

Why Document Avoided Emissions

bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit

bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by

monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test

ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

28

Calculating Avoided Emissions

Conceptually avoided emissions are calculated using net energy savings and one of two different approaches

bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)

ndash Several sources of emission factors as well as approaches for calculating the factors

ndash Avoided emissions = (net energy savings) x (emission factor)

bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a

grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs

ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

29

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 18: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

Evaluation Types

Evaluation Type Description Example Uses

Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)

Determines the amount of energy and demand saved

Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives

Identifies how program designs and processes can be improved

Market Effects Evaluation

Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program

Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program

Cost-Effectiveness Evaluation

Quantifies the costs of program implementation and compares them with program benefits

Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

17

Savings Cannot Be Measured - They Are Estimated

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18

Graph of Energy Consumption Before During And After Project Is Installed

Determining Energy Savings

bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as

used by the Northwest RTF

bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group

bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program

bull All or some of the above

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

19

Planning Implementing and Evaluating Efficiency Programs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

20

Evaluation is Integral to Planning-Implementation-Evaluation Process

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21

When to Evaluate

bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle

bull In timely manner with feedback for bull Ongoing program improvement

bull Supporting portfolio assessments

bull Support the planning of future portfolio cycles load forecastsenergy resource plans

bull Can also be used to inform future evaluations in particular through updating deemed savings values

Structure for Defining Evaluation Activities

bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input

bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle

bull Evaluation Research Plans ndash Created for the major EMampV activities

bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22

The Big Issues of EMampV

bull Fundamental issue of EMampV

bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty

bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

23

bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual

bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings

How good is good enough As compared to what

EMampV is About Risk Management

How Good is Good Enough amp As Compared to What

bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources

bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example

ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or

ndash Expending too many resources on ineffective actions or the opposite or

ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24

Side Note Energy Savings vs Energy Avoidance ndash

Consumer vs Regulator Perspectives

bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments

bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments

bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26

Non-Energy Benefits (NEBs)

NEB categories

bull NEBs can be categorized as those accruing to

ndash utilities (energy providers)

ndash society as a whole (such as avoided emissions)

ndash individual participants

bull Can be positive or negative (NEIs)

bull Might exceed energy benefits

Documenting NEBs

bull Measurement of benefits

bull Modeling

bull Surveys

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

27

Why Document Avoided Emissions

bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit

bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by

monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test

ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

28

Calculating Avoided Emissions

Conceptually avoided emissions are calculated using net energy savings and one of two different approaches

bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)

ndash Several sources of emission factors as well as approaches for calculating the factors

ndash Avoided emissions = (net energy savings) x (emission factor)

bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a

grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs

ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

29

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 19: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

Savings Cannot Be Measured - They Are Estimated

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18

Graph of Energy Consumption Before During And After Project Is Installed

Determining Energy Savings

bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as

used by the Northwest RTF

bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group

bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program

bull All or some of the above

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

19

Planning Implementing and Evaluating Efficiency Programs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

20

Evaluation is Integral to Planning-Implementation-Evaluation Process

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21

When to Evaluate

bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle

bull In timely manner with feedback for bull Ongoing program improvement

bull Supporting portfolio assessments

bull Support the planning of future portfolio cycles load forecastsenergy resource plans

bull Can also be used to inform future evaluations in particular through updating deemed savings values

Structure for Defining Evaluation Activities

bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input

bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle

bull Evaluation Research Plans ndash Created for the major EMampV activities

bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22

The Big Issues of EMampV

bull Fundamental issue of EMampV

bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty

bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

23

bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual

bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings

How good is good enough As compared to what

EMampV is About Risk Management

How Good is Good Enough amp As Compared to What

bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources

bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example

ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or

ndash Expending too many resources on ineffective actions or the opposite or

ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24

Side Note Energy Savings vs Energy Avoidance ndash

Consumer vs Regulator Perspectives

bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments

bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments

bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26

Non-Energy Benefits (NEBs)

NEB categories

bull NEBs can be categorized as those accruing to

ndash utilities (energy providers)

ndash society as a whole (such as avoided emissions)

ndash individual participants

bull Can be positive or negative (NEIs)

bull Might exceed energy benefits

Documenting NEBs

bull Measurement of benefits

bull Modeling

bull Surveys

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

27

Why Document Avoided Emissions

bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit

bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by

monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test

ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

28

Calculating Avoided Emissions

Conceptually avoided emissions are calculated using net energy savings and one of two different approaches

bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)

ndash Several sources of emission factors as well as approaches for calculating the factors

ndash Avoided emissions = (net energy savings) x (emission factor)

bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a

grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs

ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

29

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 20: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

Determining Energy Savings

bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as

used by the Northwest RTF

bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group

bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program

bull All or some of the above

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

19

Planning Implementing and Evaluating Efficiency Programs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

20

Evaluation is Integral to Planning-Implementation-Evaluation Process

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21

When to Evaluate

bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle

bull In timely manner with feedback for bull Ongoing program improvement

bull Supporting portfolio assessments

bull Support the planning of future portfolio cycles load forecastsenergy resource plans

bull Can also be used to inform future evaluations in particular through updating deemed savings values

Structure for Defining Evaluation Activities

bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input

bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle

bull Evaluation Research Plans ndash Created for the major EMampV activities

bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22

The Big Issues of EMampV

bull Fundamental issue of EMampV

bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty

bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

23

bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual

bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings

How good is good enough As compared to what

EMampV is About Risk Management

How Good is Good Enough amp As Compared to What

bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources

bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example

ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or

ndash Expending too many resources on ineffective actions or the opposite or

ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24

Side Note Energy Savings vs Energy Avoidance ndash

Consumer vs Regulator Perspectives

bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments

bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments

bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26

Non-Energy Benefits (NEBs)

NEB categories

bull NEBs can be categorized as those accruing to

ndash utilities (energy providers)

ndash society as a whole (such as avoided emissions)

ndash individual participants

bull Can be positive or negative (NEIs)

bull Might exceed energy benefits

Documenting NEBs

bull Measurement of benefits

bull Modeling

bull Surveys

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

27

Why Document Avoided Emissions

bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit

bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by

monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test

ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

28

Calculating Avoided Emissions

Conceptually avoided emissions are calculated using net energy savings and one of two different approaches

bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)

ndash Several sources of emission factors as well as approaches for calculating the factors

ndash Avoided emissions = (net energy savings) x (emission factor)

bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a

grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs

ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

29

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 21: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

Planning Implementing and Evaluating Efficiency Programs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

20

Evaluation is Integral to Planning-Implementation-Evaluation Process

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21

When to Evaluate

bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle

bull In timely manner with feedback for bull Ongoing program improvement

bull Supporting portfolio assessments

bull Support the planning of future portfolio cycles load forecastsenergy resource plans

bull Can also be used to inform future evaluations in particular through updating deemed savings values

Structure for Defining Evaluation Activities

bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input

bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle

bull Evaluation Research Plans ndash Created for the major EMampV activities

bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22

The Big Issues of EMampV

bull Fundamental issue of EMampV

bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty

bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

23

bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual

bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings

How good is good enough As compared to what

EMampV is About Risk Management

How Good is Good Enough amp As Compared to What

bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources

bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example

ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or

ndash Expending too many resources on ineffective actions or the opposite or

ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24

Side Note Energy Savings vs Energy Avoidance ndash

Consumer vs Regulator Perspectives

bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments

bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments

bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26

Non-Energy Benefits (NEBs)

NEB categories

bull NEBs can be categorized as those accruing to

ndash utilities (energy providers)

ndash society as a whole (such as avoided emissions)

ndash individual participants

bull Can be positive or negative (NEIs)

bull Might exceed energy benefits

Documenting NEBs

bull Measurement of benefits

bull Modeling

bull Surveys

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

27

Why Document Avoided Emissions

bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit

bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by

monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test

ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

28

Calculating Avoided Emissions

Conceptually avoided emissions are calculated using net energy savings and one of two different approaches

bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)

ndash Several sources of emission factors as well as approaches for calculating the factors

ndash Avoided emissions = (net energy savings) x (emission factor)

bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a

grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs

ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

29

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 22: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

Evaluation is Integral to Planning-Implementation-Evaluation Process

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21

When to Evaluate

bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle

bull In timely manner with feedback for bull Ongoing program improvement

bull Supporting portfolio assessments

bull Support the planning of future portfolio cycles load forecastsenergy resource plans

bull Can also be used to inform future evaluations in particular through updating deemed savings values

Structure for Defining Evaluation Activities

bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input

bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle

bull Evaluation Research Plans ndash Created for the major EMampV activities

bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22

The Big Issues of EMampV

bull Fundamental issue of EMampV

bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty

bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

23

bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual

bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings

How good is good enough As compared to what

EMampV is About Risk Management

How Good is Good Enough amp As Compared to What

bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources

bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example

ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or

ndash Expending too many resources on ineffective actions or the opposite or

ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24

Side Note Energy Savings vs Energy Avoidance ndash

Consumer vs Regulator Perspectives

bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments

bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments

bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26

Non-Energy Benefits (NEBs)

NEB categories

bull NEBs can be categorized as those accruing to

ndash utilities (energy providers)

ndash society as a whole (such as avoided emissions)

ndash individual participants

bull Can be positive or negative (NEIs)

bull Might exceed energy benefits

Documenting NEBs

bull Measurement of benefits

bull Modeling

bull Surveys

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

27

Why Document Avoided Emissions

bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit

bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by

monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test

ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

28

Calculating Avoided Emissions

Conceptually avoided emissions are calculated using net energy savings and one of two different approaches

bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)

ndash Several sources of emission factors as well as approaches for calculating the factors

ndash Avoided emissions = (net energy savings) x (emission factor)

bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a

grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs

ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

29

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 23: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

Structure for Defining Evaluation Activities

bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input

bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle

bull Evaluation Research Plans ndash Created for the major EMampV activities

bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22

The Big Issues of EMampV

bull Fundamental issue of EMampV

bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty

bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

23

bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual

bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings

How good is good enough As compared to what

EMampV is About Risk Management

How Good is Good Enough amp As Compared to What

bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources

bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example

ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or

ndash Expending too many resources on ineffective actions or the opposite or

ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24

Side Note Energy Savings vs Energy Avoidance ndash

Consumer vs Regulator Perspectives

bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments

bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments

bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26

Non-Energy Benefits (NEBs)

NEB categories

bull NEBs can be categorized as those accruing to

ndash utilities (energy providers)

ndash society as a whole (such as avoided emissions)

ndash individual participants

bull Can be positive or negative (NEIs)

bull Might exceed energy benefits

Documenting NEBs

bull Measurement of benefits

bull Modeling

bull Surveys

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

27

Why Document Avoided Emissions

bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit

bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by

monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test

ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

28

Calculating Avoided Emissions

Conceptually avoided emissions are calculated using net energy savings and one of two different approaches

bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)

ndash Several sources of emission factors as well as approaches for calculating the factors

ndash Avoided emissions = (net energy savings) x (emission factor)

bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a

grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs

ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

29

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 24: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

The Big Issues of EMampV

bull Fundamental issue of EMampV

bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty

bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

23

bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual

bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings

How good is good enough As compared to what

EMampV is About Risk Management

How Good is Good Enough amp As Compared to What

bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources

bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example

ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or

ndash Expending too many resources on ineffective actions or the opposite or

ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24

Side Note Energy Savings vs Energy Avoidance ndash

Consumer vs Regulator Perspectives

bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments

bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments

bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26

Non-Energy Benefits (NEBs)

NEB categories

bull NEBs can be categorized as those accruing to

ndash utilities (energy providers)

ndash society as a whole (such as avoided emissions)

ndash individual participants

bull Can be positive or negative (NEIs)

bull Might exceed energy benefits

Documenting NEBs

bull Measurement of benefits

bull Modeling

bull Surveys

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

27

Why Document Avoided Emissions

bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit

bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by

monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test

ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

28

Calculating Avoided Emissions

Conceptually avoided emissions are calculated using net energy savings and one of two different approaches

bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)

ndash Several sources of emission factors as well as approaches for calculating the factors

ndash Avoided emissions = (net energy savings) x (emission factor)

bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a

grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs

ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

29

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 25: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

How Good is Good Enough amp As Compared to What

bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources

bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example

ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or

ndash Expending too many resources on ineffective actions or the opposite or

ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24

Side Note Energy Savings vs Energy Avoidance ndash

Consumer vs Regulator Perspectives

bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments

bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments

bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26

Non-Energy Benefits (NEBs)

NEB categories

bull NEBs can be categorized as those accruing to

ndash utilities (energy providers)

ndash society as a whole (such as avoided emissions)

ndash individual participants

bull Can be positive or negative (NEIs)

bull Might exceed energy benefits

Documenting NEBs

bull Measurement of benefits

bull Modeling

bull Surveys

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

27

Why Document Avoided Emissions

bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit

bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by

monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test

ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

28

Calculating Avoided Emissions

Conceptually avoided emissions are calculated using net energy savings and one of two different approaches

bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)

ndash Several sources of emission factors as well as approaches for calculating the factors

ndash Avoided emissions = (net energy savings) x (emission factor)

bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a

grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs

ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

29

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 26: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

Side Note Energy Savings vs Energy Avoidance ndash

Consumer vs Regulator Perspectives

bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments

bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments

bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26

Non-Energy Benefits (NEBs)

NEB categories

bull NEBs can be categorized as those accruing to

ndash utilities (energy providers)

ndash society as a whole (such as avoided emissions)

ndash individual participants

bull Can be positive or negative (NEIs)

bull Might exceed energy benefits

Documenting NEBs

bull Measurement of benefits

bull Modeling

bull Surveys

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

27

Why Document Avoided Emissions

bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit

bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by

monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test

ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

28

Calculating Avoided Emissions

Conceptually avoided emissions are calculated using net energy savings and one of two different approaches

bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)

ndash Several sources of emission factors as well as approaches for calculating the factors

ndash Avoided emissions = (net energy savings) x (emission factor)

bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a

grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs

ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

29

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 27: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26

Non-Energy Benefits (NEBs)

NEB categories

bull NEBs can be categorized as those accruing to

ndash utilities (energy providers)

ndash society as a whole (such as avoided emissions)

ndash individual participants

bull Can be positive or negative (NEIs)

bull Might exceed energy benefits

Documenting NEBs

bull Measurement of benefits

bull Modeling

bull Surveys

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

27

Why Document Avoided Emissions

bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit

bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by

monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test

ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

28

Calculating Avoided Emissions

Conceptually avoided emissions are calculated using net energy savings and one of two different approaches

bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)

ndash Several sources of emission factors as well as approaches for calculating the factors

ndash Avoided emissions = (net energy savings) x (emission factor)

bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a

grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs

ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

29

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 28: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

Non-Energy Benefits (NEBs)

NEB categories

bull NEBs can be categorized as those accruing to

ndash utilities (energy providers)

ndash society as a whole (such as avoided emissions)

ndash individual participants

bull Can be positive or negative (NEIs)

bull Might exceed energy benefits

Documenting NEBs

bull Measurement of benefits

bull Modeling

bull Surveys

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

27

Why Document Avoided Emissions

bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit

bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by

monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test

ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

28

Calculating Avoided Emissions

Conceptually avoided emissions are calculated using net energy savings and one of two different approaches

bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)

ndash Several sources of emission factors as well as approaches for calculating the factors

ndash Avoided emissions = (net energy savings) x (emission factor)

bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a

grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs

ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

29

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 29: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

Why Document Avoided Emissions

bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit

bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by

monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test

ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

28

Calculating Avoided Emissions

Conceptually avoided emissions are calculated using net energy savings and one of two different approaches

bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)

ndash Several sources of emission factors as well as approaches for calculating the factors

ndash Avoided emissions = (net energy savings) x (emission factor)

bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a

grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs

ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

29

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 30: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

Calculating Avoided Emissions

Conceptually avoided emissions are calculated using net energy savings and one of two different approaches

bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)

ndash Several sources of emission factors as well as approaches for calculating the factors

ndash Avoided emissions = (net energy savings) x (emission factor)

bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a

grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs

ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

29

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 31: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

Emissions Factors and Models

bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC

bull For electricity it can be more complex options include ndash Regional non-baseload emission rates

(using EPArsquos eGRID database)

ndash Regional marginal baseload emission rates (using capacity factors or equivalent)

ndash Regional historical hourly emission rates

ndash Energy scenario modeling

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

30

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 32: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

Calculating Avoided Emissions ndash More in the Guide and on the Web

bull Special Considerations ndash Additionality

ndash Assessment Boundaries

ndash Cap and Trade Programs

bull Emission factors for Grid-Connected Electric Generating Units

bull Resources

ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)

ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

31

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 33: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

Looking to the Future ndash EMampV for Documenting Avoided Emissions

(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for

efficiency as a mitigation option

bull Scope and Metrics of a Standard including net versus gross savings requirements

bull Baselines

bull EMampV Approaches

bull Certainty of Savings Determination

bull Who Conducts the Evaluation Activities

bull Reporting and Schedules

bull Dispute Resolution

bull Regulatory Audiences and Requirements for StandardsProtocols

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013

32

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom

Page 34: Ev a l ua ti on G ui de Pr ogr a m Im pa c t e rgy -Ef fi ...€¦ · Ev a l ua ti on G ui de An i ntroducti on to and summary of the practices, planning, ... May 2013 1 . SEE Action

SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33

From Albert Einstein

ldquoEverything should be as simple

as it is but not simplerrdquo

ldquoEverything that can be counted

does not necessarily count

everything that counts cannot

necessarily be countedrdquo

Steve Schiller Contact srschillerlblgov or steveschillercom