Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
Presentation by Steve Schiller Senior Advisor - Lawrence Berkeley National Laboratory
Electricity Markets and Policy Group
Principal - Schiller Consulting Inc
Topics
Guide Background and Objectives
Topics Covered in Guide
EMampV Objectives and Basic Issues Discussed
in Guide
Determining Avoided Emissions
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 0 i
Energy-Efficiency Program Impact Evaluation Guide An introduction to and summary of
the practices planning and associated
issues of documenting energy savings
demand savings avoided emissions
and other non-energy benefits
resulting from end-use energy-
efficiency programs
A RESOURCE OF THE STATE AND LOCAL ENERGY EFFICIENCY ACTION NETWORK
Why Evaluate bull Document impacts Document the energy savings of
projects and programs in order to determine how well they have met their goals eg has there been a good use of the invested money and time Provide PROOF of the effectiveness of energy management
bull Resource Planning To support energy resource planning by understanding the historical and future resource contributions of energy efficiency as compared to other energy resources Provide data to support efficiency as a reliable resource
bull Understand why the effects occurred Identify ways to improve current and future projects and programs as well as select future projects ldquoYou canrsquot manage what you donrsquot measurerdquo and ldquoThings that are measured tend to improverdquo
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 1
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 2
Update to the 2007 NAPEE Impact Evaluation Guide
ldquoNAPEE Guiderdquo bull Widely used and cited both
domestically and internationally
bull Used for training and as a resource by many federal agencies and several state agencies and utilities
bull Cited in state and utility EMampV rules and frameworks
bull Buthellipneeded updating
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
3
The New Guide
bull Describes common terminology structures and approaches used for determining (evaluating)
ndash energy and demand savings
ndash avoided emissions
ndash other non-energy benefits
bull Does not recommend specific approaches - it provides
ndash context
ndash planning guidance
ndash discussion of issues
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
4 i
Energy-Efficiency Program Impact Evaluation Guide An introduction to and summary of
the practices planning and associated
issues of documenting energy savings
demand savings avoided emissions
and other non-energy benefits
resulting from end-use energy-
efficiency programs
A RESOURCE OF THE STATE AND LOCAL ENERGY EFFICIENCY ACTION NETWORK
Intended Audience
bull Primary audience ndash energy agencies and regulators
ndash public and private energy efficiency portfolio administrators such as utilities nonprofit organizations and government agencies
ndash program implementers bull And
ndash policymakers seeking general information
ndash evaluation practitioners - introductions to and summaries of evaluation topics that can be useful for explaining concepts and standard practices
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
5
Outcomes
bull Standard evaluation terminologystructures and best practices - support the adoption continuation and expansion of effective efficiency actions
bull For those just starting or ramping up their efficiency and evaluation activities ndash reduce costs for starting up an evaluation effort and moving ldquoup the learning curverdquo
bull Use of common approaches and terminology can support comparisons across different jurisdictions and facilitate ldquocross-borderrdquo energy efficiency andor greenhouse gas and other air emissions mitigation programs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
6
Expectations From Using the Guide ndash State Energy Officials
After reading this guide the reader from a State Energy Office will be able to
bull Define the basic objectives structure and evaluation approaches that can be used to plan and conduct impact evaluations of programs
bull Support and provide input for an energy efficiency evaluation framework (general guidance documents) and review impact evaluation plans and reports
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
7
The Guide supports lsquocapacity buildingrsquo and also provides input for some
forthcoming EMampV Topics
bull What is the future of EMampV for EE ndash more rigor or less rigor more consistency more deemed savings
bull How will energy savings and emissions reductions be documented for SIPsclimate programs
bull What matters net savings or gross savings just energy savings or NEBs
bull Is there a future for top-down evaluation macro data analyses
bull What is the best role of EMampV and evaluators
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
8
But First A Commercial Interruption
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 9
Jointly managed by the DOE Office of Electricity Delivery and Energy Reliability and the Office of Energy Efficiency and Renewable Energy
UNIFORM METHODS PROJECT
bull Addresses seven common residential and commercial efficiency measures
bull Step-by-step calculations for determining gross first-year savings using MampV approach
bull Also some cross cutting chapters on some EMampV topics (sampling metering persistence peak savings)
bull Adoption is voluntary bull httpumppnnlgov DOE Goals
1113089 Strengthen credibility EE savings calculations 1113089 Provide clear accessible step-by-step protocols 1113089 Support consistency and transparency 1113089 Reduce costs of EMampV 1113089 Allow for comparison of savings
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 11
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
12
Part Chapters Intended Audience
Part 1
Executive Summary
Readers interested in a brief summary and
introduction to impact evaluation
Part 2
Chapter 1 Introduction
Chapter 2 Energy Efficiency Program Evaluation
Overview
Chapter 3 Impact Evaluation Basics
Readers who want an overview of evaluation and
the key aspects of impact evaluation
Part 3
Chapter 4 Calculating Energy Savings
Chapter 5 Determining Net Energy Savings
Chapter 6 Calculating Avoided Air Emissions
Readers who want additional detail on impact
evaluation approaches ndash deemed savings MampV
large-scale consumption data analysis
Part 4
Chapter 7 Impact Evaluation Considerations
Chapter 8 Impact Evaluation Planning
Program implementers evaluators and
managersregulators of evaluations looking for
guidance on key evaluation issues and planning of
evaluations as well as readers with a background in
evaluation may want to go directly to these chapters
Part 5
Appendix A Glossary
Appendix B Other Evaluation Categories and
Approaches ndash Market Process Cost-Effectiveness
Appendix C Resources
References
Readers interested in standard energy efficiency
evaluation definitions and reference materials as
well as summaries of process market evaluations
cost-effectiveness analyses and top-down
evaluation
EXAMPLE Chapter 7 Impact Evaluation Considerations
bull First portion covers issues that often arise in impact evaluations ndash determining baselines
ndash determining demand savings
ndash calculating persistence of savings
ndash addressing uncertainty of savings estimates
ndash setting evaluation budgets
ndash establishing evaluation principles and ethics
bull Second portion provides brief overviews of ndash program feedback through evaluation
ndash resource planning through evaluation
ndash calculating non-energy benefits
ndash evaluating some relatively unique program types (residential behaviors education and training market transformation codes and standards demand response and GHG mitigation)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
13
Glossary ndash a combined effort with several groups to
create more consistency in how we describe EMampV
bull Accuracy A concept that refers to the relationship between the true value of a variable and an estimate of
the value The term can also be used in reference to a model or a set of measured data or to describe a measuring instrumentrsquos capability
bull helliphelliphelliphelliphellip
bull Incremental Annual Savings The difference between the amount of energy savings acquired or planned
to be acquired as a result of energy efficiency activities in one year and the amount of energy savings acquired or planned to be acquired as a result of the energy efficiency activities in the prior year
bull Independent Variables The explanatory factors (eg weather or occupancy) in a regression model that
are assumed to affect the variable under study (eg energy use)
bull Indirect Emissions Emissions that are a consequence of activities that take place within the organizational
boundaries of an entity but occur at sources owned or controlled by another entity For example emissions of electricity used by a manufacturing entity that occur at a power plant represent the manufacturerrsquos indirect emissions Indirect emissions are typically the source of avoided emissions for electric energy efficiency measures
bull helliphelliphelliphellip
bull Workforce Education and Training Programs Programs primarily intended for building maintenance engineers
HVAC contractors engineers architects maintenance personnel and others These programs provide information about energy efficiency concepts recommended energy-efficient technologies or behaviors andor programs that offer energy efficiency or load-reduction
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
14
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16
RDampD
bull Research
bull Development
bull Demonstration
Deployment
Voluntary implementation of projects and programs ndash outreach education and subsidies For example
bull Incenting early actors
bull Incenting consumers distributers and manufacturers
bull Mass market and individual market strategies
Transformed Markets
Standard practice
or
Codes and standards
There is a continuum of efficiency actions
Our goal is to greatly accelerate and
ldquofeedrdquo (widen) this continuum of actions
EMampV is a tool to support these actions
Evaluation Types
Evaluation Type Description Example Uses
Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)
Determines the amount of energy and demand saved
Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives
Identifies how program designs and processes can be improved
Market Effects Evaluation
Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program
Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program
Cost-Effectiveness Evaluation
Quantifies the costs of program implementation and compares them with program benefits
Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
17
Savings Cannot Be Measured - They Are Estimated
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18
Graph of Energy Consumption Before During And After Project Is Installed
Determining Energy Savings
bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as
used by the Northwest RTF
bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group
bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program
bull All or some of the above
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
19
Planning Implementing and Evaluating Efficiency Programs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
20
Evaluation is Integral to Planning-Implementation-Evaluation Process
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21
When to Evaluate
bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle
bull In timely manner with feedback for bull Ongoing program improvement
bull Supporting portfolio assessments
bull Support the planning of future portfolio cycles load forecastsenergy resource plans
bull Can also be used to inform future evaluations in particular through updating deemed savings values
Structure for Defining Evaluation Activities
bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input
bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle
bull Evaluation Research Plans ndash Created for the major EMampV activities
bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22
The Big Issues of EMampV
bull Fundamental issue of EMampV
bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty
bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
23
bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual
bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings
How good is good enough As compared to what
EMampV is About Risk Management
How Good is Good Enough amp As Compared to What
bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources
bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example
ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or
ndash Expending too many resources on ineffective actions or the opposite or
ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24
Side Note Energy Savings vs Energy Avoidance ndash
Consumer vs Regulator Perspectives
bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments
bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments
bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26
Non-Energy Benefits (NEBs)
NEB categories
bull NEBs can be categorized as those accruing to
ndash utilities (energy providers)
ndash society as a whole (such as avoided emissions)
ndash individual participants
bull Can be positive or negative (NEIs)
bull Might exceed energy benefits
Documenting NEBs
bull Measurement of benefits
bull Modeling
bull Surveys
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
27
Why Document Avoided Emissions
bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit
bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by
monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test
ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
28
Calculating Avoided Emissions
Conceptually avoided emissions are calculated using net energy savings and one of two different approaches
bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)
ndash Several sources of emission factors as well as approaches for calculating the factors
ndash Avoided emissions = (net energy savings) x (emission factor)
bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a
grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs
ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
29
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
Why Evaluate bull Document impacts Document the energy savings of
projects and programs in order to determine how well they have met their goals eg has there been a good use of the invested money and time Provide PROOF of the effectiveness of energy management
bull Resource Planning To support energy resource planning by understanding the historical and future resource contributions of energy efficiency as compared to other energy resources Provide data to support efficiency as a reliable resource
bull Understand why the effects occurred Identify ways to improve current and future projects and programs as well as select future projects ldquoYou canrsquot manage what you donrsquot measurerdquo and ldquoThings that are measured tend to improverdquo
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 1
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 2
Update to the 2007 NAPEE Impact Evaluation Guide
ldquoNAPEE Guiderdquo bull Widely used and cited both
domestically and internationally
bull Used for training and as a resource by many federal agencies and several state agencies and utilities
bull Cited in state and utility EMampV rules and frameworks
bull Buthellipneeded updating
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
3
The New Guide
bull Describes common terminology structures and approaches used for determining (evaluating)
ndash energy and demand savings
ndash avoided emissions
ndash other non-energy benefits
bull Does not recommend specific approaches - it provides
ndash context
ndash planning guidance
ndash discussion of issues
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
4 i
Energy-Efficiency Program Impact Evaluation Guide An introduction to and summary of
the practices planning and associated
issues of documenting energy savings
demand savings avoided emissions
and other non-energy benefits
resulting from end-use energy-
efficiency programs
A RESOURCE OF THE STATE AND LOCAL ENERGY EFFICIENCY ACTION NETWORK
Intended Audience
bull Primary audience ndash energy agencies and regulators
ndash public and private energy efficiency portfolio administrators such as utilities nonprofit organizations and government agencies
ndash program implementers bull And
ndash policymakers seeking general information
ndash evaluation practitioners - introductions to and summaries of evaluation topics that can be useful for explaining concepts and standard practices
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
5
Outcomes
bull Standard evaluation terminologystructures and best practices - support the adoption continuation and expansion of effective efficiency actions
bull For those just starting or ramping up their efficiency and evaluation activities ndash reduce costs for starting up an evaluation effort and moving ldquoup the learning curverdquo
bull Use of common approaches and terminology can support comparisons across different jurisdictions and facilitate ldquocross-borderrdquo energy efficiency andor greenhouse gas and other air emissions mitigation programs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
6
Expectations From Using the Guide ndash State Energy Officials
After reading this guide the reader from a State Energy Office will be able to
bull Define the basic objectives structure and evaluation approaches that can be used to plan and conduct impact evaluations of programs
bull Support and provide input for an energy efficiency evaluation framework (general guidance documents) and review impact evaluation plans and reports
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
7
The Guide supports lsquocapacity buildingrsquo and also provides input for some
forthcoming EMampV Topics
bull What is the future of EMampV for EE ndash more rigor or less rigor more consistency more deemed savings
bull How will energy savings and emissions reductions be documented for SIPsclimate programs
bull What matters net savings or gross savings just energy savings or NEBs
bull Is there a future for top-down evaluation macro data analyses
bull What is the best role of EMampV and evaluators
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
8
But First A Commercial Interruption
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 9
Jointly managed by the DOE Office of Electricity Delivery and Energy Reliability and the Office of Energy Efficiency and Renewable Energy
UNIFORM METHODS PROJECT
bull Addresses seven common residential and commercial efficiency measures
bull Step-by-step calculations for determining gross first-year savings using MampV approach
bull Also some cross cutting chapters on some EMampV topics (sampling metering persistence peak savings)
bull Adoption is voluntary bull httpumppnnlgov DOE Goals
1113089 Strengthen credibility EE savings calculations 1113089 Provide clear accessible step-by-step protocols 1113089 Support consistency and transparency 1113089 Reduce costs of EMampV 1113089 Allow for comparison of savings
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 11
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
12
Part Chapters Intended Audience
Part 1
Executive Summary
Readers interested in a brief summary and
introduction to impact evaluation
Part 2
Chapter 1 Introduction
Chapter 2 Energy Efficiency Program Evaluation
Overview
Chapter 3 Impact Evaluation Basics
Readers who want an overview of evaluation and
the key aspects of impact evaluation
Part 3
Chapter 4 Calculating Energy Savings
Chapter 5 Determining Net Energy Savings
Chapter 6 Calculating Avoided Air Emissions
Readers who want additional detail on impact
evaluation approaches ndash deemed savings MampV
large-scale consumption data analysis
Part 4
Chapter 7 Impact Evaluation Considerations
Chapter 8 Impact Evaluation Planning
Program implementers evaluators and
managersregulators of evaluations looking for
guidance on key evaluation issues and planning of
evaluations as well as readers with a background in
evaluation may want to go directly to these chapters
Part 5
Appendix A Glossary
Appendix B Other Evaluation Categories and
Approaches ndash Market Process Cost-Effectiveness
Appendix C Resources
References
Readers interested in standard energy efficiency
evaluation definitions and reference materials as
well as summaries of process market evaluations
cost-effectiveness analyses and top-down
evaluation
EXAMPLE Chapter 7 Impact Evaluation Considerations
bull First portion covers issues that often arise in impact evaluations ndash determining baselines
ndash determining demand savings
ndash calculating persistence of savings
ndash addressing uncertainty of savings estimates
ndash setting evaluation budgets
ndash establishing evaluation principles and ethics
bull Second portion provides brief overviews of ndash program feedback through evaluation
ndash resource planning through evaluation
ndash calculating non-energy benefits
ndash evaluating some relatively unique program types (residential behaviors education and training market transformation codes and standards demand response and GHG mitigation)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
13
Glossary ndash a combined effort with several groups to
create more consistency in how we describe EMampV
bull Accuracy A concept that refers to the relationship between the true value of a variable and an estimate of
the value The term can also be used in reference to a model or a set of measured data or to describe a measuring instrumentrsquos capability
bull helliphelliphelliphelliphellip
bull Incremental Annual Savings The difference between the amount of energy savings acquired or planned
to be acquired as a result of energy efficiency activities in one year and the amount of energy savings acquired or planned to be acquired as a result of the energy efficiency activities in the prior year
bull Independent Variables The explanatory factors (eg weather or occupancy) in a regression model that
are assumed to affect the variable under study (eg energy use)
bull Indirect Emissions Emissions that are a consequence of activities that take place within the organizational
boundaries of an entity but occur at sources owned or controlled by another entity For example emissions of electricity used by a manufacturing entity that occur at a power plant represent the manufacturerrsquos indirect emissions Indirect emissions are typically the source of avoided emissions for electric energy efficiency measures
bull helliphelliphelliphellip
bull Workforce Education and Training Programs Programs primarily intended for building maintenance engineers
HVAC contractors engineers architects maintenance personnel and others These programs provide information about energy efficiency concepts recommended energy-efficient technologies or behaviors andor programs that offer energy efficiency or load-reduction
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
14
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16
RDampD
bull Research
bull Development
bull Demonstration
Deployment
Voluntary implementation of projects and programs ndash outreach education and subsidies For example
bull Incenting early actors
bull Incenting consumers distributers and manufacturers
bull Mass market and individual market strategies
Transformed Markets
Standard practice
or
Codes and standards
There is a continuum of efficiency actions
Our goal is to greatly accelerate and
ldquofeedrdquo (widen) this continuum of actions
EMampV is a tool to support these actions
Evaluation Types
Evaluation Type Description Example Uses
Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)
Determines the amount of energy and demand saved
Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives
Identifies how program designs and processes can be improved
Market Effects Evaluation
Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program
Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program
Cost-Effectiveness Evaluation
Quantifies the costs of program implementation and compares them with program benefits
Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
17
Savings Cannot Be Measured - They Are Estimated
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18
Graph of Energy Consumption Before During And After Project Is Installed
Determining Energy Savings
bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as
used by the Northwest RTF
bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group
bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program
bull All or some of the above
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
19
Planning Implementing and Evaluating Efficiency Programs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
20
Evaluation is Integral to Planning-Implementation-Evaluation Process
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21
When to Evaluate
bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle
bull In timely manner with feedback for bull Ongoing program improvement
bull Supporting portfolio assessments
bull Support the planning of future portfolio cycles load forecastsenergy resource plans
bull Can also be used to inform future evaluations in particular through updating deemed savings values
Structure for Defining Evaluation Activities
bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input
bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle
bull Evaluation Research Plans ndash Created for the major EMampV activities
bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22
The Big Issues of EMampV
bull Fundamental issue of EMampV
bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty
bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
23
bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual
bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings
How good is good enough As compared to what
EMampV is About Risk Management
How Good is Good Enough amp As Compared to What
bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources
bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example
ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or
ndash Expending too many resources on ineffective actions or the opposite or
ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24
Side Note Energy Savings vs Energy Avoidance ndash
Consumer vs Regulator Perspectives
bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments
bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments
bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26
Non-Energy Benefits (NEBs)
NEB categories
bull NEBs can be categorized as those accruing to
ndash utilities (energy providers)
ndash society as a whole (such as avoided emissions)
ndash individual participants
bull Can be positive or negative (NEIs)
bull Might exceed energy benefits
Documenting NEBs
bull Measurement of benefits
bull Modeling
bull Surveys
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
27
Why Document Avoided Emissions
bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit
bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by
monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test
ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
28
Calculating Avoided Emissions
Conceptually avoided emissions are calculated using net energy savings and one of two different approaches
bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)
ndash Several sources of emission factors as well as approaches for calculating the factors
ndash Avoided emissions = (net energy savings) x (emission factor)
bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a
grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs
ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
29
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 2
Update to the 2007 NAPEE Impact Evaluation Guide
ldquoNAPEE Guiderdquo bull Widely used and cited both
domestically and internationally
bull Used for training and as a resource by many federal agencies and several state agencies and utilities
bull Cited in state and utility EMampV rules and frameworks
bull Buthellipneeded updating
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
3
The New Guide
bull Describes common terminology structures and approaches used for determining (evaluating)
ndash energy and demand savings
ndash avoided emissions
ndash other non-energy benefits
bull Does not recommend specific approaches - it provides
ndash context
ndash planning guidance
ndash discussion of issues
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
4 i
Energy-Efficiency Program Impact Evaluation Guide An introduction to and summary of
the practices planning and associated
issues of documenting energy savings
demand savings avoided emissions
and other non-energy benefits
resulting from end-use energy-
efficiency programs
A RESOURCE OF THE STATE AND LOCAL ENERGY EFFICIENCY ACTION NETWORK
Intended Audience
bull Primary audience ndash energy agencies and regulators
ndash public and private energy efficiency portfolio administrators such as utilities nonprofit organizations and government agencies
ndash program implementers bull And
ndash policymakers seeking general information
ndash evaluation practitioners - introductions to and summaries of evaluation topics that can be useful for explaining concepts and standard practices
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
5
Outcomes
bull Standard evaluation terminologystructures and best practices - support the adoption continuation and expansion of effective efficiency actions
bull For those just starting or ramping up their efficiency and evaluation activities ndash reduce costs for starting up an evaluation effort and moving ldquoup the learning curverdquo
bull Use of common approaches and terminology can support comparisons across different jurisdictions and facilitate ldquocross-borderrdquo energy efficiency andor greenhouse gas and other air emissions mitigation programs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
6
Expectations From Using the Guide ndash State Energy Officials
After reading this guide the reader from a State Energy Office will be able to
bull Define the basic objectives structure and evaluation approaches that can be used to plan and conduct impact evaluations of programs
bull Support and provide input for an energy efficiency evaluation framework (general guidance documents) and review impact evaluation plans and reports
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
7
The Guide supports lsquocapacity buildingrsquo and also provides input for some
forthcoming EMampV Topics
bull What is the future of EMampV for EE ndash more rigor or less rigor more consistency more deemed savings
bull How will energy savings and emissions reductions be documented for SIPsclimate programs
bull What matters net savings or gross savings just energy savings or NEBs
bull Is there a future for top-down evaluation macro data analyses
bull What is the best role of EMampV and evaluators
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
8
But First A Commercial Interruption
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 9
Jointly managed by the DOE Office of Electricity Delivery and Energy Reliability and the Office of Energy Efficiency and Renewable Energy
UNIFORM METHODS PROJECT
bull Addresses seven common residential and commercial efficiency measures
bull Step-by-step calculations for determining gross first-year savings using MampV approach
bull Also some cross cutting chapters on some EMampV topics (sampling metering persistence peak savings)
bull Adoption is voluntary bull httpumppnnlgov DOE Goals
1113089 Strengthen credibility EE savings calculations 1113089 Provide clear accessible step-by-step protocols 1113089 Support consistency and transparency 1113089 Reduce costs of EMampV 1113089 Allow for comparison of savings
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 11
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
12
Part Chapters Intended Audience
Part 1
Executive Summary
Readers interested in a brief summary and
introduction to impact evaluation
Part 2
Chapter 1 Introduction
Chapter 2 Energy Efficiency Program Evaluation
Overview
Chapter 3 Impact Evaluation Basics
Readers who want an overview of evaluation and
the key aspects of impact evaluation
Part 3
Chapter 4 Calculating Energy Savings
Chapter 5 Determining Net Energy Savings
Chapter 6 Calculating Avoided Air Emissions
Readers who want additional detail on impact
evaluation approaches ndash deemed savings MampV
large-scale consumption data analysis
Part 4
Chapter 7 Impact Evaluation Considerations
Chapter 8 Impact Evaluation Planning
Program implementers evaluators and
managersregulators of evaluations looking for
guidance on key evaluation issues and planning of
evaluations as well as readers with a background in
evaluation may want to go directly to these chapters
Part 5
Appendix A Glossary
Appendix B Other Evaluation Categories and
Approaches ndash Market Process Cost-Effectiveness
Appendix C Resources
References
Readers interested in standard energy efficiency
evaluation definitions and reference materials as
well as summaries of process market evaluations
cost-effectiveness analyses and top-down
evaluation
EXAMPLE Chapter 7 Impact Evaluation Considerations
bull First portion covers issues that often arise in impact evaluations ndash determining baselines
ndash determining demand savings
ndash calculating persistence of savings
ndash addressing uncertainty of savings estimates
ndash setting evaluation budgets
ndash establishing evaluation principles and ethics
bull Second portion provides brief overviews of ndash program feedback through evaluation
ndash resource planning through evaluation
ndash calculating non-energy benefits
ndash evaluating some relatively unique program types (residential behaviors education and training market transformation codes and standards demand response and GHG mitigation)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
13
Glossary ndash a combined effort with several groups to
create more consistency in how we describe EMampV
bull Accuracy A concept that refers to the relationship between the true value of a variable and an estimate of
the value The term can also be used in reference to a model or a set of measured data or to describe a measuring instrumentrsquos capability
bull helliphelliphelliphelliphellip
bull Incremental Annual Savings The difference between the amount of energy savings acquired or planned
to be acquired as a result of energy efficiency activities in one year and the amount of energy savings acquired or planned to be acquired as a result of the energy efficiency activities in the prior year
bull Independent Variables The explanatory factors (eg weather or occupancy) in a regression model that
are assumed to affect the variable under study (eg energy use)
bull Indirect Emissions Emissions that are a consequence of activities that take place within the organizational
boundaries of an entity but occur at sources owned or controlled by another entity For example emissions of electricity used by a manufacturing entity that occur at a power plant represent the manufacturerrsquos indirect emissions Indirect emissions are typically the source of avoided emissions for electric energy efficiency measures
bull helliphelliphelliphellip
bull Workforce Education and Training Programs Programs primarily intended for building maintenance engineers
HVAC contractors engineers architects maintenance personnel and others These programs provide information about energy efficiency concepts recommended energy-efficient technologies or behaviors andor programs that offer energy efficiency or load-reduction
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
14
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16
RDampD
bull Research
bull Development
bull Demonstration
Deployment
Voluntary implementation of projects and programs ndash outreach education and subsidies For example
bull Incenting early actors
bull Incenting consumers distributers and manufacturers
bull Mass market and individual market strategies
Transformed Markets
Standard practice
or
Codes and standards
There is a continuum of efficiency actions
Our goal is to greatly accelerate and
ldquofeedrdquo (widen) this continuum of actions
EMampV is a tool to support these actions
Evaluation Types
Evaluation Type Description Example Uses
Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)
Determines the amount of energy and demand saved
Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives
Identifies how program designs and processes can be improved
Market Effects Evaluation
Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program
Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program
Cost-Effectiveness Evaluation
Quantifies the costs of program implementation and compares them with program benefits
Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
17
Savings Cannot Be Measured - They Are Estimated
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18
Graph of Energy Consumption Before During And After Project Is Installed
Determining Energy Savings
bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as
used by the Northwest RTF
bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group
bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program
bull All or some of the above
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
19
Planning Implementing and Evaluating Efficiency Programs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
20
Evaluation is Integral to Planning-Implementation-Evaluation Process
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21
When to Evaluate
bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle
bull In timely manner with feedback for bull Ongoing program improvement
bull Supporting portfolio assessments
bull Support the planning of future portfolio cycles load forecastsenergy resource plans
bull Can also be used to inform future evaluations in particular through updating deemed savings values
Structure for Defining Evaluation Activities
bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input
bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle
bull Evaluation Research Plans ndash Created for the major EMampV activities
bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22
The Big Issues of EMampV
bull Fundamental issue of EMampV
bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty
bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
23
bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual
bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings
How good is good enough As compared to what
EMampV is About Risk Management
How Good is Good Enough amp As Compared to What
bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources
bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example
ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or
ndash Expending too many resources on ineffective actions or the opposite or
ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24
Side Note Energy Savings vs Energy Avoidance ndash
Consumer vs Regulator Perspectives
bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments
bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments
bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26
Non-Energy Benefits (NEBs)
NEB categories
bull NEBs can be categorized as those accruing to
ndash utilities (energy providers)
ndash society as a whole (such as avoided emissions)
ndash individual participants
bull Can be positive or negative (NEIs)
bull Might exceed energy benefits
Documenting NEBs
bull Measurement of benefits
bull Modeling
bull Surveys
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
27
Why Document Avoided Emissions
bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit
bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by
monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test
ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
28
Calculating Avoided Emissions
Conceptually avoided emissions are calculated using net energy savings and one of two different approaches
bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)
ndash Several sources of emission factors as well as approaches for calculating the factors
ndash Avoided emissions = (net energy savings) x (emission factor)
bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a
grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs
ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
29
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
Update to the 2007 NAPEE Impact Evaluation Guide
ldquoNAPEE Guiderdquo bull Widely used and cited both
domestically and internationally
bull Used for training and as a resource by many federal agencies and several state agencies and utilities
bull Cited in state and utility EMampV rules and frameworks
bull Buthellipneeded updating
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
3
The New Guide
bull Describes common terminology structures and approaches used for determining (evaluating)
ndash energy and demand savings
ndash avoided emissions
ndash other non-energy benefits
bull Does not recommend specific approaches - it provides
ndash context
ndash planning guidance
ndash discussion of issues
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
4 i
Energy-Efficiency Program Impact Evaluation Guide An introduction to and summary of
the practices planning and associated
issues of documenting energy savings
demand savings avoided emissions
and other non-energy benefits
resulting from end-use energy-
efficiency programs
A RESOURCE OF THE STATE AND LOCAL ENERGY EFFICIENCY ACTION NETWORK
Intended Audience
bull Primary audience ndash energy agencies and regulators
ndash public and private energy efficiency portfolio administrators such as utilities nonprofit organizations and government agencies
ndash program implementers bull And
ndash policymakers seeking general information
ndash evaluation practitioners - introductions to and summaries of evaluation topics that can be useful for explaining concepts and standard practices
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
5
Outcomes
bull Standard evaluation terminologystructures and best practices - support the adoption continuation and expansion of effective efficiency actions
bull For those just starting or ramping up their efficiency and evaluation activities ndash reduce costs for starting up an evaluation effort and moving ldquoup the learning curverdquo
bull Use of common approaches and terminology can support comparisons across different jurisdictions and facilitate ldquocross-borderrdquo energy efficiency andor greenhouse gas and other air emissions mitigation programs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
6
Expectations From Using the Guide ndash State Energy Officials
After reading this guide the reader from a State Energy Office will be able to
bull Define the basic objectives structure and evaluation approaches that can be used to plan and conduct impact evaluations of programs
bull Support and provide input for an energy efficiency evaluation framework (general guidance documents) and review impact evaluation plans and reports
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
7
The Guide supports lsquocapacity buildingrsquo and also provides input for some
forthcoming EMampV Topics
bull What is the future of EMampV for EE ndash more rigor or less rigor more consistency more deemed savings
bull How will energy savings and emissions reductions be documented for SIPsclimate programs
bull What matters net savings or gross savings just energy savings or NEBs
bull Is there a future for top-down evaluation macro data analyses
bull What is the best role of EMampV and evaluators
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
8
But First A Commercial Interruption
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 9
Jointly managed by the DOE Office of Electricity Delivery and Energy Reliability and the Office of Energy Efficiency and Renewable Energy
UNIFORM METHODS PROJECT
bull Addresses seven common residential and commercial efficiency measures
bull Step-by-step calculations for determining gross first-year savings using MampV approach
bull Also some cross cutting chapters on some EMampV topics (sampling metering persistence peak savings)
bull Adoption is voluntary bull httpumppnnlgov DOE Goals
1113089 Strengthen credibility EE savings calculations 1113089 Provide clear accessible step-by-step protocols 1113089 Support consistency and transparency 1113089 Reduce costs of EMampV 1113089 Allow for comparison of savings
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 11
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
12
Part Chapters Intended Audience
Part 1
Executive Summary
Readers interested in a brief summary and
introduction to impact evaluation
Part 2
Chapter 1 Introduction
Chapter 2 Energy Efficiency Program Evaluation
Overview
Chapter 3 Impact Evaluation Basics
Readers who want an overview of evaluation and
the key aspects of impact evaluation
Part 3
Chapter 4 Calculating Energy Savings
Chapter 5 Determining Net Energy Savings
Chapter 6 Calculating Avoided Air Emissions
Readers who want additional detail on impact
evaluation approaches ndash deemed savings MampV
large-scale consumption data analysis
Part 4
Chapter 7 Impact Evaluation Considerations
Chapter 8 Impact Evaluation Planning
Program implementers evaluators and
managersregulators of evaluations looking for
guidance on key evaluation issues and planning of
evaluations as well as readers with a background in
evaluation may want to go directly to these chapters
Part 5
Appendix A Glossary
Appendix B Other Evaluation Categories and
Approaches ndash Market Process Cost-Effectiveness
Appendix C Resources
References
Readers interested in standard energy efficiency
evaluation definitions and reference materials as
well as summaries of process market evaluations
cost-effectiveness analyses and top-down
evaluation
EXAMPLE Chapter 7 Impact Evaluation Considerations
bull First portion covers issues that often arise in impact evaluations ndash determining baselines
ndash determining demand savings
ndash calculating persistence of savings
ndash addressing uncertainty of savings estimates
ndash setting evaluation budgets
ndash establishing evaluation principles and ethics
bull Second portion provides brief overviews of ndash program feedback through evaluation
ndash resource planning through evaluation
ndash calculating non-energy benefits
ndash evaluating some relatively unique program types (residential behaviors education and training market transformation codes and standards demand response and GHG mitigation)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
13
Glossary ndash a combined effort with several groups to
create more consistency in how we describe EMampV
bull Accuracy A concept that refers to the relationship between the true value of a variable and an estimate of
the value The term can also be used in reference to a model or a set of measured data or to describe a measuring instrumentrsquos capability
bull helliphelliphelliphelliphellip
bull Incremental Annual Savings The difference between the amount of energy savings acquired or planned
to be acquired as a result of energy efficiency activities in one year and the amount of energy savings acquired or planned to be acquired as a result of the energy efficiency activities in the prior year
bull Independent Variables The explanatory factors (eg weather or occupancy) in a regression model that
are assumed to affect the variable under study (eg energy use)
bull Indirect Emissions Emissions that are a consequence of activities that take place within the organizational
boundaries of an entity but occur at sources owned or controlled by another entity For example emissions of electricity used by a manufacturing entity that occur at a power plant represent the manufacturerrsquos indirect emissions Indirect emissions are typically the source of avoided emissions for electric energy efficiency measures
bull helliphelliphelliphellip
bull Workforce Education and Training Programs Programs primarily intended for building maintenance engineers
HVAC contractors engineers architects maintenance personnel and others These programs provide information about energy efficiency concepts recommended energy-efficient technologies or behaviors andor programs that offer energy efficiency or load-reduction
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
14
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16
RDampD
bull Research
bull Development
bull Demonstration
Deployment
Voluntary implementation of projects and programs ndash outreach education and subsidies For example
bull Incenting early actors
bull Incenting consumers distributers and manufacturers
bull Mass market and individual market strategies
Transformed Markets
Standard practice
or
Codes and standards
There is a continuum of efficiency actions
Our goal is to greatly accelerate and
ldquofeedrdquo (widen) this continuum of actions
EMampV is a tool to support these actions
Evaluation Types
Evaluation Type Description Example Uses
Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)
Determines the amount of energy and demand saved
Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives
Identifies how program designs and processes can be improved
Market Effects Evaluation
Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program
Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program
Cost-Effectiveness Evaluation
Quantifies the costs of program implementation and compares them with program benefits
Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
17
Savings Cannot Be Measured - They Are Estimated
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18
Graph of Energy Consumption Before During And After Project Is Installed
Determining Energy Savings
bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as
used by the Northwest RTF
bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group
bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program
bull All or some of the above
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
19
Planning Implementing and Evaluating Efficiency Programs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
20
Evaluation is Integral to Planning-Implementation-Evaluation Process
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21
When to Evaluate
bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle
bull In timely manner with feedback for bull Ongoing program improvement
bull Supporting portfolio assessments
bull Support the planning of future portfolio cycles load forecastsenergy resource plans
bull Can also be used to inform future evaluations in particular through updating deemed savings values
Structure for Defining Evaluation Activities
bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input
bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle
bull Evaluation Research Plans ndash Created for the major EMampV activities
bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22
The Big Issues of EMampV
bull Fundamental issue of EMampV
bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty
bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
23
bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual
bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings
How good is good enough As compared to what
EMampV is About Risk Management
How Good is Good Enough amp As Compared to What
bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources
bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example
ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or
ndash Expending too many resources on ineffective actions or the opposite or
ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24
Side Note Energy Savings vs Energy Avoidance ndash
Consumer vs Regulator Perspectives
bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments
bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments
bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26
Non-Energy Benefits (NEBs)
NEB categories
bull NEBs can be categorized as those accruing to
ndash utilities (energy providers)
ndash society as a whole (such as avoided emissions)
ndash individual participants
bull Can be positive or negative (NEIs)
bull Might exceed energy benefits
Documenting NEBs
bull Measurement of benefits
bull Modeling
bull Surveys
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
27
Why Document Avoided Emissions
bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit
bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by
monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test
ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
28
Calculating Avoided Emissions
Conceptually avoided emissions are calculated using net energy savings and one of two different approaches
bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)
ndash Several sources of emission factors as well as approaches for calculating the factors
ndash Avoided emissions = (net energy savings) x (emission factor)
bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a
grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs
ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
29
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
The New Guide
bull Describes common terminology structures and approaches used for determining (evaluating)
ndash energy and demand savings
ndash avoided emissions
ndash other non-energy benefits
bull Does not recommend specific approaches - it provides
ndash context
ndash planning guidance
ndash discussion of issues
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
4 i
Energy-Efficiency Program Impact Evaluation Guide An introduction to and summary of
the practices planning and associated
issues of documenting energy savings
demand savings avoided emissions
and other non-energy benefits
resulting from end-use energy-
efficiency programs
A RESOURCE OF THE STATE AND LOCAL ENERGY EFFICIENCY ACTION NETWORK
Intended Audience
bull Primary audience ndash energy agencies and regulators
ndash public and private energy efficiency portfolio administrators such as utilities nonprofit organizations and government agencies
ndash program implementers bull And
ndash policymakers seeking general information
ndash evaluation practitioners - introductions to and summaries of evaluation topics that can be useful for explaining concepts and standard practices
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
5
Outcomes
bull Standard evaluation terminologystructures and best practices - support the adoption continuation and expansion of effective efficiency actions
bull For those just starting or ramping up their efficiency and evaluation activities ndash reduce costs for starting up an evaluation effort and moving ldquoup the learning curverdquo
bull Use of common approaches and terminology can support comparisons across different jurisdictions and facilitate ldquocross-borderrdquo energy efficiency andor greenhouse gas and other air emissions mitigation programs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
6
Expectations From Using the Guide ndash State Energy Officials
After reading this guide the reader from a State Energy Office will be able to
bull Define the basic objectives structure and evaluation approaches that can be used to plan and conduct impact evaluations of programs
bull Support and provide input for an energy efficiency evaluation framework (general guidance documents) and review impact evaluation plans and reports
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
7
The Guide supports lsquocapacity buildingrsquo and also provides input for some
forthcoming EMampV Topics
bull What is the future of EMampV for EE ndash more rigor or less rigor more consistency more deemed savings
bull How will energy savings and emissions reductions be documented for SIPsclimate programs
bull What matters net savings or gross savings just energy savings or NEBs
bull Is there a future for top-down evaluation macro data analyses
bull What is the best role of EMampV and evaluators
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
8
But First A Commercial Interruption
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 9
Jointly managed by the DOE Office of Electricity Delivery and Energy Reliability and the Office of Energy Efficiency and Renewable Energy
UNIFORM METHODS PROJECT
bull Addresses seven common residential and commercial efficiency measures
bull Step-by-step calculations for determining gross first-year savings using MampV approach
bull Also some cross cutting chapters on some EMampV topics (sampling metering persistence peak savings)
bull Adoption is voluntary bull httpumppnnlgov DOE Goals
1113089 Strengthen credibility EE savings calculations 1113089 Provide clear accessible step-by-step protocols 1113089 Support consistency and transparency 1113089 Reduce costs of EMampV 1113089 Allow for comparison of savings
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 11
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
12
Part Chapters Intended Audience
Part 1
Executive Summary
Readers interested in a brief summary and
introduction to impact evaluation
Part 2
Chapter 1 Introduction
Chapter 2 Energy Efficiency Program Evaluation
Overview
Chapter 3 Impact Evaluation Basics
Readers who want an overview of evaluation and
the key aspects of impact evaluation
Part 3
Chapter 4 Calculating Energy Savings
Chapter 5 Determining Net Energy Savings
Chapter 6 Calculating Avoided Air Emissions
Readers who want additional detail on impact
evaluation approaches ndash deemed savings MampV
large-scale consumption data analysis
Part 4
Chapter 7 Impact Evaluation Considerations
Chapter 8 Impact Evaluation Planning
Program implementers evaluators and
managersregulators of evaluations looking for
guidance on key evaluation issues and planning of
evaluations as well as readers with a background in
evaluation may want to go directly to these chapters
Part 5
Appendix A Glossary
Appendix B Other Evaluation Categories and
Approaches ndash Market Process Cost-Effectiveness
Appendix C Resources
References
Readers interested in standard energy efficiency
evaluation definitions and reference materials as
well as summaries of process market evaluations
cost-effectiveness analyses and top-down
evaluation
EXAMPLE Chapter 7 Impact Evaluation Considerations
bull First portion covers issues that often arise in impact evaluations ndash determining baselines
ndash determining demand savings
ndash calculating persistence of savings
ndash addressing uncertainty of savings estimates
ndash setting evaluation budgets
ndash establishing evaluation principles and ethics
bull Second portion provides brief overviews of ndash program feedback through evaluation
ndash resource planning through evaluation
ndash calculating non-energy benefits
ndash evaluating some relatively unique program types (residential behaviors education and training market transformation codes and standards demand response and GHG mitigation)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
13
Glossary ndash a combined effort with several groups to
create more consistency in how we describe EMampV
bull Accuracy A concept that refers to the relationship between the true value of a variable and an estimate of
the value The term can also be used in reference to a model or a set of measured data or to describe a measuring instrumentrsquos capability
bull helliphelliphelliphelliphellip
bull Incremental Annual Savings The difference between the amount of energy savings acquired or planned
to be acquired as a result of energy efficiency activities in one year and the amount of energy savings acquired or planned to be acquired as a result of the energy efficiency activities in the prior year
bull Independent Variables The explanatory factors (eg weather or occupancy) in a regression model that
are assumed to affect the variable under study (eg energy use)
bull Indirect Emissions Emissions that are a consequence of activities that take place within the organizational
boundaries of an entity but occur at sources owned or controlled by another entity For example emissions of electricity used by a manufacturing entity that occur at a power plant represent the manufacturerrsquos indirect emissions Indirect emissions are typically the source of avoided emissions for electric energy efficiency measures
bull helliphelliphelliphellip
bull Workforce Education and Training Programs Programs primarily intended for building maintenance engineers
HVAC contractors engineers architects maintenance personnel and others These programs provide information about energy efficiency concepts recommended energy-efficient technologies or behaviors andor programs that offer energy efficiency or load-reduction
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
14
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16
RDampD
bull Research
bull Development
bull Demonstration
Deployment
Voluntary implementation of projects and programs ndash outreach education and subsidies For example
bull Incenting early actors
bull Incenting consumers distributers and manufacturers
bull Mass market and individual market strategies
Transformed Markets
Standard practice
or
Codes and standards
There is a continuum of efficiency actions
Our goal is to greatly accelerate and
ldquofeedrdquo (widen) this continuum of actions
EMampV is a tool to support these actions
Evaluation Types
Evaluation Type Description Example Uses
Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)
Determines the amount of energy and demand saved
Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives
Identifies how program designs and processes can be improved
Market Effects Evaluation
Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program
Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program
Cost-Effectiveness Evaluation
Quantifies the costs of program implementation and compares them with program benefits
Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
17
Savings Cannot Be Measured - They Are Estimated
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18
Graph of Energy Consumption Before During And After Project Is Installed
Determining Energy Savings
bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as
used by the Northwest RTF
bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group
bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program
bull All or some of the above
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
19
Planning Implementing and Evaluating Efficiency Programs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
20
Evaluation is Integral to Planning-Implementation-Evaluation Process
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21
When to Evaluate
bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle
bull In timely manner with feedback for bull Ongoing program improvement
bull Supporting portfolio assessments
bull Support the planning of future portfolio cycles load forecastsenergy resource plans
bull Can also be used to inform future evaluations in particular through updating deemed savings values
Structure for Defining Evaluation Activities
bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input
bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle
bull Evaluation Research Plans ndash Created for the major EMampV activities
bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22
The Big Issues of EMampV
bull Fundamental issue of EMampV
bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty
bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
23
bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual
bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings
How good is good enough As compared to what
EMampV is About Risk Management
How Good is Good Enough amp As Compared to What
bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources
bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example
ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or
ndash Expending too many resources on ineffective actions or the opposite or
ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24
Side Note Energy Savings vs Energy Avoidance ndash
Consumer vs Regulator Perspectives
bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments
bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments
bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26
Non-Energy Benefits (NEBs)
NEB categories
bull NEBs can be categorized as those accruing to
ndash utilities (energy providers)
ndash society as a whole (such as avoided emissions)
ndash individual participants
bull Can be positive or negative (NEIs)
bull Might exceed energy benefits
Documenting NEBs
bull Measurement of benefits
bull Modeling
bull Surveys
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
27
Why Document Avoided Emissions
bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit
bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by
monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test
ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
28
Calculating Avoided Emissions
Conceptually avoided emissions are calculated using net energy savings and one of two different approaches
bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)
ndash Several sources of emission factors as well as approaches for calculating the factors
ndash Avoided emissions = (net energy savings) x (emission factor)
bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a
grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs
ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
29
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
Intended Audience
bull Primary audience ndash energy agencies and regulators
ndash public and private energy efficiency portfolio administrators such as utilities nonprofit organizations and government agencies
ndash program implementers bull And
ndash policymakers seeking general information
ndash evaluation practitioners - introductions to and summaries of evaluation topics that can be useful for explaining concepts and standard practices
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
5
Outcomes
bull Standard evaluation terminologystructures and best practices - support the adoption continuation and expansion of effective efficiency actions
bull For those just starting or ramping up their efficiency and evaluation activities ndash reduce costs for starting up an evaluation effort and moving ldquoup the learning curverdquo
bull Use of common approaches and terminology can support comparisons across different jurisdictions and facilitate ldquocross-borderrdquo energy efficiency andor greenhouse gas and other air emissions mitigation programs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
6
Expectations From Using the Guide ndash State Energy Officials
After reading this guide the reader from a State Energy Office will be able to
bull Define the basic objectives structure and evaluation approaches that can be used to plan and conduct impact evaluations of programs
bull Support and provide input for an energy efficiency evaluation framework (general guidance documents) and review impact evaluation plans and reports
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
7
The Guide supports lsquocapacity buildingrsquo and also provides input for some
forthcoming EMampV Topics
bull What is the future of EMampV for EE ndash more rigor or less rigor more consistency more deemed savings
bull How will energy savings and emissions reductions be documented for SIPsclimate programs
bull What matters net savings or gross savings just energy savings or NEBs
bull Is there a future for top-down evaluation macro data analyses
bull What is the best role of EMampV and evaluators
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
8
But First A Commercial Interruption
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 9
Jointly managed by the DOE Office of Electricity Delivery and Energy Reliability and the Office of Energy Efficiency and Renewable Energy
UNIFORM METHODS PROJECT
bull Addresses seven common residential and commercial efficiency measures
bull Step-by-step calculations for determining gross first-year savings using MampV approach
bull Also some cross cutting chapters on some EMampV topics (sampling metering persistence peak savings)
bull Adoption is voluntary bull httpumppnnlgov DOE Goals
1113089 Strengthen credibility EE savings calculations 1113089 Provide clear accessible step-by-step protocols 1113089 Support consistency and transparency 1113089 Reduce costs of EMampV 1113089 Allow for comparison of savings
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 11
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
12
Part Chapters Intended Audience
Part 1
Executive Summary
Readers interested in a brief summary and
introduction to impact evaluation
Part 2
Chapter 1 Introduction
Chapter 2 Energy Efficiency Program Evaluation
Overview
Chapter 3 Impact Evaluation Basics
Readers who want an overview of evaluation and
the key aspects of impact evaluation
Part 3
Chapter 4 Calculating Energy Savings
Chapter 5 Determining Net Energy Savings
Chapter 6 Calculating Avoided Air Emissions
Readers who want additional detail on impact
evaluation approaches ndash deemed savings MampV
large-scale consumption data analysis
Part 4
Chapter 7 Impact Evaluation Considerations
Chapter 8 Impact Evaluation Planning
Program implementers evaluators and
managersregulators of evaluations looking for
guidance on key evaluation issues and planning of
evaluations as well as readers with a background in
evaluation may want to go directly to these chapters
Part 5
Appendix A Glossary
Appendix B Other Evaluation Categories and
Approaches ndash Market Process Cost-Effectiveness
Appendix C Resources
References
Readers interested in standard energy efficiency
evaluation definitions and reference materials as
well as summaries of process market evaluations
cost-effectiveness analyses and top-down
evaluation
EXAMPLE Chapter 7 Impact Evaluation Considerations
bull First portion covers issues that often arise in impact evaluations ndash determining baselines
ndash determining demand savings
ndash calculating persistence of savings
ndash addressing uncertainty of savings estimates
ndash setting evaluation budgets
ndash establishing evaluation principles and ethics
bull Second portion provides brief overviews of ndash program feedback through evaluation
ndash resource planning through evaluation
ndash calculating non-energy benefits
ndash evaluating some relatively unique program types (residential behaviors education and training market transformation codes and standards demand response and GHG mitigation)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
13
Glossary ndash a combined effort with several groups to
create more consistency in how we describe EMampV
bull Accuracy A concept that refers to the relationship between the true value of a variable and an estimate of
the value The term can also be used in reference to a model or a set of measured data or to describe a measuring instrumentrsquos capability
bull helliphelliphelliphelliphellip
bull Incremental Annual Savings The difference between the amount of energy savings acquired or planned
to be acquired as a result of energy efficiency activities in one year and the amount of energy savings acquired or planned to be acquired as a result of the energy efficiency activities in the prior year
bull Independent Variables The explanatory factors (eg weather or occupancy) in a regression model that
are assumed to affect the variable under study (eg energy use)
bull Indirect Emissions Emissions that are a consequence of activities that take place within the organizational
boundaries of an entity but occur at sources owned or controlled by another entity For example emissions of electricity used by a manufacturing entity that occur at a power plant represent the manufacturerrsquos indirect emissions Indirect emissions are typically the source of avoided emissions for electric energy efficiency measures
bull helliphelliphelliphellip
bull Workforce Education and Training Programs Programs primarily intended for building maintenance engineers
HVAC contractors engineers architects maintenance personnel and others These programs provide information about energy efficiency concepts recommended energy-efficient technologies or behaviors andor programs that offer energy efficiency or load-reduction
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
14
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16
RDampD
bull Research
bull Development
bull Demonstration
Deployment
Voluntary implementation of projects and programs ndash outreach education and subsidies For example
bull Incenting early actors
bull Incenting consumers distributers and manufacturers
bull Mass market and individual market strategies
Transformed Markets
Standard practice
or
Codes and standards
There is a continuum of efficiency actions
Our goal is to greatly accelerate and
ldquofeedrdquo (widen) this continuum of actions
EMampV is a tool to support these actions
Evaluation Types
Evaluation Type Description Example Uses
Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)
Determines the amount of energy and demand saved
Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives
Identifies how program designs and processes can be improved
Market Effects Evaluation
Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program
Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program
Cost-Effectiveness Evaluation
Quantifies the costs of program implementation and compares them with program benefits
Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
17
Savings Cannot Be Measured - They Are Estimated
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18
Graph of Energy Consumption Before During And After Project Is Installed
Determining Energy Savings
bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as
used by the Northwest RTF
bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group
bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program
bull All or some of the above
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
19
Planning Implementing and Evaluating Efficiency Programs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
20
Evaluation is Integral to Planning-Implementation-Evaluation Process
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21
When to Evaluate
bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle
bull In timely manner with feedback for bull Ongoing program improvement
bull Supporting portfolio assessments
bull Support the planning of future portfolio cycles load forecastsenergy resource plans
bull Can also be used to inform future evaluations in particular through updating deemed savings values
Structure for Defining Evaluation Activities
bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input
bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle
bull Evaluation Research Plans ndash Created for the major EMampV activities
bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22
The Big Issues of EMampV
bull Fundamental issue of EMampV
bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty
bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
23
bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual
bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings
How good is good enough As compared to what
EMampV is About Risk Management
How Good is Good Enough amp As Compared to What
bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources
bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example
ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or
ndash Expending too many resources on ineffective actions or the opposite or
ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24
Side Note Energy Savings vs Energy Avoidance ndash
Consumer vs Regulator Perspectives
bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments
bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments
bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26
Non-Energy Benefits (NEBs)
NEB categories
bull NEBs can be categorized as those accruing to
ndash utilities (energy providers)
ndash society as a whole (such as avoided emissions)
ndash individual participants
bull Can be positive or negative (NEIs)
bull Might exceed energy benefits
Documenting NEBs
bull Measurement of benefits
bull Modeling
bull Surveys
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
27
Why Document Avoided Emissions
bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit
bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by
monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test
ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
28
Calculating Avoided Emissions
Conceptually avoided emissions are calculated using net energy savings and one of two different approaches
bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)
ndash Several sources of emission factors as well as approaches for calculating the factors
ndash Avoided emissions = (net energy savings) x (emission factor)
bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a
grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs
ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
29
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
Outcomes
bull Standard evaluation terminologystructures and best practices - support the adoption continuation and expansion of effective efficiency actions
bull For those just starting or ramping up their efficiency and evaluation activities ndash reduce costs for starting up an evaluation effort and moving ldquoup the learning curverdquo
bull Use of common approaches and terminology can support comparisons across different jurisdictions and facilitate ldquocross-borderrdquo energy efficiency andor greenhouse gas and other air emissions mitigation programs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
6
Expectations From Using the Guide ndash State Energy Officials
After reading this guide the reader from a State Energy Office will be able to
bull Define the basic objectives structure and evaluation approaches that can be used to plan and conduct impact evaluations of programs
bull Support and provide input for an energy efficiency evaluation framework (general guidance documents) and review impact evaluation plans and reports
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
7
The Guide supports lsquocapacity buildingrsquo and also provides input for some
forthcoming EMampV Topics
bull What is the future of EMampV for EE ndash more rigor or less rigor more consistency more deemed savings
bull How will energy savings and emissions reductions be documented for SIPsclimate programs
bull What matters net savings or gross savings just energy savings or NEBs
bull Is there a future for top-down evaluation macro data analyses
bull What is the best role of EMampV and evaluators
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
8
But First A Commercial Interruption
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 9
Jointly managed by the DOE Office of Electricity Delivery and Energy Reliability and the Office of Energy Efficiency and Renewable Energy
UNIFORM METHODS PROJECT
bull Addresses seven common residential and commercial efficiency measures
bull Step-by-step calculations for determining gross first-year savings using MampV approach
bull Also some cross cutting chapters on some EMampV topics (sampling metering persistence peak savings)
bull Adoption is voluntary bull httpumppnnlgov DOE Goals
1113089 Strengthen credibility EE savings calculations 1113089 Provide clear accessible step-by-step protocols 1113089 Support consistency and transparency 1113089 Reduce costs of EMampV 1113089 Allow for comparison of savings
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 11
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
12
Part Chapters Intended Audience
Part 1
Executive Summary
Readers interested in a brief summary and
introduction to impact evaluation
Part 2
Chapter 1 Introduction
Chapter 2 Energy Efficiency Program Evaluation
Overview
Chapter 3 Impact Evaluation Basics
Readers who want an overview of evaluation and
the key aspects of impact evaluation
Part 3
Chapter 4 Calculating Energy Savings
Chapter 5 Determining Net Energy Savings
Chapter 6 Calculating Avoided Air Emissions
Readers who want additional detail on impact
evaluation approaches ndash deemed savings MampV
large-scale consumption data analysis
Part 4
Chapter 7 Impact Evaluation Considerations
Chapter 8 Impact Evaluation Planning
Program implementers evaluators and
managersregulators of evaluations looking for
guidance on key evaluation issues and planning of
evaluations as well as readers with a background in
evaluation may want to go directly to these chapters
Part 5
Appendix A Glossary
Appendix B Other Evaluation Categories and
Approaches ndash Market Process Cost-Effectiveness
Appendix C Resources
References
Readers interested in standard energy efficiency
evaluation definitions and reference materials as
well as summaries of process market evaluations
cost-effectiveness analyses and top-down
evaluation
EXAMPLE Chapter 7 Impact Evaluation Considerations
bull First portion covers issues that often arise in impact evaluations ndash determining baselines
ndash determining demand savings
ndash calculating persistence of savings
ndash addressing uncertainty of savings estimates
ndash setting evaluation budgets
ndash establishing evaluation principles and ethics
bull Second portion provides brief overviews of ndash program feedback through evaluation
ndash resource planning through evaluation
ndash calculating non-energy benefits
ndash evaluating some relatively unique program types (residential behaviors education and training market transformation codes and standards demand response and GHG mitigation)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
13
Glossary ndash a combined effort with several groups to
create more consistency in how we describe EMampV
bull Accuracy A concept that refers to the relationship between the true value of a variable and an estimate of
the value The term can also be used in reference to a model or a set of measured data or to describe a measuring instrumentrsquos capability
bull helliphelliphelliphelliphellip
bull Incremental Annual Savings The difference between the amount of energy savings acquired or planned
to be acquired as a result of energy efficiency activities in one year and the amount of energy savings acquired or planned to be acquired as a result of the energy efficiency activities in the prior year
bull Independent Variables The explanatory factors (eg weather or occupancy) in a regression model that
are assumed to affect the variable under study (eg energy use)
bull Indirect Emissions Emissions that are a consequence of activities that take place within the organizational
boundaries of an entity but occur at sources owned or controlled by another entity For example emissions of electricity used by a manufacturing entity that occur at a power plant represent the manufacturerrsquos indirect emissions Indirect emissions are typically the source of avoided emissions for electric energy efficiency measures
bull helliphelliphelliphellip
bull Workforce Education and Training Programs Programs primarily intended for building maintenance engineers
HVAC contractors engineers architects maintenance personnel and others These programs provide information about energy efficiency concepts recommended energy-efficient technologies or behaviors andor programs that offer energy efficiency or load-reduction
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
14
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16
RDampD
bull Research
bull Development
bull Demonstration
Deployment
Voluntary implementation of projects and programs ndash outreach education and subsidies For example
bull Incenting early actors
bull Incenting consumers distributers and manufacturers
bull Mass market and individual market strategies
Transformed Markets
Standard practice
or
Codes and standards
There is a continuum of efficiency actions
Our goal is to greatly accelerate and
ldquofeedrdquo (widen) this continuum of actions
EMampV is a tool to support these actions
Evaluation Types
Evaluation Type Description Example Uses
Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)
Determines the amount of energy and demand saved
Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives
Identifies how program designs and processes can be improved
Market Effects Evaluation
Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program
Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program
Cost-Effectiveness Evaluation
Quantifies the costs of program implementation and compares them with program benefits
Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
17
Savings Cannot Be Measured - They Are Estimated
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18
Graph of Energy Consumption Before During And After Project Is Installed
Determining Energy Savings
bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as
used by the Northwest RTF
bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group
bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program
bull All or some of the above
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
19
Planning Implementing and Evaluating Efficiency Programs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
20
Evaluation is Integral to Planning-Implementation-Evaluation Process
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21
When to Evaluate
bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle
bull In timely manner with feedback for bull Ongoing program improvement
bull Supporting portfolio assessments
bull Support the planning of future portfolio cycles load forecastsenergy resource plans
bull Can also be used to inform future evaluations in particular through updating deemed savings values
Structure for Defining Evaluation Activities
bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input
bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle
bull Evaluation Research Plans ndash Created for the major EMampV activities
bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22
The Big Issues of EMampV
bull Fundamental issue of EMampV
bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty
bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
23
bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual
bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings
How good is good enough As compared to what
EMampV is About Risk Management
How Good is Good Enough amp As Compared to What
bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources
bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example
ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or
ndash Expending too many resources on ineffective actions or the opposite or
ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24
Side Note Energy Savings vs Energy Avoidance ndash
Consumer vs Regulator Perspectives
bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments
bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments
bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26
Non-Energy Benefits (NEBs)
NEB categories
bull NEBs can be categorized as those accruing to
ndash utilities (energy providers)
ndash society as a whole (such as avoided emissions)
ndash individual participants
bull Can be positive or negative (NEIs)
bull Might exceed energy benefits
Documenting NEBs
bull Measurement of benefits
bull Modeling
bull Surveys
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
27
Why Document Avoided Emissions
bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit
bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by
monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test
ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
28
Calculating Avoided Emissions
Conceptually avoided emissions are calculated using net energy savings and one of two different approaches
bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)
ndash Several sources of emission factors as well as approaches for calculating the factors
ndash Avoided emissions = (net energy savings) x (emission factor)
bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a
grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs
ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
29
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
Expectations From Using the Guide ndash State Energy Officials
After reading this guide the reader from a State Energy Office will be able to
bull Define the basic objectives structure and evaluation approaches that can be used to plan and conduct impact evaluations of programs
bull Support and provide input for an energy efficiency evaluation framework (general guidance documents) and review impact evaluation plans and reports
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
7
The Guide supports lsquocapacity buildingrsquo and also provides input for some
forthcoming EMampV Topics
bull What is the future of EMampV for EE ndash more rigor or less rigor more consistency more deemed savings
bull How will energy savings and emissions reductions be documented for SIPsclimate programs
bull What matters net savings or gross savings just energy savings or NEBs
bull Is there a future for top-down evaluation macro data analyses
bull What is the best role of EMampV and evaluators
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
8
But First A Commercial Interruption
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 9
Jointly managed by the DOE Office of Electricity Delivery and Energy Reliability and the Office of Energy Efficiency and Renewable Energy
UNIFORM METHODS PROJECT
bull Addresses seven common residential and commercial efficiency measures
bull Step-by-step calculations for determining gross first-year savings using MampV approach
bull Also some cross cutting chapters on some EMampV topics (sampling metering persistence peak savings)
bull Adoption is voluntary bull httpumppnnlgov DOE Goals
1113089 Strengthen credibility EE savings calculations 1113089 Provide clear accessible step-by-step protocols 1113089 Support consistency and transparency 1113089 Reduce costs of EMampV 1113089 Allow for comparison of savings
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 11
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
12
Part Chapters Intended Audience
Part 1
Executive Summary
Readers interested in a brief summary and
introduction to impact evaluation
Part 2
Chapter 1 Introduction
Chapter 2 Energy Efficiency Program Evaluation
Overview
Chapter 3 Impact Evaluation Basics
Readers who want an overview of evaluation and
the key aspects of impact evaluation
Part 3
Chapter 4 Calculating Energy Savings
Chapter 5 Determining Net Energy Savings
Chapter 6 Calculating Avoided Air Emissions
Readers who want additional detail on impact
evaluation approaches ndash deemed savings MampV
large-scale consumption data analysis
Part 4
Chapter 7 Impact Evaluation Considerations
Chapter 8 Impact Evaluation Planning
Program implementers evaluators and
managersregulators of evaluations looking for
guidance on key evaluation issues and planning of
evaluations as well as readers with a background in
evaluation may want to go directly to these chapters
Part 5
Appendix A Glossary
Appendix B Other Evaluation Categories and
Approaches ndash Market Process Cost-Effectiveness
Appendix C Resources
References
Readers interested in standard energy efficiency
evaluation definitions and reference materials as
well as summaries of process market evaluations
cost-effectiveness analyses and top-down
evaluation
EXAMPLE Chapter 7 Impact Evaluation Considerations
bull First portion covers issues that often arise in impact evaluations ndash determining baselines
ndash determining demand savings
ndash calculating persistence of savings
ndash addressing uncertainty of savings estimates
ndash setting evaluation budgets
ndash establishing evaluation principles and ethics
bull Second portion provides brief overviews of ndash program feedback through evaluation
ndash resource planning through evaluation
ndash calculating non-energy benefits
ndash evaluating some relatively unique program types (residential behaviors education and training market transformation codes and standards demand response and GHG mitigation)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
13
Glossary ndash a combined effort with several groups to
create more consistency in how we describe EMampV
bull Accuracy A concept that refers to the relationship between the true value of a variable and an estimate of
the value The term can also be used in reference to a model or a set of measured data or to describe a measuring instrumentrsquos capability
bull helliphelliphelliphelliphellip
bull Incremental Annual Savings The difference between the amount of energy savings acquired or planned
to be acquired as a result of energy efficiency activities in one year and the amount of energy savings acquired or planned to be acquired as a result of the energy efficiency activities in the prior year
bull Independent Variables The explanatory factors (eg weather or occupancy) in a regression model that
are assumed to affect the variable under study (eg energy use)
bull Indirect Emissions Emissions that are a consequence of activities that take place within the organizational
boundaries of an entity but occur at sources owned or controlled by another entity For example emissions of electricity used by a manufacturing entity that occur at a power plant represent the manufacturerrsquos indirect emissions Indirect emissions are typically the source of avoided emissions for electric energy efficiency measures
bull helliphelliphelliphellip
bull Workforce Education and Training Programs Programs primarily intended for building maintenance engineers
HVAC contractors engineers architects maintenance personnel and others These programs provide information about energy efficiency concepts recommended energy-efficient technologies or behaviors andor programs that offer energy efficiency or load-reduction
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
14
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16
RDampD
bull Research
bull Development
bull Demonstration
Deployment
Voluntary implementation of projects and programs ndash outreach education and subsidies For example
bull Incenting early actors
bull Incenting consumers distributers and manufacturers
bull Mass market and individual market strategies
Transformed Markets
Standard practice
or
Codes and standards
There is a continuum of efficiency actions
Our goal is to greatly accelerate and
ldquofeedrdquo (widen) this continuum of actions
EMampV is a tool to support these actions
Evaluation Types
Evaluation Type Description Example Uses
Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)
Determines the amount of energy and demand saved
Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives
Identifies how program designs and processes can be improved
Market Effects Evaluation
Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program
Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program
Cost-Effectiveness Evaluation
Quantifies the costs of program implementation and compares them with program benefits
Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
17
Savings Cannot Be Measured - They Are Estimated
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18
Graph of Energy Consumption Before During And After Project Is Installed
Determining Energy Savings
bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as
used by the Northwest RTF
bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group
bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program
bull All or some of the above
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
19
Planning Implementing and Evaluating Efficiency Programs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
20
Evaluation is Integral to Planning-Implementation-Evaluation Process
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21
When to Evaluate
bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle
bull In timely manner with feedback for bull Ongoing program improvement
bull Supporting portfolio assessments
bull Support the planning of future portfolio cycles load forecastsenergy resource plans
bull Can also be used to inform future evaluations in particular through updating deemed savings values
Structure for Defining Evaluation Activities
bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input
bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle
bull Evaluation Research Plans ndash Created for the major EMampV activities
bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22
The Big Issues of EMampV
bull Fundamental issue of EMampV
bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty
bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
23
bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual
bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings
How good is good enough As compared to what
EMampV is About Risk Management
How Good is Good Enough amp As Compared to What
bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources
bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example
ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or
ndash Expending too many resources on ineffective actions or the opposite or
ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24
Side Note Energy Savings vs Energy Avoidance ndash
Consumer vs Regulator Perspectives
bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments
bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments
bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26
Non-Energy Benefits (NEBs)
NEB categories
bull NEBs can be categorized as those accruing to
ndash utilities (energy providers)
ndash society as a whole (such as avoided emissions)
ndash individual participants
bull Can be positive or negative (NEIs)
bull Might exceed energy benefits
Documenting NEBs
bull Measurement of benefits
bull Modeling
bull Surveys
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
27
Why Document Avoided Emissions
bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit
bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by
monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test
ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
28
Calculating Avoided Emissions
Conceptually avoided emissions are calculated using net energy savings and one of two different approaches
bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)
ndash Several sources of emission factors as well as approaches for calculating the factors
ndash Avoided emissions = (net energy savings) x (emission factor)
bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a
grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs
ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
29
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
The Guide supports lsquocapacity buildingrsquo and also provides input for some
forthcoming EMampV Topics
bull What is the future of EMampV for EE ndash more rigor or less rigor more consistency more deemed savings
bull How will energy savings and emissions reductions be documented for SIPsclimate programs
bull What matters net savings or gross savings just energy savings or NEBs
bull Is there a future for top-down evaluation macro data analyses
bull What is the best role of EMampV and evaluators
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
8
But First A Commercial Interruption
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 9
Jointly managed by the DOE Office of Electricity Delivery and Energy Reliability and the Office of Energy Efficiency and Renewable Energy
UNIFORM METHODS PROJECT
bull Addresses seven common residential and commercial efficiency measures
bull Step-by-step calculations for determining gross first-year savings using MampV approach
bull Also some cross cutting chapters on some EMampV topics (sampling metering persistence peak savings)
bull Adoption is voluntary bull httpumppnnlgov DOE Goals
1113089 Strengthen credibility EE savings calculations 1113089 Provide clear accessible step-by-step protocols 1113089 Support consistency and transparency 1113089 Reduce costs of EMampV 1113089 Allow for comparison of savings
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 11
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
12
Part Chapters Intended Audience
Part 1
Executive Summary
Readers interested in a brief summary and
introduction to impact evaluation
Part 2
Chapter 1 Introduction
Chapter 2 Energy Efficiency Program Evaluation
Overview
Chapter 3 Impact Evaluation Basics
Readers who want an overview of evaluation and
the key aspects of impact evaluation
Part 3
Chapter 4 Calculating Energy Savings
Chapter 5 Determining Net Energy Savings
Chapter 6 Calculating Avoided Air Emissions
Readers who want additional detail on impact
evaluation approaches ndash deemed savings MampV
large-scale consumption data analysis
Part 4
Chapter 7 Impact Evaluation Considerations
Chapter 8 Impact Evaluation Planning
Program implementers evaluators and
managersregulators of evaluations looking for
guidance on key evaluation issues and planning of
evaluations as well as readers with a background in
evaluation may want to go directly to these chapters
Part 5
Appendix A Glossary
Appendix B Other Evaluation Categories and
Approaches ndash Market Process Cost-Effectiveness
Appendix C Resources
References
Readers interested in standard energy efficiency
evaluation definitions and reference materials as
well as summaries of process market evaluations
cost-effectiveness analyses and top-down
evaluation
EXAMPLE Chapter 7 Impact Evaluation Considerations
bull First portion covers issues that often arise in impact evaluations ndash determining baselines
ndash determining demand savings
ndash calculating persistence of savings
ndash addressing uncertainty of savings estimates
ndash setting evaluation budgets
ndash establishing evaluation principles and ethics
bull Second portion provides brief overviews of ndash program feedback through evaluation
ndash resource planning through evaluation
ndash calculating non-energy benefits
ndash evaluating some relatively unique program types (residential behaviors education and training market transformation codes and standards demand response and GHG mitigation)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
13
Glossary ndash a combined effort with several groups to
create more consistency in how we describe EMampV
bull Accuracy A concept that refers to the relationship between the true value of a variable and an estimate of
the value The term can also be used in reference to a model or a set of measured data or to describe a measuring instrumentrsquos capability
bull helliphelliphelliphelliphellip
bull Incremental Annual Savings The difference between the amount of energy savings acquired or planned
to be acquired as a result of energy efficiency activities in one year and the amount of energy savings acquired or planned to be acquired as a result of the energy efficiency activities in the prior year
bull Independent Variables The explanatory factors (eg weather or occupancy) in a regression model that
are assumed to affect the variable under study (eg energy use)
bull Indirect Emissions Emissions that are a consequence of activities that take place within the organizational
boundaries of an entity but occur at sources owned or controlled by another entity For example emissions of electricity used by a manufacturing entity that occur at a power plant represent the manufacturerrsquos indirect emissions Indirect emissions are typically the source of avoided emissions for electric energy efficiency measures
bull helliphelliphelliphellip
bull Workforce Education and Training Programs Programs primarily intended for building maintenance engineers
HVAC contractors engineers architects maintenance personnel and others These programs provide information about energy efficiency concepts recommended energy-efficient technologies or behaviors andor programs that offer energy efficiency or load-reduction
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
14
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16
RDampD
bull Research
bull Development
bull Demonstration
Deployment
Voluntary implementation of projects and programs ndash outreach education and subsidies For example
bull Incenting early actors
bull Incenting consumers distributers and manufacturers
bull Mass market and individual market strategies
Transformed Markets
Standard practice
or
Codes and standards
There is a continuum of efficiency actions
Our goal is to greatly accelerate and
ldquofeedrdquo (widen) this continuum of actions
EMampV is a tool to support these actions
Evaluation Types
Evaluation Type Description Example Uses
Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)
Determines the amount of energy and demand saved
Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives
Identifies how program designs and processes can be improved
Market Effects Evaluation
Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program
Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program
Cost-Effectiveness Evaluation
Quantifies the costs of program implementation and compares them with program benefits
Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
17
Savings Cannot Be Measured - They Are Estimated
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18
Graph of Energy Consumption Before During And After Project Is Installed
Determining Energy Savings
bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as
used by the Northwest RTF
bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group
bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program
bull All or some of the above
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
19
Planning Implementing and Evaluating Efficiency Programs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
20
Evaluation is Integral to Planning-Implementation-Evaluation Process
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21
When to Evaluate
bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle
bull In timely manner with feedback for bull Ongoing program improvement
bull Supporting portfolio assessments
bull Support the planning of future portfolio cycles load forecastsenergy resource plans
bull Can also be used to inform future evaluations in particular through updating deemed savings values
Structure for Defining Evaluation Activities
bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input
bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle
bull Evaluation Research Plans ndash Created for the major EMampV activities
bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22
The Big Issues of EMampV
bull Fundamental issue of EMampV
bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty
bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
23
bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual
bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings
How good is good enough As compared to what
EMampV is About Risk Management
How Good is Good Enough amp As Compared to What
bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources
bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example
ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or
ndash Expending too many resources on ineffective actions or the opposite or
ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24
Side Note Energy Savings vs Energy Avoidance ndash
Consumer vs Regulator Perspectives
bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments
bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments
bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26
Non-Energy Benefits (NEBs)
NEB categories
bull NEBs can be categorized as those accruing to
ndash utilities (energy providers)
ndash society as a whole (such as avoided emissions)
ndash individual participants
bull Can be positive or negative (NEIs)
bull Might exceed energy benefits
Documenting NEBs
bull Measurement of benefits
bull Modeling
bull Surveys
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
27
Why Document Avoided Emissions
bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit
bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by
monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test
ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
28
Calculating Avoided Emissions
Conceptually avoided emissions are calculated using net energy savings and one of two different approaches
bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)
ndash Several sources of emission factors as well as approaches for calculating the factors
ndash Avoided emissions = (net energy savings) x (emission factor)
bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a
grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs
ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
29
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
But First A Commercial Interruption
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 9
Jointly managed by the DOE Office of Electricity Delivery and Energy Reliability and the Office of Energy Efficiency and Renewable Energy
UNIFORM METHODS PROJECT
bull Addresses seven common residential and commercial efficiency measures
bull Step-by-step calculations for determining gross first-year savings using MampV approach
bull Also some cross cutting chapters on some EMampV topics (sampling metering persistence peak savings)
bull Adoption is voluntary bull httpumppnnlgov DOE Goals
1113089 Strengthen credibility EE savings calculations 1113089 Provide clear accessible step-by-step protocols 1113089 Support consistency and transparency 1113089 Reduce costs of EMampV 1113089 Allow for comparison of savings
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 11
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
12
Part Chapters Intended Audience
Part 1
Executive Summary
Readers interested in a brief summary and
introduction to impact evaluation
Part 2
Chapter 1 Introduction
Chapter 2 Energy Efficiency Program Evaluation
Overview
Chapter 3 Impact Evaluation Basics
Readers who want an overview of evaluation and
the key aspects of impact evaluation
Part 3
Chapter 4 Calculating Energy Savings
Chapter 5 Determining Net Energy Savings
Chapter 6 Calculating Avoided Air Emissions
Readers who want additional detail on impact
evaluation approaches ndash deemed savings MampV
large-scale consumption data analysis
Part 4
Chapter 7 Impact Evaluation Considerations
Chapter 8 Impact Evaluation Planning
Program implementers evaluators and
managersregulators of evaluations looking for
guidance on key evaluation issues and planning of
evaluations as well as readers with a background in
evaluation may want to go directly to these chapters
Part 5
Appendix A Glossary
Appendix B Other Evaluation Categories and
Approaches ndash Market Process Cost-Effectiveness
Appendix C Resources
References
Readers interested in standard energy efficiency
evaluation definitions and reference materials as
well as summaries of process market evaluations
cost-effectiveness analyses and top-down
evaluation
EXAMPLE Chapter 7 Impact Evaluation Considerations
bull First portion covers issues that often arise in impact evaluations ndash determining baselines
ndash determining demand savings
ndash calculating persistence of savings
ndash addressing uncertainty of savings estimates
ndash setting evaluation budgets
ndash establishing evaluation principles and ethics
bull Second portion provides brief overviews of ndash program feedback through evaluation
ndash resource planning through evaluation
ndash calculating non-energy benefits
ndash evaluating some relatively unique program types (residential behaviors education and training market transformation codes and standards demand response and GHG mitigation)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
13
Glossary ndash a combined effort with several groups to
create more consistency in how we describe EMampV
bull Accuracy A concept that refers to the relationship between the true value of a variable and an estimate of
the value The term can also be used in reference to a model or a set of measured data or to describe a measuring instrumentrsquos capability
bull helliphelliphelliphelliphellip
bull Incremental Annual Savings The difference between the amount of energy savings acquired or planned
to be acquired as a result of energy efficiency activities in one year and the amount of energy savings acquired or planned to be acquired as a result of the energy efficiency activities in the prior year
bull Independent Variables The explanatory factors (eg weather or occupancy) in a regression model that
are assumed to affect the variable under study (eg energy use)
bull Indirect Emissions Emissions that are a consequence of activities that take place within the organizational
boundaries of an entity but occur at sources owned or controlled by another entity For example emissions of electricity used by a manufacturing entity that occur at a power plant represent the manufacturerrsquos indirect emissions Indirect emissions are typically the source of avoided emissions for electric energy efficiency measures
bull helliphelliphelliphellip
bull Workforce Education and Training Programs Programs primarily intended for building maintenance engineers
HVAC contractors engineers architects maintenance personnel and others These programs provide information about energy efficiency concepts recommended energy-efficient technologies or behaviors andor programs that offer energy efficiency or load-reduction
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
14
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16
RDampD
bull Research
bull Development
bull Demonstration
Deployment
Voluntary implementation of projects and programs ndash outreach education and subsidies For example
bull Incenting early actors
bull Incenting consumers distributers and manufacturers
bull Mass market and individual market strategies
Transformed Markets
Standard practice
or
Codes and standards
There is a continuum of efficiency actions
Our goal is to greatly accelerate and
ldquofeedrdquo (widen) this continuum of actions
EMampV is a tool to support these actions
Evaluation Types
Evaluation Type Description Example Uses
Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)
Determines the amount of energy and demand saved
Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives
Identifies how program designs and processes can be improved
Market Effects Evaluation
Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program
Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program
Cost-Effectiveness Evaluation
Quantifies the costs of program implementation and compares them with program benefits
Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
17
Savings Cannot Be Measured - They Are Estimated
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18
Graph of Energy Consumption Before During And After Project Is Installed
Determining Energy Savings
bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as
used by the Northwest RTF
bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group
bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program
bull All or some of the above
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
19
Planning Implementing and Evaluating Efficiency Programs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
20
Evaluation is Integral to Planning-Implementation-Evaluation Process
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21
When to Evaluate
bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle
bull In timely manner with feedback for bull Ongoing program improvement
bull Supporting portfolio assessments
bull Support the planning of future portfolio cycles load forecastsenergy resource plans
bull Can also be used to inform future evaluations in particular through updating deemed savings values
Structure for Defining Evaluation Activities
bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input
bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle
bull Evaluation Research Plans ndash Created for the major EMampV activities
bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22
The Big Issues of EMampV
bull Fundamental issue of EMampV
bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty
bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
23
bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual
bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings
How good is good enough As compared to what
EMampV is About Risk Management
How Good is Good Enough amp As Compared to What
bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources
bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example
ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or
ndash Expending too many resources on ineffective actions or the opposite or
ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24
Side Note Energy Savings vs Energy Avoidance ndash
Consumer vs Regulator Perspectives
bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments
bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments
bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26
Non-Energy Benefits (NEBs)
NEB categories
bull NEBs can be categorized as those accruing to
ndash utilities (energy providers)
ndash society as a whole (such as avoided emissions)
ndash individual participants
bull Can be positive or negative (NEIs)
bull Might exceed energy benefits
Documenting NEBs
bull Measurement of benefits
bull Modeling
bull Surveys
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
27
Why Document Avoided Emissions
bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit
bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by
monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test
ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
28
Calculating Avoided Emissions
Conceptually avoided emissions are calculated using net energy savings and one of two different approaches
bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)
ndash Several sources of emission factors as well as approaches for calculating the factors
ndash Avoided emissions = (net energy savings) x (emission factor)
bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a
grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs
ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
29
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
Jointly managed by the DOE Office of Electricity Delivery and Energy Reliability and the Office of Energy Efficiency and Renewable Energy
UNIFORM METHODS PROJECT
bull Addresses seven common residential and commercial efficiency measures
bull Step-by-step calculations for determining gross first-year savings using MampV approach
bull Also some cross cutting chapters on some EMampV topics (sampling metering persistence peak savings)
bull Adoption is voluntary bull httpumppnnlgov DOE Goals
1113089 Strengthen credibility EE savings calculations 1113089 Provide clear accessible step-by-step protocols 1113089 Support consistency and transparency 1113089 Reduce costs of EMampV 1113089 Allow for comparison of savings
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 11
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
12
Part Chapters Intended Audience
Part 1
Executive Summary
Readers interested in a brief summary and
introduction to impact evaluation
Part 2
Chapter 1 Introduction
Chapter 2 Energy Efficiency Program Evaluation
Overview
Chapter 3 Impact Evaluation Basics
Readers who want an overview of evaluation and
the key aspects of impact evaluation
Part 3
Chapter 4 Calculating Energy Savings
Chapter 5 Determining Net Energy Savings
Chapter 6 Calculating Avoided Air Emissions
Readers who want additional detail on impact
evaluation approaches ndash deemed savings MampV
large-scale consumption data analysis
Part 4
Chapter 7 Impact Evaluation Considerations
Chapter 8 Impact Evaluation Planning
Program implementers evaluators and
managersregulators of evaluations looking for
guidance on key evaluation issues and planning of
evaluations as well as readers with a background in
evaluation may want to go directly to these chapters
Part 5
Appendix A Glossary
Appendix B Other Evaluation Categories and
Approaches ndash Market Process Cost-Effectiveness
Appendix C Resources
References
Readers interested in standard energy efficiency
evaluation definitions and reference materials as
well as summaries of process market evaluations
cost-effectiveness analyses and top-down
evaluation
EXAMPLE Chapter 7 Impact Evaluation Considerations
bull First portion covers issues that often arise in impact evaluations ndash determining baselines
ndash determining demand savings
ndash calculating persistence of savings
ndash addressing uncertainty of savings estimates
ndash setting evaluation budgets
ndash establishing evaluation principles and ethics
bull Second portion provides brief overviews of ndash program feedback through evaluation
ndash resource planning through evaluation
ndash calculating non-energy benefits
ndash evaluating some relatively unique program types (residential behaviors education and training market transformation codes and standards demand response and GHG mitigation)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
13
Glossary ndash a combined effort with several groups to
create more consistency in how we describe EMampV
bull Accuracy A concept that refers to the relationship between the true value of a variable and an estimate of
the value The term can also be used in reference to a model or a set of measured data or to describe a measuring instrumentrsquos capability
bull helliphelliphelliphelliphellip
bull Incremental Annual Savings The difference between the amount of energy savings acquired or planned
to be acquired as a result of energy efficiency activities in one year and the amount of energy savings acquired or planned to be acquired as a result of the energy efficiency activities in the prior year
bull Independent Variables The explanatory factors (eg weather or occupancy) in a regression model that
are assumed to affect the variable under study (eg energy use)
bull Indirect Emissions Emissions that are a consequence of activities that take place within the organizational
boundaries of an entity but occur at sources owned or controlled by another entity For example emissions of electricity used by a manufacturing entity that occur at a power plant represent the manufacturerrsquos indirect emissions Indirect emissions are typically the source of avoided emissions for electric energy efficiency measures
bull helliphelliphelliphellip
bull Workforce Education and Training Programs Programs primarily intended for building maintenance engineers
HVAC contractors engineers architects maintenance personnel and others These programs provide information about energy efficiency concepts recommended energy-efficient technologies or behaviors andor programs that offer energy efficiency or load-reduction
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
14
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16
RDampD
bull Research
bull Development
bull Demonstration
Deployment
Voluntary implementation of projects and programs ndash outreach education and subsidies For example
bull Incenting early actors
bull Incenting consumers distributers and manufacturers
bull Mass market and individual market strategies
Transformed Markets
Standard practice
or
Codes and standards
There is a continuum of efficiency actions
Our goal is to greatly accelerate and
ldquofeedrdquo (widen) this continuum of actions
EMampV is a tool to support these actions
Evaluation Types
Evaluation Type Description Example Uses
Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)
Determines the amount of energy and demand saved
Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives
Identifies how program designs and processes can be improved
Market Effects Evaluation
Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program
Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program
Cost-Effectiveness Evaluation
Quantifies the costs of program implementation and compares them with program benefits
Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
17
Savings Cannot Be Measured - They Are Estimated
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18
Graph of Energy Consumption Before During And After Project Is Installed
Determining Energy Savings
bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as
used by the Northwest RTF
bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group
bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program
bull All or some of the above
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
19
Planning Implementing and Evaluating Efficiency Programs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
20
Evaluation is Integral to Planning-Implementation-Evaluation Process
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21
When to Evaluate
bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle
bull In timely manner with feedback for bull Ongoing program improvement
bull Supporting portfolio assessments
bull Support the planning of future portfolio cycles load forecastsenergy resource plans
bull Can also be used to inform future evaluations in particular through updating deemed savings values
Structure for Defining Evaluation Activities
bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input
bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle
bull Evaluation Research Plans ndash Created for the major EMampV activities
bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22
The Big Issues of EMampV
bull Fundamental issue of EMampV
bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty
bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
23
bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual
bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings
How good is good enough As compared to what
EMampV is About Risk Management
How Good is Good Enough amp As Compared to What
bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources
bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example
ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or
ndash Expending too many resources on ineffective actions or the opposite or
ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24
Side Note Energy Savings vs Energy Avoidance ndash
Consumer vs Regulator Perspectives
bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments
bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments
bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26
Non-Energy Benefits (NEBs)
NEB categories
bull NEBs can be categorized as those accruing to
ndash utilities (energy providers)
ndash society as a whole (such as avoided emissions)
ndash individual participants
bull Can be positive or negative (NEIs)
bull Might exceed energy benefits
Documenting NEBs
bull Measurement of benefits
bull Modeling
bull Surveys
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
27
Why Document Avoided Emissions
bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit
bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by
monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test
ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
28
Calculating Avoided Emissions
Conceptually avoided emissions are calculated using net energy savings and one of two different approaches
bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)
ndash Several sources of emission factors as well as approaches for calculating the factors
ndash Avoided emissions = (net energy savings) x (emission factor)
bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a
grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs
ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
29
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 11
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
12
Part Chapters Intended Audience
Part 1
Executive Summary
Readers interested in a brief summary and
introduction to impact evaluation
Part 2
Chapter 1 Introduction
Chapter 2 Energy Efficiency Program Evaluation
Overview
Chapter 3 Impact Evaluation Basics
Readers who want an overview of evaluation and
the key aspects of impact evaluation
Part 3
Chapter 4 Calculating Energy Savings
Chapter 5 Determining Net Energy Savings
Chapter 6 Calculating Avoided Air Emissions
Readers who want additional detail on impact
evaluation approaches ndash deemed savings MampV
large-scale consumption data analysis
Part 4
Chapter 7 Impact Evaluation Considerations
Chapter 8 Impact Evaluation Planning
Program implementers evaluators and
managersregulators of evaluations looking for
guidance on key evaluation issues and planning of
evaluations as well as readers with a background in
evaluation may want to go directly to these chapters
Part 5
Appendix A Glossary
Appendix B Other Evaluation Categories and
Approaches ndash Market Process Cost-Effectiveness
Appendix C Resources
References
Readers interested in standard energy efficiency
evaluation definitions and reference materials as
well as summaries of process market evaluations
cost-effectiveness analyses and top-down
evaluation
EXAMPLE Chapter 7 Impact Evaluation Considerations
bull First portion covers issues that often arise in impact evaluations ndash determining baselines
ndash determining demand savings
ndash calculating persistence of savings
ndash addressing uncertainty of savings estimates
ndash setting evaluation budgets
ndash establishing evaluation principles and ethics
bull Second portion provides brief overviews of ndash program feedback through evaluation
ndash resource planning through evaluation
ndash calculating non-energy benefits
ndash evaluating some relatively unique program types (residential behaviors education and training market transformation codes and standards demand response and GHG mitigation)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
13
Glossary ndash a combined effort with several groups to
create more consistency in how we describe EMampV
bull Accuracy A concept that refers to the relationship between the true value of a variable and an estimate of
the value The term can also be used in reference to a model or a set of measured data or to describe a measuring instrumentrsquos capability
bull helliphelliphelliphelliphellip
bull Incremental Annual Savings The difference between the amount of energy savings acquired or planned
to be acquired as a result of energy efficiency activities in one year and the amount of energy savings acquired or planned to be acquired as a result of the energy efficiency activities in the prior year
bull Independent Variables The explanatory factors (eg weather or occupancy) in a regression model that
are assumed to affect the variable under study (eg energy use)
bull Indirect Emissions Emissions that are a consequence of activities that take place within the organizational
boundaries of an entity but occur at sources owned or controlled by another entity For example emissions of electricity used by a manufacturing entity that occur at a power plant represent the manufacturerrsquos indirect emissions Indirect emissions are typically the source of avoided emissions for electric energy efficiency measures
bull helliphelliphelliphellip
bull Workforce Education and Training Programs Programs primarily intended for building maintenance engineers
HVAC contractors engineers architects maintenance personnel and others These programs provide information about energy efficiency concepts recommended energy-efficient technologies or behaviors andor programs that offer energy efficiency or load-reduction
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
14
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16
RDampD
bull Research
bull Development
bull Demonstration
Deployment
Voluntary implementation of projects and programs ndash outreach education and subsidies For example
bull Incenting early actors
bull Incenting consumers distributers and manufacturers
bull Mass market and individual market strategies
Transformed Markets
Standard practice
or
Codes and standards
There is a continuum of efficiency actions
Our goal is to greatly accelerate and
ldquofeedrdquo (widen) this continuum of actions
EMampV is a tool to support these actions
Evaluation Types
Evaluation Type Description Example Uses
Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)
Determines the amount of energy and demand saved
Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives
Identifies how program designs and processes can be improved
Market Effects Evaluation
Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program
Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program
Cost-Effectiveness Evaluation
Quantifies the costs of program implementation and compares them with program benefits
Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
17
Savings Cannot Be Measured - They Are Estimated
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18
Graph of Energy Consumption Before During And After Project Is Installed
Determining Energy Savings
bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as
used by the Northwest RTF
bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group
bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program
bull All or some of the above
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
19
Planning Implementing and Evaluating Efficiency Programs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
20
Evaluation is Integral to Planning-Implementation-Evaluation Process
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21
When to Evaluate
bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle
bull In timely manner with feedback for bull Ongoing program improvement
bull Supporting portfolio assessments
bull Support the planning of future portfolio cycles load forecastsenergy resource plans
bull Can also be used to inform future evaluations in particular through updating deemed savings values
Structure for Defining Evaluation Activities
bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input
bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle
bull Evaluation Research Plans ndash Created for the major EMampV activities
bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22
The Big Issues of EMampV
bull Fundamental issue of EMampV
bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty
bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
23
bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual
bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings
How good is good enough As compared to what
EMampV is About Risk Management
How Good is Good Enough amp As Compared to What
bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources
bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example
ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or
ndash Expending too many resources on ineffective actions or the opposite or
ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24
Side Note Energy Savings vs Energy Avoidance ndash
Consumer vs Regulator Perspectives
bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments
bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments
bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26
Non-Energy Benefits (NEBs)
NEB categories
bull NEBs can be categorized as those accruing to
ndash utilities (energy providers)
ndash society as a whole (such as avoided emissions)
ndash individual participants
bull Can be positive or negative (NEIs)
bull Might exceed energy benefits
Documenting NEBs
bull Measurement of benefits
bull Modeling
bull Surveys
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
27
Why Document Avoided Emissions
bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit
bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by
monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test
ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
28
Calculating Avoided Emissions
Conceptually avoided emissions are calculated using net energy savings and one of two different approaches
bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)
ndash Several sources of emission factors as well as approaches for calculating the factors
ndash Avoided emissions = (net energy savings) x (emission factor)
bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a
grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs
ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
29
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
12
Part Chapters Intended Audience
Part 1
Executive Summary
Readers interested in a brief summary and
introduction to impact evaluation
Part 2
Chapter 1 Introduction
Chapter 2 Energy Efficiency Program Evaluation
Overview
Chapter 3 Impact Evaluation Basics
Readers who want an overview of evaluation and
the key aspects of impact evaluation
Part 3
Chapter 4 Calculating Energy Savings
Chapter 5 Determining Net Energy Savings
Chapter 6 Calculating Avoided Air Emissions
Readers who want additional detail on impact
evaluation approaches ndash deemed savings MampV
large-scale consumption data analysis
Part 4
Chapter 7 Impact Evaluation Considerations
Chapter 8 Impact Evaluation Planning
Program implementers evaluators and
managersregulators of evaluations looking for
guidance on key evaluation issues and planning of
evaluations as well as readers with a background in
evaluation may want to go directly to these chapters
Part 5
Appendix A Glossary
Appendix B Other Evaluation Categories and
Approaches ndash Market Process Cost-Effectiveness
Appendix C Resources
References
Readers interested in standard energy efficiency
evaluation definitions and reference materials as
well as summaries of process market evaluations
cost-effectiveness analyses and top-down
evaluation
EXAMPLE Chapter 7 Impact Evaluation Considerations
bull First portion covers issues that often arise in impact evaluations ndash determining baselines
ndash determining demand savings
ndash calculating persistence of savings
ndash addressing uncertainty of savings estimates
ndash setting evaluation budgets
ndash establishing evaluation principles and ethics
bull Second portion provides brief overviews of ndash program feedback through evaluation
ndash resource planning through evaluation
ndash calculating non-energy benefits
ndash evaluating some relatively unique program types (residential behaviors education and training market transformation codes and standards demand response and GHG mitigation)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
13
Glossary ndash a combined effort with several groups to
create more consistency in how we describe EMampV
bull Accuracy A concept that refers to the relationship between the true value of a variable and an estimate of
the value The term can also be used in reference to a model or a set of measured data or to describe a measuring instrumentrsquos capability
bull helliphelliphelliphelliphellip
bull Incremental Annual Savings The difference between the amount of energy savings acquired or planned
to be acquired as a result of energy efficiency activities in one year and the amount of energy savings acquired or planned to be acquired as a result of the energy efficiency activities in the prior year
bull Independent Variables The explanatory factors (eg weather or occupancy) in a regression model that
are assumed to affect the variable under study (eg energy use)
bull Indirect Emissions Emissions that are a consequence of activities that take place within the organizational
boundaries of an entity but occur at sources owned or controlled by another entity For example emissions of electricity used by a manufacturing entity that occur at a power plant represent the manufacturerrsquos indirect emissions Indirect emissions are typically the source of avoided emissions for electric energy efficiency measures
bull helliphelliphelliphellip
bull Workforce Education and Training Programs Programs primarily intended for building maintenance engineers
HVAC contractors engineers architects maintenance personnel and others These programs provide information about energy efficiency concepts recommended energy-efficient technologies or behaviors andor programs that offer energy efficiency or load-reduction
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
14
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16
RDampD
bull Research
bull Development
bull Demonstration
Deployment
Voluntary implementation of projects and programs ndash outreach education and subsidies For example
bull Incenting early actors
bull Incenting consumers distributers and manufacturers
bull Mass market and individual market strategies
Transformed Markets
Standard practice
or
Codes and standards
There is a continuum of efficiency actions
Our goal is to greatly accelerate and
ldquofeedrdquo (widen) this continuum of actions
EMampV is a tool to support these actions
Evaluation Types
Evaluation Type Description Example Uses
Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)
Determines the amount of energy and demand saved
Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives
Identifies how program designs and processes can be improved
Market Effects Evaluation
Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program
Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program
Cost-Effectiveness Evaluation
Quantifies the costs of program implementation and compares them with program benefits
Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
17
Savings Cannot Be Measured - They Are Estimated
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18
Graph of Energy Consumption Before During And After Project Is Installed
Determining Energy Savings
bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as
used by the Northwest RTF
bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group
bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program
bull All or some of the above
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
19
Planning Implementing and Evaluating Efficiency Programs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
20
Evaluation is Integral to Planning-Implementation-Evaluation Process
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21
When to Evaluate
bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle
bull In timely manner with feedback for bull Ongoing program improvement
bull Supporting portfolio assessments
bull Support the planning of future portfolio cycles load forecastsenergy resource plans
bull Can also be used to inform future evaluations in particular through updating deemed savings values
Structure for Defining Evaluation Activities
bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input
bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle
bull Evaluation Research Plans ndash Created for the major EMampV activities
bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22
The Big Issues of EMampV
bull Fundamental issue of EMampV
bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty
bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
23
bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual
bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings
How good is good enough As compared to what
EMampV is About Risk Management
How Good is Good Enough amp As Compared to What
bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources
bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example
ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or
ndash Expending too many resources on ineffective actions or the opposite or
ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24
Side Note Energy Savings vs Energy Avoidance ndash
Consumer vs Regulator Perspectives
bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments
bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments
bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26
Non-Energy Benefits (NEBs)
NEB categories
bull NEBs can be categorized as those accruing to
ndash utilities (energy providers)
ndash society as a whole (such as avoided emissions)
ndash individual participants
bull Can be positive or negative (NEIs)
bull Might exceed energy benefits
Documenting NEBs
bull Measurement of benefits
bull Modeling
bull Surveys
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
27
Why Document Avoided Emissions
bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit
bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by
monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test
ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
28
Calculating Avoided Emissions
Conceptually avoided emissions are calculated using net energy savings and one of two different approaches
bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)
ndash Several sources of emission factors as well as approaches for calculating the factors
ndash Avoided emissions = (net energy savings) x (emission factor)
bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a
grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs
ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
29
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
EXAMPLE Chapter 7 Impact Evaluation Considerations
bull First portion covers issues that often arise in impact evaluations ndash determining baselines
ndash determining demand savings
ndash calculating persistence of savings
ndash addressing uncertainty of savings estimates
ndash setting evaluation budgets
ndash establishing evaluation principles and ethics
bull Second portion provides brief overviews of ndash program feedback through evaluation
ndash resource planning through evaluation
ndash calculating non-energy benefits
ndash evaluating some relatively unique program types (residential behaviors education and training market transformation codes and standards demand response and GHG mitigation)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
13
Glossary ndash a combined effort with several groups to
create more consistency in how we describe EMampV
bull Accuracy A concept that refers to the relationship between the true value of a variable and an estimate of
the value The term can also be used in reference to a model or a set of measured data or to describe a measuring instrumentrsquos capability
bull helliphelliphelliphelliphellip
bull Incremental Annual Savings The difference between the amount of energy savings acquired or planned
to be acquired as a result of energy efficiency activities in one year and the amount of energy savings acquired or planned to be acquired as a result of the energy efficiency activities in the prior year
bull Independent Variables The explanatory factors (eg weather or occupancy) in a regression model that
are assumed to affect the variable under study (eg energy use)
bull Indirect Emissions Emissions that are a consequence of activities that take place within the organizational
boundaries of an entity but occur at sources owned or controlled by another entity For example emissions of electricity used by a manufacturing entity that occur at a power plant represent the manufacturerrsquos indirect emissions Indirect emissions are typically the source of avoided emissions for electric energy efficiency measures
bull helliphelliphelliphellip
bull Workforce Education and Training Programs Programs primarily intended for building maintenance engineers
HVAC contractors engineers architects maintenance personnel and others These programs provide information about energy efficiency concepts recommended energy-efficient technologies or behaviors andor programs that offer energy efficiency or load-reduction
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
14
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16
RDampD
bull Research
bull Development
bull Demonstration
Deployment
Voluntary implementation of projects and programs ndash outreach education and subsidies For example
bull Incenting early actors
bull Incenting consumers distributers and manufacturers
bull Mass market and individual market strategies
Transformed Markets
Standard practice
or
Codes and standards
There is a continuum of efficiency actions
Our goal is to greatly accelerate and
ldquofeedrdquo (widen) this continuum of actions
EMampV is a tool to support these actions
Evaluation Types
Evaluation Type Description Example Uses
Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)
Determines the amount of energy and demand saved
Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives
Identifies how program designs and processes can be improved
Market Effects Evaluation
Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program
Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program
Cost-Effectiveness Evaluation
Quantifies the costs of program implementation and compares them with program benefits
Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
17
Savings Cannot Be Measured - They Are Estimated
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18
Graph of Energy Consumption Before During And After Project Is Installed
Determining Energy Savings
bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as
used by the Northwest RTF
bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group
bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program
bull All or some of the above
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
19
Planning Implementing and Evaluating Efficiency Programs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
20
Evaluation is Integral to Planning-Implementation-Evaluation Process
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21
When to Evaluate
bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle
bull In timely manner with feedback for bull Ongoing program improvement
bull Supporting portfolio assessments
bull Support the planning of future portfolio cycles load forecastsenergy resource plans
bull Can also be used to inform future evaluations in particular through updating deemed savings values
Structure for Defining Evaluation Activities
bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input
bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle
bull Evaluation Research Plans ndash Created for the major EMampV activities
bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22
The Big Issues of EMampV
bull Fundamental issue of EMampV
bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty
bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
23
bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual
bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings
How good is good enough As compared to what
EMampV is About Risk Management
How Good is Good Enough amp As Compared to What
bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources
bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example
ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or
ndash Expending too many resources on ineffective actions or the opposite or
ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24
Side Note Energy Savings vs Energy Avoidance ndash
Consumer vs Regulator Perspectives
bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments
bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments
bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26
Non-Energy Benefits (NEBs)
NEB categories
bull NEBs can be categorized as those accruing to
ndash utilities (energy providers)
ndash society as a whole (such as avoided emissions)
ndash individual participants
bull Can be positive or negative (NEIs)
bull Might exceed energy benefits
Documenting NEBs
bull Measurement of benefits
bull Modeling
bull Surveys
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
27
Why Document Avoided Emissions
bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit
bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by
monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test
ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
28
Calculating Avoided Emissions
Conceptually avoided emissions are calculated using net energy savings and one of two different approaches
bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)
ndash Several sources of emission factors as well as approaches for calculating the factors
ndash Avoided emissions = (net energy savings) x (emission factor)
bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a
grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs
ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
29
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
Glossary ndash a combined effort with several groups to
create more consistency in how we describe EMampV
bull Accuracy A concept that refers to the relationship between the true value of a variable and an estimate of
the value The term can also be used in reference to a model or a set of measured data or to describe a measuring instrumentrsquos capability
bull helliphelliphelliphelliphellip
bull Incremental Annual Savings The difference between the amount of energy savings acquired or planned
to be acquired as a result of energy efficiency activities in one year and the amount of energy savings acquired or planned to be acquired as a result of the energy efficiency activities in the prior year
bull Independent Variables The explanatory factors (eg weather or occupancy) in a regression model that
are assumed to affect the variable under study (eg energy use)
bull Indirect Emissions Emissions that are a consequence of activities that take place within the organizational
boundaries of an entity but occur at sources owned or controlled by another entity For example emissions of electricity used by a manufacturing entity that occur at a power plant represent the manufacturerrsquos indirect emissions Indirect emissions are typically the source of avoided emissions for electric energy efficiency measures
bull helliphelliphelliphellip
bull Workforce Education and Training Programs Programs primarily intended for building maintenance engineers
HVAC contractors engineers architects maintenance personnel and others These programs provide information about energy efficiency concepts recommended energy-efficient technologies or behaviors andor programs that offer energy efficiency or load-reduction
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
14
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16
RDampD
bull Research
bull Development
bull Demonstration
Deployment
Voluntary implementation of projects and programs ndash outreach education and subsidies For example
bull Incenting early actors
bull Incenting consumers distributers and manufacturers
bull Mass market and individual market strategies
Transformed Markets
Standard practice
or
Codes and standards
There is a continuum of efficiency actions
Our goal is to greatly accelerate and
ldquofeedrdquo (widen) this continuum of actions
EMampV is a tool to support these actions
Evaluation Types
Evaluation Type Description Example Uses
Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)
Determines the amount of energy and demand saved
Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives
Identifies how program designs and processes can be improved
Market Effects Evaluation
Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program
Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program
Cost-Effectiveness Evaluation
Quantifies the costs of program implementation and compares them with program benefits
Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
17
Savings Cannot Be Measured - They Are Estimated
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18
Graph of Energy Consumption Before During And After Project Is Installed
Determining Energy Savings
bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as
used by the Northwest RTF
bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group
bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program
bull All or some of the above
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
19
Planning Implementing and Evaluating Efficiency Programs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
20
Evaluation is Integral to Planning-Implementation-Evaluation Process
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21
When to Evaluate
bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle
bull In timely manner with feedback for bull Ongoing program improvement
bull Supporting portfolio assessments
bull Support the planning of future portfolio cycles load forecastsenergy resource plans
bull Can also be used to inform future evaluations in particular through updating deemed savings values
Structure for Defining Evaluation Activities
bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input
bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle
bull Evaluation Research Plans ndash Created for the major EMampV activities
bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22
The Big Issues of EMampV
bull Fundamental issue of EMampV
bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty
bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
23
bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual
bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings
How good is good enough As compared to what
EMampV is About Risk Management
How Good is Good Enough amp As Compared to What
bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources
bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example
ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or
ndash Expending too many resources on ineffective actions or the opposite or
ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24
Side Note Energy Savings vs Energy Avoidance ndash
Consumer vs Regulator Perspectives
bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments
bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments
bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26
Non-Energy Benefits (NEBs)
NEB categories
bull NEBs can be categorized as those accruing to
ndash utilities (energy providers)
ndash society as a whole (such as avoided emissions)
ndash individual participants
bull Can be positive or negative (NEIs)
bull Might exceed energy benefits
Documenting NEBs
bull Measurement of benefits
bull Modeling
bull Surveys
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
27
Why Document Avoided Emissions
bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit
bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by
monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test
ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
28
Calculating Avoided Emissions
Conceptually avoided emissions are calculated using net energy savings and one of two different approaches
bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)
ndash Several sources of emission factors as well as approaches for calculating the factors
ndash Avoided emissions = (net energy savings) x (emission factor)
bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a
grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs
ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
29
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 15
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16
RDampD
bull Research
bull Development
bull Demonstration
Deployment
Voluntary implementation of projects and programs ndash outreach education and subsidies For example
bull Incenting early actors
bull Incenting consumers distributers and manufacturers
bull Mass market and individual market strategies
Transformed Markets
Standard practice
or
Codes and standards
There is a continuum of efficiency actions
Our goal is to greatly accelerate and
ldquofeedrdquo (widen) this continuum of actions
EMampV is a tool to support these actions
Evaluation Types
Evaluation Type Description Example Uses
Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)
Determines the amount of energy and demand saved
Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives
Identifies how program designs and processes can be improved
Market Effects Evaluation
Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program
Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program
Cost-Effectiveness Evaluation
Quantifies the costs of program implementation and compares them with program benefits
Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
17
Savings Cannot Be Measured - They Are Estimated
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18
Graph of Energy Consumption Before During And After Project Is Installed
Determining Energy Savings
bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as
used by the Northwest RTF
bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group
bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program
bull All or some of the above
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
19
Planning Implementing and Evaluating Efficiency Programs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
20
Evaluation is Integral to Planning-Implementation-Evaluation Process
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21
When to Evaluate
bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle
bull In timely manner with feedback for bull Ongoing program improvement
bull Supporting portfolio assessments
bull Support the planning of future portfolio cycles load forecastsenergy resource plans
bull Can also be used to inform future evaluations in particular through updating deemed savings values
Structure for Defining Evaluation Activities
bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input
bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle
bull Evaluation Research Plans ndash Created for the major EMampV activities
bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22
The Big Issues of EMampV
bull Fundamental issue of EMampV
bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty
bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
23
bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual
bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings
How good is good enough As compared to what
EMampV is About Risk Management
How Good is Good Enough amp As Compared to What
bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources
bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example
ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or
ndash Expending too many resources on ineffective actions or the opposite or
ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24
Side Note Energy Savings vs Energy Avoidance ndash
Consumer vs Regulator Perspectives
bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments
bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments
bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26
Non-Energy Benefits (NEBs)
NEB categories
bull NEBs can be categorized as those accruing to
ndash utilities (energy providers)
ndash society as a whole (such as avoided emissions)
ndash individual participants
bull Can be positive or negative (NEIs)
bull Might exceed energy benefits
Documenting NEBs
bull Measurement of benefits
bull Modeling
bull Surveys
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
27
Why Document Avoided Emissions
bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit
bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by
monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test
ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
28
Calculating Avoided Emissions
Conceptually avoided emissions are calculated using net energy savings and one of two different approaches
bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)
ndash Several sources of emission factors as well as approaches for calculating the factors
ndash Avoided emissions = (net energy savings) x (emission factor)
bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a
grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs
ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
29
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 16
RDampD
bull Research
bull Development
bull Demonstration
Deployment
Voluntary implementation of projects and programs ndash outreach education and subsidies For example
bull Incenting early actors
bull Incenting consumers distributers and manufacturers
bull Mass market and individual market strategies
Transformed Markets
Standard practice
or
Codes and standards
There is a continuum of efficiency actions
Our goal is to greatly accelerate and
ldquofeedrdquo (widen) this continuum of actions
EMampV is a tool to support these actions
Evaluation Types
Evaluation Type Description Example Uses
Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)
Determines the amount of energy and demand saved
Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives
Identifies how program designs and processes can be improved
Market Effects Evaluation
Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program
Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program
Cost-Effectiveness Evaluation
Quantifies the costs of program implementation and compares them with program benefits
Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
17
Savings Cannot Be Measured - They Are Estimated
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18
Graph of Energy Consumption Before During And After Project Is Installed
Determining Energy Savings
bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as
used by the Northwest RTF
bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group
bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program
bull All or some of the above
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
19
Planning Implementing and Evaluating Efficiency Programs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
20
Evaluation is Integral to Planning-Implementation-Evaluation Process
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21
When to Evaluate
bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle
bull In timely manner with feedback for bull Ongoing program improvement
bull Supporting portfolio assessments
bull Support the planning of future portfolio cycles load forecastsenergy resource plans
bull Can also be used to inform future evaluations in particular through updating deemed savings values
Structure for Defining Evaluation Activities
bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input
bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle
bull Evaluation Research Plans ndash Created for the major EMampV activities
bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22
The Big Issues of EMampV
bull Fundamental issue of EMampV
bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty
bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
23
bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual
bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings
How good is good enough As compared to what
EMampV is About Risk Management
How Good is Good Enough amp As Compared to What
bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources
bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example
ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or
ndash Expending too many resources on ineffective actions or the opposite or
ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24
Side Note Energy Savings vs Energy Avoidance ndash
Consumer vs Regulator Perspectives
bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments
bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments
bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26
Non-Energy Benefits (NEBs)
NEB categories
bull NEBs can be categorized as those accruing to
ndash utilities (energy providers)
ndash society as a whole (such as avoided emissions)
ndash individual participants
bull Can be positive or negative (NEIs)
bull Might exceed energy benefits
Documenting NEBs
bull Measurement of benefits
bull Modeling
bull Surveys
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
27
Why Document Avoided Emissions
bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit
bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by
monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test
ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
28
Calculating Avoided Emissions
Conceptually avoided emissions are calculated using net energy savings and one of two different approaches
bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)
ndash Several sources of emission factors as well as approaches for calculating the factors
ndash Avoided emissions = (net energy savings) x (emission factor)
bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a
grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs
ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
29
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
Evaluation Types
Evaluation Type Description Example Uses
Impact Evaluation Quantifies direct and indirect changes associated with the subject program(s)
Determines the amount of energy and demand saved
Process Evaluation Indicates how the procedures associated with program design and implementation are performing from both the administratorrsquos and the participantsrsquo perspectives
Identifies how program designs and processes can be improved
Market Effects Evaluation
Analyzes how the overall supply chain and market for energy efficiency products have been affected by the program
Characterizes changes that have occurred in efficiency markets and whether they are attributable to and sustainable with or without the program
Cost-Effectiveness Evaluation
Quantifies the costs of program implementation and compares them with program benefits
Determines whether an energy efficiency program is a cost-effective investment compared with other programs and energy supply resources
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
17
Savings Cannot Be Measured - They Are Estimated
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18
Graph of Energy Consumption Before During And After Project Is Installed
Determining Energy Savings
bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as
used by the Northwest RTF
bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group
bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program
bull All or some of the above
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
19
Planning Implementing and Evaluating Efficiency Programs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
20
Evaluation is Integral to Planning-Implementation-Evaluation Process
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21
When to Evaluate
bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle
bull In timely manner with feedback for bull Ongoing program improvement
bull Supporting portfolio assessments
bull Support the planning of future portfolio cycles load forecastsenergy resource plans
bull Can also be used to inform future evaluations in particular through updating deemed savings values
Structure for Defining Evaluation Activities
bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input
bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle
bull Evaluation Research Plans ndash Created for the major EMampV activities
bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22
The Big Issues of EMampV
bull Fundamental issue of EMampV
bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty
bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
23
bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual
bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings
How good is good enough As compared to what
EMampV is About Risk Management
How Good is Good Enough amp As Compared to What
bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources
bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example
ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or
ndash Expending too many resources on ineffective actions or the opposite or
ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24
Side Note Energy Savings vs Energy Avoidance ndash
Consumer vs Regulator Perspectives
bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments
bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments
bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26
Non-Energy Benefits (NEBs)
NEB categories
bull NEBs can be categorized as those accruing to
ndash utilities (energy providers)
ndash society as a whole (such as avoided emissions)
ndash individual participants
bull Can be positive or negative (NEIs)
bull Might exceed energy benefits
Documenting NEBs
bull Measurement of benefits
bull Modeling
bull Surveys
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
27
Why Document Avoided Emissions
bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit
bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by
monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test
ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
28
Calculating Avoided Emissions
Conceptually avoided emissions are calculated using net energy savings and one of two different approaches
bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)
ndash Several sources of emission factors as well as approaches for calculating the factors
ndash Avoided emissions = (net energy savings) x (emission factor)
bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a
grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs
ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
29
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
Savings Cannot Be Measured - They Are Estimated
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 18
Graph of Energy Consumption Before During And After Project Is Installed
Determining Energy Savings
bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as
used by the Northwest RTF
bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group
bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program
bull All or some of the above
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
19
Planning Implementing and Evaluating Efficiency Programs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
20
Evaluation is Integral to Planning-Implementation-Evaluation Process
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21
When to Evaluate
bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle
bull In timely manner with feedback for bull Ongoing program improvement
bull Supporting portfolio assessments
bull Support the planning of future portfolio cycles load forecastsenergy resource plans
bull Can also be used to inform future evaluations in particular through updating deemed savings values
Structure for Defining Evaluation Activities
bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input
bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle
bull Evaluation Research Plans ndash Created for the major EMampV activities
bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22
The Big Issues of EMampV
bull Fundamental issue of EMampV
bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty
bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
23
bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual
bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings
How good is good enough As compared to what
EMampV is About Risk Management
How Good is Good Enough amp As Compared to What
bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources
bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example
ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or
ndash Expending too many resources on ineffective actions or the opposite or
ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24
Side Note Energy Savings vs Energy Avoidance ndash
Consumer vs Regulator Perspectives
bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments
bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments
bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26
Non-Energy Benefits (NEBs)
NEB categories
bull NEBs can be categorized as those accruing to
ndash utilities (energy providers)
ndash society as a whole (such as avoided emissions)
ndash individual participants
bull Can be positive or negative (NEIs)
bull Might exceed energy benefits
Documenting NEBs
bull Measurement of benefits
bull Modeling
bull Surveys
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
27
Why Document Avoided Emissions
bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit
bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by
monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test
ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
28
Calculating Avoided Emissions
Conceptually avoided emissions are calculated using net energy savings and one of two different approaches
bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)
ndash Several sources of emission factors as well as approaches for calculating the factors
ndash Avoided emissions = (net energy savings) x (emission factor)
bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a
grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs
ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
29
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
Determining Energy Savings
bull Deemed savings that are based on historical and verified data are applied to conventional energy efficiency measures implemented in the program A variant of this is the deemed calculated savings as
used by the Northwest RTF
bull Large-scale consumption data analysis of metered energy use data to compare the energy use of the program participants with the energy use of a control group
bull One or more measurement and verification (MampV) options (A B C andor D) from the IPMVP are used to determine the savings from a sample of projects These savings are then applied to all of the projects in the program
bull All or some of the above
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
19
Planning Implementing and Evaluating Efficiency Programs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
20
Evaluation is Integral to Planning-Implementation-Evaluation Process
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21
When to Evaluate
bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle
bull In timely manner with feedback for bull Ongoing program improvement
bull Supporting portfolio assessments
bull Support the planning of future portfolio cycles load forecastsenergy resource plans
bull Can also be used to inform future evaluations in particular through updating deemed savings values
Structure for Defining Evaluation Activities
bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input
bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle
bull Evaluation Research Plans ndash Created for the major EMampV activities
bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22
The Big Issues of EMampV
bull Fundamental issue of EMampV
bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty
bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
23
bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual
bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings
How good is good enough As compared to what
EMampV is About Risk Management
How Good is Good Enough amp As Compared to What
bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources
bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example
ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or
ndash Expending too many resources on ineffective actions or the opposite or
ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24
Side Note Energy Savings vs Energy Avoidance ndash
Consumer vs Regulator Perspectives
bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments
bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments
bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26
Non-Energy Benefits (NEBs)
NEB categories
bull NEBs can be categorized as those accruing to
ndash utilities (energy providers)
ndash society as a whole (such as avoided emissions)
ndash individual participants
bull Can be positive or negative (NEIs)
bull Might exceed energy benefits
Documenting NEBs
bull Measurement of benefits
bull Modeling
bull Surveys
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
27
Why Document Avoided Emissions
bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit
bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by
monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test
ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
28
Calculating Avoided Emissions
Conceptually avoided emissions are calculated using net energy savings and one of two different approaches
bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)
ndash Several sources of emission factors as well as approaches for calculating the factors
ndash Avoided emissions = (net energy savings) x (emission factor)
bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a
grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs
ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
29
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
Planning Implementing and Evaluating Efficiency Programs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
20
Evaluation is Integral to Planning-Implementation-Evaluation Process
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21
When to Evaluate
bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle
bull In timely manner with feedback for bull Ongoing program improvement
bull Supporting portfolio assessments
bull Support the planning of future portfolio cycles load forecastsenergy resource plans
bull Can also be used to inform future evaluations in particular through updating deemed savings values
Structure for Defining Evaluation Activities
bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input
bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle
bull Evaluation Research Plans ndash Created for the major EMampV activities
bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22
The Big Issues of EMampV
bull Fundamental issue of EMampV
bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty
bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
23
bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual
bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings
How good is good enough As compared to what
EMampV is About Risk Management
How Good is Good Enough amp As Compared to What
bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources
bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example
ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or
ndash Expending too many resources on ineffective actions or the opposite or
ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24
Side Note Energy Savings vs Energy Avoidance ndash
Consumer vs Regulator Perspectives
bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments
bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments
bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26
Non-Energy Benefits (NEBs)
NEB categories
bull NEBs can be categorized as those accruing to
ndash utilities (energy providers)
ndash society as a whole (such as avoided emissions)
ndash individual participants
bull Can be positive or negative (NEIs)
bull Might exceed energy benefits
Documenting NEBs
bull Measurement of benefits
bull Modeling
bull Surveys
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
27
Why Document Avoided Emissions
bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit
bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by
monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test
ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
28
Calculating Avoided Emissions
Conceptually avoided emissions are calculated using net energy savings and one of two different approaches
bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)
ndash Several sources of emission factors as well as approaches for calculating the factors
ndash Avoided emissions = (net energy savings) x (emission factor)
bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a
grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs
ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
29
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
Evaluation is Integral to Planning-Implementation-Evaluation Process
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 21
When to Evaluate
bull Evaluations should be produced within a portfolio cycle or very soon after the completion of a cycle
bull In timely manner with feedback for bull Ongoing program improvement
bull Supporting portfolio assessments
bull Support the planning of future portfolio cycles load forecastsenergy resource plans
bull Can also be used to inform future evaluations in particular through updating deemed savings values
Structure for Defining Evaluation Activities
bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input
bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle
bull Evaluation Research Plans ndash Created for the major EMampV activities
bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22
The Big Issues of EMampV
bull Fundamental issue of EMampV
bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty
bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
23
bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual
bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings
How good is good enough As compared to what
EMampV is About Risk Management
How Good is Good Enough amp As Compared to What
bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources
bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example
ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or
ndash Expending too many resources on ineffective actions or the opposite or
ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24
Side Note Energy Savings vs Energy Avoidance ndash
Consumer vs Regulator Perspectives
bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments
bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments
bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26
Non-Energy Benefits (NEBs)
NEB categories
bull NEBs can be categorized as those accruing to
ndash utilities (energy providers)
ndash society as a whole (such as avoided emissions)
ndash individual participants
bull Can be positive or negative (NEIs)
bull Might exceed energy benefits
Documenting NEBs
bull Measurement of benefits
bull Modeling
bull Surveys
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
27
Why Document Avoided Emissions
bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit
bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by
monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test
ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
28
Calculating Avoided Emissions
Conceptually avoided emissions are calculated using net energy savings and one of two different approaches
bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)
ndash Several sources of emission factors as well as approaches for calculating the factors
ndash Avoided emissions = (net energy savings) x (emission factor)
bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a
grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs
ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
29
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
Structure for Defining Evaluation Activities
bull EMampV Framework ndashPrimary document that lays out top level structure This is perhaps the principle document that all stakeholders can focus on and provide high level input
bull Annual Plans ndash Indicates major evaluation activities that will be conducted during the evaluation cycle
bull Evaluation Research Plans ndash Created for the major EMampV activities
bull Site Specific MampV Plans ndash For custom project sites that are analyzed and inspected
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 22
The Big Issues of EMampV
bull Fundamental issue of EMampV
bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty
bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
23
bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual
bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings
How good is good enough As compared to what
EMampV is About Risk Management
How Good is Good Enough amp As Compared to What
bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources
bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example
ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or
ndash Expending too many resources on ineffective actions or the opposite or
ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24
Side Note Energy Savings vs Energy Avoidance ndash
Consumer vs Regulator Perspectives
bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments
bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments
bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26
Non-Energy Benefits (NEBs)
NEB categories
bull NEBs can be categorized as those accruing to
ndash utilities (energy providers)
ndash society as a whole (such as avoided emissions)
ndash individual participants
bull Can be positive or negative (NEIs)
bull Might exceed energy benefits
Documenting NEBs
bull Measurement of benefits
bull Modeling
bull Surveys
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
27
Why Document Avoided Emissions
bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit
bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by
monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test
ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
28
Calculating Avoided Emissions
Conceptually avoided emissions are calculated using net energy savings and one of two different approaches
bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)
ndash Several sources of emission factors as well as approaches for calculating the factors
ndash Avoided emissions = (net energy savings) x (emission factor)
bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a
grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs
ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
29
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
The Big Issues of EMampV
bull Fundamental issue of EMampV
bull How certain does one have to be of savings estimates and is that certainty balanced against the amount of effort utilized to obtain that level of certainty
bull EMampV investments should consider risk management principles - balance the costs and value of information derived from EMampV (ie EMampV should be cost-effective)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
23
bull First - Defining a baseline against which efficiency actions are compared for determining energy savings and whether attribution should be considered ndash the counterfactual
bull Second ndash Establishing level of performance confidence and risk for efficiency relative to other options for reducing savings and risk of not getting the savings
How good is good enough As compared to what
EMampV is About Risk Management
How Good is Good Enough amp As Compared to What
bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources
bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example
ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or
ndash Expending too many resources on ineffective actions or the opposite or
ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24
Side Note Energy Savings vs Energy Avoidance ndash
Consumer vs Regulator Perspectives
bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments
bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments
bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26
Non-Energy Benefits (NEBs)
NEB categories
bull NEBs can be categorized as those accruing to
ndash utilities (energy providers)
ndash society as a whole (such as avoided emissions)
ndash individual participants
bull Can be positive or negative (NEIs)
bull Might exceed energy benefits
Documenting NEBs
bull Measurement of benefits
bull Modeling
bull Surveys
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
27
Why Document Avoided Emissions
bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit
bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by
monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test
ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
28
Calculating Avoided Emissions
Conceptually avoided emissions are calculated using net energy savings and one of two different approaches
bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)
ndash Several sources of emission factors as well as approaches for calculating the factors
ndash Avoided emissions = (net energy savings) x (emission factor)
bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a
grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs
ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
29
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
How Good is Good Enough amp As Compared to What
bull Other resources will also include uncertainty thus energy efficiency will likely need to establish a level of confidence and relative risks that is comparable to other energy resources
bull Tolerance for uncertainty is driven by how much risk is associated with getting the wrong answer andor spending resources on EMampV For example
ndash Crediting too much savings or too little savings to the actions that have been taken to comply with a savings goal or
ndash Expending too many resources on ineffective actions or the opposite or
ndash Expending too many resources on EMampV and thus wasting time and money or stifling energy efficiency activities
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 24
Side Note Energy Savings vs Energy Avoidance ndash
Consumer vs Regulator Perspectives
bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments
bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments
bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26
Non-Energy Benefits (NEBs)
NEB categories
bull NEBs can be categorized as those accruing to
ndash utilities (energy providers)
ndash society as a whole (such as avoided emissions)
ndash individual participants
bull Can be positive or negative (NEIs)
bull Might exceed energy benefits
Documenting NEBs
bull Measurement of benefits
bull Modeling
bull Surveys
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
27
Why Document Avoided Emissions
bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit
bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by
monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test
ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
28
Calculating Avoided Emissions
Conceptually avoided emissions are calculated using net energy savings and one of two different approaches
bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)
ndash Several sources of emission factors as well as approaches for calculating the factors
ndash Avoided emissions = (net energy savings) x (emission factor)
bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a
grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs
ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
29
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
Side Note Energy Savings vs Energy Avoidance ndash
Consumer vs Regulator Perspectives
bull Regulatorsutilitiesresource planners focus on how much energy would have been consumed if the consumers had not taken the energy efficiency action They want to know how much energy is avoided To do so they make adjustments
bull Energy consumers often use the word lsquosavingsrsquo to describe lsquocost reductionsrsquo They might make lsquotechnicalrsquo adjustments but certainly not lsquoresourcersquo adjustments
bull Result ndash savings for consumers might be different than savings determined from a resourceregulatory perspective
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 25
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26
Non-Energy Benefits (NEBs)
NEB categories
bull NEBs can be categorized as those accruing to
ndash utilities (energy providers)
ndash society as a whole (such as avoided emissions)
ndash individual participants
bull Can be positive or negative (NEIs)
bull Might exceed energy benefits
Documenting NEBs
bull Measurement of benefits
bull Modeling
bull Surveys
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
27
Why Document Avoided Emissions
bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit
bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by
monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test
ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
28
Calculating Avoided Emissions
Conceptually avoided emissions are calculated using net energy savings and one of two different approaches
bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)
ndash Several sources of emission factors as well as approaches for calculating the factors
ndash Avoided emissions = (net energy savings) x (emission factor)
bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a
grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs
ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
29
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 26
Non-Energy Benefits (NEBs)
NEB categories
bull NEBs can be categorized as those accruing to
ndash utilities (energy providers)
ndash society as a whole (such as avoided emissions)
ndash individual participants
bull Can be positive or negative (NEIs)
bull Might exceed energy benefits
Documenting NEBs
bull Measurement of benefits
bull Modeling
bull Surveys
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
27
Why Document Avoided Emissions
bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit
bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by
monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test
ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
28
Calculating Avoided Emissions
Conceptually avoided emissions are calculated using net energy savings and one of two different approaches
bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)
ndash Several sources of emission factors as well as approaches for calculating the factors
ndash Avoided emissions = (net energy savings) x (emission factor)
bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a
grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs
ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
29
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
Non-Energy Benefits (NEBs)
NEB categories
bull NEBs can be categorized as those accruing to
ndash utilities (energy providers)
ndash society as a whole (such as avoided emissions)
ndash individual participants
bull Can be positive or negative (NEIs)
bull Might exceed energy benefits
Documenting NEBs
bull Measurement of benefits
bull Modeling
bull Surveys
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
27
Why Document Avoided Emissions
bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit
bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by
monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test
ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
28
Calculating Avoided Emissions
Conceptually avoided emissions are calculated using net energy savings and one of two different approaches
bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)
ndash Several sources of emission factors as well as approaches for calculating the factors
ndash Avoided emissions = (net energy savings) x (emission factor)
bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a
grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs
ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
29
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
Why Document Avoided Emissions
bull Historically emission reductions from efficiency activities were usually only described subjectively in program evaluations as a non-quantified (non-monetized) benefit
bull This is changing with increasing interest in quantifying these benefits for at least two purposes ndash Determining the cost-effectiveness of efficiency programs by
monetizing the environmental benefits of efficiency ndash such as in a Societal Cost Test
ndash Supporting states claiming of emissions benefits in state air pollution plans (eg State Implementation PlansmdashSIPSmdashfor criteria pollutants) or GHG reduction requirements (eg Californiarsquos Assembly Bill 32 Global Warming Solutions Act)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
28
Calculating Avoided Emissions
Conceptually avoided emissions are calculated using net energy savings and one of two different approaches
bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)
ndash Several sources of emission factors as well as approaches for calculating the factors
ndash Avoided emissions = (net energy savings) x (emission factor)
bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a
grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs
ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
29
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
Calculating Avoided Emissions
Conceptually avoided emissions are calculated using net energy savings and one of two different approaches
bull Emission Factor Approach ndash Multiplying net energy savings by emission factors (eg pounds of CO2 per MWh)
ndash Several sources of emission factors as well as approaches for calculating the factors
ndash Avoided emissions = (net energy savings) x (emission factor)
bull Scenario Analysis Approach ndash Calculating a base case of source (eg electricity generating units connected to a
grid) emissions without the efficiency programs and comparing that with the emissions of those sources operating with the reduced energy consumption associated with the efficiency programs
ndash Avoided emissions = (base case emissions) ndash (reporting period emissions)
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
29
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
Emissions Factors and Models
bull For onsite emissions (eg natural gas in boilers) it is pretty simple use default factors from EPA or IPCC
bull For electricity it can be more complex options include ndash Regional non-baseload emission rates
(using EPArsquos eGRID database)
ndash Regional marginal baseload emission rates (using capacity factors or equivalent)
ndash Regional historical hourly emission rates
ndash Energy scenario modeling
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
30
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
Calculating Avoided Emissions ndash More in the Guide and on the Web
bull Special Considerations ndash Additionality
ndash Assessment Boundaries
ndash Cap and Trade Programs
bull Emission factors for Grid-Connected Electric Generating Units
bull Resources
ndash State and regional efforts such as efforts by Northeast States for Coordinated Air Use Management (NESCAUM)
ndash EPA guidance documents which can be found at wwwepagovairqualityeere including EERE Roadmap for SIPs
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
31
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
Looking to the Future ndash EMampV for Documenting Avoided Emissions
(at least) nine factors to consider for designing EMampV bull Legislative or Regulatory Mandate Structure for
efficiency as a mitigation option
bull Scope and Metrics of a Standard including net versus gross savings requirements
bull Baselines
bull EMampV Approaches
bull Certainty of Savings Determination
bull Who Conducts the Evaluation Activities
bull Reporting and Schedules
bull Dispute Resolution
bull Regulatory Audiences and Requirements for StandardsProtocols
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013
32
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom
SEE Action Impact Evaluation Guide NASEO Presentation - Steven Schiller - May 2013 33
From Albert Einstein
ldquoEverything should be as simple
as it is but not simplerrdquo
ldquoEverything that can be counted
does not necessarily count
everything that counts cannot
necessarily be countedrdquo
Steve Schiller Contact srschillerlblgov or steveschillercom