Upload
sharleen-roberts
View
214
Download
0
Tags:
Embed Size (px)
Citation preview
CPUC Meeting on 2010 – 2012 Evaluation
WO033 Custom Impact Evaluation Results
May 2, 2014
2
TOPICS
WO033 Custom Impact Overview
WO033 Custom Impact Elements
Evaluation Results
• Gross Impact Findings
• Net Impact Findings
• Lower Rigor Assessment (LRA) Findings
Selected WO033 Findings and Recommendations
Q&A/Discussion
Emphasis on new sampling and evaluation approaches, tailored scope allowing for measure- and program-level results, and new recommendations
WO033 Slides - Stakeholders Mtg 5/2/2014
3
WO033 CUSTOM IMPACT OVERVIEW
• Evaluation of Custom (non-deemed measures) across multiple programs and the four major California IOUs Core Calculated Programs Third Party Programs Local Government Partnerships and Universities / Colleges And More!! Over 100 programs within the WO033 scope
• Why is Custom Impact Evaluation Important? Custom measures account for 19% of electric [1,754 GWh] and
65% of statewide natural gas savings claims [134 Gigatherms] History of low realization rates for savings claims
Not Just Another Custom Impact Evaluation
WO033 Slides - Stakeholders Mtg 5/2/2014
4
WO033 CUSTOM IMPACT ELEMENTS
• What’s New? Two Periods, BD and AD, based on D.11-07-030 Large Gross Impact Sample Expanded Net Impact Sample Lower Rigor Assessment for Program-Level Impact
Feedback Interim Report for Early Feedback Stand-Alone Net-to-Gross Report (forthcoming) New Findings and Recommendations
These evaluation elements support program-level results for “programs of interest” and some additional measure-specific results
Not Just Another Custom Impact Evaluation
WO033 Slides - Stakeholders Mtg 5/2/2014
5
WO033 CUSTOM IMPACT SAMPLE SIZES
Accomplishments
(1) 429 gross impact points across five IOU-fuel analysis domains,
(2) telephone survey supporting net to gross (NTG) estimation and industry standard practice (ISP) determinations for a total of 1,388 projects, and
(3) a total of 536 engineering reviews supporting LRA assessment
IOU/Fuel Sampling Domain
Number of Completes
Gross Impact (M&V)* NTG LRA
PG&E Electric 112 558 155
PG&E Gas 75 230 97
SCE Electric 100 367 139
SDG&E Electric 73 125 73
SDG&E/SCG Gas 69 108 72
Total 429 1,388 536
* The 429 M&V sample points account for 495 electric and gas fuel-differentiated results. This is because some sample points included claims for both electric and gas savings.
WO033 Slides - Stakeholders Mtg 5/2/2014
6
GROSS IMPACT FINDINGS Project Lifecycle Gross Realization Rates by Fuel Domain and Energy Metric (kWh, kW, and Therms)
0
0.2
0.4
0.6
0.8
1
1.2
1.4
SDGE ElectricSCG andSDG&E Gas*
SCE ElectricPG&E GasPG&E Electric
kWh kW Therms
1.6
n=139n=91 n=100
n=118
n=88
n=77
n=59
n=94
n=84
WO033 Slides - Stakeholders Mtg 5/2/2014
* Lower value excludes extreme points.
7
PROJECT LIFECYCLE GROSS REALIZATION RATES BY SAMPLE DOMAIN AND ENERGY METRIC
Energy MetricSample Size (n)
Mean Gross Realization
Rate
Population (N)
Error Ratio90% Confidence
Interval
PG&E Electric
kWh* 139 0.59 6,994 1.27 0.49 - 0.70
kW 118 0.46 6,248 1.67 0.35 - 0.58
PG&E Gas
Therms* 91 0.64 1,270 0.58 0.58 - 0.70
SCE Electric
kWh* 100 0.61 3,052 1.03 0.51 - 0.71
kW 94 0.57 2,748 1.08 0.47 - 0.67
SDGE Electric
kWh* 77 0.55 1,469 1.08 0.44 - 0.66
kW 59 0.78 790 2.20 0.43 - 1.14
SCG and SDG&E Gas
Therms, all points 88 1.45 1,077 4.82 0.28 - 2.62
Therms, less extreme points
84 0.63 1,077 0.84 0.54 - 0.73
WO033 Slides – Stakeholders Mtg 5/2/2014
Note: First year savings are also reported in Chapter 5.
8
LIFECYCLE EX-ANTE AND EX-POST ELECTRIC SAVINGS (KWH) FOR SAMPLED PROJECTS
WO033 Slides – Stakeholders Mtg 5/2/2014
9
LIFECYCLE EX-ANTE AND EX-POST GAS SAVINGS (THERMS) FOR SAMPLED PROJECTS
WO033 Slides – Stakeholders Mtg 5/2/2014
10
GROSS IMPACT FINDINGS (CONT’D)
Program-specific and Measure–specific Findings
• GRR results for program and measure groups with substantial sample sizes examined in detail
• Core, third party and new construction programs generally have above average GRR results
• While quite sensitive to outliers, lower GRR results for compressed air, HVAC and HVAC controls measure groups
• Higher GRR results for HVAC controls (PG&E
Gas), HVAC (SCE), steam traps (SCG/SDGE)
WO033 Slides - Stakeholders Mtg 5/2/2014
11
GROSS IMPACT FINDINGS (CONT’D)Reasons for Discrepancies in Realization Rates
Three principal reasons observed operating conditions baseline specification the IOUs calculation methods
• Explain differences for 44%, 17% and 19% of the records studied • Impact on savings
observed operating conditions -17% (electric), -14% (gas) baseline specification -16% (electric), -14% (gas) the IOUs calculation methods -2% (electric), -1% (gas)
• Other reasons for differences incorrect equipment specifications ineligible measures incorrect measure counts tracking database discrepancy
WO033 Slides - Stakeholders Mtg 5/2/2014
12
FREQUENCY OF DISCREPANCY FACTORS
WO033 Slides – Stakeholders Mtg 5/2/2014
44%
19%17%
6%
3% 3%1%
4% 2%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
Operating Conditions
Calculation Method
Inappropriate Baseline
Ineligible Measure
Equipment Specifications
Measure Count Tracking Data Discrepancy
Other No Significant Discrepancy
13
EFFECT OF DOMINANT DISCREPANCY FACTORS ON TOTAL EX-ANTE KWH SAVINGS CLAIMS
WO033 Slides – Stakeholders Mtg 5/2/2014
14
GROSS IMPACT FINDINGS (CONT’D)Non-Residential New Construction
• 43 sites in the NRNC measure group (25 EnergyPro)• GRR: 81% (kWh), 72% (kW) and 51% (therms)
• Primary discrepancies Operating conditions Calculation methods Inappropriate baselines
• EnergyPro-related issues All whole building sites used T-24 schedules instead of the as-
designed scheduled to estimate the ex-ante savings Ex ante energy models were not created or updated to match the
physical as-built conditions of the building Operating under sub-optimal conditions, e.g., low occupancy
• Non EnergyPro-related issues Use of inappropriate baseline on data center projects Estimated I.T. loads on data centers were higher than observed
WO033 Slides - Stakeholders Mtg 5/2/2014
15
GROSS IMPACT FINDINGS (CONT’D)Monitoring-Based Commissioning
• 10 sites in the MBCx measure group GRR: 58% (kWh), 108% (kW) and 34% (therms)
• Primary discrepancies Validity of ex ante regression model Retrofit measures implemented during and after MBCx Benchmarking of project sites Reliability of energy meters and flawed metered data Changes incompatible with the building equipment capability Negative energy savings that may not be due to MBCx
measures Adjustments to energy usage baseline
WO033 Slides - Stakeholder's Mtg 5/2/2014
16
NET IMPACT FINDINGSNet to Gross Completes
• Mix of CATI and In Depth/Professional Interviews • Weighted NTGRs by program or program groupings
0
0.2
0.4
0.6
0.8
1.0
SDGE ElectricSCG andSDG&E Gas
SCE ElectricPG&E GasPG&E Electric
kWh Therms
n=558n=230
n=367 n=108 n=125
WO033 Slides - Stakeholders Mtg 5/2/2014
17
NET IMPACT FINDINGSNet to Gross Results
Results
Electric NTGRs Gas NTGRs
Statewide PGE SCE SDG&E Statewide PGE SDG&E/SCG
Weighted NTGR 0.47 0.46 0.49 0.45 0.53 0.56 0.50
90 Percent Confidence
Interval 0.46 - 0.48
0.45 -
0.48 0.47 - 0.50 0.43 - 0.48 0.50 - 0.57 0.53 - 0.58
0.42 -
0.57
Relative Precision 0.03 0.04 0.04 0.06 0.07 0.05 0.15
n NTGR Completes 1,050 558 367 125 338 230 108
N Sampling Units 11,515 6,994 3,052 1,469 2,347 1,270 1,077
Error ratio (ER) 0.52 0.59 0.44 0.43 0.79 0.46 0.99
WO033 Slides - Stakeholders Mtg 5/2/2014
18
NET IMPACT FINDINGSNet to Gross Findings
• Significant levels of free ridership have persisted Statewide NTGRs of 0.47 (electric) and 0.53 (gas) Similar in magnitude to NTGRs from prior cycles (Significant improvement in PG&E gas NTGR from
PY2006-2008)• Little evidence of actions taken to reduce free ridership via
changes in program design or implementation procedures • Factors causing non-zero levels of free ridership:
Size and sophistication of customers. Custom program design
• Despite these challenges, many strategies available to reduce free ridership
Many recommended in previous cycles
WO033 Slides - Stakeholders Mtg 5/2/2014
19
OVERALL NET REALIZATION RATES
• PG&E kWh – 0.27, kW – 0.22, Therms – 0.36 Evaluated as % of claimed – kWh – 44%, kW – 34%,
Therms – 59%
• SCE kWh – 0.30, kW – 0.28 Evaluated as % of claimed – kWh – 49%, kW – 46%
• SDG&E/SCG kWh – 0.25, kW – 0.36, Therms – 0.72 Evaluated as % of claimed – kWh – 50%, kW – 53%,
Therms – 124%*
* 54% excluding extreme points.
WO033 Slides - Stakeholders Mtg 5/2/2014
20
LRA Overview
LOWER RIGOR ASSESSMENT FINDINGS
• Uses a desk review process similar to EAR
• Provides early feedback based on qualitative information
• Findings by program/program group, fuel domain, and IOU
• Expanded LRA form in AD period
• Results supported by a minimum of 8 assessments/category
• Low-frequency issues (n<100) use “GNP” denominator, see next slide
• Tables highlight strengths of project documentation related patterns, trends areas for improvement
• LRA Score in Final Report – program-level LRA performance
WO033 Slides - Stakeholders Mtg 5/2/2014
21
LOWER RIGOR ASSESSMENT FINDINGS (CONT’D)
WO033 Slides - Stakeholders Mtg 5/2/2014
Issue Assessed Assessment Results (n)
Assessment Results (%)
Good Neutral Poor (OD) Poor (GNP)
Project Documentation and Specification
IOU application documentation complete and accurate 513 39% 33% 26% 28%
IOU tracking data complete and accurate 520 38% 42% 20% 20%
Project utilized pre-installation M&V 390 43% 27% 22% 30%
Appropriate baseline 507 64% 14% 21% 22%
Early replacement claim: Valid RUL / EUL approach used 249 48% 12% 18% 40%
Appropriate Calculation Method
Appropriate impact calculation method 492 43% 34% 21% 23%
All relevant inputs considered 458 63% 16% 18% 21%
Adequate values for all inputs 470 37% 40% 21% 23%
Appropriate HVAC interactive effects calculation method 67 55% 12% 4% 33%
Appropriate non-HVAC interactive effects calculation method 77 62% 17% 3% 21%
Project utilized post-installation M&V 505 33% 34% 30% 32%
Compliance with Program Rules Measures are IOU program eligible 520 84% 13% 3% 3%
Measures exceed code or industry standard practice 473 75% 12% 11% 13%
Multiple IOU fuel impacts properly accounted for (includes fuel switching and cogeneration)
44 43% 5% 4% 52%
If applicable, fuel switching supported with three prong test 16 31% 0% 2% 69%
Non-IOU fuel and ancillary impacts of project properly accounted for (cogen/waste heat recovery/ refinery gas, etc.)
94 27% 0% 13% 73%
Customer installation meets all program rules 518 75% 12% 13% 13%
Final Report Table 7-1: LRA Complete Portfolio Sample
Issue areas with fewer than 100 responses included only LRAs with valid responses in the denominator as shown in “Assessment Results” column, otherwise the count of all LRAs (536).
22
LOWER RIGOR ASSESSMENT FINDINGS (CONT’D)LRA Results Wrap-up
Issues with high rate of deficiencies (percent “poor” shown in parentheses):
• Project documentation (26%)• Pre- and post-retrofit M&V (22% and 30%)• Calculation methods and inputs, specifically HVAC interactive
effects (33%)
Issues with high rates of poor assessments within lower-frequency issue areas
Accounting for non-IOU fuel and ancillary impacts (73%)Accounting for multiple IOU fuel impacts (52%)Fuel switching supported with 3-prong test (69%)
Lower rigor assessment qualitative outcomes are not necessarily indicative of gross impact evaluation results (LRA Score vs GRR: R2=0.04 for kWh)
WO033 Slides - Stakeholders Mtg 5/2/2014
23
LOWER RIGOR ASSESSMENT FINDINGS (CONT’D)Final Report Table 7-3: IOU Fuel Domain Program Performance
WO033 Slides - Stakeholders Mtg 5/2/2014
Issue Assessed IOU Fuel Domain
PG&E Electric
PG&E Gas SCE SCG and
SDG&E Gas SDG&E Electric All LRA Average
Number of Assessments 172 97 139 72 73 536
Number of “above average” issues 5 6 3 3 1 –
Number of “below average” issues 1 1 1 6 11 –
Overall Assessment Score 0.73 0.93 0.33 -0.50 -1.83 -0.50
Project Documentation and Specification IOU application documentation complete and accurate 26% 23% 30% 17% 36% 26%
IOU tracking data complete and accurate 13% 22% 20% 24% 25% 20%
Project utilized pre-installation M&V 20% 15% 24% 25% 30% 22%
Appropriate baseline 14% 15% 22% 24% 37% 21%
Early replacement claim: valid RUL / EUL approach used 20% 13% 16% 17% 29% 18%
Appropriate Calculation Method
Appropriate impact calculation method 18% 14% 22% 21% 33% 21%
All relevant inputs considered 12% 19% 20% 21% 25% 18%
Adequate values for all inputs 20% 21% 21% 22% 18% 21%
Appropriate HVAC interactive effects calculation method 18% 11% 56% Small Sample 33% 33%
Appropriate non-HVAC interactive effects calculation method
29% 12% 17% 36% 0% 21%
Project utilized post-installation M&V 23% 26% 33% 38% 40% 30%
Compliance with Program Rules Measures are IOU program eligible 3% 7% 1% 1% 3% 3%
Measures exceed code or industry standard practice 12% 11% 6% 22% 10% 11%
Multiple IOU fuel impacts properly accounted for 60% 42% Small Sample 38% 70% 52%
If applicable, fuel switching supported with three prong test Small Sample 57% Zero Sample Small Sample 100% 69% Non-IOU fuel and ancillary impacts accounted for (cogen/waste heat recovery/ refinery gas, etc.)
84% 56% 88% 64% 71% 73%
Customer installation meets program rules 10% 14% 9% 17% 19% 13%
24
WO033 REPORT RECOMMENDATIONS
• Project Savings Estimation Improve implementation and QC of project operating
conditions, ex-ante baselines, calculation methods Improve accuracy of savings in IOU tracking databases
based on post-installation inspections and M&V Improve engineering and simulation modeling for novel
measures and projects with limited access to data• Sufficient Quality Control for Documentation and Tracking
Data Increase consistency between project files and tracking
data Minimize miscommunication of project claims. Provide additional scrutiny and QC
o Accurate, comprehensive project documentationo Internally reviewedo Consistent with tracking system entries
Provide electronic access to complete documentation
WO033 Slides - Stakeholders Mtg 5/2/2014
25
WO033 REPORT RECOMMENDATIONS (CONT’D)
• IOU Project Verification Perform pre and post-retrofit verification and M&V
more frequently, document results completely Goal to accurately identify
o pre-existing equipmento ex-ante operating conditionso baseline considerations, and other issues
• LRA Evaluation Reduce documentation inconsistencies Improve impact calculation methods Comply with program rules or justify exceptions Conduct LRA-type reviews to enhance project
reviews
WO033 Slides - Stakeholders Mtg 5/2/2014
26
WO033 REPORT RECOMMENDATIONS (CONT’D)
• Nonresidential New Construction Use as-built schedule to update estimates of ex-ante
savings and use “non-compliance” mode calculations for savings
Submit a Title-24 acceptance test report, perform site visits to verify key measures and revise energy model to physical “as-built” conditions
Provide technical outreach assistance to all program participants
Thoroughly review and make appropriate use of the data center baseline document
Conduct an ISP study focused on changes in load over time for various types and sizes of data centers
WO033 Slides - Stakeholders Mtg 5/2/2014
27
WO033 REPORT RECOMMENDATIONS (CONT’D)
• MBCX
Develop guidelines for regression modeling, review early stage projects
ensure conformance, applicability, exceptions Keep centralized records of all building operation changes,
especially for higher education campuses For project benchmarking, use historical data for buildings with
similar characteristics, test modeling with independent variables Use meters that conform to the adopted metering guidelines Verify that MBCx recommendations address the control system.
Recommended changes must be compatible with existing system
Retain records of each measure implemented, along with post-implementation functional performance tests
List specific activities done, verify work was done correctly Perform more extensive and thorough data collection, analysis,
model development and validation of the engineering models
WO033 Slides - Stakeholders Mtg 5/2/2014
28
WO033 REPORT RECOMMENDATIONS (CONT’D)
• Program Influence/NTG Adopt procedures to screen for and increase
efficiency levels for high likelihood free ridership projects. Advise customers to
o move to a higher level of efficiencyo undertake a bundled retrofit to ensure
deeper savingso move to another EE project under
consideration Carefully review the list of qualifying measures;
eliminate eligibility for standard practice measures Make changes to the incentive design. Tier
incentives by technology class. Consider adopting a payback floor.
WO033 Slides - Stakeholders Mtg 5/2/2014
THANK YOU
www.itron.com
CONSULTING & ANALYSIS GROUP
1111 Broadway
Oakland, CA