Development of RTF Research Plans (and Research Strategies):

RTF RoleDHP for FAF Research Plan

CC&S Research Plan

Adam HadleyRegional Technical Forum

March 17, 2015

2

Overview

1. Today, we are seeking to clarify the RTF’s role in developing research plans and “research strategies”

Also, we are seeking approval of research for two UES measures (which provide good examples of new approach for research):2. DHP for FAF3. Ducted Heat Pump Commissioning, Controls,

and Sizing (CC&S)

3

1. RTF’s Role: Research Plans

• Area of RTF where we most frequently run into issues

• Priority for this year to better define the RTF role in research plans and improve the process

Key Challenges• Funding

– When there isn’t a funder or owner, we struggle:• Potential funders have a difficult time signing off on a plan that might fall in

their lap• Research plans perceived to be too “baked” before reaching the potential

funders, not allowing flexibility for what is actually feasible• Risk approving something that programs use to claim savings, but research is

never initiated/completed

• Not always funding: even when there is funding or an owner, the RTF has struggled– What does RTF approval mean for the planned research?

• Previous approach put RTF in a quasi-policing role– Threat of deactivating measure if research wasn’t carried out– Inappropriate role for RTF

4 – Research Plans

Yet, non-Proven Measures Have Value

• Provide flexibility, enabling implementers to move forward with programs while data is collected to improve the reliability of the savings estimate

• Research Plan provides direction and clarity about what is needed to bring measure to Proven

• RTF development and review provides credibility and conveys regional importance

Therefore: We need to figure out a path to enable this flexibility and provide the direction and credibility needed, while not getting bogged down with each measure.

5 – Research Plans

Proposal:redefine Planning and Provisional measure categories

and acknowledge an interim discussion period

6 – Research Plans

7

Subcommittee Discussion

• Previewed this plan with the R&E Subcommittee on March 12

• Generally supportive of this direction• Identified a few aspects that need to be

further addressed (we will address each of these as we go through the presentation):– Role of RTF in funding discussion– What does the Research Strategy look like?– Role and timing of sunset dates

ParticipantsAndie BakerMike BakerDave BaylonTodd BlackmanLarry BlaufusEli CaudillRachel ClarkTom EckhartLauren GageBen HannasMark JohnsonCheryn MetzgerPeter MillerGraham ParkerBob RamirezDan RubadoJustin SpencerRobert WeberSarah WidderJennifer AnzianoAdam HadleyJosh Rushton

• Category: Planning– Define Research Strategy, Funders Not Yet Identified

• RTF Role: Describe research needs and estimated cost range• Method: Put together a “Research Strategy” = A research plan that is expected to meet the research

needs at the lowest expected cost to meet the guidelines; may be in more of an outline form than a formal Research Plan, but contains specifics of sample size, data to be collected, analysis methodology, and an estimated cost range

• RTF Approval: Means RTF agrees with the research needs and agrees the proposed method of data collection and analysis provides a reasonable way to meet the research goals; RTF acknowledges funders may alter research design to meet their needs

• Funding Discussions– Funders Refine the Research Strategy into a Research Plan

• RTF Role: CAT involved in discussions of edits to research plan, research status tracking (will seek guidance from RTF PAC around an addition role: hosting these discussions?)

• Method: Potential funders weigh research costs against measure savings potential; edit research plan as needed to fit within budget constraints

• RTF Approval: No RTF review/approval in this phase

• Category: Provisional– Finalize Research Plan, Funders Identified

• RTF Role: Provide opinion on whether the (revised) research is likely to meet the research goals for the measure

• Method: CAT to incorporate funders’ research design into an RTF Research Plan document• RTF Approval: Means RTF agrees the (revised) method of data collection and analysis will help with some

or all of the research goals, while acknowledging any potential challenges and limitations of the research. Also means results of research can be used to evaluate measure (no further primary data collection needed).

8 – Research Plans

RTF Decision: Approval of UES Measure or Standard Protocol to “Planning” Category, “Active” Status.

RTF Decision: Approval of UES Measure or Standard Protocol to “Provisional” Category, “Active” Status.

RTF Decision: Approval of UES Measure or Standard Protocol to “Proven” Category, “Active” Status.

9 – Research Plans

How would this work with respect to the Guidelines?• This is a case where the guidelines language would be lagging for a short

while• Communicating our guidelines is an ongoing task• Guidelines Edits

– Planning Category• “Research Strategy” required at time of measure approval– Add discussion of RTF role at adoption of research strategy

• Savings Estimate Reliability – Same as before– RTF Judgment Allowed (no data required)

• Impact Evaluation – Same as before– Treated as “Other UES”, where savings are estimated using one or more studies that may require

site-specific data collection (i.e. RTF UES estimate not used in evaluation).

– Provisional Category• Savings Estimate Reliability – Set to same as Planning– Removes requirement for reliable data being available to estimate the baseline– Removes requirement for model to be calibrated to baseline energy use

• Sponsored Research Plan required at time of measure approval– Add discussion of RTF role at adoption of research plan

• Impact Evaluation – Same as before– Savings estimates use the results of the research plan (i.e. use a research-adjusted RTF UES

estimate in evaluation); but if results not available, treat as “Other UES”

10 – Research Plans

How will this work with check ins?• Establish a process for checking in on the status of all Planning and

Provisional measures at one time– Enables a review of the full suite of measures to better weigh priorities

– Ensures that measures are not forgotten and provides an opportunity to check in on interest (RTF could choose to deactivate measures at this time based on lack of regional interest/support)

– Expectations: Review twice a year with R&E subcommittee, RTF, and RTF PAC (try to align with RTF work planning and utility budget cycles)

• What is the role of the sunset date?– Near term (ex: 1 year): Use only as a check in on the status of the

measure (similar to above)

– Longer term (ex: 3 years): Use as an opportunity to revisit regional interest and, if interest, revisit the savings estimate

R&E Subcommittee discussed this

Staff recommendation: Check in process above allows for streamlined check in and keeps sunset date focused on revisiting energy savings estimates

Decision

• We’re not looking for a decision on this approach, that will happen when we have specific guidelines language to review (later this year)

• Instead, support of this approach would be implied through acceptance of the following measure proposals

12 – Research Plans

13

2. DHP for FAF Research Strategy

Today, we are seeking RTF approval of the Research Strategy for the DHP for FAF UES Measure• At the January 2015 RTF meeting, the RTF approved this measure but with

Under Review status (sunset date of April 2015) to allow time for further review and revision of the Research Plan

Research Strategy

• Research Goals– Estimate the mix of DHP and electric FAF heating system operation under varying conditions of

control strategy, outside temperature, house heat loss rate, or other driving factors– Estimate the impact supplemental fuel usage has on measure savings

• Data Collection– Utility Billing Data (pre and post) – ~1,200 sites– Phone Survey - ~1,000 sites

• Supplemental fuel use, control strategy, house size, etc.

– Field Metering Study – 40 sites• Same as 2012 study, but these sites will all be in heating zones 2 and 3

• Analysis– Use Field Research data to develop and calibrate SEEM simulation (for houses with a solid heat

signature)

– Billing Analysis: Pre/post VBDD for each house• Use Billing Analysis and Phone Survey results to estimate impact of supplemental fuels (similar to method used

in zonal study: energy consumption pre and post of those with supplemental fuels vs those without)

• Also used as a QC check on resulting UES values after many manipulations through SEEM

– Use Phone Survey results to establish a population-weighted control strategy mix (and same for other critical variables)

– Using newly calibrated version of SEEM, estimate UES• Estimated Cost Range:

– (< $25k) ($25k-$100k) ($100k-$250k) ($250k-$500k) ($500k-$1million) ($1million-$2million) (> $2million)

14 – DHP FAF

Research Plan/Strategy Timeline

• 1st Review by Research and Evaluation Subcommittee: December 2014– Written comments from 14 participants

• Revisions by Ecotope and CAT: January 2015– Significant revisions were made to improve clarity; scope remained the same– CAT felt comments were addressed– Did not have time to circle back with commenters

• RTF meeting: January 2015– Measure approved, Under Review to allow time for working through any remaining Research Plan

issues

• Most Recent Version of Draft Research Plan• 2nd Review by Research and Evaluation Subcommittee: February 2015

– Conference Call with 22 participants– Main Discussion:

• Who will fund? • Whether to include (require?) expensive field study

– Technical Input• Plan needs clarity on how the usable billing analysis sample interacts with the usable phone survey sample• Plan needs clarity on what the field research achieves• Discussion whether field research is optional for the UES measure, or necessary per guidelines

• RTF meeting: March 17, 2015– That’s today!

15 – DHP FAF

Funders Not Yet Identified*, so “Planning” Category would apply

• RTF Role: Describe research needs and estimated cost range• Method: Put together a “Research Strategy” = A research plan that is

expected to meet the research needs at the lowest expected cost to meet the guidelines; may be in more of an outline form than a formal Research Plan, but contains specifics of sample size, data to be collected, analysis methodology, and an estimated cost range

• RTF Approval: Means RTF agrees with the research needs and agrees the proposed method of data collection and analysis provides a reasonable way to meet the research goals; RTF acknowledges funders may alter research design to meet their needs

*No specific funder has come forward, but we’ve started into the “Funding Discussions” with the likely suspects and haven’t finalized the research plan, yet. “Funding not yet committed” is probably a more appropriate statement here, but “Planning” is appropriate at this stage to allow the discussions to happen without the added pressure of RTF presentations, sunset dates, etc.

16 – DHP FAF

Proposed Motion“I _________ move the RTF, for the UES measure Residential Existing MH and SF

Ductless Heat Pumps for Electric Forced-Air• accept the ‘Research Strategy’ (slide 14);• set the Category to ‘Planning’;• set the Status to ‘Active’; and • set the Sunset Date to March [2016 or 2018].”

Note: If selected, the intention of the 1-year sunset date is to update the RTF on the status of funding discussions. It is not intended to be a trigger for revisiting the savings estimate or used for forcing research funding. If selected, the intention of the 3-year sunset date is for the RTF to check in on whether the measure is still of interest to the Region, and if so, update the savings estimate with any new information. RTF discussion of moving measure to Provisional would be triggered by funders having a Research Plan for the RTF to review.

Reminder on Savings estimates approved in January: • 3,722 kWh for Single Family• 5,696 kWh for Manufactured Homes

17 – DHP FAF

18

3. Heat Pump Commissioning, Controls, and Sizing Research Strategy

Today, we are seeking RTF approval of the Research Plan for the ASHP CC&S UES Measure• At the December 2015 RTF meeting, the RTF approved this measure but

with Under Review status (sunset date of April 2015) to allow time for further review and revision of the Research Plan

Research Strategy

• Research Goal– Estimate the baseline and efficient-case heat pump installation practices in the Region

as they relate to heat pump sizing, indoor airflow, and compressor/resistance heat control logic

• Data Collection– Field Audits: 120 baseline + 120 efficient-case

• House and duct audit, compressor and resistance heat controls, heat pump capacity, indoor airflow

– Utility Bills (post-only): 240 (for field audit sites)• Analysis

– Use field data to develop simulation modeling inputs to represent baseline and efficient-case heat pump installation practices with respect to HP/resistance control settings, heat pump sizing, and indoor airflow

– Use utility billing data as a rough check on the outputs of the simulation model, adjust heat pump installation input parameters as justified

• Estimated Cost Range:– (< $25k) ($25k-$100k) ($100k-$250k) ($250k-$500k) ($500k-$1million) ($1million-$2million) (> $2million)

Note: Other reasonable strategies have been proposed to replace the billing data collection and analysis. For example: collecting heat pump and resistance heat run-time data on a sub-sample of field audit sites to verify the understanding of the installation practice observed in the field audit. This isn’t expected to appreciably change the research goals or the estimated cost range for the “Research Strategy” level decision.

19 – CC&S

R esearch P lan /Strategy Tim elin e

• Technical Subcommittee Review (call): December 2014– Quick review – general agreement Research Plan will get the measure closer to proven

• 1st R&E Subcommittee Review (email): December 2014– Questions raised, not enough time to discuss prior to…

• RTF Meeting: December 2014– Measure approved, Under Review status to allow time for working through questions on Research

Plan• CAT, Ecotope, BPA, Navigant Discussion (call): January 2015

– Discussion of alternative approaches• No less expensive ideas identified• Reduction in sample size considered

– Agreement plan needs to be re-written to improve clarity• CAT and Ecotope make significant revisions to plan: February 2015

– Re-submit to BPA/Navigant team for review• CAT, BPA, Navigant, SBW Discussion (call): March 4, 2015

– Discussion of alternative approaches; Agreement to further discuss and refine plan– Discussion of comments/edits by Navigant and SBW; Agreement to make edits

• Revisions by CAT and Navigant: March 2015– Revisions per March 2015 discussion

• 2nd R&E Subcommittee Review (call): March 12, 2015– This was originally planned to cover the CC&S Research Plan, but instead focused on this entire

presentation and the new approach to research planning• Most Recent Version of Draft Research Plan• RTF meeting: March 17, 2015

– That’s today!

20 – CC&S

Funders Not Yet Identified*, so “Planning” Category would apply

• RTF Role: Describe research needs and estimated cost• Method: Put together a “Research Strategy” = A research plan that is

expected to meet the research needs at the lowest possible cost; may be in more of an outline form than a formal Research Plan, but contains specifics of sample size, data collected, analysis methodology, and an estimated cost range

• RTF Approval: Means RTF agrees with the research needs and agrees the proposed method of data collection and analysis provides a reasonable way to meet the research goals; RTF acknowledges funders may alter research design to meet their needs

*BPA is the likely funder for this measure, but we’ve started into the “Funding Discussions” and haven’t finalized the research plan, yet. “Funding not yet committed” is probably a more appropriate statement here, but “Planning” is appropriate at this stage to allow the discussions to happen without the added pressure of RTF presentations, sunset dates, etc.

21 – CC&S

Proposed Motion“I _________ move the RTF, for the UES measure Residential Heat Pump

Commissioning Controls and Sizing• accept the ‘Research Strategy’ (slide 19);• set the Category to ‘Planning’;• set the Status to ‘Active’; and • set the Sunset Date to March [2016 or 2018].”

Note: If selected, the intention of the 1-year sunset date is to update the RTF on the status of funding discussions. It is not intended to be a trigger for revisiting the savings estimate or used for forcing research funding. If selected, the intention of the 3-year sunset date is for the RTF to check in on whether the measure is still of interest to the Region, and if so, update the savings estimate with any new information. RTF discussion of moving measure to Provisional would be triggered by funders having a Research Plan for the RTF to review.

22 – CC&S

23

Backup Slides

GuidelinesRoadmap Volume5.3.1. Provisional Measures UES and Standard Protocol measures may be approved by the RTF in the Provisional category. Provisional measures are those that require additional data collection and measure assessment work before they can be approved by the RTF in the Proven category. The RTF collaborates with other parties to develop research plans for gathering the required data and completing the assessment work. Approval for the Provisional category requires RTF approval of such plans.

Savings Volume2.6.2. Research on Measures Research plans must be developed for each Planning and Provisional UES measure. These plans are approved as part of RTF measure approval and the date that the research is expected to be complete will be considered by the RTF in setting the sunset date. The plans should conform to the Research Plan Template (see Supporting Documents) and address the following topics: • Design and selection of a sample from which data would be collected. • Collaboration strategy for implementing the sample design. • Analysis plan for developing data needed to advance the measure to the Provisional or Proven category.Comparable plans may also be needed to support the update of Proven measures, such as data needed to update baseline characteristics to correctly model measure interactions.

3.6.2. Research on Provisional Measures The RTF in collaboration with program operators must develop a research plan for each Provisional Standard Protocol. The plan must be documented separately from the Standard Protocol so that the users of the Standard Protocol do not confuse data analyses needed to develop a Proven Standard Protocol with data analyses required by the Provisional Standard Protocol itself. The plan should conform to the Research Plan Template (see Supporting Documents) and address the following topics: • Design and selection of a sample from which Provisional data would be collected. The sample should represent the range of eligible

measure applications, so that the reliability of the simplified method can be thoroughly examined. • Collaboration strategy for implementing the sample design. • Analysis plan for developing best practice savings estimates for each of the Provisional data collection sites. • Comparison of the best practice and candidate simplest reliable savings estimates, across the range of eligible measure applications to

determine the reliability of the simplified estimation method. • As needed, revision to the simplest reliable savings estimation methods.