1

1

Closing the Loop Between Evaluators and Implementers

&

Innovation and Experimental Design

Edward Vine

Lawrence Berkeley National Laboratory

&

California Institute for Energy and Environment

Behavior, Energy and Climate Change Conference

Sacramento, CA

November 9, 2007

2

Topics

!Closing the Loop

! Importance of evaluation & what is at stake

! Different behavioral (institutional) models &

implications

! Recommendations

! Innovation and Experimental Design

! Importance of innovation and experimental design

! Past and current examples

! New opportunities

2

3

Closing the Loop Between

Evaluators and Implementers

4

Importance of Evaluation

Planning

ProgramDesign

ProgramImplementation

ProgramEvaluation

3

5

Use of Evaluation! Instrumental

! Directly influences decision-making or program

! Conceptual! Influences thinking about the program

! Legitimative! Justifies decisions made or responds to policy concerns

! Process (capacity building)! Cognitive and behavioral changes result from the user’s

involvement in the evaluation process

" Enhanced skills in evaluation, routine incorporation ofevaluation in the organization’s processes, and programimprovement

Source: Peters et al., IEPEC, August 2007

6

Evaluation Uses byProgram & Portfolio Managers

! Evaluation results can be used for:

! Evaluating performance (“report card”)

" Accountability and demonstration of success

! Improving program design and implementation

! Improving engineering & behavioral assumptions

! Prioritizing program and portfolio budgets

! Finalizing incentive payments

! Resource (strategic) planning

4

7

What Is At Stake

! California example

! IOU program budget: ~ $2 billion (2006-2008)

" $2.7 billion in net ratepayer benefits (resource savings minus

investment costs)

" Avoiding 3 giant (500 MW) power plants

" Reduction of 3.4 Mt CO2 (= 650,000 cars off the road)

! EM&V budget: $163 million (2006-2008)

" ~8% of total portfolio funding

! Incentives to utilities (CPUC Decision 2007)

" Shareholder earnings & penalties: $450 million (capped)

8

Different Behavioral (Institutional) Models:New York

! New York! All evaluation and implementation - managed by NYSERDA

! Implementers and evaluators work together

! Use of evaluation results! NYPSC places a high priority on evaluation, but does not have

a formal policy regarding the use of evaluation results

! NYPSC staff reviews evaluation reports

! NYPSC emphasizes to NYSERDA the importance of obtainingprogram recommendations from the evaluators andimplementing the recommendations as appropriate

5

9

Different Behavioral (Institutional) Models:New York

! NYSERDA’s evaluation of evaluations

! Review of recommendations over two years (2005 and 2006) via in-depth interviews and email survey

! 244 actionable recommendations across three cycles (2003-2005)

" Actions taken on 48% for 2003 & 2004 recommendations

" Actions taken on 67% for 2005 recommendations

! Evaluation capacity has increased for program and evaluation staff

! Evaluation is increasingly seen as useful

Source: Peters et al., IEPEC, August 2007

10

Different Behavioral (Institutional) Models:California

! EE programs - implemented by IOUs

! Impact evaluation - managed by CPUC! Process evaluation - managed by IOUs

! Firewall between impact evaluators & programimplementers! Has created perverse incentives: an incentive that has unintended and

negative consequences

! Example: evaluators cannot rely on data from implementers - must becollected twice

! How to communicate with IOUs so that evaluationfindings will be utilized?! CPUC and IOU relationship - cooperative or adversarial?

6

11

Different Behavioral (Institutional) Models:California

! Use of evaluation now! CPUC Ruling in place

" Within 60 days of public release, program administratorswill respond in writing to the final report findings andrecommendations indicating what action, if any, will betaken as a result of study findings

" In this follow-up response to each study, administratorsshould note any concerns they have over specific reportfindings and indicate whether they agree with the finalload impact estimates for the programs in question

12

Source: Shel Feldman et al presentation, IEPEC, August 2007

7

13

Different Behavioral (Institutional) Models:California

! How to build more trust in this model?

! Provide more timely feedback from evaluation

" Early feedback process memo is being prepared by CPUC

" “Early warning system” (on results & methodologies) possible?

" DEER updates

! Improve communication

" Presenting findings (not just delivering a report)

" Have a dialogue for discussing recommendations

" Study groups (implementers and evaluators) - Technical AdvisoryGroups - possible to develop methodologies together?

! Increase capacity building of CPUC evaluation staff

" Improve skill sets of staff (conferences, workshops)

" Work with trained evaluators

14

Recommendations[Based on CA, NY, WI experience]

! Portfolio /program design input: use evaluators’experience to ensure that programs incorporate lessonsfrom other programs and best practices:! General program concepts

! Strategies to reduce free ridership

! Design of tracking system to meet evaluation needs [next slide]

! Review of definitions (peak period), baselines

! Review of deemed impacts

! Review algorithms and inputs used by program implementersto estimate impacts of custom measures

8

15

Recommendations

! Integrate evaluation data collection within program implementationactivities

! Collect data when it is most cost-effective to do so

" Program application form, field installation form, nameplate data when installingequipment

! Collect data when it is most likely it will be accurate

" Survey participants immediately after training, get information on old equipmentbefore it is gone to support the baseline

! Avoid lost opportunities

" Get baseline data on the pre-existing equipment before it is gone

" Record key inputs and assumptions when calculating savings

" Ask “Why” when participants call to drop out

! Provide timely feedback and course correction indicators

" Send a process survey out with the rebate check

" Hand out a mail-back exit survey after training

Source: Erickson and Keneipp, AESP, May 2007

16

Recommendations

! Program delivery feedback! Increase likelihood of obtaining real-time feedback so that

adjustments can be made quickly - Heads up! Don’t wait for thefinal report!

! Alert implementer of:

" Potential issues/problems so they can be avoided in thefuture

" Improved calculations

" Documentation gaps or improved documentation

! Conduct small pilot projects - experiment!

9

17

Recommendations

! Establish assumptions & evaluation processes from theoutset to minimize potential conflicts between evaluators& implementers

! Ensure program design & implementation planning satisfyevaluation data needs

! Foster rapport between evaluation & implementationcontractors through open communication (convey trust)! Weekly meetings

! Annual or semi-annual program update meetings

! Working groups

! Invite evaluation team to early design & implementationdiscussions

18

Recommendations

! Develop an organizational culture that is characterizedby trust, transparency and a constructive approach tomistakes

! Have leaders endorse learning from evaluation! Present evaluation as a win-win collaboration (sharing of

information)

! Have sufficient resources dedicated to evaluationpractice

! Provide guidance on how implementers shouldrespond to evaluation recommendations

! Develop a strategic evaluation plan

10

19

Remaining Question

20

Innovation &

Experimental Design

11

21

Importance of Innovation

! Innovation inducement prizes (NSF 2007)! Focus on high risk/high payoff [research] projects

! Example: 3-4 prizes in first two years

! Prizes: $200,000 to $2 million each; plan for $5-50 millionannually

! Need to take risks! Jeff Bezos - Amazon

! Are we too risk averse?! Regulatory structure limiting risk

! Need to think outside of the box

22

Some Examples of Past and

Current Innovations (1)! Branding and ratings

! Energy Star

! HERS

! LEED

! Community programs (whole communities)

! PG&E’s Community Energy Management Program (Davis, Lodi)

! Hood River (Oregon)

! Espanola (Ontario, Canada)

! Cambridge Energy Alliance (now)

! Competitions for reducing energy use (carbon emissions)

! Madison Gas & Electric - competed with three other firms

! Rutgers: four prizes of $2500 each will be awarded to undergraduate students

12

23

Some Examples of Past and

Current Innovations (2)

! Market transformation initiatives

! Golden Carrot (refrigerator)

! NE-ISO Forward Capacity Market

! California Shareholder Incentives

! On-bill financing (OBF)

! New organizations

! NEEA, NEEP, SWEEP, SEEA, Efficiency Vermont, Energy Trust of Oregon,

24

Importance of Experimental Design

! Go beyond trial and error

! Systematic, empirical analysis

! What works, what does not work

! Causal analysis: isolate the effects of specific policy

interventions (programs)

! Ideally: true random experimental design

! Small-scale experimentation - experimental and control group

! Alternate: Quasi-experimental design

! Comparison groups

13

25

Recent and Ongoing Experiments

! San Marcos, CA social norms experiment (Schultz et al 2007)

! Information provided to 290 households (telling homeowners how their energyconsumption compared to the neighborhood average) with a mixed factorial design:

" 2 types of feedback (descriptive norm only vs. descriptive plus injunctiveinformation) X

" 2 consumption levels (consumption above- vs. below-average energy consumption) X

" 3 time periods (baseline, short-term follow-up, longer-term follow-up)

! Time of use pilot (BC Hydro)

! Customers randomly assigned to a control group (TOU meter, no TOU price) with a mixedfactorial design:

" 3 communication groups (low communication, high communication (includesemails), and high communication plus Blue Line Monitors)

" 3 rate classes (off-peak $0.063 and on-peak $0.19; off-peak $0.063 and on-peak $0.25;off-peak $0.045 and on-peak $0.28)

26

Recent and Ongoing Experiments

! Home energy displays (Sierra Pacific Resources)

! Residential customers receive one of 6 different displays

! Six month evaluation of energy savings, device performance, and customerexperiences

! Incentive offerings (BC Hydro)

! Residential customers receive one of 5 different incentive offers

" 20% reduction in electricity use will result in a 20% discount on bill

" 10% reduction may get a prize (TV)

" 5% reduction will result in a 5% discount on bill

" 10% reduction will result in a 5% discount on bill

" Voluntary reduction via information and education - no incentive (control)

! On-line tool (website)

" Customers can track their electricity reduction progress

" Includes specific recommendations to help teach customers to save

14

27

New Opportunities

What are your suggestions?