2013 STATEWIDE AC CYCLING PROGRAMS PROCESS EVALUATION PG&E SMARTAC, SCE SUMMER DISCOUNT PLAN, SDG&E SUMMER SAVER PROGRAMS Findings & Recommendations November

2013 STATEWIDE AC CYCLING PROGRAMS PROCESS EVALUATION

PG&E SMARTAC, SCE SUMMER DISCOUNT PLAN, SDG&E SUMMER SAVER PROGRAMS

Findings & Recommendations

November 12, 2014

Presentation Agenda

PY2013 AC Cycling Process Evaluation Findings 2

Overview of Evaluation Effort Key Observations Evaluation Tasks Program Descriptions Integrated Findings Recommendations

Overview of PY2013 AC Cycling Process Evaluation Effort

3

Primary research objectives include: Document the AC Cycling programs’ administration and

delivery strategies Assessing the effectiveness of the program administration Evaluating participant experience with the programs and their

DR events

Secondary research objectives included: Review marketing costs per enrolled customer Examine the range of appropriate costs for AC Cycling programs

This presentation focuses on providing highlights of these overarching topics across all utilities, their

programs, and their customers

PY2013 AC Cycling Process Evaluation Findings

Key Observations

4PY2013 AC Cycling Process Evaluation Findings

Key Observations

5

The IOUs are implementing the three programs as designed Program is triggered on days with peak temperatures or high demand

Overall customer satisfaction is high among all three programs

Although each program has marked differences in the sub-groups studied

Customers who are notified of events are more satisfied and take more action to reduce energy consumption

But there was no clear strategy for notifications

Satisfaction with the program incentives were consistent for all IOUs

Despite the fact that the incentive structures vary significantly

Technologies are generally as effective as other programs in the US

Devices could be upgraded and override capability is a benefit

Marketing costs are slightly higher than industry average


Evaluation Tasks


Opinion Dynamics spoke with 1,160 active and lapsed customers to assess their experience with the program

7

Evaluation Task Description

Program Materials and Database Review

Analyzed the program databases to characterize and understand the participant population

Reviewed program marketing and outreach materials

Program Manager Interviews

Conducted telephone / in-person interviews with program and EM&V staff from PG&E (3), SCE (3) and SDG&E (2); and SCE account representatives (2)

Implementer/Contractor/ Aggregator Interviews

Conducted telephone interviews with GoodCents (PG&E), GoodCents and NRG (SCE) and Comverge (SDG&E)

Participant Survey

Fielded telephone survey with 880 participants ensuring representation by technology type, dual-enrollment, geography and duty cycle selection, where relevant

PG&E SmartAC (328) SCE Summer Discount (264) SDG&E Summer Saver (288)

Lapsed Customer Survey

Fielded telephone survey with 280 lapsed customers ensuring representation by geography, technology and duty cycle selection

PG&E SmartAC (68) SCE SDP (71) SDG&E Summer Saver (141: 96 residential and 45 nonresidential)

AC Cycling Process Evaluation Tasks

Program Descriptions


Combined, the AC Cycling programs have half a million participants and installed load control devices

9

AC Cycling Program Participants and Load Control Devices(As of December 2013)


AC Cycling Programs

PG&E SmartAC

SCE Summer Discount Plan

SDG&E Summer Saver

Residential Participants

Yes X X

Small Nonres Close to new enrollment X XLarge Nonres Not available X

Technology Offerings

Adaptive Load Switch Programmable

Thermostat Direct Load Switch Direct Load Switch

Event Triggers

Emergencies Economic dispatch Sub-lap System testing

Emergencies Economic dispatch Sub-lap System testing

Emergencies Economic dispatch System testing

Duty Cycle Options

All: 50% Residential: 100% and 50% Nonres: 100%, 50% and 30%

Residential: 100% and 50%Nonres: 50% and 30%

ContractorsScheduling and conducting installations, maintenance GoodCents

Scheduling and conducting installations, maintenance GoodCents and NRG

Aggregator model Comverge

Number of 2013 Events

3 sub-lap events 1 test system-wide test

event (all participants) 1 test sub-lap event

12 residential events (11 during the summer and 1 in the winter)

4 summer nonres events

6 system wide events

Event Periods in 2013

3-4 hours in evening for sub-lap events;

Varied for tests, with 1.5 hours in duration

Varied: most residential in late afternoon, with most with 2 hours duration; all nonresidential 1 hour in duration

All events to all segments four hours in duration

Incentive Structure

One-time incentive payment at enrollment

Summer monthly credits that vary according to duty cycle selection, AC tonnage controlled, and whether customers choose the option to override events

Bill credit calculated based on duty cycle option in at the end of year

Participant’s Experience


Despite limited knowledge of program design features and low awareness of AC Cycling events, satisfaction levels are high overall

12

While we present statistical significance when comparing within program results, no statistical tests were established for across program comparisons.

Residential Nonresidential

PG&E SmartAC 8.0 Not Applicable


8.5 8.2

SDG&E Summer Saver 8.3 7.3

Overall Program Satisfaction Ratings (0 to 10 point scale)


Dually Enrolled (PG&E)

Residential-Highest

Duty Cycles

Nonresidential Lowest Duty Cycle

Characteristics of Program Satisfaction


Effectiveness of Program Administration and Delivery Strategies


The program technology – adaptive/traditional load control switches and programmable thermostats - are appropriate for set-it and forget-it design

Failure rates for load switches are in line with national ranges.

All program technologies are one-way communicating devices, which provide limitations on:

Visibility into failed devices and available load as well as ability for the program to evolve to provide ancillary service and increase costs of QA/QC.

The Effectiveness of Load Control Technology

14

Failure rates: Within range of

similar programs for load switches (7% to 12%)

PG&E discontinued programmable

thermostats due to paging errors that lead to high failure

rates (40%)

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

Load Switches (12%)

Programmable Thermostats

(40%)

Fai

lure

Rat

es

National Average


Program options are not always clearly explained, potentially reducing potential savings from program

15

Lack of awareness of multiple duty cycle options were more pronounced among 50% and 30% Duty Cycle participants

Awareness of Multiple Duty Cycle Options

SegmentClearly

Explained

Not Clearly Explained/ Don’t

know

SDG&E

Residential 54% 46%

Nonresidential

28% 72%

SCEResidential 69% 31%

Nonresidential 62% 38%

Program options were clearly explained

30% Duty Cycle

(n=26)

50% Duty Cycle

(n=40)

50% Duty Cycle

(n=47)

15%

13%

5%

21%

18%

6%

RES

NO

N R

ES

SDG&E SCE

50% Duty Cycle (n=45)

50% Duty Cycle (n=105)

23%

45%

56%

40%


Note: Might be a self-selected group; satisfaction may be due to SmartRate program characteristics

- It might lead to additional savings

- Participants might feel discomfort when notified of events and leave the program

PROS

CONS

Event notifications are either non-existent (PG&E) or are opt-in. Evidence indicates those who receive notifications are more satisfied

16

Those who receive notifications (dually-enrolled) are more satisfied (PG&E)

Although some of this may also be attributed to other SmartRate features

Those who receive notification take action (PG&E)

Participants report taking action upon receiving notification (from other DR programs)

There are pros and cons to notifications:

Turn off lights (21%)

Delay use of appliances

(26%)

Unplug devices (20%)


Some participants report that they are likely to engage in actions that lead to snapback had they been aware of AC Cycling events

17

PG&E SmartAC

SCE Summer Discount

Plan

SDG&E Summer Saver

Res Nonres

Change temp settings

23% 10% 23% 18%

18% 6% 4% 16%

TOTAL 41% 16% 27% 34%

Precool

15% 22% 10% 16%

11% 8% 22% 10%

TOTAL 26% 30% 32% 26%

Participant Self-Reported Behavior had they been aware of an AC Cycling Events

Always

Sometimes


Satisfaction ratings are similar across utilities irrespective of incentive structure

18

(*) $ /calculated AC ton per day

Segment CategoryPG&E

SmartAC


(bill credit)

SDG&E Summer

Saver (bill credit)

Residential

Incentive

One-time $50 per load switch at installation

Maximum Savings (100%):• Standard: $0.36/AC

ton/day• Override: $0.18/AC

ton/dayMaximum Comfort (50%)• Standard: $0.18/AC

ton/day• Override: $0.09/AC

ton/day

• 100% duty cycle: $38/ton paid at year end

• 50% duty cycle: $11.50/ton paid at year end

Satisfaction with

Incentive7.6 7.9 7.5

Commercial

Incentive NA

• Maximum Savings (100%): $12.69/AC ton

• Good Value (50%): $4.44/AC ton

• Maximum Comfort (30%): $0.89/ AC ton

• (all per month)

• 50% duty cycle: $15/ton per month

• 30% duty cycle: $9/ton per month

Satisfaction with

IncentiveNA 7.7 6.8


Program Marketing


In general, marketing costs are slightly higher than industry average practices

20

Marketing costs Interviewed 12 program managers of AC Cycling programs across the

country Marketing costs are slightly higher (average of $68.00-69.00 per

enrolled customer) than industry average practices (average of $61.00/per enrolled customer for print mail)

Vary by the stage of the program, with newer programs having much higher marketing costs than established programs that are focusing on replacement of attrition

Marketing effectiveness Program marketing campaigns have, for the most part, included

customers in the entire service territories. This results in high costs and potentially suboptimal customer reach.

Targeted marketing, with the support of data analytics to identify optimal participants, can both reduce marketing costs per enrolled customer by increasing conversion rates, while simultaneously increasing program overall impact performance.


Program Recommendations


Leverage Smart Meter Data/Advanced Analytics

22

Use data analytics to actively target participants more likely to provide load reduction:

Target by geography and participant profile Emphasize warmer climates / avoid coastal areas Emphasize those with peaky load during likely event days (warm

afternoons) / Avoid those who do not Target areas of locational grid constraints/historically high Locational

Marginal Price (LMP)

Leverage data analytics to more readily identify failed load control devices:

Identify whether participants consistently do not respond to AC Cycling events (indication of failure device or a non-contributor)

This option may be limited where the events are mostly sub-Lap limiting participants’ exposure to multiple events

Will reduce costs of random inspections and unnecessary truck rolls and increase the reliability of program capacity


Upgrade technology for two-way communications to allow for provision of ancillary services (if desired)

23

Requires both an upgrade to the program technology and program design (to allow events outside of current afternoon windows). Options for upgrading the devices to allow for ancillary services include, where feasible:

Technology Upgrade Options

Short-term,Low cost

• Replace load switch one-way door with a two-way communication chip.

Medium-term,High cost

• Replace load devices with upgraded two-way communicating devices.

Long-term• Leverage the “internet-of-things” and allow for web-addressable technology-agnostic options driven by the market. Utilities move away from providing technology to enable demand response services.


Consider introducing widespread event notification

24

If the utilities choose to keep the program as currently designed and thus, not use AC Cycling programs for ancillary and other grid-regulation services:

Notify participants of AC Cycling events: this may increase satisfaction and load shifted (where participants may engage in additional energy-saving behaviors)

Consider testing event notification with a customer subset to establish whether notification yields additional savings and higher satisfaction, or lead participants to leave the program at a higher rate

Enhance education about program duty cycle options Where multiple duty cycle options offered, enhance materials explaining

options to allow participants to select a higher-level duty cycle: Account representatives should clearly explain these tradeoffs when

marketing the programs Call center representatives should be trained to encourage higher duty

cycle options provided the customer feels comfortable with that choice Consider introducing a higher duty cycle option (PG&E only).


Revisit incentive structure

25

The lag between event experience and the process surveys potentially introduces recollection biases in survey results.

Ideally, participants would receive a survey about a program event within a few days of experiencing the event.

The programs can use pre-programmed phone and/or internet surveys to collect participant responses immediately following an AC Cycling event.

To address false positives, surveys could also be administered in a typical AC Cycling day (hot afternoon temperature/humidity) where no events are called to normalize results.

Survey customers immediately following an AC Cycling event

Given similar satisfaction ratings across programs despite different incentive structures, it may be that multi-year incentives and/or incentive amounts could be revisited without a significant adverse effect on satisfaction.

We recognize that incentives needed to maintain capacity may be different from incentives needed to recruit participants.


Detailed Program Slides


One-time Incentive

$50 (since 2012; $25

earlier)None

Duty Cycle 50% for residential and 33% for previously enrolled nonresidential

SmartAC Only74%

Dually Enrolled

26%(n=154,398)

Load Switches

81%

PCTs19% (n=181,976)

Enrollment Option

Participants

Load Switche

sPCTs

Total Devices

SmartAC Only 114,751 109,032 18,117 127,149

Dually Enrolled (SmartRate)

39,647 37,912 6,086 43,998

Total 154,398 146,944 24,203 171,147

Participan

ts

Load Switche

sPCTs

Total Devices

Total 5,762 1,001 9,828 10,829

Number of 2013 events

5 5

Scope of Events3 sub-Lap /2

test3 sub-Lap /2

test

Event triggersEmergency/

TestEmergency/

TestAlignment with system peaks

Aligned Aligned

TechnologiesAdaptive Load Switch

PCTAdaptive Load Switch

PCT

Override Option

Offered* Offered*

Event Notification

Not offered Not offered

*online or through SmartAC hotline


Control Devices

Program Design Features Event Characteristics

PG&E SmartAC Program

Cycling Option

Participants

Control Devices

Standard

Incentive

Override Incentiv

e

Maximum Savings (100%)

268,424 273,022 $0.36 $0.18

Maximum Comfort (50%)

27,856 28,572 $0.18 $0.09

Total 296,280 301,594 Per calculated AC ton per day Cycling Option Participants

Control Devices

Incentive

Maximum Savings (100%) 6,034 7,309 $12.69

Good Value (50%) 1,775 2,417 $4.44

Maximum Comfort (30%) 694 913 $0.89

Total 8,503 10,639 Per

calculated AC ton per day


12 4

Scope of Events Most sub-LapSystem-

wideMost common event triggers

Economic Economic

Alignment with system peaks

Data not provided

Data not provided

TechnologiesDirect Load

SwitchT-Stat Pilot *

Direct Load Switch

T-Stat Pilot *Override Option

With capable load switch

Not offered

Event Notification

Not offered Opt-In

*1,300 thermostats tested in 2013

Maximum Sav-ings 91%

Maximum Comfort 9%

(n=296,280)

Maximum Savings

71%

Good Value 21%

Maximum Com-fort 8%

(n=8,503)



Duty Cycle Choices Among Residential Participants

Duty Cycle Choices Among Nonresidential

Participants

50% Duty Cycle(69%)

30% Duty Cycle(31%)

(n=4,620)

100% Duty Cycle(49%)

50% Duty Cycle(51%)

(n=23,085) Cycling

OptionParticipant

sControl Devices

IncentiveIncentive Limit

100% Duty Cycle 11,302 3,929 $38/tonup to $152

50% Duty Cycle 11,783 13,787 $11.50/tonup to $46

Total 23,085 27,716 Annual bill credit in December

Cycling Option

Participants

Control Devices

Incentive

Incentive Limit

50% Duty Cycle 3,201 7,511 $15/ton n/a

30% Duty Cycle 1,419 3,839 $9/ton n/a

Total 4,620 11,350 Annual bill credit in December


6 6

Scope of Events System-wideSystem-

wideMost common event triggers

System loadSystem

load

Alignment with system peaks

Aligned Aligned

TechnologiesDirect Load

SwitchDirect Load

SwitchOverride Option

Not offered Not offered

Event Notification

Opt-in by phone Opt-In by phone

Duty Cycle Choices Among Residential Participants

Duty Cycle Choices Among Nonresidential Participants


SDG&E Summer Saver Program

Documents

2013 STATEWIDE AC CYCLING PROGRAMS PROCESS EVALUATION PG&E SMARTAC, SCE SUMMER DISCOUNT PLAN, SDG&E SUMMER SAVER PROGRAMS Findings & Recommendations November