Findings from the Alpha Test of the Opower/Honeywell ... · • That PG&E work with Opower, Honeywell and FSC to revise the supporting materials that installers leave behind based

PG&E’s Emerging Technologies Program

Findings from the Alpha Test of the Opower/Honeywell Thermostat Pilot ET Project Number: ET11PGE1072

Project Manager: Sarah Bresko Pacific Gas and Electric Company Prepared By: Michael Perry Freeman, Sullivan & Co. 101 Montgomery set, 15th floor San Francisco, CA 94104

Issued: April 13, 2012

Copyright, 2012, Pacific Gas and Electric Company. All rights reserved.


ACKNOWLEDGEMENTS

Pacific Gas and Electric Company’s Emerging Technologies Program is responsible for this project. It was developed under internal project number ET11PGE1072. Michael Perry conducted this technology evaluation for Pacific Gas and Electric Company with overall guidance and management from Sarah Bresko. For more information on this project, contact [email protected].

LEGAL NOTICE

This report was prepared for Pacific Gas and Electric Company for use by its employees and agents. Neither Pacific Gas and Electric Company nor any of its employees and agents:

(1) makes any written or oral warranty, expressed or implied, including, but not limited to those concerning merchantability or fitness for a particular purpose;

(2) assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, process, method, or policy contained herein; or

(3) represents that its use would not infringe any privately owned rights, including, but not limited to, patents, trademarks, or copyrights.


ABBREVIATIONS AND ACRONYMS

HVAC Heating, Ventilation and Air Conditioning

PC Personal Computer

App Software application—either on an iPhone or a PC


TABLES

Table 1: Frequency of Responses to Overall Impression Questions

and Average Response ................................................ 12

Table 2: Degree of Agreement Questions About Supporting Materials ................................................................... 14

Table 3: Number of Times Respondents Have Used Each App .......... 15

Table 4: Tallies of Responses to Degree of Agreement Questions ..... 18

Table 5: Ranking of Reasons for Saving Energy .............................. 19

Table 6: Frequency of Responses to Overall Impression Questions

and Average Response ................................................ 19

Table 7: Number of Times Respondents Have Used Each App .......... 20


CONTENTS

EXECUTIVE SUMMARY 6

INTRODUCTION 8

BACKGROUND 8

Emerging Technology/Product 9

ASSESSMENT OBJECTIVES 10

TECHNOLOGY/PRODUCT EVALUATION 10

TECHNICAL APPROACH/TEST METHODOLOGY 10

Field Testing of Technology 11

Test Plan 11

Instrumentation Plan 11

RESULTS 11

First Survey of Participants 12

Overall Impression Responses and Comments 12

Web Portal Enrollment 13

Call Center 13

Installation and the Installer 13

iPhone App 13

Internet App 13

Supporting Materials Ratings 14

Frequency of use of Apps 15

Final open-ended Comments 15

Second Survey of Participants 16

High priority Issues 16

Overall Impression Responses and Comments 19

Frequency of use of Apps 20

Focus Group Results 20

First Question 20

Second Question 22

Third Question 22

Fourth Question 23

Fifth Question 23

Sixth Question 24

Seventh Question 24

Eighth Question 25

Ninth Question 25

Tenth Question 25

Leader Board 25

Data Analysis 25

EVALUATIONS 26


RECOMMENDATIONS 26

APPENDIX A SURVEY INSTRUMENTS 27

First Survey 27

Second Survey 29


EXECUTIVE SUMMARY

This emerging technology report documents the results from the Alpha test of the Honeywell communicating thermostat pilot and Opower behavioral software. In this wave,

17 employees of PG&E had the thermostat installed at their home. The thermostat includes programming that allows it to be remotely programmed through an iPhone app or a PC-based app. It also has the functionality to push messages to participants through the

iPhone app, alerting them to important aspects of their energy use.

Honeywell technicians installed the devices and made sure that participants’ iPhones were set up with the app. Participants received instructions on how to use the device and the apps.

Participants were surveyed twice over the internet. The first survey took place after the device had been in place for about two weeks and the second survey took place when the device had been in place for about five weeks. Participants also took part in a focus group

one week after the second survey in which they were asked for their opinions about the device along many dimensions.

The Alpha test is a first step in a broader effort to test the Opower/Honeywell solution in PG&E’s service territory.

PROJECT GOAL

The goal of the Alpha test is to test functional readiness of the thermostat and behavioral

software solution, to assess customer satisfaction with the installation process and with the thermostat and to assess customer engagement with the thermostat.

Goals of the broader project include:

• Driving a higher percentage of customers to program their thermostats with a custom schedule;

• Motivating customers to set a more efficient temperature when home;

• Increasing set-back temperatures (i.e., automating systems when customers are away from home); and

• Expanding the length of the set-back period.

PROJECT DESCRIPTION

The evaluation of the Alpha test consisted of four main steps, three of which are

documented here. The first step consisted of recruiting and installing Honeywell thermostat devices at the homes of 17 volunteers. Each volunteer is a PG&E employee. The recruitment and installation processes are documented here only insofar as participants provided information about them through surveys and focus groups.

The other three steps in the project consisted of two surveys about the customer experience and a focus group, also about the customer experience.

The Alpha test is not focused on energy savings, load balance or cost savings. It is focused

on fixing problems prior to the larger Beta test of the project. The Beta test may provide


energy savings by allowing customers to better control their energy use associated with climate control.

PROJECT FINDINGS/RESULTS

The main findings of the first survey are that:

• Respondents were impressed with the installation process and the installers, although some thought the installers should be more knowledgeable about the internet app and iPhone app;

• Respondents would like to have that ability to program the thermostat using the

device itself;1

• Several respondents had difficulties getting the iPhone app up and running at the beginning;2

• Call center reps came off as inexperienced; and

• Several respondents thought the way the thermostat responded to programming and the messages it sent could be improved.

The main findings of the second survey are that:

• 13 out of 15 respondents thought that they had reduced energy usage;

• 14 out of 15 respondents thought that the pilot should be rolled out to 750 customers;

• Customers were generally glad to have been part of the pilot;

• My Thermostat was the most popular name;

• The internet app is less well-liked and well-understood than the iPhone app; and

• The iPhone app is both well-liked and the most popular reason given for why the

respondent was able to save energy. Respondents also generally say that it is easy to use.

Focus group findings were varied. Overall ratings of the device by participants ranged from a 2 on a scale of 10 to an 8 on a scale of 10. Participants ranged from quite positive to

mildly negative in their view of how exciting, engaging and high-tech the device was. Most participants failed to notice messages being pushed to them through the iPhone app.3

PROJECT RECOMMENDATIONS

Recommendations from the Alpha test are:

1 This was mentioned by respondents to the first survey and in the focus group. 2 The process for loading the iPhone app was a temporary, testing solution only. For the Beta, the smart phone application will be downloaded in

the typical way all apps are downloaded.

3 Participants were specifically asked if they had noticed the messages being sent to them and most had not. This question was a lead-in to a

question about what types of messaging would be most useful.


• That Opower or Honeywell follow up with respondents who indicated either that the thermostat isn’t working or that one of the apps isn’t working. The most important

steps are to identify the problem and communicate back what the problem is, how it can be fixed and whether that problem is likely to occur often in the broader pilot;

• That Honeywell add a step to the installation process in which the installer fully verifies that the iPhone app controls the thermostat before the installation is

considered complete;

• That PG&E work with Opower, Honeywell and FSC to revise the supporting materials that installers leave behind based on the suggestions below;

• That the solution be modified to work with Android smart phones as well (a step that

is being taken for the Beta test).

INTRODUCTION

This report describes the procedures and results obtained in the Alpha test of the

Opower/Honeywell solution. The results from this evaluation will support PG&E, Honeywell, and Opower in the launch of Beta test of the same thermostat by providing useful user feedback, and evaluation of the thermostat’s functionality and communication quality.

The Alpha test assessment was designed to allow for testing of the thermostat in a friendly

environment in which PG&E could simulate the customer experience and receive important feedback about device operation, while not risking customer relations problems that might arise due to technical problems with the technology being tested.

BACKGROUND

Honeywell and Opower, Inc. have collaborated to develop a solution that includes an internet-enabled thermostat and behavioral software that allows residential customers to better manage their climate-control energy use.

The Alpha test of the project, documented in this report, consists of a pilot program in which 17 PG&E employees were recruited and had the thermostat installed at their home. The feedback from these employees will be used to modify the recruitment and installation processes and to alert Honeywell and Opower to technical problems that must be addressed

prior to the Beta test, which will be comprised of PG&E customers rather than employees.

The two main goals for the Beta test will be to develop understanding of:

• Energy savings potential from reducing residential heating, ventilation and air conditioning (HVAC) consumption through behavioral changes; and

• Residential consumer preferences and attitudes towards enabling technologies.


HVAC consumption is targeted for this project because it represents approximately 9% of electric consumption and 37% of gas consumption for residential customers in California.4

However, the percentage of total electric and gas consumption can vary drastically depending on the climate zone. HVAC consumption represents 30% of total electric consumption in several of PG&E’s climate zones, but can range as high as 45% for 2-story homes in specific climate zones.5

While many customers have programmable thermostats, research shows that the majority of customers do not know how to properly set their thermostats.6 This suggests that there

may be substantial savings available by making it easier for customers to set their thermostats and by providing reminders when it might be advantageous for them to change their settings.

The thermostat and software are meant to address this possibility by providing an intuitive programming interface and messaging, each delivered through an app on the customer’s iPhone or PC.

EMERGING TECHNOLOGY/PRODUCT

The solution is a combination of a Honeywell internet enabled thermostat with Opower

behavioral solution software, which includes normative messaging and recommended schedules through an iPhone Application (referred to as an app). The thermostat can also be programmed through an app on a participants PC. Through this messaging, the thermostat keeps customers better informed about their energy choices and the cost

ramifications of those choices. The solution design is intended to make it easy for customers to act and, by using psychological mechanisms and consistent encouragement and feedback, to motivate customers to reduce energy consumption.

The product in the home works as a standard programmable thermostat. The innovation is the simplicity of its programming process and its ability to interact with Opower software. The software nudges customers towards using less HVAC energy by making suggestions. The suggestions alert customers to times when the customer could save energy by allowing

the home to heat up more in the summer or cool down more in the winter.

The thermostat would replace either a conventional non-programmable thermostat, programmable thermostat or a thermostat.

The primary market barrier currently is that the customer must have an iPhone to use with

the thermostat, if they want to take full advantage of the mobile control. Opower is currently developing the software that will allow the thermostat to work with other types of smartphone. Though a smartphone will always be required for mobile control, a web

interface is also available for remote access and control.

4 California Energy Commission. “California Statewide Residential Appliance Saturation Study Final Report Executive Summary.” 2010 5 eQuest – Building Energy Use and Cost Analysis Software, developed by James J. Hirsch & Associates (JJH), version 3.64 was the latest release,

http://www.doe2.com/ 6 California Energy Commission. “California Statewide Residential Appliance Saturation Study Final Report Executive Summary.” 2010


ASSESSMENT OBJECTIVES

The data from the Alpha test of the project consists of responses to two online surveys and a focus group, each of which included subsets of the 17 PG&E employees who had the

thermostat installed at their home. Each survey and the focus group included a different subset of participants.

The data is evaluated through summary tables of survey responses as well as qualitative

judgments of comments made to open-ended survey responses and focus group questions. The primary objectives of the data evaluation are to test functional readiness, customer satisfaction and customer engagement of the thermostat.

Energy savings will not be determined in the Alpha test of the project, but will be

determined as part of the broader project.

TECHNOLOGY/PRODUCT EVALUATION

The evaluation of the thermostat in the Alpha test was done by PG&E employees in their homes. This was an ideal test bed as it reflects what PG&E customers will experience if they

choose to take part in a broader pilot of this technology.

The criteria that PG&E employees had to satisfy in order to take part were that they had to have central heating and air-conditioning and an iPhone, and live in a single-family home

with only one zone for climate control.

The assessment of the Alpha test has been performed by a team including staff from both PG&E and FSC. FSC is a contractor providing advice and assistance with many aspects of the Alpha and Beta tests, including sampling, recruiting and installation. FSC will be the

evaluator of the Beta test, and therefore will also produce estimates of energy savings impacts from the technology. For the Alpha test assessment, FSC has extensive experience in survey research and focus groups evaluating customer-facing programs in the utility

industry.

TECHNICAL APPROACH/TEST METHODOLOGY

The technical approach to the test is that of a field test in which the primary data of interest are all aspects of the participant experience. These include:

• How satisfied participants were with the recruitment and installation process;

• How much they liked the thermostat device and how useful they found it;

• How much they liked the iPhone and PC apps and how useful participants found the apps;

• Whether participants liked the messages that were sent to them through their iPhones;

• Whether participants thought that the technology would lead them to save

energy; and


• Whether participants thought that the technology should be introduced to PG&E customers in a Beta test.

This data has been evaluated mainly qualitatively to assess the technology and its installation process. Due to the small sample size and the qualitative nature of the data collected, there is no formal quantitative analysis of results from the Alpha test. Quantitative analysis will be performed as part of the Beta test evaluation.

FIELD TESTING OF TECHNOLOGY

This is a test of a thermostat technology, its associated iPhone and PC apps and the installation process for all three.

The sites for testing were homes of PG&E employees that satisfy similar eligibility criteria to the customers who will take part in the Beta test. Therefore this test bed

provides unbiased results.

Test sites may have a variety of different incumbent technologies, ranging from traditional thermostats to other types of programmable thermostat.

The system is controlled by the participant.

TEST PLAN

There is no baseline measurement for comparison. The implicit technology baseline is each participant’s previous thermostat and the test will rely on participants to make that comparison.

The energy consumption associated with the HVAC being controlled by the test technology will not be measured in the Alpha test.

The variables used to measure performance are responses to survey and focus group questions about the technology and its associated installation process. These

questions are detailed in the results section.

The monitoring period for the Alpha test is 6 weeks.

Data are gathered via online survey and through voice recordings and notes taken during focus group sessions.

INSTRUMENTATION PLAN

This section is not applicable because the only data collected are survey and focus group data.

RESULTS

This section is divided into three subsections describing results from the first survey, second

survey and focus groups, respectively. Copies of the survey instruments are provided in Appendix A.


FIRST SURVEY OF PARTICIPANTS

Survey invitations were sent out to all 17 Alpha test participants on December 23, 2011. Reminder emails were sent to non-respondents on December 27. A total of 15 employees responded by January 3, 2012. Those responses are summarized here.

The main findings of the first survey are that:

• Respondents were impressed with the installation process and the installers, although some thought the installers should be more knowledgeable about the

internet app and iPhone app;

• Respondents would like to have that ability to program the thermostat using the device itself; Several respondents had technical problems with the iPhone app;7

• Call center reps will require additional training to understand nuances of the

program; and

• Several respondents thought the way the thermostat responded to programming and the messages it sent could be improved.

OVERALL IMPRESSION RESPONSES AND COMMENTS

The survey contained several questions about participants’ overall impressions of aspects of the installation and features of the device and supporting software. Table 1 summarizes the responses to those questions. It shows the tally of the number of times respondents gave each rating on a scale from 1 to 10, with 1 meaning “very negative” and 10 meaning “very

positive.” There were no responses below 3 for any aspect of the installation or features.

TABLE 1: FREQUENCY OF RESPONSES TO OVERALL IMPRESSION QUESTIONS AND AVERAGE RESPONSE

Rating (1-10)

Feature 3 4 5 6 7 8 9 10 Average Rating

Web portal enrollment

| || ||| |||| | ||| 8.5

Call center | |||| ||| || ||||| 8.4

Installation of

your thermostat

| | |||| |||| | ||| 8.6

Installer of

your thermostat

| |||| |||| |||| 8.8

Thermostat || | | ||| ||| |||| 8.0

iPhone app | ||| ||| ||| |||| 7.9

Internet app

on your PC | ||| | || 8.6

7 The process for loading the iPhone app was a temporary, testing solution only. For the Beta test, the smart phone application will be

downloaded in the typical way all apps are downloaded.


Overall, responses were quite positive. The two lowest average scores were for the thermostat and the iPhone app. Each of those had more than one rating of 5 or below.

None of the other aspects or features received any rating below 5.

For all but one of the overall impression questions, respondents were asked to provide comments on why they gave the score they did. Only respondents giving scores of seven or lower responded to these questions (this was not a restriction, it just happened that way).

There were many useful comments which are discussed below.

WEB PORTAL ENROLLMENT

There were three comments about web portal enrollment:

• One respondent said that her husband signed up for the program using an email

address different from the one associated with their iPhone. This caused a problem at installation, and the respondent felt that it wasn’t clear that it needed to be the same address;

• Another respondent went through the enrollment process, and then when the installer showed up the app had disappeared from his/her phone, forcing the respondent to go through the process again; and

• A third respondent had trouble getting the Opower site to operate on his/her

iPhone. The respondent loaded it two or three times to “get it to be resident” and then it stopped working.

CALL CENTER

All three comments about the call center indicated that the representatives needed

additional training about the details of the program

INSTALLATION AND THE INSTALLER

Although the survey instrument differentiated between installers and the installation process, respondents did not. Respondents focused their comments on the installers.

Respondents were quite pleased with installers’ punctuality and respectfulness, and with the speed of installation. However, three respondents were disappointed that the installer was unable to help them install or use the iPhone app and could only refer the respondents to email instructions they had received. One of these respondents was specifically

disappointed because they said they had been told that “you will know everything about this program when the installer leaves.” Another of these respondents indicated that their device was still non-functional for controlling their heat and that that was costing them a lot

of money.

IPHONE APP

Two respondents said that they downloaded the iPhone app and it did not work. Another found the app messages to be confusing.

INTERNET APP

Only one respondent commented on the internet app as part of this section. The comment was that the app does not provide insight into the ramifications of the settings selected.


SUPPORTING MATERIALS RATINGS

The installers left behind a set of supporting materials to help participants use and understand the thermostat and apps. In the survey, respondents were asked to give

numerical answers indicating on a scale from 1 to 10 how strongly they agreed with a set of statements about the materials. A response of 1 indicates “strongly disagree” and a response of 10 indicates “strongly agree.” The tallies of these responses are shown in Table 2.

TABLE 2: DEGREE OF AGREEMENT QUESTIONS ABOUT SUPPORTING MATERIALS

Rating (1-10, Strong Disagree-Strongly Agree)

Characteristic of Materials 1 2 3 4 5 6 7 8 9 10 Average

Provide useful information

| ||| || |||| || || 7.2

Are easy to understand | || ||| || |||| | | 7.5

Do a good job describing

how the Smart Thermostat can help me control my energy use

and save money

| ||| || || ||| ||| | 6.9

Are missing important information

| ||| ||| ||| | | ||| 5.1

Responses to these questions were less positive than the overall impression questions. It appears that there is room to improve the materials along every dimension respondents were asked about.

As a follow-up, respondents were asked what additional information should be added to the supporting materials. Respondents gave these suggestions for additions to the material:

• How to install and use the app on the iPhone;

• Material indicating that the thermostat can only be programmed via the iPhone app or the internet app—that programming on the device itself is disabled (two respondents mentioned this issue);

• Material indicating that the thermostat literally interprets “wake-up time” as time

to turn the heat on. If the respondent wants the house to be warm before that, then he/she must set a wake-up time before the actual wake-up time. A similar point applies to “arrive home time;”

• Instructions on how to have different temperatures at different times while at home;

• Instructions on how to use the Vera router (FSC note: not sure what this is);

• A one-pager with all necessary steps for set-up;

• Clear examples of when to use the iPhone app and when to control the unit using its own controls; and

• Examples of when the HVAC will go on and off based on programmed arrival times, going to bed times, etc.


FREQUENCY OF USE OF APPS

Respondents were asked how many times so far they had used the iPhone app and internet app to control their thermostat. The tallies of these responses are shown in Table 3. In

general, respondents use the iPhone app much more than the internet app.

TABLE 3: NUMBER OF TIMES RESPONDENTS HAVE USED EACH APP

App 5+

times

3-4

times

1-2

times

Not at

all

iPhone App 12 1 2 0

Internet App

2 3 2 8

FINAL OPEN-ENDED COMMENTS

At the end of the survey, respondents were asked for anything else they would like to share about their experience so far. They gave the following responses:

• “The app installation process was very confusing. Also, there wasn't really any explanation of how to install the app on multiple devices;”

• “Suggested feature for Opower: leverage the geolocation feature of the iPhone to

offer option to let system detect that we are outside the home;”

• “Overall everything went pretty smoothly;”

• “The phone app is great. Easy and makes sense. The only thing I don't like is the physical thermostat. It is too big and it is too GREEN. Why does the light

not completely go out if I am not using it? I don't need the glowing nightlight in my kitchen;”

• “I made a mistake when I was setting my schedule, and I thought I had set the temperature very low during the day while we are usually out of the house. But

we had some contractors working replacing a bunch of windows, and they left doors open, one of them told me the heat was continually blasting the whole time they were working. This went on for a few days before I found out. I felt secure

that I had done the settings correctly. It would be great if the app could send you a text message, something like ‘Your heat is set at 68 degrees while you are gone from home today.’ No explanation necessary, just a fact that looks right or wrong. You could even let people choose the kinds of messages they'd like to

receive.”

• “Enrollment and putting the app on my iPhone was very easy. I don't have the URL for the PC app but would like to try that. The 2 guys who installed my

thermostat were very professional;”

• “So far the use / controls have been flawless;”

• “I would add a feature to dim the light on the thermostat;”

• “Outside of the installation of the app and the issue I had there, everything has

been very favorable. I appreciated the reminder call for the appointment, the warning that the crew was 45 minutes away and the fact that the crew sent to my home was courteous (even wore booties inside my home so as not to scratch of floors - nice touch that showed me they cared);”


• “As a customer, I was initially a bit apprehensive about the installation process because the guys installing it were being trained. This apprehensiveness quickly

faded because the gentlemen supervising them were very meticulous and supervised closely. I'd like to say that the installers were VERY professional. They arrived at the beginning of the 4-hour window, they wore shoe covers to protect my carpets, they explained everything very thoroughly and I felt

comfortable having them in my home. On the iPhone application it's not 100% clear to me how the leaderboard is calculated. Although this is a cool feature, I'm not sure I would change my behavior much to influence my position on the board. I'm already trying to maximize savings;”

• “Did not know that we had control on PC - entire program seemed to be iPhone focused. Having a good video to watch - about how to use would be helpful. Need to call help desk during non-business hours when I am home;”

• “Very easy, The web interface could be more informative, but I am a tech person and like more detail;”

• “The thermostat should have the option to program it directly. Versus stating when I get up, leave, return, etc...I would rather state when I would like the

equipment to turn off and on.”

SECOND SURVEY OF PARTICIPANTS

Invitations for the second survey were sent to participants on January 13, 2012. Reminder emails were sent to non-respondents on January 20. A total of 15 employees responded by January 24, 2012. Those responses are summarized here.

The main findings of the survey are that:

• 13 out of 15 respondents thought that they had reduced energy usage through use of the thermostat and apps;

• 14 out of 15 respondents thought that the pilot should be rolled out to 750 customers;

• Customers were generally glad to have been part of the pilot;

• My Thermostat was the most popular name;

• The iPhone app was more well-liked and heavily used compared to the internet app; and

• The iPhone app is both well-liked and the most popular reason given for why the

respondent was able to save energy. Respondents also generally say that it is easy to use.

HIGH PRIORITY ISSUES

There were several high priority questions that are addressed individually. All follow up comments to these questions are shown in Appendix B.

First, to the question of whether customers believed that they had reduced energy usage as a result of the thermostat. The tally was 13 yes and 2 no.


Next, Question 24 asked whether the pilot should be expanded to 750. The tally was 13 yes and 2 no. One of the no responses was from a respondent who fairly clearly meant to say

yes based on his follow-up comment.

The most popular name (Q22) by a wide margin was My Thermostat, with 8 first place votes and 1 second place vote. Next were EasySet Thermostat (0 first, 6 second, 3 third), and Mobile Access Thermostat (MAT; 1 first, 4 second, 3 third); followed by MobileStat (2 first, 1

second, 3 third) and Intellistat (1 first, 2 second, 3 third).

Question 15 asked “Besides the support materials left at your home after installation, would you like to receive additional educational communications or information regarding your Smart Thermostat?” The responses were 3 yes, 7 no and 3 don’t know. See the Appendix

B for further details on what customers would like to see.

Table 1 below contains the tallies of the answers to question 8, which is about each respondent’s degree of agreement with a set of statements. The answers were mixed,

although the most positive response came to the question about whether the respondent was glad to have signed up.


TABLE 4: TALLIES OF RESPONSES TO DEGREE OF AGREEMENT QUESTIONS

Statement

Degree of Agreement Rating Tally

Average 1 (Strongly Disagree) 2 3 4 5 6 7 8 9

10 (Strongly Agree)

The thermostat, iPhone, and web applications raise my concerns about data privacy.

|||| ||||| || | | 2.7

The thermostat, iPhone, and web applications have met my expectations.

| || | || ||| ||||| 8.4

The thermostat, iPhone, and web applications give me greater control over my energy usage.

| | | || || || ||||| 8.1

The thermostat on the wall is easy to use.

| || | | ||| |||||| 7.9

The iPhone application is easy to use.

| | ||| |||| ||||| 8.7

The web application is easy to use.

| ||||| | ||| | ||| 6.4

The iPhone app and web applications provide useful information.

||| | | ||| || || ||| 7.0

I’m glad I signed up for the trial.

| ||| || |||||||| 9.3

I make adjustments to my temperature settings when coming home early or late.

| || | |||| | || |||| 7.0

My initial pre-programmed settings have kept me comfortable.

| || | | |||| || || | 6.3

I like the messages I receive when I make temperature adjustments.

|| || ||| | ||| | | || 5.9

The thermostat, iPhone, and web applications have made me more aware of how much energy I use.

|| || | |||| || || 6.3

Those who answered yes to whether they had saved energy were asked to rank in order the

reasons they thought they had done so. Table 2 shows the tally of responses. Remote access through the iPhone, followed by the initial temperature profile were the main reasons. The respondent who ranked “other” first gave this reason: “I was able to reduce my temperature setting to 62 at night with no discomfort, versus the 65 previous night time

setting.” There were two other written in reasons provided under other.


They were both ranked third and they were:

• “can check my temp and adjust if I am not coming home.”

• “Easy Scheduling Control”

TABLE 5: RANKING OF REASONS FOR SAVING ENERGY

Feature Ranking Average

Ranking 1st 2nd 3rd 4th 5th

Setting up my temperature profile

upon enrollment || |||| ||| ||| 2.1

Remote access to my thermostat through my iPhone

|||| |||| ||| 1.2

The messages I have received || |||| ||| ||| 4.1

The Leader Board feature of the iPhone app

|| |||| |||| || 3.5

Other | |||| |||| ||| 4.1

Four of 15 respondents had problems connecting with or controlling their thermostat

remotely. Four of 15 respondents had to contact customer service—three of which were

included in the customers who had trouble connecting.

OVERALL IMPRESSION RESPONSES AND COMMENTS

The survey contained four questions about participants’ overall impressions of the

thermostat, supporting software and customer service. Table 3 summarizes the responses

to those questions. It shows the tally of the number of times respondents gave each rating

on a scale from 1 to 10, with 1 meaning “very negative” and 10 meaning “very positive.”

There were no responses below 5.

TABLE 6: FREQUENCY OF RESPONSES TO OVERALL IMPRESSION QUESTIONS AND AVERAGE RESPONSE

Rating (1-10)

Feature 3 4 5 6 7 8 9 10 Average Rating

Thermostat

| |||| | |||| |||| 8.5

iPhone app

|| | |||| |||| | || 8.3

Internet app on your PC

| | | ||

| 7.3

Customer Service

||

| | 8.2

Overall, responses were quite positive, although there were fewer total responses about the

internet app and customer service. The iPhone app, thermostat and customer service

received very high ratings. The internet app received the lowest scores.


For all but one of the overall impression questions, respondents were asked to provide

comments on why they gave the score they did. These are shown in Appendix B.

FREQUENCY OF USE OF APPS

Respondents were asked how many times so far they had used the iPhone app and internet

app to control their thermostat. The tallies of these responses are shown in Table 3. In

general, respondents used the iPhone app much more than the internet app.

TABLE 7: NUMBER OF TIMES RESPONDENTS HAVE USED EACH APP

App 15+

times 6-15 times

2-5 times

Not at all

iPhone App 7 6 2 0

Internet App 2 3 2 8

FOCUS GROUP RESULTS

On January 30 and 31, FSC conducted focus groups with Alpha test participants. Eight

members of the Alpha test participated in the January 30 group and one member

participated on January 31. Results from the focus groups are inherently non-quantitative

and results are presented as synopses and paraphrases of what was actually said.

The responses of each participant are paraphrased below. Participants were called on in a

random order for their response. Respondents are referred to by a randomly-assigned

number for anonymity. Respondent 9 was the participant from the January 31 group.

FIRST QUESTION

For the first question of the focus group, the FSC moderator asked participants “On a scale

from 1 to 10, 1 is boring, 10 is exciting, how would you rate it, both hardware and

software?”

• Respondent 6: rating: 8, but only if it would interact with her more. Would like more messaging on iPhone, displaying progress toward a goal. She would like to look at how she was consuming energy relative to how she was in the past. She learned she had set her thermostat wrong and she didn’t get a message that told

her she shouldn’t have been running as much as she was.


• Respondent 4: rating: 5. It would be nice to have a two hour default so she could raise a couple of degrees and then go back. She also likes the idea of

having a comparison to prior usage.

• Respondent 3: rating: Hardware was a 2 to a 3. Could be higher if it was something like a Nest product that didn’t look and feel like a traditional thermostat. Application gets a 5. The product did a good job of getting him to

set up settings. Would be better if it pushed notices to him that told him that he had a message over the IP. After he set it up, the only time he went back into it was to show a friend he could do it. He didn’t realize that there was a leader board until Sarah [Bresko] told him later. He didn’t know where he stood relative

to others. The product didn’t push useful information to him. He would like to have control with other end-uses through the device, not just the thermostat, such being able to lock front doors. It would be better if it was something that

gave him a reason to go into it other than just adjusting the thermostat.

• Respondent 2: rating: Hardware a 7 and application an 8. Could improve if he could ask questions and get answers, if it was really smart and gave some guidance and feedback. He had a bad experience on the programming side. His

daughter was there when it was installed and she didn’t really get the education and when he tried to do the programming himself it wasn’t obvious how to program it. He would like to be able to tailor his settings to be a bit off of when

he wakes up or goes to sleep. It would be good to get a text message when he’s away asking whether he wants to change his setting. It would be good if tied into other applications. He thinks the thermostat is low touch and maybe could/should be high touch. The comparison stuff didn’t make much sense to

him. He didn’t understand it.

• Respondent 1: rating: solid 8 for both hardware and software. He really liked the leader board aspect but the more effort he put into saving gas, the worse that he was getting on the leader board. He liked the built-in intelligence, it was

very understandable. He liked the cool factor of controlling his thermostat remotely. One thing to think about implementing is a geo location feature that knows when everyone is away from the house and the software would know this

and adjust accordingly.

• Respondent 8: rating: – Software and 8, hardware a 7. He liked the cool factor of remote control. His wife complained that she wanted to control things as well but it only allowed one iPhone in this pilot. After awhile he didn’t see the benefit

of the tool unless they were out of schedule – such as away at a movie and they would turn off the schedule. Also, he could preheat house when coming home off-schedule. He would really like an estimated bill feature for the end of the

month that would tell him where he lies relative to a budget goal. He had similar experience with leader board as Respondent 1. It wasn’t clear who he was being compared to. He had to have people come back and install a repeater because it didn’t work well from the beginning. Then in the last week, his iPhone app failed.

So he has had two failures.

• Respondent 5: rating: Device a 3 and the app a 5. The device could be a lot more high-tech and engaging. It looks like every other thermostat. The device is big and bulky, not sleek. She had trouble programming, and she said her

furnace accidentally ran 24/7 for 3 days. She wants override ability. She would like to be able to program it on the thermostat rather than just remotely or on the PC. She thinks energy use is actually increasing rather than decreasing

because of lack of control of settings. She is looking for a lot more flexibility. The leader board didn’t provide any context. What are you being measured on.


The recommendations it gave she had already implemented. She didn’t see any value in the information.

• Respondent 7: rating: none – More lenient because it’s an evolving journey. It’s a nicer thermostat but it’s not Nest. Integration into the iPhone was very helpful as a family. He and wife were very engaged with the application. The leader board – not sure if there is value yet because the comparison group is small. The

ability to know what the temperature is and engage with it remotely is very good. Overall experience is an 8, but to get to a 10, get rid of the bugs. It needs more push messaging – there is no push messaging. Needs to push notifications either of something going wrong or to draw you in. Eventually you will need a self-

aware thermostat, like a Nest. Hardware shouldn’t be dependent on human interaction.

• Respondent 9: Rate both app and thermostat as an 8. She would want the

thermostat to show the time. Her prior thermostat let her set things on four periods M through F and then on Saturday and Sunday. Would like thermostat to be more interactive like her prior thermostat. Likes the leader board on the app. Would like app to be a little more intuitive – if you lowered temperature by 3

degrees, you would save $X. Push more information out. Could do more with the leader board (smiley face when doing well). Wants it to show what temperatures are when they are set.

SECOND QUESTION “At what point would more push messaging or tracking of where you are start to bother

you? Who would worry about this?”

Respondents 8 and 5 had worries about the security of the data. Push messaging doesn’t

bother them as long as information is secure. As long as the benefits are there in terms of

reduced bills they are in favor of it.

Respondent 7 says we are constantly giving away information but those who will be

involved will consider the balance between intrusion and value.

Respondent 1 says the amount of interaction you want needs to be part of the design.

Some want more, some want less. Need to be able to self select.

THIRD QUESTION “Knowing what you know now, would you recommend to your friends that they replace their

current thermostat with this new one? If yes, what would you say its advantages are and if

you wouldn’t recommend it, why?”

• Respondent 3: Depends on the cost. If it was free, he would recommend it because

most people don’t know how to program their thermostat and this would be enough

if free. He would recommend to friends that are technologically challenged. It gets

people over the hardware challenge that a standard thermostat has.

• Respondent 2: Yes, but based on some caveats. If a family has a very diverse

schedule.

• Respondent 1: Yes. It saves some energy and provides education for the kids,

especially if leader board is working correctly. He also likes the cool factor.


• Respondent 4: Yes, he would recommend it but would like to see something that

covers the whole house. Also he would like to have something that zones the house.

• Respondent 5: Maybe. If they didn’t already have a programmable thermostat to

begin with, she would recommend it but would also suggest looking at other things

on the market, such as Nest. She would probably recommend Nest instead.

• Respondent 8: Yes, at a reasonable cost, but it depends on current usage and how

much they would save. He likes the remote capability. H would think that this is a

step toward controlling other end uses.

• Respondent 6: if there was no cost and nothing to lose, she would say yes. The

highest value items are the remote control features. It is also easier to program

than on thermostat itself.

• Respondent 9: when away on vacation, she could see what the house temperatures

were. She has cats so she could set the temperature up in the evenings to take care

of the cats. Also she likes to know what current temperature is when away.

FOURTH QUESTION The fourth question asked about preferences for the iPhone app versus the PC app. All participants preferred the iPhone app.

FIFTH QUESTION “Is this ready to be tested on a larger group of 750 customers? If not, what needs to be fixed before doing this?”

• Respondent 7: yes, but the software must be bullet proof. Can be NO hiccups

between hardware and software.

• Respondent 3 doesn’t think we have a good handle on whether this really saves

energy. In his case, he thought it cost him more and not knowing this it could be a

customer nightmare. He would develop better functionality before going larger. He

wants push messaging to be part of it and he would look for a different, better

thermostat. He also would want to see broader control of other end uses. By the

time they deploy this, he is afraid this will be obsolete.

• Respondent 4 would like to see it add other end-uses first. She wants to see cost

broken down by end-use. Her bill went up and she wants to know why.

• Respondent 8: roll it out, but he wants more features. Installations need to be

correct the first time. Need to have installs on the weekend. Taking days off work

to get installed is bad. If there are problems, need to get technical people out there

very quickly, like that day. He would also like to see color coded information saying

whether you are or aren’t improving relative to goal. He says yes to next step but

doesn’t know if next step should really be focused on energy savings, but maybe

about other things like remote control—maybe that is the sell. There is also a need

to work through troubling installations. He also wants more interface options and

more messaging. He called help desk during the day when at work but it would be

easier if he was standing in front of thermostat.

• Respondent 6: yes, this is a good thing. She would love to see a test group for

SmartRate customers.

• Respondent 9: It would depend on the demographics of customers. If we rolled it

out to her dad, it’s not going to happen. If we’re looking at Silicon Valley, gen Y and

maybe some gen X’s, they would do better. The performance of the device and


installation are ready for prime time. They went smoothly and the installer did a

good job.

SIXTH QUESTION “If you are a non-employee customer, how can you be marketed to, to say this is available?”

• Respondent 8: Home depot, Lowes, PG&E MyAccount and home market expos.

Coupons or kiosk in thermostat aisle.

• Respondent 6: Find out who you want and go after them. High bill folks? Tech

savvy people? Use a screening survey.

• Respondent 5: For a pilot, plug into existing channels for residential customers to

minimize impact to customers. We have an HVAC QM program. Can we tie into

these customers so don’t have to schedule another appointment – do it when HVAC

vendor is already there. For larger group, Home Depot, social media, people that

already doing service calls for something else. Don’t want to have to wait in a four

hour window to get installation.

• Respondent 1: Random, outbound calls from call center.

• Respondent 3: retail alliances are important but small. A bigger channel will be the

HVAC contractors. Piggy back on existing programs. Information at the Apple Store.

• Respondent 2: Send marketing with high bills notices. People investing in retrofit

programs. Focus in areas that have negatively complained about Smart Meters.

Movie theaters.

• Respondent 4: Should add to other smart phones, but for now should sell through

the app store.

• Respondent 9: Once again, it depends on demographics. Her dad reads the paper

and that would be a better way to reach him. She has an iPad and iPhone and she

gets information from there. Facebook and twitter could work, depending on target

market. If you’re pursuing tech savvy people, she thinks you should tweet it and

push it out that way. She doesn’t think Craig’s list would be good here.

SEVENTH QUESTION “Will rolling this out now enhance PG&E’s image or not?” (Respondent numbers not

recorded) • Respondent: Depends on how we market it. It should help if done well.

• Respondent: It all goes back to what expectation we set up front and do we meet or

exceed that expectation. Are we setting an expectation that they will save energy.

If they don’t exceed it they may be unhappy.

• Respondent: Don’t sell on saving energy, sell on education and remote control. It’s

empowering.

• Respondent: In general it will enhance PG&E’s image but only if done well.

• Respondent: needs to be lowest common denominator technology. Can’t just be for

the techies—need to be simple for everyone (like the color coding of the EV

mentioned).

• Respondent: Focus on control and comfort, not energy savings.


• Respondent: What is the goal – energy savings, customer satisfaction, choice?

Does PG&E have to show savings to put the product into play?

• Respondent 9: It will enhance it. Putting information in the hands of customers is

important.

EIGHTH QUESTION Participants were asked whether they liked the messaging they had been receiving. No one remembered any messages.

NINTH QUESTION “What kind of messages would be useful?”

• Respondent (number not recorded): Remind them to use it.

• Respondent 6: Wants to know how she is doing relative to a goal and to alert her if

something looks unusual or wrong so she can check it out. But give people choice

regarding the type and amount of messaging.

• Respondent 5: If you are setting your thermostat out of the norm for the

population, ask whether that’s what you really meant.

• Respondent 1: Messages should only emphasize those that entail significant dollar

amounts. He would like to see a message that gives tips for saving energy and

money and ties behavior change to dollar amount.

TENTH QUESTION

In the tenth question participants were asked whether they would like having the ability to program the thermostat itself rather than having to program it through the iPhone or PC

apps. Participants agreed that that would be an important and useful feature, specifically for times when they cannot access the apps.

LEADER BOARD

Although not part of any particular focus group question, several participants expressed frustration with the leader board function of the iPhone app. This part of the app compares

a customer’s HVAC usage and climate control choices to other customers and places customers in a ranking based on who is most efficient. Participants had trouble understanding the rankings and more than one mentioned frustration at undertaking an

effort to improve their leader board rank only to fall in the rankings.8

DATA ANALYSIS

Based on their analysis of their own operations, Opower provided certain statistics on technical problems that affected participants and on usage of the thermostat and the apps. Major findings were that:

• All participants were affected during a 17-minute outage of the Opower database. During this time, the thermostat would not respond to commands;

8 The leader board function was removed from the app and will not be shown to Beta customers.


• Due to other problems between the Opower database and the thermostat operating system, seven users experienced total outages greater than 1 hour in length and two

users experienced total outages greater than 2 hours in length;

• Due to problems between the operating system and the thermostat itself, one user experienced 20 hours of downtime and another experienced 82 hours of downtime. During these periods, the thermostat did not respond to user commands. This was

caused by a poor connection between the Gateway (which houses the operating system) and the thermostat itself;

• Aside from these issues, there was very little downtime for participants;

• Participants viewed the mobile app an average of 0.8 times per day and the PC app

about four times less frequently;

• The mobile app usage was steady over the course of the study period;

• The average time spent on the mobile app was 2-3 minutes per use; and

• The thermostat page was the most frequently viewed page on the mobile app, followed by the schedule page.

EVALUATIONS

The main results of the evaluation are that the technology works well, although there are

certain refinements that would be useful. The new technology appears to be an improvement on existing thermostats in many ways.

The primary marketing barrier is that customers must have iPhones. This limitation is being addressed in the Beta test by expanding the technology scope to include Android smart

phones.

Energy savings due to the technology has not yet been measured, but seems plausible. It will depend completely on participants in the Beta test using the thermostat to set lower

temperatures at home during the winter and higher temperatures during the summer, primarily when no one is at home.

RECOMMENDATIONS Recommendations from the Alpha test are:

• That Opower or Honeywell follow up with respondents who indicated either that the thermostat isn’t working or that one of the apps isn’t working. The most important steps are to identify the problem and communicate back what the problem is, how it

can be fixed and whether that problem is likely to occur often in the broader pilot;

• That Honeywell add a step to the installation process in which the installer fully verifies that the iPhone app controls the thermostat before the installation is

considered complete;

• That PG&E work with Opower, Honeywell and FSC to revise the supporting materials that installers leave behind based on the suggestions below;

• That PG&E adopt the name My Thermostat for the product; and


• That the thermostat be modified to work with Android smart phones as well (a step that is being taken for the Beta test).

APPENDIX A SURVEY INSTRUMENTS

First Survey Smart Thermostat Employee Trial

Survey 1: Installation and Initial Impressions

Introduction:

Thank you for taking the time to participate in this trial and completing this survey. This survey should take around 10 minutes to complete. We encourage your candid feedback

and there are many optional open-ended questions to enable you to expand on your impressions. Your responses will be used for research purposes only.

To begin, please click on the button below.

1. We are interested in understanding your overall impression of the web portal

enrollment that you used to sign up for the Smart Thermostat Program. Using the following 10-point scale, where “1” means “very negative” and “10” means “very

positive,” please rate your overall experience with the web portal enrollment process. 1 2 3 4 5 6 7 8 9 10

Very negative Very positive

2. [if Q1 is not 8-10] Why did you give a rating of [insert rating from above] for the online

enrollment process? Please be as specific and detailed as possible. [Open end] 3. We are interested in understanding your overall impression of the call center, which

you used to schedule your installation and may have used if you had any other issues.

Using the following 10-point scale, where “1” means “very negative” and “10” means “very positive,” please rate your overall experience for any interactions with the call center.

1 2 3 4 5 6 7 8 9 10


4. [if Q3 is not 8-10] Why did you give a rating of [insert rating from above] for the call

center experience? Please be as specific and detailed as possible. [Open end and not required]

5. We are interested in understanding your overall impression of the installation of your thermostat. Using the following 10-point scale, where “1” means “very negative” and “10” means “very positive,” please rate your overall experience with the installation

process of the thermostat. 1 2 3 4 5 6 7 8 9 10


6. [if Q5 is not 8-10] Why did you give a rating of [insert rating from above] for the

installation process? Please be as specific and detailed as possible. [Open end and not required]

7. Did the installer arrive at the agreed upon time? (yes/no)


8. [If no to 7] You indicated that the installer did not arrive at the agreed upon time. Please provide us some more information about when your installer arrived. [Open end

and not required] 9. Did you feel that the amount of time it took to install the thermostat solution was

reasonable? (yes/no) 10. [If no to 9] You indicated that the amount of time it took to install the thermostat

solution was not reasonable. What do you think would be a more reasonable amount of time? [Open end and not required]

11. We are interested in understanding your overall impression of the installer of your thermostat. Using the following 10-point scale, where “1” means “very negative” and

“10” means “very positive,” please rate your overall experience with the thermostat installer.

1 2 3 4 5 6 7 8 9 10


12. [if Q11 is not 8-10] Why did you give a rating of [insert rating from above] for the installer? Please be as specific and detailed as possible. [Open end and not required]

13. Please indicate your level of agreement with the following statements using a 10-point scale where “1” means you “strongly disagree”, “10” means you “strongly agree”, and

you can use any number between 1 and 10… (Randomize statements)

1 2 3 4 5 6 7 8 9 10

Strongly disagree Strongly agree

The support materials left at your home after installation: Provide useful information

Are easy to understand Do a good job describing how the Smart Thermostat can help you control your energy use

and save money Are missing important information

13b. [If Q13 is not 8-10] What additional information would you suggest could be added to the support materials? Please be as specific and detailed as possible. (open end and not required)

14. We are interested in your initial impression of the thermostat. Again, using a 10-point scale, would you say you…?

1 2 3 4 5 6 7 8 9 10

Do not like this thermostat

Like this thermostat very much

15. About how often have you tried using the iPhone app to control your thermostat? 5 times or more

3-4 times 1-2 times Not at all

16. [If more than not at all to prior question] We are interested in your impression of the iPhone app. Again, using a 10-point scale, would you say you…?

1 2 3 4 5 6 7 8 9 10

Do not like the iPhone app

Like the iPhone app very much


iPhone app? Please be as specific and detailed as possible. [Open end and not required] 18. About how often have you tried using the internet app on your PC to control your

thermostat?


5 times or more 3-4 times

1-2 times Not at all 19. [if more than not at all to prior question] We are interested in your impression of the

internet app on your PC. Again, using a 10-point scale, would you say you…

1 2 3 4 5 6 7 8 9 10

Do not like the internet app on your PC

Like the internet app your PC


internet app on your PC? Please be as specific and detailed as possible. [Open end and not required]

21. Is there anything else you would like to share about your experience (enrollment,

installation, use of devices, etc.) so far with the Smart Thermostat Program? [Open end] 22. If we have follow-up questions about your responses, what is the best phone number

and time for us to reach you? [Open end] We’re almost finished. Our last question is for classification purposes only. D1. How many people, including yourself, live in your household? (Choose only one)

1, just me 2 3 4

5 or more That is the end of our survey.

Thank you very much!

Second Survey Smart Thermostat Employee Trial

Survey 2: Thermostat Usability

Introduction:

Thank you for taking the time to participate in this trial and completing this survey. This

survey should take less than 10 minutes to complete. We encourage your candid feedback and there are many OPTIONAL open-ended questions to enable you to expand on your impressions. Your responses will remain anonymous and will be used for research purposes

only. To begin, please click on the button below.

1. We are interested in your overall impression of the thermostat to date. Using a 10-point scale, would you say you…?

1 2 3 4 5 6 7 8 9 10

Do not like this thermostat

Like this thermostat very much


2. About how often have you tried using the iPhone app to control your thermostat? a. More than 16 times

b. 6-15 times c. 2-5 times d. Once e. never

3. [If not e] We are interested in your impression of the iPhone app. Again, using a

10-point scale, would you say you…?

1 2 3 4 5 6 7 8 9 10

Do not like the iPhone app

Like the iPhone app very much

4. [if prior question answered] Why did you give a rating of [insert rating from

above] for the iPhone app? Please be as specific and detailed as possible. [open-

end and not required]

5. About how often have you tried using the internet app on your PC to control your thermostat?

a. More than 16 times b. 6-15 times c. 2-5 times

d. Once e. never

6. [if not e to prior question] We are interested in your impression of the internet

app on your PC. Again, using a 10-point scale, would you say you…

1 2 3 4 5 6 7 8 9 10

Do not like the internet app on your PC

Like the internet app your PC

7. [if prior question answered] Why did you give a rating of [insert rating from

above] for the internet app on your PC? Please be as specific and detailed as possible. [open-end and not required]

8. Please indicate whether you agree or disagree with the following statements using a

10-point scale where “1” means you “strongly disagree”, “10” means you “strongly

agree”, and you can use any number between 1 and 10... [Randomize

statements]

1 2 3 4 5 6 7 8 9 10

Strongly disagree

Strongly agree

a. The thermostat, iPhone, and web applications raise my concerns about data privacy.

b. The thermostat, iPhone, and web applications have met my expectations.

c. The thermostat, iPhone, and web applications give me greater control over my energy usage.

d. The thermostat on the wall is easy to use. e. The iPhone application is easy to use.

f. The web application is easy to use. g. The iPhone app and web applications provide useful information. h. I’m glad I signed up for the trial.


i. I make adjustments to my temperature settings when coming home early or late.

j. My initial pre-programmed settings have kept me comfortable. k. I like the messages I receive when I make temperature adjustments. l. The thermostat, iPhone, and web applications have made me more aware of

how much energy I use.

9. If you have any comments about the ratings you just provided, please note them

here. Please be as detailed as possible. [open-end and not required]

10. Have you had any problems connecting with or controlling your thermostat remotely?

a. Yes

b. No

11. [if yes to prior question] Please describe the problems you had. Please be as specific and detailed as possible. [open-end and not required]

12. Did you need to contact customer service for support at any point?

a. Yes

b. No

13. [if yes to prior question] Did you feel that your customer service experience was positive? Using a 10-point scale, would you say you had a very positive customer

experience (10) or a negative customer experience (10)?

1 2 3 4 5 6 7 8 9 10

Negative customer experience

Positive customer experience

14. [if Q14 is not 8-10] Why did you give a rating of [insert rating from above] for

your customer service experience? Please be as specific and detailed as possible (Optional). [Open end and not required]

15. Besides the support materials left at your home after installation, would you like to

receive additional educational communications or information regarding your Smart Thermostat?

a. Yes b. No c. Don’t know

16. [If 15 is yes] What additional communications or information would you like to

receive? Please be as specific and detailed as possible. [open-end and not

required]

17. [If 16 is yes] How would you like to receive the additional information? (Choose all

that apply) [Randomize options]

a. By mail, in the same envelope as my PG&E bill b. By mail, but in a separate envelope from my PG&E bill c. Via email d. Online at pge.com

e. Mobile application on your SmartPhone f. Phone call from PG&E g. Text message on your mobile phone from PG&E


h. Other (Specify)

18. Do you believe that you have reduced your energy use as a result of the thermostat

and messaging provided through this pilot? [Yes/no]

19. If no, please provide feedback on why you feel the thermostat and messaging have not helped you reduce your energy use. [open-end and not required]

20. If yes, please rank in order of importance the following reasons why you think you

have been able to reduce your energy use (where 1 is the most important reason and 5 is the least important reason):

Reason Ranking

Setting up my temperature profile upon enrollment _______ Remote access to my thermostat through my iPhone _______ The messages I have received _______ The Leader Board feature of the iPhone app _______

Other _______ Please explain ___________________________________________________

21. If you have any comments about the ratings you just provided, please note them here. Please be as detailed as possible. [open-end and not required]

22. If PG&E decides to roll this out beyond the employee pilot, a name will need to be created to market the program. Based on your experiences with the thermostat solution please rank your top three choices for a name: [Randomize]

a. My Thermostat b. My Therm c. Intelistat

d. Mobilestat e. Remotestat f. Customstat g. Mobile Access Thermostat (MAT)

h. EasySet Thermostat i. OptiTemp Thermostat

23. What other feedback, if any, do you have concerning your participation in this pilot (Optional)? [Open end and not required]

24. Based on your overall experience with the participating in this pilot so far, would you

recommend that PG&E expand the pilot to include participation by up to 750 customers?

a. Yes

b. No (if no, Please indicate the primary reason or reasons why you think PG&E should not expand this to a broader customer population at this time.).

25. If we have follow-up questions about your responses, may we contact you? a. Yes b. No


26. [if yes to prior question] What is the best e-mail/phone number and time for us to reach you? (We will contact you only if we have follow-up questions to this survey.)

That is the end of our survey. Thank you.

Documents

Findings from the Alpha Test of the Opower/Honeywell ... · • That PG&E work with Opower, Honeywell and FSC to revise the supporting materials that installers leave behind based