"Siri, did I leave the oven on?" UX for the connected home (updated for IA Summit 2013)

“Siri, did I leave the oven on?”Mundane UX for the connected home

@clurr #connectedhomeTuesday, 4 June 2013

Weʼve become accustomed to using technology like the web and mobiles to keep us in touch with the people and activities that are important in our lives. Still one thing thatʼs relatively unconnected - home - big dumb box of mostly dumb things that donʼt talk to us, or each other. But thatʼs changing.

This talk is about the challenge of making new technologies make sense to the mass market.

Tuesday, 4 June 2013

Wiredʼs number 1 London startup 2012@alertmesays


You probably havenʼt heard of AlertMe. Last year we were nominated Wiredʼs number 1 startup in London.


One reason you might not have heard of us because we are mostly B2B: we create hardware and services that other companies sell on to their end customers. Currently, you can get our products and services from British Gas, Loweʼs and Essent. We cover areas like heating, energy monitoring, home controls and monitoring, and data analytics. Weʼll find out a bit more about some of this later.

Opinions are entirely my own :)


And I canʼt take credit for many of interfaces I will be able to show today.

Whatʼs a connected home?


Embedded computing in everyday objects...

...connected up to the internet


Sensors and controllers around the home, embedded computing in everyday objects, and connecting it all up to the internet so you can access and control it via web and phone. Lots of things you can do here...

Understand energy use...

Energy clamp

In-home display

Smart plug

Web and mobile interfaces


For sake of time, Iʼll show some AlertMe examples to give you a flavour of the kind of thing I do. This is AlertMeʼs current energy service. Itʼs made up of sensors, displays, smartphone and web apps.

(We make apps because they are a better experience for a control system right now.)

Control your heating...

Thermostat

Web and mobile interfaces


This is AlertMeʼs current remote heating controller. Stuff magazine gadget of the year.

Camera

Contact sensor

Key fobs

Motion sensor

...secure your home...Tuesday, 4 June 2013

AlertMe home security.

Thereʼs more... lighting, switches, locks, appliances, catflaps...

...on a combined platformTuesday, 4 June 2013

This is the web UI for the Iris system we make for Loweʼs. I canʼt take any credit for this UI, which is a custom version for Loweʼs and was well developed before I joined AlertMe.

This home automation stuff has been around for ages though, hasnʼt it?


Connected home technology has existed since at least as far back as 1975...

This is X10 Powerhouse for the Commodore 64, from 1986.


It let you schedule lights and appliances to turn on and off, control a burglar alarm and thermostat, and could be operated remotely by telephone. Those are pretty much the things Iʼm working on right now. Except the telephoneʼs got a bit smaller and now we have the internet.

...but you had to be rich...


...and/or a geek


For most people, the benefits just didnʼt outweigh the cost

“Little bits of smartness”


Things are changing... getting cheaper, wireless... better designed...Big smart home concept isnʼt with us yet but you can already buy a range of everyday things like bathroom scales, baby monitors and electrical sockets with connectivity and even intelligence built in.

We have a metaphor for the “remote control for your life”


Donʼt underestimate how powerful this is. Mobile is central control platform: whether youʼre out and about, or on the sofa. whatʼs the point of embedding connectivity and smartness around your home only to have to sit down at a desk to use it on your Commodore 64?

Interconnectivity is still a challenge...

but so is understanding and delivering what the mass market actually needs


NB: big UX opportunity.

4 key UX challenges


There are many, here are 4 of the big, general ones for designing interoperable systems for the mass market.

UX challenge 1:

Make it feel like home


Home is a very personal context.

AT&T Digital LifeTuesday, 4 June 2013

Greenwave Reality


Home Automation Ltd(yes, really)


Bit unfair to pick on these guys, they are a relatively small outfit. But this screen sums up a lot of what I think is wrong with home automation.

Spot the design metaphor?


Spot the design metaphor?


•System has users and peripheral devices•Users have access permissions and are IN or OUT•Their goal is to program the home for optimal efficiency


Um, yeah.

ʻRomantikʼ modean engineering solution to a human non-problem


Modes are a common smart home feature. But they require a lot of planning and advance configuration. Which isnʼt very sexy.

Real life is too messy to program•People are generally a bit disorganised and bad at predicting their future needs

•Life is full of contradictions and exceptions

•Devices are shared, and lent

•Whoʼs allowed to do what is negotiated and flexible, not completely codified


e.g. Little Jack isnʼt normally allowed to watch that much TV, but today heʼs ill so youʼre feeling sorry for hime.g. The sheets ought to be washed but everyoneʼs busy so theyʼll do for a bit longer

“My teenagers skulk in their bedrooms. Theyʼre not out, but theyʼre not really in either...”


We already have a perfectly good metaphor for the home:

Itʼs the home This one happens to be my home. I donʼt want to log into it, become a super user, or worry that itʼs going to crash or need debugging.

sudo open-window


Last place in the world I want to feel out of control... and we all know how people often feel out of control of computers when they are too hard to use or do things we donʼt understand.

This is not a mass market solutionNew systems like WeMo are neat but basically better-designed early adopter kit


You define your own problem and configure the algorithms to execute it.

Iʼm prepared to be proven wrong, but I donʼt think this is the mass market solution

UX challenge 2:

Making data visible has social consequences


•There is often more than one person in a house

•They have interpersonal dynamics

•They may want different things

•Relationships are smoothed by not necessarily knowing everything about each other


The home is a complex social context

Connected home technology surfaces information about what is happening in the home


These are the unknown measures from my Withings scales: the ones it wasnʼt able to recognise as a known user.

This data is anonymous but based on time of use and estimates of mass I can infer two things from this

My cat sits on the scales


My cleaner is watching her weight


and neither of them have any idea that I know this.

• Itʼs often possible to work out who is in, out, turning the heating up all the time, or on the Xbox at 4am

•When parties have different ideas about how things should be, that surfaces tensions

21 °C 19 °C


Thatʼs all pretty innocuous, but add in other simple data like when the burglar alarm was set, and energy monitoring, and you can figure out...

Itʼs a healthy and necessary part of most relationships to have the right to some private space, and to ignore or pretend not to notice some of the other personʼs behaviours. Technology makes this harder.

Tension between the person who uses the energy monitor and the people who use the appliances is common


Tumble dryers are a particular source of angst.

Who came in at what time? (Did they look drunk? Was anyone with them??!)

How long did the cleaner really stay?

If this information is up on the internet, who might get access to it?

Presence surfaces trust and privacy issues


UX challenge 3:

The mundane should not demand too much attention


•A lot of what goes on in the home is actually pretty unremarkable and mundane

•We develop routines to help us stay on top of the boring stuff without too much conscious effort

•This allows us to save our attention for important or interesting things


My washing machine is as needy as a burglar alarm


It thinks washing is most important and interesting thing in my life. It beeps when itʼs finished a load. Itʼs a bit aggressive, but I could let that go. But it doesnʼt stop beeping until you empty it. It expects you to drop everything and come running, right now, because the washing must come out IMMEDIATELY. This is appropriate behaviour from a burglar alarm, but not a washing machine.

What if you had a whole home full of attention seeking devices?...


one device, irritating, but we accept it.

a whole home full of devices with no manners.... developing some new and interesting ways to break down?

This is attention seekingTuesday, 4 June 2013

Too much work for most people.

User instructions:

1) Ignore it


We need to design things that need less attention, not more.

Wattbox - intelligent heating controller (prototype hardware shown).

Nest: works off motion, light, and settings you choose in first week or two. If you donʼt bother to turn the heat down when you go to bed, or your heating controller sits somewhere people donʼt pass by, it might not learn correctly. Wattbox uses electrical activity to infer whether anyone is in, and whether they are up and about. When we're in and awake we're usually using electricity above baseload, itʼs a good proxy for occupancy and activity.

“Donʼt make me think” harder about my heating... HEATING IS BORING!!!

UX challenge 4:

We donʼt understand how to use half the stuff in our homes anyway


Most of us understand our heating systems about as well as this guy does


This is a ring tailed lemur. Itʼs a native of Madagascar, and it knows about as much about domestic heating as the average human. I am not being flippant here.

Ring-tailed lemurs in a zoo in south west England learned to turn up the temperature on their heating thermostat when it got cold


http://www.thisissomerset.co.uk/Smart-lemurs-learn-turn-thermostat-cold-snap/story-17929368-detail/story.html#axzz2Nod8uQoU

“When itʼs cold you need to turn the thermostat up.”

This mental model is completely wrong

This is what many humans do too:


A thermostat is an automatic switch.

Most people treat it like a valve: turn up dial, get more.

“My thermostat is too confusing to use so when I want to turn the heating up I put it in the fridge.”

NB: this might sound silly but itʼs far more logical:


Sometimes people just have illogical habits or beliefs that challenge our assumptions about what to design:

“I donʼt set my burglar alarm when Iʼm only going out for a few hours.”


People are sometimes going to use it in the ways you may consider irrational. Engineers find this baffling, but you have to deal with it.

So how do we fix this mess?


Iʼve given this talk in the past and had the response ʻyeah, what are you actually doing about this?ʼ.

I am working on design concepts for a more humanistic connected home experience.

I canʼt show you most of that, but I can talk about some of the practical design experience Iʼve gained along the way.

Some of this is applicable to more general internet of things/ubicomp type UX design.

UI/visual designscreen layout, look and feel

Platform designconceptual architecture spanning multiple

services, devices, common design principles

CX designcustomer lifecycle, customer services, integration with non digital touchpoints

Productisationaudience, proposition, objectives, functionality of a specific service

Industrial designphysical hardware: capabilities and

form factor

UX/interaction designarchitecture and behaviours per

service, per device

Interusabilityinteractions spanning multiple

devices with different capabilities

Many layers of connected home UX


[talk through]

I tend to use term ʻservice designʼ to encompass the lot.

A lot of my work is centred on the web and mobile UIs because these are the touchpoints users will interact directly with the most. But if you just think of it as doing web and mobile UI design, you miss a lot and risk creating a lot of problems.

Hence why Iʼm called a service design manager.

Iʼm going to talk about 3 of these, reflecting specific challenges I deal with...

“People have to understand it before they can want it”

Denise Wilton, BERG







form factor


service, per device




this is about making it make sense to end users.

Productisation is the extent to which the supplier makes the user value explicit


for some specific bits of hardware, like an energy monitor, thereʼs a close mapping between function and value. it does one thing, hopefully well. itʼs easy for people to understand what they do.

This is not a contact sensorthis is a thing that tells you:

•when an intruder has forced your front door open•when your child has opened her window in the middle of the night•when someone is trying to steal your guns

This is hard to do for general purpose devices


In areas where they donʼt have expert knowledge consumers tend to buy products, not tools

Product Tool


A product says ʻHere is the valueʼ, and comes preconfigured to deliver it.

A tool provides functionality. You figure out the value, and how to get it. (Weʼre back to the computer metaphor.)

Nothing wrong with making tools but they are less likely to go mass market

You can productise the box...but you also need to productise the service

• 1 SmartThings Hub• 2 SmartSense Multi (Open/

Closed, Temperature, Vibration)• 2 SmartSense Presence• 1 SmartPower Outlet• 1 SmartSense Motion Detector

• 1 SmartThings Hub• 2 SmartSense Moisture

Detectors• 1 SmartSense Motion• 1 SmartSense Presence• 3 SmartSense Multi (Open/

Closed, Temperature, Vibration)

• 1 SmartThings Hub• 2 SmartSense Multi (Open/

Closed, Temperature, Vibration)

• 4 SmartSense Presence• 1 SmartSense Motion Detector


Smart Things are trying to do this.

Most of the contents of these kits are pretty similar:

hub, open-closed/vibration/temperature multi-sensors, presence, motion.

Productising the box is a start. Iʼm very interested to see what the service UIs are like when you connect this all up. Is it a generic UI for all 3? Or does it have customised functionality for each service: so for home security you get specific instructions on setting up an alarm, for home watch you get a flood alarm, temperature warnings, an earthquake alarm; and for family life you get the “oh crap the dogʼs escaped” alarm?







form factor


service, per device




Interusability:• composition

• consistency

• continuity

Cross-Platform Service User Experience: A Field Study and an Initial Framework. Minna Wäljas, Katarina Segerståhl, Kaisa Väänänen-Vainio-Mattila, Harri Oinas-Kukkonen MobileHCI'10


Model I want to draw on here which I have found incredibly useful: interusability.

I know we like to kind of put usability in a box now and not use it as a catchall for broad UX, but bear with me and donʼt be put off. I use the word “interusability” because the people who came up with it called it interusability. I like to think of it as really talking about a type of cross-device digital service design.

Anyone who works in cross platform design should read the paper cited here, if you havenʼt already. The examples are a bit out of date now but the principles are still highly valid.

Talks about 3 components: composition, consistency, continuity

Composition• Figuring out which devices your service needs

• Figuring out what each device does

vs


As a designer of smart services, one of your first tasks is to think about composition: what devices will you have, and which ones will do what.

Your decision will be influenced by whether any parts of the system need to have particular form factors/be used in certain contexts, cost, whether any parts of the system need to work if they are offline, user expectations.

Displays and controls usually add to the cost, so itʼs often cheaper to handle user inputs and outputs on a remote mobile or web UI.

http://www.tado.com/en/example: tado thermostat has no UI, itʼs all on the phone. probable reasons: itʼs expensive to make a good thermostat UI, (and no-one understands the bad ones), so just make a good phone UI, which is relatively cheap to do (and gets you round some of the UI consistency challenges Iʼll talk about in a minute). Thereʼs a certain purist elegance to this decision but tiʼs a brave move: if you donʼt have your phone to hand, or itʼs not working, or youʼre a guest in the house without access to the phone UI, you canʼt adjust the heating.

AlertMe chose differently: we have a fairly standard thermostat with a conventionally bad UI but also the phone and web apps, which offer a much better experience (the one you see here looks rather plain as itʼs our unbranded version). This means that you, and your guests or other residents without smartphones, can still use it as a conventional thermostat. Itʼs less elegant (you will at some point encounter a UI that has been compromised by the need to keep the thermostat price down), but itʼs pragmatic.

ConsistencyAdapting interfaces for different types of device, but still making them feel like a family


Nest wall stat: twiddly knob on the wall that clicks. Touchscreen: up and down arrow. (Twiddly knobs are inefficient and inaccurate on touchscreens). BUT it still makes the same click :)

ContinuityUp to date data and content across all platforms. Fluent cross platform interactions.

Battery limitations impose possible 2 minute delay!


Perhaps the biggest challenge is continuity.

If I interact with the service on one device, all other devices reflect that change in state. e.g. if I turn the target heating temperature up on my wall thermostat, youʼd expect the new temperature to be immediately reflected on the smartphone too.

But sometimes this isnʼt technically possible.

In the case of the AlertMe heating system, there can be a delay of up to two minutes before the smartphone app is updated. This is because the wall thermostat runs off a battery (as is normal in the UK), and sending data to the network uses a lot of power so it only does it every two minutes. If it sent it more frequently than that, it would run the battery down very fast. We could make mains powered controllers, but engineers donʼt like those in the UK as they are more complicated to install. So for the time being, the UX is a compromise, albeit a small one as the main use of the smartphone app is when you are not standing in front of the wall thermostat, and 2 minutes isnʼt a long delay in turning the heating on.

The important thing is to ensure that users are as informed as possible about whatʼs going on.

Thermostat > hub > service > phone UIbut can be separate API calls

Boiler (furnace) > thermostat > hub > service > phone UI

3rd party weather service > phone UI

A complex service can have many potential points of failure


many points of potential connectivity failure: hub offline, thermostat offline, thermostat lost connection to boiler;

also individual API calls can fail like target temperature, current temp. and sometimes some are slower to load than others and that can be outside our control.

so there are times when you effectively have missing parts of the service, or are waiting for things to respond, and you have to deal with this in the UI.

itʼs not like many of the apps many of us work with, where cached data may still be useful. out of date data can be a big problem. it can lead you to believe something is on when itʼs off, or ok when itʼs not ok. it's perhaps not a disaster if it's 5 mins out of date for a heating app, but what if it's your burglar alarm,or an emergency alarm for an old person?

rule of thumb: donʼt show old data as this can be misleading, donʼt imply that a change has been made before it is completed,

figure out which data can be missing without rendering the service useless (like weather)

previous app loaded screen and then filled it with data. [screenshot]

i think this feels mainframey, and wanted the screen not to load until the data was there. my interaction is with the service, not interface plus data.

but sometimes that would mean that it didnʼt load at all for ages and that would be really frustrating. decided what we could live with (E.g. weather not updated) and what was essential to service experience

then what happens if you change one setting, e.g. turn from off to auto? more than one thing may update (e.g. mode, and . interface needs to update to reflect status change but dontʼ want to show this change until you know itʼs been applied. but some data not available, so end up with some blank data. it's not great (see loading on RHC homescreen when changing thermostat setting).

hardware constraints can be limiting...nest is mains powered so can use wifi and connect more instantaneously (file under hardware constraints?)







form factor


service, per device



aka the really big IA challenge


If you’re just making a single service that supports a limited set of devices, your platform can consist of device control/data APIs available via web/mobile interface


But remember, this can get complexTuesday, 4 June 2013

look at this again for a minute...

If you want to offer multiple, overlapping services, in which devices can do different things as part of different services and users can have different sets of services, then you start to need some kind of underlying logic to tie it all together.

Empty space = more future devices?


Nest looks like it may be the beginnings of a platform.

• services: intruder alarm, lighting, garden sprinkler, heating/cooling, gun cabinet, smoke alarm, energy usage, window blinds, Grannyʼs alarm...

• devices: motion sensors, lighting, sprinkler, thermostat, cabinet sensor, smoke detector, energy monitor, blind controls, panic button...

• controls: on/off, up/down, less/more, timer/schedule, hot/cold, set/unset

• notifications: alarm, message, status...

• presence: whoʼs in/out, nearby/far away, available/unavailable, authorised/not authorised

• contacts: people who live in the house, have access permissions to the physical property or the service UI

Across a range of home services you will have constructs like:


Within a service like this we have... [these things]

They are interrelated in potentially complex ways.

Add to: lighting controls?security system?both?


If you want to be smart in the ways that you offer services up...

... the more this conceptual model needs to be codified somewhere.

User tasks are heterogeneous and overlapping

• device/device group based: turn up the TV, turn off all the lights

• location based: set alarm downstairs, turn off outside lights, lower blinds on west side of the house in afternoon

• time/state based: activate security lights when iʼm away

• optimisation: keep the house temperature comfortable, use energy efficiently

• authorisation/presence based: lock the gun cabinet when the adults are not at home

• person based: tell me if Granny hasnʼt got out of bed, tell me when Jake gets home from school


There is no one hierarchy that supports all of this. You either force people to think in terms of your hierarchy... or you design something that supports the way they think... without overloading them with options.

notificationsservices

controls

devices presence

contacts

user needs

The big IA challenge:creating the UX logic that bridges the two


Creating the UX logic that allows you to support all these things, and create great UXes, for services that you have already and those you donʼt have yet, is the big IA challenge

Itʼs not stretching it too much to call it the ontology/domain model of the home

Until we make some headway here, most people wonʼt consider it worth the pain of buying wholesale into the technology.

Interoperability beyond the walled garden makes this an even bigger challenge


Right now all these things are walled gardens... realistically, who wants their entire house to come from Samsung, AlertMe, even Apple?

There isnʼt an open platform: there are some open network protocols for connecting devices, but Iʼm not aware of anything that helps figure out how they work together.

But even with open services, something is going to have to happen to ensure you donʼt end up in a mess of different UIs and metadata and control structures from different providers.

Thatʼs a really hard problem.

1 brilliant quote


“People donʼt want more control of their homes.

“They want more control of their lives.”Scott Davidoff, Min Kyung Lee, Charles Yiu, John Zimmerman, Anind K. Dey: Principles of Smart Home Control (Ubicomp 2006)


Burnt pie by Jet LimX10 Powerhouse from commodore.caInternet fridge from fuckyeahinternetfridge.tumblr.comMessy House by Elizabeth Table4FiveTrapped by MerinaComputer by Phil GoldCrying child by eggonstiltsArmy from hdwallpapers.comTea cosy by Brixton MakerhoodTeeth by ktpuppSleeping by StanFrustration by dieselbug2007Washing machine firmware error by Adam CrickettHouses by Peter O, Clive Darr, hollandhistory.net Usabilty lab by Leanne WaldalBurglar by homesecurityfocus.comMongkok advertising by Slices of LightPosh house by Savant TorontoTeenage bedroom by WendizzleHAL smarthome by james.lipsit.comJack Black from bradley.chattablogs.comHoliday home: geograph.co.ukOlder woman: soylentgreen23Ringtailed lemur by digidaveRingtailed lemur 2 by Tropiquaria Zoo

Than

ks fo

r th

e ph

otos


S Intille, The goal: Smart people not smart homes (2006)http://web.media.mit.edu/~intille/papers-files/IntilleICOST06.pdf

Minna Wäljas, Katarina Segerståhl, Kaisa Väänänen-Vainio-Mattila, Harri Oinas-Kukkonen: Cross-Platform Service User Experience: A Field Study and an Initial Framework (Nordichi 2010)http://bugi.oulu.fi/~ksegerst/publications/p219-waljas.pdf

Colin Dixon, Ratul Mahajan, Sharad Agarwal, AJ Brush, Bongshin Lee, Stefan Saroiu, and Victor Bahl, An Operating System for the Home (NSDI, USENIX, April 2012)

Pertti Huuskonen: Run to the Hills! Ubiquitous Computing Meltdown (Advances in Ambient Intelligence, 2007)

Peter Tolmie, James Pycock, Tim Diggins. Allan Maclean, Alain Karsenty, Unremarkable Computing (Ubiquity, 2002).

Genevieve Bell & Paul Dourish: Yesterdayʼs tomorrows: notes on ubiquitous computingʼs dominant vision (Personal and Ubiquitous Computing, 2006)http://www.ics.uci.edu/~jpd/ubicomp/BellDourish-YesterdaysTomorrows.pdf

Scott Davidoff, Min Kyung Lee, Charles Yiu, John Zimmerman, and Anind K. Dey: Principles of Smart Home Control (Ubicomp 2006)

T Saizmaa, A Holistic Understanding of HCI Perspectives on Smart Home, Networked Computing and Advanced Information Management, 2008. NCM '08Th

anks

for t

he

rese

arch


Thank you

@[email protected]

Thanks to: Alex von Feldmann, Fraser Hamilton, Martin Storey, Naintara Land and Anna Kuriakose who have contributed insights, thinking and research to this presentation


Design

"Siri, did I leave the oven on?" UX for the connected home (updated for IA Summit 2013)