Updated: Design beyond the glowing rectangle (Polish IA Summit 2011)

Design beyond the glowing rectangle.What does the internet of things mean for UX designers?

Claire Rowland & Chris BrowneApril 2011

Sunday, 10 April 2011

Cześć :)

Claire RowlandHead of Research

Chris BrowneTechnical Design Lead

Sunday, 10 April 2011Cheshcht :)

Hello, we are ... from Fjord in London. Thanks for having us!

Slide © Fjord 2010 | Confidential

A bit about Fjord...

rough slide

Sunday, 10 April 2011We’re a digital service design agency.

That means we design user experiences for services that are used on more than one platform - often mobile, tablets like the iPad, digital TV as well as web.


A 3 year EU project across 16 partners & 7 countries

Investigating user-centred design of interconnected embedded objects

Fjord’s role covers developing design, concepting and research methods, running trials

We’re currently at the stage of identifying challenges and generating initial concepts

Sunday, 10 April 2011All of our client work to date has been around designing for things with screens. But we’re now working on an EU funded research project that has given us the opportunity to think beyond the screen.

Although we are working with the above partners, this presentation represents Fjord’s view.

The rest of this talk is a quick tour of some of the issues we’ve been thinking about on this project.

We’re at an early stage and this is a massive area of ongoing research, but we are starting to propose some challenges and suggestions as to how we might overcome them

“Today’s multimedia machine makes the computer screen into a demanding focus of attention rather than allowing it to fade into the background.”Mark Weiser


In 1991, Mark Weiser (the ‘father of ubiquitous computing’) said...

“I hope we don’t end up in a world filled solely with

slick, glowing rectangles” Timo Arnall

image - The Onion

Sunday, 10 April 2011...20 years later, we’ve not really solved this, have we?

Our day to day lives involve many interactions with objects, but most of our interactions with computing still happen through the abstracted world of what Timo Arnall calls ‘slick, glowing rectangles’.

UX is moving beyond the screen.

Sunday, 10 April 2011However, we’re starting to see digital dissolve more into the physical world.

Of course, people like Weiser have been talking about this for a long time now.

But things are now starting to happen in the mainstream, here and now, which pose new challenges for UX.

We think that in the next couple of years, UX designers are going to have the opportunity to design things that involve not just screens, but services and physical objects for the world around them.


We thought it was going to be like this...


A lot of the classic ubicomp (ubiquitous computing) research has been focused on trying to understand the ‘bigger picture’ such as smart environments and homes.

There’s been lots of promise in this but very few good examples have made it out of research and into the mainstream.


But it’s actually more like this...rough slide

When your shipment is vital, know it’s vital signs.


‘The internet of things’: everyday objects becoming interconnected and increasingly able to sense their environment.

Currently 98% of all processors are in embedded devices, not multi-purpose computers.... and they are becoming increasingly connected.

An example of this is Fedex Senseaware: sensors track the location, temperature of packages and whether they have been opened. For example, if temperature sensitive medical supplies are in danger of being destroyed en route, the package can be returned and a new one sent out, saving vital time.


...and thisrough slide


Everyday objects getting smarter

This is RTMM: real time medication monitoring - made by Evalan, one of our partners on Smarcos...

...a smart pill bottle that knows when the user has taken their pills and can share this data with the user’s doctor or family. It can also remind the user to take them, via SMS or phone call

However many of these smart objects are still closed and proprietary systems.


Why should UX designers care?

because lots of other things are coming online...

Sunday, 10 April 2011Who works on PC apps/websites?


It’s not just PCs...


Sunday, 10 April 2011Who works on mobile, interactive TV... tablet?


It’s everything


Sunday, 10 April 2011Screen based media aren’t going away. But lots of other types of object are becoming connected and will be forming part of digital

services and the experiences we design.

Some of them will be objects we interact with directly.

They might take lots of forms, and some of them won’t have screens.

Some of these objects will just talk to other objects, but will provide data that’s used by things we do interact with.

Real (and perhaps crazy) examples of these that someone somewhere has made include dog tags that tweet, connected umbrellas, coffee machines smart home heating and lighting systems.

We’re still experimenting with what can be done here and some of the examples might seem flippant.

The point is that objects that were not connected can now be connected, and even if we’re not designing those objects, they can have an impact on what we do.

For example, right now, you might design websites that sell washing machines. Sooner or later, you may have sensor data from those machines that enables much better after sales service.. how does that change the service you provide? Or you may work for an environmental organisation. What difference would widespread live pollution data make to you?

These are examples of ways in which connected devices enable more complex services, which will make designing those services more complex, and, we would say, more exciting.

Either way, IA has a lot to offer in terms of both top down experience design and bottom up information sensemaking.

Devices are becoming connected faster than people


New Verizon and AT&T subscriptionsMay - August 2010


By way of evidence...

As of February 2011, both these networks now have more machine subscribers than human subscribers. That’s currently things like e readers, vehicle tracking systems and alarm monitoring services.

http://www.economist.com/blogs/babbage/2010/08/internet_things

22 billion devices connected by 2020?


IMS Research, Ericsson

... or 50 billion?

Sunday, 10 April 2011... that’s TVs, cars, mobiles and lots and lots of other random things.

This is actually a conservative estimate - Ericsson predict 50 billion connected devices.

300 000 000 000 000 000 000 000 IP addresses


image by Flickr user !iLauriin

"If the earth were made entirely out of 1 cubic millimetre grains of sand, then you could give a unique [IPv6] address to each grain in 300 million planets the size of the earth" Wikipedia


New standard of IPv6 will provide over 300 sextillion (that’s 300 billion billion billions) IP addresses, is currently being

implemented, which will allow every single object you can think of to have an internet connection.

What does this mean for design?

Sunday, 10 April 2011Over the next couple of years, this stuff is due to hit the mainstream and will affect the work UX designers do on an increasing basis

Here’s what we think this might mean for design...


Give a digital service a physical form


We can give digital services a physical embodiment.

A simple example is London’s Oyster card - a passive RFID chip embedded in a plastic card, linked up to a service which allows you to charge it up with credit.

The card itself is just an identifier, it has no intrinsic value. You wave it over readers and credit is deducted from your account, which lives remotely in a database.

But in users’ minds, the card *is* the service.


Give a physical product a digital form


Another way to approach design is enhancing existing products with digital services.

“Webkinz are toy stuffed animals that were originally released by the Canadian Ganz company on April 29, 2005. The toys are similar to many other small plush toys, however, each Webkinz toy has an attached tag with a unique "Secret Code" printed on it that allows access to the "Webkinz World" website. On Webkinz World, the Secret Code allows the user to own a virtual versionof the pet for online play for a limited time (one year).” - http://en.wikipedia.org/wiki/Webkinz


Click to edit Master text styles

Live data enables smarter services.

Sunday, 10 April 2011We design for and with live data to enrich our services.

“SFpark works by collecting and distributing real-time information about where parking is available so drivers can quickly find open spaces.To help achieve the right level of parking availability, SFpark will periodically adjust meter pricing up and down to match demand. Demand-responsive pricing encourages drivers to park in underused areas and garages, reducing demand in overused areas. With SFpark, real-time data and demand-responsive pricing work together to readjust parking patterns in the City so that parking is easier to find.” http://sfpark.org/

Key design challenges.

Sunday, 10 April 2011Here are a few of the key challenges we think UX designers will have to be prepared for, and some suggested ways to do things differently.

We’re just working this stuff out ourselves... these are some of the issues we hope to be able to research over the next couple of years.

These touch on bigger issues - they’re important for this but each is a huge topic in its own right

1. Device - service relationship gets more complex.


Classic usability tends to focus on one user using one device and one service to do one task at a time.

That’s increasingly not what’s actually going on. Our relationship with devices and services is getting more complicated.

It’s really important to think not just about device, but service design: how your user experience works across multiple devices. The device is no longer the unit of experience... the service is.

As embedded components come online, digital services will have to cope with increasing complexity in several ways...

One device, multiple services


image: Vivotech.com

Sunday, 10 April 2011Some objects will be touchpoints or data sources for multiple services.

Mobiles will play a huge role in this.

- Apps are a way to access different services

- NFC, beginning to appear in phones, enables multiple types of payment service. One interaction could take money from your credit card, bank account, Paypal... how do you make sure it’s the one you intended to use?

How do you know which service you’re using?


Sunday, 10 April 2011But there will be others which may not have screens to show us what they are up to.

For example, in New Songdo, a Korean ubiquitous city in development, there is a student smartcard that can be used to get into school, pay for food, as a library card, and as a bus pass (which can be used to notify students’ parents that the student is on the bus and expected home soon- no more secretly sloping off to hang out with friends after school).

Although it’s a simple object, and not even terribly smart, it’s a good example of an object with dynamic functionality, dependent on context.

Users have to use context of use to infer what it’s doing, and what information it is exchanging, every time it’s swiped.

Service coherence across multiple devices


Sunday, 10 April 2011Individual services are being delivered across multiple devices

Service coherence is a key part of UX- the service being in sync between multiple devices.

Facebook notifications are a current example of this working badly.

BBC used to talk about PC web as second screen for iPlayer, then mobile as third screen, but is now talking about creating orchestrated media - a synchronised iPlayer experience across all screens at once.

This would allow users to start an interaction on one device and continue it on another without losing their place in the task, and deliver different and complementary experiences across different devices.

Multiple devices, one service experience


Sunday, 10 April 2011Non screened device example... Ford Sync: an in car connectivity system that enables the car to be used as a controller for the phone,

so users can control the phone’s music player with the car’s voice recognition system, and more.

http://www.ford.com/technology/sync/about/

2. Services become interoperable.



Sunday, 10 April 2011For a while now, we’ve been seeing a trend for services to integrate with other services for the bits those services already do well,

such as 1Password, which integrates with Dropbox for synchronisation across devices.


Sunday, 10 April 2011Services can be used to integrate objects, too. Withings, the WiFi body scale shown in our video, shares data with a variety of

different fitness and weight loss services, such as RunKeeper, Lose It!, Daily Burn.

http://www.withings.com/en/bodyscale/sharing

3. New platforms for services.


Data overload


More and more data is being produced in both the physical and digital space, and can be shared in near real time.

How do we as designers leverage this huge amount of increasing complex data to help enrich the services we design, and aid us in designing new forms of services?


Aggregation


The traditional web 2.0 way of leveraging this data would be that of aggregation.

Example: Flipboard, which aggregate various web feeds such as news sites RSS, Twitter, Facebook etc into a magazine format.


Organic platforms: user driven

Text


This aggregation is also happening on an organic level, as in the example of the pdxboom hashtag.

Just after 8 PM on Sunday, March 28th 2010, a deafening BOOM was heard throughout SE Portland. Quickly the twitter community had adopted the hash tag #pdxboom. In less than 30mins there was a google map displaying where individuals heard the bang and how loud it was. This quickly began to show a pattern and allowed the police to establish the source and cause (it was a pipebomb).

Granted it wasn’t the most scientific approach and there are other established platforms such as Ushahidi that would have been better suited. Instead a new platform was created around this individual event in almost real time.


Organic platforms: corporate data

Text


The previous example shows these platforms being formed by the community/users but they can also be created out of leveraging existing data beyond its intended purpose.

This was shown recently after the earthquake and tsunami in Japan, Honda opened up their traffic data service and plotted the data onto a public Google map. This information showed in near real time the conditions of the roads and which roads were passable and which ones were not.

This is a service that you wouldn’t have expected from Honda as it leveraged a Honda asset beyond its intended purpose.

---http://www.thetruthaboutcars.com/2011/03/honda-gives-devastated-japan-something-badly-needed-information/


Physical platforms


We can also leverage hardware beyond its intended use, as it becomes more connected and its data more accessible.

If you were designing a wireless security system for a home, the same sensors used for the security system could be used as a health system for the elderly e.g pick up on if the person has fallen over or has an unusual behaviour/movement pattern (key in diagnosing the early signs of dementia). do we need to start thinking about designing ‘apps for you house’?


Software changes the purpose of hardware

http://www.flickr.com/photos/nolifebeforecoffee/


We can agree to hardware for one reason but it can be used in other ways.

“ in Wellington, for example, we were told that the surveillance cameras that voters approved to help manage traffic congestion had been repurposed for crime prevention, without a corresponding degree of public consultation.Let the image stream coming off of them be provided with a facial-recognition algorithm, and you’ve got an entirely different kind of

system on your hands, with entirely different potentials and vastly expanded implications. Yet the cameras, domed or otherwise, look

no different from one day to the next.” Adam Greenfield

As both designers and users we need to be aware of how data is and also could be used in the future, and ensure that the right

safeguards are in place and their is transparency in the service of how data is collected and used.

http://speedbird.wordpress.com/2010/04/28/neopanoptical/http://www.flickr.com/photos/nolifebeforecoffee/



4. Ensuring users retain control of their data.


“There’s a fine line between pervasive computing and invasive computing.”Victor Rozek


Privacy management is much more than a bunch of tick boxes and security settings. It incorporates less tangible elements such as 'appropriate use' and a user’s ‘comfort level’ etc.

Many people already find managing privacy too difficult on Facebook and share things with people they didn’t mean to share them with. It’s going to get a lot more complicated.

Passive vs active control


Sunday, 10 April 2011Passive vs active - person to service can be active.|| service to service is going to be passive ||

Google Latitude didn’t take off partly because users didn’t always want passively to share their location. On Foursquare, which requires active check in, you can simply choose not to check in. But active controls won’t scale up to really complex services or services that talk to each other.

We can design individual services that use a user’s data ‘appropriately’. However, gaps between individual services can form where personal data may be leaked.

With more objects becoming connected and sharing what’s deemed 'appropriate' data dependent on their service, the potential for gaps that can leak personal data increases, especially when data we share with one service ends up being used by another service. How do you know where your data is going to end up? How can service designers design for transparency - so that users can see when their data is being shared, and where it will end up?


When aggregated, data can get creepier

http://ilektrojohn.github.com/creepy/

Sunday, 10 April 2011“Creepy is an application that allows you to gather geolocation related information about users from social networking platforms and image hosting services. The information is presented in a map inside the application where all the retrieved data is shown accompanied with relevant information (i.e. what was posted from that specific location) to provide context to the presentation.”

Information that seemed innocent when shared can be aggregated to build up a scarily detailed picture of someone’s daily life.

Will we see the emergence of privacy brokers to help us manage our data in the face of all this complexity?

If brokers do emerge, what kind of organisations would we trust enough to manage this? Some have suggested the mobile operators... Also, what degree of control would we be happy to hand over to them?




The benefits can outweigh the risks

Sunday, 10 April 2011...there’s huge potential in all the data that can be gathered and mined... to deliver new types of service and create better user experiences.

... but we need to do more work to understand how to help people manage this.

5. Interactions become tangible.


Thinking is physical

Sunday, 10 April 2011Cognitive scientists now talk about ‘embodied cognition’*: the idea that the way we think is shaped by, and inseparable from, our physical experiences of interacting with the world. (Dourish, McCullough)

For example, the idea that up is good and down is bad is rooted in your physical experience of living with gravity. “I’m feeling down today” is bad. Up (and fast) is good... “I’m feeling upbeat”. We say someone is ‘boiling over with rage‘ or ‘steam is coming out of their ears’: understanding anger through containment of liquids. These are English language examples, but the principles seem to be universal.

Cognitive scientists would argue that this perception of up and down is a very fundamental basic level category or building block of thought used to make sense of other, more abstract things.

Embodied interaction seeks to make physical designs make sense to us through harnessing the way we understand the world through physical experience. At the moment, much tangible interaction work is happening in R&D labs...

Physical interactions can convey information


Sunday, 10 April 2011MIT’s information design ecology unit’s Proverbial Wallet aims to ‘unabstract virtual assets’ and provide a tangible link to virtual money. Connects to phone via Bluetooth, and then to your online banking service.

It has an interface through physical actuators, but no screen.

It gets fatter or thinner depending on how much money you have in the bank - direct physical representation of a metaphor (fat wallet).

The hinge resistance increases when you have less money, making it harder to open. This is a nice example of encouraging positive behaviour without controlling the user. [metaphor of good angel?]

The wallet buzzes when your bank processes a transaction, creating a conscious connection between handing over your credit card and the money actually leaving your account. This is a nice example of seamful design... communicating interconnections between services back to the user.

http://eco.media.mit.edu/proverbialwallets/

Physical interactions can enable product design


image: www.fabianhemmert.com

Sunday, 10 April 2011This is a prototype phone-shaped box by the tangible design researcher Fabian Hemmert at Deutsche Telekom labs.

It has an actuated back plate (controlled by servo motors) which can be tilted. so it can be thin in your pocket, but tapered

downwards when held in the hand.

this can also be used to convey contextual information: for example to make the phone thick or thin on one side when reading an

ebook to indicate how much has been read and is still to be read. the shape of the phone can also be used to indicate direction when

providing navigation...

http://fabianhemmert.com/projects/shape-changing-mobiles

There’s a lot we can do with tangible design to convey information without screens. As digital designers who’ve only worked with screens, we should start to think about these new interaction channels and how they can help us reclaim digital interactions into the real world.

Also, physical things are fun :)

6. What happens when it all goes wrong?


When interconnected services go wrong, they will go very wrong.


...it will go badly wrong

Sunday, 10 April 2011Technology is not reliable... we were trying to share files with a ubicomp researcher over Bluetooth the other day and we couldn’t even get that working or figure out why it wasn’t working.

Imagine that scaled up to a smarthome consisting of hundreds or thousands of connected objects...

A friend’s Danish friend lives in a smarthome with a system to decides when you could or could not open windows. it broke.

Thankfully, there was an override.

Then the override failed, and the house took over, opening and closing windows on its own. No-one could figure out how to fix it.

So the more interconnected things are, the harder it is to know which bit’s gone wrong. and the harder it is for the system to give useful information on overcoming errors.


congratulations!

Sunday, 10 April 2011If technology becomes a part of your everyday life, reliability is critical.

“A place where one can solve one’s worries” – a Korean euphemism for toilet

In Korea, there are toilets with sensors for female hormones and the signs of diabetes.

Right now, they don’t seem to include a pregnancy test but it’s not too much of a stretch to imagine.

Home pregnancy tests are 99% accurate.

Should you decide to test yourself once in that year, there’s only a 1 in 100 chance you’ll get a false positive.

If you’re tested 1000 times a year by a smart toilet, your toilet will wrongly congratulate you almost once a month.

You’d stop taking it seriously, wouldn’t you?

How much trust would you place in these things? How dependent will we become along them?

7. Digital business models hit the real world.


As the boundaries between the physical and digital world blur, we’ll see digital business models starting to appear in the physical world.

Some of these may be more or less acceptable to users...


Closer relationships with customers


People will pay more for a cheap toy if a digital service is added.

Engagement models last longer - longer engagement with product and customer

Ability to upsell over time, and have a closer direct relationship with the customer


Freemium objectsFridge will open

after this message from our sponsors!


Rethink how we own objects

Freemium services - get basic service free with ads, pay for ad free. Will we see this extended to everyday objects? Will Tesco provide fridges free to anyone willing to share all their food consumption data, buy food from Tesco, and watch adverts on a regular basis?


Be an individual, not a demographic

image: kihonmedia/ compensationclaims.com

Sunday, 10 April 2011Pay for what you’re using/what you need, e.g. sensors in cars enable drivers to pay car insurance based on how they actually drive, not on how the average person of their demographic drives.

8. User research methods.


There are two big challenges here...

Understanding needs for things people are not yet familiar with

Prototyping and testing complex systems


Emergent behaviour and expectations

image: mp.natlib.govt.nz

image: techpin.com

Sunday, 10 April 2011We recently interviewed some designers of smart objects as part of our project, about their methods. Numbers were small, but tendency was to feel that insight research was difficult to do before prototyping.

We often can’t predict the changing behaviours and attitudes that will emerge from new technology until people actually have it.

Putting washing machines in homes, plus bathrooms, changed ideas of acceptable hygiene - expectations of cleanliness rose.

People didn’t used to get mobiles for social reasons, they claimed it was for emergencies, but once they had them they quickly adapt to using them for social reasons. (e.g. Edwards and Grinter).

But observing current tasks can help us undersatnd cultural values, may not always help generate detailed requirements for highly novel systems, and you certainly can’t ask people what they want from something totally new.

But there may be methods we can use...


Participatory design/enactment

Sunday, 10 April 2011Participatory drama methods can be used to help shape designs... for example giving users scenarios to imagine novel interactions with an everyday object, invent interactions with an ambiguous object, or a wand that allows them to imbue other objects with magical powers. Also bodystorming - acting as if the product existed.


Prototyping

Sunday, 10 April 2011Prototyping is an important part of exploration in tangible design. Need early stage iteration as it’s hard to change complex systems! but hard to create real enough experiences.

Getting into hardware is a little daunting but there are ways to get feedback before going that far.

You *can* use paper models to mock up and get feedback on quick ideas/key interactions, wizard of oz style. Also useful for concepting.

You can also use video to mock up things that don’t yet exist to get feedback... this Microsoft Office 2019 one is for marketing but you can make versions in ways that support concept testing.

Need to work closely with technologists... helps for designers to get some understanding of tech, and for developers to move closer to understanding user needs.

Testing


Sunday, 10 April 2011Testing methods are geared towards one user, one task, one(ish) devices, that the user is highly focused on. We don’t even have a good definition of usability for more ambient interactions.

With testing, ecological validity is an issue: you can do very controlled simulations in the lab which are good for key interactions, but when you’re designing something to work in an environment context of use is really important.

Need to study usage over time as well - services are dynamic, and people may react differently to things once they become normal than they will to novel technology.

So context, and time are important.

One solution to this is the paratype: a modified diary method (Abowd, 05).

This involves a simulated interaction with a certain technological artifact within a specific setting of real social action, and documenting the effects of this combination.

- focuses on the social - samples experience using a simulation in real social contexts, e.g. personal audio loop - memory device - allowed them to study 3rd party reactions to the context.

There’s more work to do to develop methods here, but academic ubicomp literature is trying to tackle this issue.

How can UX people get started?

Sunday, 10 April 2011Everything we’ve talked about is happening now, somewhere.

We think this is about to affect the work that many of us do, even if just in small ways.

We’d like to suggest a few ways in which UX designers can start to think about this.


Look for potential in existing projects...

Sunday, 10 April 2011Look for potential in other projects by thinking of the ecosystem beyond your service and how this might be leveraged to extend or

enrich your service

When everything around you has the potential to be hardware and run software, the challenge lies in looking for the relationships between objects and services that provide real value to the user experience.

A simple example is to start with mobile. Mobile phones are essentially sensor platforms - many of them have GPs that tells them where they are, motion sensors that tell them when they are moving, and some have compasses so they know which way they are facing. They also know what time it is, and can take pictures with the camera. If you’re delivering something on a mobile, even if it’s a website, you can usually access this data from the phone. How can you use that to make the service better?


Anthropomorphise objects

Sunday, 10 April 2011A possible starter for creating design briefs...

Use objects as a basis for concepting... if they could tell a story, what would it be?

what personality would they have?what emotions would they feel?what information would they know, or share?how would they talk to us?how could they have good manners? (i.e. some rules for interaction)

Learn some hardware prototyping


Sunday, 10 April 2011Learn some hardware prototyping...


Sunday, 10 April 2011Be open to other skillsets. service design, interaction and product design will have to work together to address new challenges

We’ll be doing a lot more work in this area over the next couple of years and would love to connect with anyone else who’s interested.


Mike Kuniavsky: “Smart Things: Ubiquitous Computing User Experience Design”

Timo Arnall: “Designing for an internet of things” (http://www.slideshare.net/tmo/designing-for-an-internet-of-things)

ReadWriteWeb: Internet of Things: http://www.readwriteweb.com/archives/internet-of-things/

Adam Greenfield: “Everyware: the dawning age of ubiquitous computing”

Bruce Sterling: “Shaping Things”

BERG London: http://berglondon.com/blog/

Some reading...


http://berglondon.com/blog/

Thank [email protected]@fjord.co.uk [email protected] / @clurr

Thanks also to Alex von Feldmann, Dom Quigley, Ann Light, Alfred Lui, Ji-Hye Park, Sam Crosland, Martin Charlier, Helen Le Voi

PS: we’re hiring in London
