Rapid prototyping for Wearables

Rapid Prototyping for Wearables

January 15th 2015 TEI 2015 Studio

Mark Billinghurst

HIT Lab NZ University of Canterbury

mark.billinghurst@canterbury.ac.nz

Daniela Busse Director (UX) Citi Ventures

daniela.busse@citi.com

1: Introduction

Mark Billinghurst ▪  Director of HIT Lab NZ, University

of Canterbury

▪  PhD Univ. Washington

▪  Research on AR, mobile HCI, Collaborative Interfaces

▪  More than 250 papers in AR, VR, interface design

▪  Sabbatical in Glass team at Google [x] in 2013

● Daniela Busse ▪ UX Director

▪ Citi Ventures Global Innovation Network

▪ Previous ▪ Design Futurist, Samsung Research America ▪ UX Director/Chief UX Architect, SAP

▪ PhD, Computer Science ▪ Glasgow University

● What You Will Learn ▪ An introduction to wearable computers ▪ Key interface design guidelines ▪ Examples of good wearable design ▪ How to use a variety of rapid prototyping tools ▪ Hands on with Google Glass hardware ▪ Active areas of wearable computing research

● Schedule ▪ 9:00: Introduction ▪ 9:15: Introduction to Wearables ▪ 9:35: Introduction to Design ▪ 10:00: Tools for Low Fidelity Prototyping ▪ 10:30 Break/Design Session ▪ 11:00 UX Design Guidelines ▪ 11:30: Tools for High Fidelity prototyping ▪ 12:00 Presentation of Designs ▪ 12:15 Research Directions

2: Overview/History

A Brief History of Computing

Trend ▪  Smaller, cheaper, faster, more intimate ▪  Moving from fixed to handheld and onto body

1950’s

1980’s

1990’s

Room Desk Lap Hand Head

What is a Wearable Computer? ▪  Computer on the body that is: ▪  Always on ▪  Always accessible ▪  Always connected

▪  Other attributes ▪  Augmenting user actions ▪  Aware of user and surroundings

Rhodes, B. J. (1997). The wearable remembrance agent: A system for augmented memory. Personal Technologies, 1(4), 218-224. Mann, S. (1997). Wearable computing: A first step toward personal imaging. Computer, 30(2), 25-32.

The Ideal Wearable ▪  Persists and Provides Constant Access: Designed

for everyday and continuous user over a lifetime. ▪  Senses and Models Context: Observes and

models the users and environment. ▪  Augments and Mediates: Information support for

the user in both the physical and virtual realities. ▪  Interacts Seamlessly: Adapts its input and output

modalities to those most appropriate at the time.

Starner, T. E. (1999). Wearable computing and contextual awareness (Doctoral dissertation, Massachusetts Institute of Technology).

Wearable Attributes

▪  fafds

History of Wearables ▪  1960-90: Early Exploration ▪  Custom build devices

▪  1990 - 2000: Academic, Military Research ▪  MIT, CMU, Georgia Tech, EPFL, etc ▪  1997: ISWC conference starts

▪  1995 – 2005+: First Commercial Uses ▪  Niche industry applications, Military

▪  2010 - : Second Wave of Wearables ▪  Consumer applications, Head & Wrist Worn

The Gamblers

▪  Timing device for roulette prediction ▪  Card counting hardware (toe input)

Ed Thorp (1961)

Thorp, E. O. (1998, October). The invention of the first wearable computer. In Wearable Computers, 1998. Second International Symposium on (pp. 4-8). IEEE.

Belt computer

Shoe Input

Glasses Display

Keith Taft (1972)

● The Academics (1980’s - )

▪ MIT Media Lab – Wearable Computing (1993) ▪ CMU – Industrial wearables

http://www.media.mit.edu/wearables/ http://www.cs.cmu.edu/afs/cs/project/vuman/www/frontpage.html

Prototype Applications ▪  Remembrance Agent ▪  Rhodes (97)

▪  Augmented Reality ▪  Feiner (97), Thomas (98)

▪  Remote Collaboration ▪  Garner (97), Kraut (96) ▪ Maintenance/Factory ▪ Caudell (92), Thompson (97)

Mobile AR: Touring Machine (1997) ▪  University of Columbia ▪  Feiner, MacIntyre, Höllerer, Webster

▪  Combines ▪  See through head mounted display ▪  GPS tracking ▪  Orientation sensor ▪  Backpack PC (custom) ▪  Tablet input

Feiner, S., MacIntyre, B., Höllerer, T., & Webster, A. (1997). A touring machine: Prototyping 3D mobile augmented reality systems for exploring the urban environment. Personal Technologies, 1(4), 208-217.

Early Commercial Systems ▪  Xybernaut (1996 - 2007) ▪  Belt worn, HMD, 200 MHz

▪  ViA (1996 – 2001) ▪  Belt worn, Audio Interface ▪  700 MHz Crusoe ▪ Symbol (1998 – ) ▪ Wrist worn computer ▪ Finger scanner

● Symbol WWC 1000 (1998 - )

▪ Wrist worn wearable + finger barcode scanner ▪ $3500 USD, current price $1000 ▪ Over 30K sold in first 2 years, still selling (>100k units?) ▪ First widely deployed wearable computer

● Reasons For Success ▪ Well defined large market niche ▪ Stock pickers with holster scanners

▪ Significant usability/ergonomics effort ▪ Over 40,000 hours user testing

▪ Provided significant performance improvement ▪ Met user needs, solved existing problems

▪ Addressed social factors ▪ Company with substantial R+D resources

Stein, R., Ferrero, S., Hetfield, M., Quinn, A., & Krichever, M. (1998, October). Development of a commercially successful wearable data collection system. In Wearable Computers, 1998. Digest of Papers. Second International Symposium on (pp. 18-24). IEEE.

● Second Gen. Systems ▪ Recon (2010 - ) ▪ Head worn displays for sports ▪ Ski goggle display ▪ Investment from Intel (2013)

▪ Google (2011 - ) ▪ Google Glass ▪ Consumer focus

● Recon Use Case ▪ While skiing show: ▪ maps, ▪ speed, ▪ altitude ▪ phone calls ▪ text messages

● Demo Video

▪ https://www.youtube.com/watch?v=u24cbjqiVfE

Google Glass (2011 - )

● View Through Google Glass

Always available peripheral information display Combining computing, communications and content capture

● Google Glass User Interface

▪ dfasdf

● Timeline Metaphor

● User Experience ▪ Truly Wearable Computing ▪ Less than 46 ounces

▪ Hands-free Information Access ▪ Voice interaction, Ego-vision camera

▪ Intuitive User Interface ▪ Touch, Gesture, Speech, Head Motion

▪ Access to all Google Services ▪ Map, Search, Location, Messaging, Email, etc

Other Wearables ▪  Vuzix M-100 ▪  $999, professional

▪  Recon Jet ▪  $600, more sensors, sports

▪  Opinvent ▪  500 Euro, multi-view mode

▪  Motorola Golden-i ▪  Rugged, remote assistance

dsfh

● Projected Markets

● Business Evolution ▪ First wearable companies ▪ Targeting niche markets ▪ Expensive/poorly designed solutions ▪ Mostly low sales (< 10,000)

▪ Current generation ▪ First general consumer wearable (Glass, others) ▪ Bigger niche markets (skiing, sports) – > 50K+ sales ▪ Many diversified devices ▪ Lower costs/better design

Summary Wearables are a new class of computing

Intimate, persistent, aware, accessible, connected Evolution over 50 year history

Backpack to head worn Custom developed to consumer ready device

Enables new applications Collaboration, memory, AR, industry, etc

Many types of wearables are coming Android based, sensor package, micro-display

▪  3: Design

How do you Design for this?

● Design Thinking Process

5 modes iterated through

● Process ▪ Empathize: Understand the user needs ▪ Define: Define the problem to be solved ▪ Ideate: Brainstorm solutions ▪ Prototype: Develop sample solutions ▪ Test: User evaluation/validation of solutions

● Three Phase Model

● Understanding ▪ Understand the Problem Space ▪ What types of problems are we trying to solve? ▪ What’s our bigger goal?

▪ Understand the User ▪ Who are we solving the problem for ▪ Creating a Persona (typical end user)

▪ Define the Design Challenge ▪ Express the problem you are addressing ▪ Identify problem Insight

Celine needs & wants because

____________ ____________ ____________

Persona - Celine Story Travel writer & well-known blogger in Paris. Just published her first book of travel photography. Has 30K+ followers on Twitter, 800K+ on Instagram

Celine needs

& wants

because

_to capture and share what she is doing now_ _the experience to be shared with friends_______ she values being able to take friends with her on trips___

Persona - Celine

● The Design Challenge ▪ How can a wearable computer be used to capture and share a user’s experience with remote people? ▪ Key Insight: A wearable computer allows a person to see and hear with the eyes and ears of another

● Creating ▪ Brain storming ▪ Rapid idea generation ▪ “How might we?” questions ▪ “Yes and..” responses

▪ Body storming ▪ Act out ideas

▪ Organize ideas ▪ Idea map

It is easier to tone down a wild idea than to think up a new one. Alex Osborn

#androidwear

● Focus and Flare Design is a convergent and divergent process

● Delivering ▪ Prototyping ▪ Create physical form of idea ▪ Use for empathy, exploration, testing, inspiration

▪ Testing ▪ Evaluate prototype ▪ Validate idea ▪ Learn about user experience

● Interaction Design Process

● Design is All About the User

▪ Users should be involved throughout the Design Process ▪ Consider all the needs of the user

Task #1 1. Create or Identify a Persona 2. Complete a Needs Statement 3. Define the problem being solved 4. Brainstorm possible solutions

▪  4: Prototyping

Why Prototype? ▪  Quick visual design ▪  Capture key interactions ▪  Focus on user experience ▪  Communicate design ideas ▪  “Learn by doing/experiencing”

● Google Glass Prototyping

How can we quickly prototype Wearable

experiences with little or no coding?

● Design/Prototyping Tools

●  Typical Development Steps ▪ Sketching ▪ Storyboards ▪ UI Mockups ▪ Interaction Flows

▪ Video Prototypes ▪ Interactive Prototypes

▪ Final Native Application

Increased Fidelity & Interactivity

●  Prototyping Tools ▪ Static/Low fidelity ▪ Sketching ▪ User interface templates ▪ Storyboards/Application flows

▪ Interactive/High fidelity ▪ Wireframing tools ▪ Mobile prototyping ▪ Native Coding

▪  5: Low Fidelity Prototyping

● Storyboarding

Application Storyboard

▪  http://dsky9.com/glassfaq/google-glass-storyboard-template-download/

Sketched Interfaces

▪  Sketch + Powerpoint/Photoshop/Illustrator

GlassSim – http://glasssim.com/

▪  Simulate the view through Google Glass ▪  Multiple card templates

Glass UI Templates

▪  Google Glass Photoshop Templates ▪  http://glass-ui.com/ ▪  http://dsky9.com/glassfaq/the-google-glass-psd-template/

Sample Slides From Templates

ToolKit for Designers

▪  Vectoform Google Glass Toolkit for Designers ▪  http://blog.vectorform.com/2013/09/16/google-glass-

toolkit-for-designers-2/

▪  Sample cards, app flows, icons, etc

Application Flow

● Glassware Flow Designer

▪ Visual tool for designing application flows ▪ Drag and drop interface developers.google.com/glass/tools-downloads/glassware-flow-designer

Limitations ▪  Positives ▪  Good for documenting screens ▪  Can show application flow

▪  Negatives ▪  No interactivity/transitions ▪  Can’t be used for testing ▪  Can’t deploy on wearable ▪  Can be time consuming to create

● Task #2 1. Sketch out a storyboard showing how the user interacts with the application 2. Sketch out some sample user interfaces and application screens 3. Create an interaction flow for the application using Google Glassware Designer

▪  5: Wearable UX Design

● Consider Your User ▪ Wearable User ▪ Probably Mobile ▪ One/no hand interaction ▪ Short application use ▪ Need to be able to multitask ▪ Use in outdoor or indoor environment ▪ Want to enhance interaction with real world

● Design For Device

▪ Simple, relevant information ▪ Complement existing devices

● It's  like  a  rear  view  mirror  

 Don't  overload  the  user.  S7ck  to  the  absolutely  essen7al,  avoid  long  

interac7ons.  Be  explicit.      

● Typical Usage Times

Micro  Interac7ons  

The  posi*on  of  the  display  and  limited  input  ability  makes  longer  interac*ons  less  comfortable.  

 Using  it  shouldn’t  take  longer  than  taking  out  your  phone.  

Micro-Interactions

▪  On mobiles people split attention between display and real world

● Time Looking at Screen

Oulasvirta, A. (2005). The fragmentation of attention in mobile interaction, and what to do with it. interactions, 12(6), 16-18.

Design for MicroInteractions ▪  Design interaction less than a few seconds ▪  Tiny bursts of interaction ▪  One task per interaction ▪  One input per interaction

▪  Benefits ▪  Use limited input ▪  Minimize interruptions ▪  Reduce attention fragmentation

● Make Interface Glanceable

▪ Seek to rigorously reduce information density. ▪ Design for recognition, not reading.

Bad Good

● Reduce the number of info chunks

Reducing the total # of information chunks will increase the glanceability of your design.

1

2

3

1

2

3

4

5 (6)

● Single Interactions Faster than 4 s

1

2

3

1

2

3

4

5 (6)

Eye Movements: ~ 2 secs

Eye Movements: ~ 0.9 sec

Test the glanceability of your design ✓

● Don’t Get in the Way

▪ Enhance, not replace, real world interaction

● Design for Interruptions

▪  Gradually increase engagement and attention load ▪  Respond to user engagement

Receiving SMS on Glass

“Bing”

Tap Swipe

Glass

Show Message Start Reply User

Look Up

Say Reply

● Keep it Relevant

▪ Information at the right time and place

✓ Test your design indoors + outdoors

Interface Guidelines ▪  Design for device ▪  Use Micro Interaction ▪  Make it glanceable ▪  Do one thing at a time ▪  Reduce number of information chunks ▪  Design for indoor and outdoor use

6: Sample Use Cases

● Ideal Applications ▪ Use cases that require: ▪ Hands-free interaction ▪ Mobile information access ▪ Constant access to information ▪ Access to computing/communication ▪ Supporting activity in real world ▪ Low likelihood of social issues ▪ Enhanced view of reality

● Wearable Use Cases

▪ https://glass.google.com/glassware

Glassware Applications

Social  ac7on    First-­‐person  journalist  Tim  Pool    broadcasts  an  in*mate  view    of  Istanbul  protests.                'I  want  to  show  you  what  it's  like  to  be  there  as  best  I  can,  even  if  that  ends  with  me  running  full-­‐speed  into  a  cafe  and  rubbing  lemons  all  over  my  face  a<er  being  tear-­‐gassed'        

● CityViewAR

▪ Using AR to visualize Christchurch city buildings ▪ 3D models of buildings, 2D images, text, panoramas ▪ AR View, Map view, List view ▪ Available on Android market

● CityViewAR on Glass

▪ AR overlay of virtual buildings in Christchurch

● Virtual Exercise Companion

▪ GlassFitGames ▪ http://www.glassfitgames.com

● Example –Telemedicine

▪ Vipaar + UAB - http://www.vipaar.com ▪ Endoscopic view streamed remotely ▪ Remote expert adds hands – viewed in Glass

● Example: Virgin Atlantic

▪ Virgin Atlantic trialing Glass for customer check in ▪ Features ▪ Agent greets customer curb-side, start check-in process ▪ Provide customer details, personalized service ▪ Document verification – camera scanning of boarding pass

▪ Advantages ▪ Focus attention on customer ▪ Moves agent to customer

- Earlier engagement ▪ Reduces technology barrier between agent and customer

- Hide behind computer/desk ▪ Provides personalized service

- Name, flight details, weather, diet, translation services, etc

“The trial was a huge success with positive feedback from both our staff and customers on the usage of

wearable technology” ▪ Key findings ▪ Google Glass permitted the agent to maintain eye contact showing they were engaged and interesting in helping. ▪ Some passengers were taken aback initially by Glass wearing concierges, but, passengers responded well. ▪  Some technical challenges to overcome

- Short battery life, camera resolution, wifi issues

▪  7: High Fidelity Prototyping

●  Transitions

▪ Series of still photos in a movie format. ▪ Demonstrates the experience of the product ▪ Discover where concept needs fleshing out. ▪ Communicate experience and interface ▪ You can use many tools, from Flash to iMovie.

● Video Sketching

See https://vine.co/v/bgIaLHIpFTB

● Example: Video Sketch of Vine UI

● UI Concept Movies

● Pop - https://popapp.in/

▪ Combining sketching and interactivity on mobiles ▪ Take pictures of sketches ▪ Link pictures together

● Using Pop

Interactive Wireframing ▪  Developing interactive interfaces/wireframes ▪  Transitions, user feedback, interface design

▪  Web based tools ▪  UXpin - http://www.uxpin.com/ ▪  proto.io - http://www.proto.io/

▪  Native tools ▪  Justinmind - http://www.justinmind.com/ ▪  Axure - http://www.axure.com/

Proto.io - http://www.proto.io/ ▪  Web based mobile prototyping tool ▪  Features ▪  Prototype for multiple devices ▪  Gesture input, touch events, animations ▪  Share with collaborators ▪  Test on device

Proto.io - Interface

Demo: Building a Simple Flow

Gesture Flow

Scr1

Scr2 Scr3

Scr4 Scr5 Scr6

Tap Swipe

Start Transitions

● Android Design Preview

▪ Mirror portion of desktop to Android devices ▪ Works with Google Glass and other Android wearables ▪ Using any desktop application for prototyping

https://github.com/romannurik/AndroidDesignPreview

Wireframe Limitations ▪  Can’t deploy on Device ▪  No access to sensor data ▪  Camera, orientation sensor

▪  No multimedia playback ▪  Audio, video

▪  Simple transitions ▪  No conditional logic

▪  No networking

● Task #3 1. Create a Pop interactive sketch project showing the interface transitions 2. Develop a proto.io project showing the interface interactivity

Processing ▪  Programming tool for Artists/Designers ▪  http://processing.org ▪  Easy to code, Free, Open source, Java based ▪  2D, 3D, audio/video support

▪  Processing For Android ▪  http://wiki.processing.org/w/Android ▪  Strong Android support ▪  Generates Android .apk file

Processing - Motivation ▪  Language of Interaction ▪  Sketching with code ▪  Support for rich interaction

▪  Large developer community ▪  Active help forums ▪  Dozens of plug-in libraries

▪  Strong Android support ▪  Easy to run on wearables

http://processing.org/

http://openprocessing.org/

Basic Parts of a Processing Sketch

/* Notes comment */ //set up global variables float moveX = 50; //Initialize the Sketch void setup (){ } //draw every frame void draw(){ }

Processing and Glass ▪  One of the easiest ways to build rich

interactive wearable applications ▪  focus on interactivity, not coding

▪  Collects all sensor input ▪  camera, accelerometer, touch

▪  Can build native Android .apk files ▪  Side load onto Glass

Example: Hello World //called initially at the start of the Processing sketch void setup() { size(640, 360); background(0); } //called every frame to draw output void draw() { background(0); //draw a white text string showing Hello World fill(255); text("Hello World", 50, 50); }

Demo

Hello World Image PImage img; // Create an image variable void setup() { size(640, 360); //load the ok glass home screen image img = loadImage("okGlass.jpg"); // Load the image into the program } void draw() { // Displays the image at its actual size at point (0,0) image(img, 0, 0); }

Demo

Touch Pad Input ▪  Tap recognized as DPAD input

void keyPressed() { if (key == CODED){ if (keyCode == DPAD) {

// Do something ..

▪  Java code to capture rich motion events ▪  import android.view.MotionEvent;

Motion Event //Glass Touch Events - reads from touch pad public boolean dispatchGenericMotionEvent(MotionEvent event) { float x = event.getX(); // get x/y coords float y = event.getY(); int action = event.getActionMasked(); // get code for action switch (action) { // let us know which action code shows up case MotionEvent.ACTION_DOWN: touchEvent = "DOWN"; fingerTouch = 1; break; case MotionEvent.ACTION_MOVE: touchEvent = "MOVE"; xpos = myScreenWidth-x*touchPadScaleX; ypos = y*touchPadScaleY; break;

Demo

Sensors ▪  Ketai Library for Processing ▪  https://code.google.com/p/ketai/

▪  Support all phone sensors ▪  GPS, Compass, Light, Camera, etc

▪  Include Ketai Library ▪  import ketai.sensors.*; ▪  KetaiSensor sensor;

Using Sensors ▪  Setup in Setup( ) function

▪  sensor = new KetaiSensor(this); ▪  sensor.start(); ▪ sensor.list();

▪  Event based sensor reading void onAccelerometerEvent(…) { accelerometer.set(x, y, z); }

Sensor Demo

Using the Camera ▪  Import camera library

▪  import ketai.camera.*; ▪  KetaiCamera cam;

▪  Setup in Setup( ) function ▪ cam = new KetaiCamera(this, 640, 480, 15);

▪  Draw camera image void draw() { //draw the camera image image(cam, width/2, height/2); }

Camera Demo

WearScript

▪  JavaScript development for Glass ▪  http://www.wearscript.com/en/

▪  Script directory ▪  http://weariverse.com/

● WearScript Features ▪ Community of Developers ▪ Easy development of Glass Applications ▪ GDK card format ▪ Support for all sensor input

▪ Support for advanced features ▪ Augmented Reality ▪ Eye tracking ▪ Arduino input

● WearScript Playground

▪  Test code and run on Glass ▪  https://api.wearscript.com/

▪  8: Design Presentations

▪  9: Research Directions

Challenges for the Future (2001) ▪  Privacy ▪  Power use ▪  Networking ▪  Collaboration ▪  Heat dissipation ▪  Interface design ▪  Intellectual tools ▪  Augmented Reality systems

Starner, T. (2001). The challenges of wearable computing: Part 1. IEEE Micro,21(4), 44-52. Starner, T. (2001). The challenges of wearable computing: Part 2. IEEE Micro,21(4), 54-67.

Gesture Interaction With Glass ▪  3 Gear Systems ▪  Hand tracking

▪  Hand data sent to glass ▪  Wifi networking ▪  Hand joint position ▪  AR application rendering ▪  Vuforia tracking

Performance

▪  Full 3d hand model input ▪  10 - 15 fps tracking, 1 cm fingertip resolution

● Meta Gesture Interaction

▪ Depth sensor + Stereo see-through ▪ https://www.spaceglasses.com/

Current Collaboration

▪  First person remote conferencing/hangouts ▪  Limitations

-  Single POV, no spatial cues, no annotations, etc

Sharing Space: Social Panoramas

▪  Capture and share social spaces in real time ▪  Enable remote people to feel like they’re with you

Context Sensing ▪  Using context to manage information ▪  progressive information display as user shows

interest ▪  Context from ▪  Speech ▪  Gaze ▪  Real world

▪  Wearable AR Display

Ajanki, A., Billinghurst, M., Gamper, H., Järvenpää, T., Kandemir, M., Kaski, S., ... & Tossavainen, T. (2011). An augmented reality interface to contextual information. Virtual reality, 15(2-3), 161-173.

Gaze Interaction

AR View

More Information Over Time

Social Perception

TAT Augmented ID

▪  10: Conclusions

● Conclusions ▪ Wearable computing is a 4th generation of computing devices ▪ A range of wearables will appear in 2014 ▪ Ecosystem of devices

▪ There are many existing tools for prototyping ▪ Low fidelity, high fidelity

▪ Significant research opportunities exist ▪ User interaction, displays, social impact

More Information •  Mark Billinghurst

– Email: mark.billinghurst@hitlabnz.org

– Twitter: @marknb00 •  Daniela Busse

– Email: daniela.busse@gmail.com

•  Website – www.hitlabnz.org

▪  11: Resources

Glass Developer Resources ▪  Main Developer Website ▪  https://developers.google.com/glass/

▪  Glass Apps Developer Site ▪  http://glass-apps.org/glass-developer

▪  Google Design Guidelines Site ▪  https://developers.google.com/glass/design/

index?utm_source=tuicool ▪  Google Glass Emulator ▪  http://glass-apps.org/google-glass-emulator

Other Resources ▪  AR for Glass Website ▪  http://www.arforglass.org/

▪  Vandrico Database of wearable devices ▪  http://vandrico.com/database

● Glass UI Design Guidelines

▪ More guidelines ▪ https://developers.google.com/glass/design/index

Books ▪  Programming Google Glass ▪  Eric Redmond

▪  Rapid Android Development: Build Rich, Sensor-Based Applications with Processing ▪  Daniel Sauter

▪ Microinteractions: Designing with Details ▪ Dan Saffer ▪ http://microinteractions.com/

▪ Beginning Google Glass Development ▪ by Jeff Tang