Embed Size (px)
Citation preview
Rapid Prototyping for Wearables
January 15th 2015 TEI 2015 Studio
Mark Billinghurst
HIT Lab NZ University of Canterbury
Daniela Busse Director (UX) Citi Ventures
1: Introduction
Mark Billinghurst ▪ Director of HIT Lab NZ, University
of Canterbury
▪ PhD Univ. Washington
▪ Research on AR, mobile HCI, Collaborative Interfaces
▪ More than 250 papers in AR, VR, interface design
▪ Sabbatical in Glass team at Google [x] in 2013
● Daniela Busse ▪ UX Director
▪ Citi Ventures Global Innovation Network
▪ Previous ▪ Design Futurist, Samsung Research America ▪ UX Director/Chief UX Architect, SAP
▪ PhD, Computer Science ▪ Glasgow University
● What You Will Learn ▪ An introduction to wearable computers ▪ Key interface design guidelines ▪ Examples of good wearable design ▪ How to use a variety of rapid prototyping tools ▪ Hands on with Google Glass hardware ▪ Active areas of wearable computing research
● Schedule ▪ 9:00: Introduction ▪ 9:15: Introduction to Wearables ▪ 9:35: Introduction to Design ▪ 10:00: Tools for Low Fidelity Prototyping ▪ 10:30 Break/Design Session ▪ 11:00 UX Design Guidelines ▪ 11:30: Tools for High Fidelity prototyping ▪ 12:00 Presentation of Designs ▪ 12:15 Research Directions
2: Overview/History
A Brief History of Computing
Trend ▪ Smaller, cheaper, faster, more intimate ▪ Moving from fixed to handheld and onto body
1950’s
1980’s
1990’s
Room Desk Lap Hand Head
What is a Wearable Computer? ▪ Computer on the body that is: ▪ Always on ▪ Always accessible ▪ Always connected
▪ Other attributes ▪ Augmenting user actions ▪ Aware of user and surroundings
Rhodes, B. J. (1997). The wearable remembrance agent: A system for augmented memory. Personal Technologies, 1(4), 218-224. Mann, S. (1997). Wearable computing: A first step toward personal imaging. Computer, 30(2), 25-32.
The Ideal Wearable ▪ Persists and Provides Constant Access: Designed
for everyday and continuous user over a lifetime. ▪ Senses and Models Context: Observes and
models the users and environment. ▪ Augments and Mediates: Information support for
the user in both the physical and virtual realities. ▪ Interacts Seamlessly: Adapts its input and output
modalities to those most appropriate at the time.
Starner, T. E. (1999). Wearable computing and contextual awareness (Doctoral dissertation, Massachusetts Institute of Technology).
Wearable Attributes
▪ fafds
History of Wearables ▪ 1960-90: Early Exploration ▪ Custom build devices
▪ 1990 - 2000: Academic, Military Research ▪ MIT, CMU, Georgia Tech, EPFL, etc ▪ 1997: ISWC conference starts
▪ 1995 – 2005+: First Commercial Uses ▪ Niche industry applications, Military
▪ 2010 - : Second Wave of Wearables ▪ Consumer applications, Head & Wrist Worn
The Gamblers
▪ Timing device for roulette prediction ▪ Card counting hardware (toe input)
Ed Thorp (1961)
Thorp, E. O. (1998, October). The invention of the first wearable computer. In Wearable Computers, 1998. Second International Symposium on (pp. 4-8). IEEE.
Belt computer
Shoe Input
Glasses Display
Keith Taft (1972)
● The Academics (1980’s - )
▪ MIT Media Lab – Wearable Computing (1993) ▪ CMU – Industrial wearables
http://www.media.mit.edu/wearables/ http://www.cs.cmu.edu/afs/cs/project/vuman/www/frontpage.html
Prototype Applications ▪ Remembrance Agent ▪ Rhodes (97)
▪ Augmented Reality ▪ Feiner (97), Thomas (98)
▪ Remote Collaboration ▪ Garner (97), Kraut (96) ▪ Maintenance/Factory ▪ Caudell (92), Thompson (97)
Mobile AR: Touring Machine (1997) ▪ University of Columbia ▪ Feiner, MacIntyre, Höllerer, Webster
▪ Combines ▪ See through head mounted display ▪ GPS tracking ▪ Orientation sensor ▪ Backpack PC (custom) ▪ Tablet input
Feiner, S., MacIntyre, B., Höllerer, T., & Webster, A. (1997). A touring machine: Prototyping 3D mobile augmented reality systems for exploring the urban environment. Personal Technologies, 1(4), 208-217.
Early Commercial Systems ▪ Xybernaut (1996 - 2007) ▪ Belt worn, HMD, 200 MHz
▪ ViA (1996 – 2001) ▪ Belt worn, Audio Interface ▪ 700 MHz Crusoe ▪ Symbol (1998 – ) ▪ Wrist worn computer ▪ Finger scanner
● Symbol WWC 1000 (1998 - )
▪ Wrist worn wearable + finger barcode scanner ▪ $3500 USD, current price $1000 ▪ Over 30K sold in first 2 years, still selling (>100k units?) ▪ First widely deployed wearable computer
● Reasons For Success ▪ Well defined large market niche ▪ Stock pickers with holster scanners
▪ Significant usability/ergonomics effort ▪ Over 40,000 hours user testing
▪ Provided significant performance improvement ▪ Met user needs, solved existing problems
▪ Addressed social factors ▪ Company with substantial R+D resources
Stein, R., Ferrero, S., Hetfield, M., Quinn, A., & Krichever, M. (1998, October). Development of a commercially successful wearable data collection system. In Wearable Computers, 1998. Digest of Papers. Second International Symposium on (pp. 18-24). IEEE.
● Second Gen. Systems ▪ Recon (2010 - ) ▪ Head worn displays for sports ▪ Ski goggle display ▪ Investment from Intel (2013)
▪ Google (2011 - ) ▪ Google Glass ▪ Consumer focus
● Recon Use Case ▪ While skiing show: ▪ maps, ▪ speed, ▪ altitude ▪ phone calls ▪ text messages
● Demo Video
▪ https://www.youtube.com/watch?v=u24cbjqiVfE
Google Glass (2011 - )
● View Through Google Glass
Always available peripheral information display Combining computing, communications and content capture
● Google Glass User Interface
▪ dfasdf
● Timeline Metaphor
● User Experience ▪ Truly Wearable Computing ▪ Less than 46 ounces
▪ Hands-free Information Access ▪ Voice interaction, Ego-vision camera
▪ Intuitive User Interface ▪ Touch, Gesture, Speech, Head Motion
▪ Access to all Google Services ▪ Map, Search, Location, Messaging, Email, etc
Other Wearables ▪ Vuzix M-100 ▪ $999, professional
▪ Recon Jet ▪ $600, more sensors, sports
▪ Opinvent ▪ 500 Euro, multi-view mode
▪ Motorola Golden-i ▪ Rugged, remote assistance
dsfh
● Projected Markets
● Business Evolution ▪ First wearable companies ▪ Targeting niche markets ▪ Expensive/poorly designed solutions ▪ Mostly low sales (< 10,000)
▪ Current generation ▪ First general consumer wearable (Glass, others) ▪ Bigger niche markets (skiing, sports) – > 50K+ sales ▪ Many diversified devices ▪ Lower costs/better design
Summary Wearables are a new class of computing
Intimate, persistent, aware, accessible, connected Evolution over 50 year history
Backpack to head worn Custom developed to consumer ready device
Enables new applications Collaboration, memory, AR, industry, etc
Many types of wearables are coming Android based, sensor package, micro-display
▪ 3: Design
How do you Design for this?
● Design Thinking Process
5 modes iterated through
● Process ▪ Empathize: Understand the user needs ▪ Define: Define the problem to be solved ▪ Ideate: Brainstorm solutions ▪ Prototype: Develop sample solutions ▪ Test: User evaluation/validation of solutions
● Three Phase Model
● Understanding ▪ Understand the Problem Space ▪ What types of problems are we trying to solve? ▪ What’s our bigger goal?
▪ Understand the User ▪ Who are we solving the problem for ▪ Creating a Persona (typical end user)
▪ Define the Design Challenge ▪ Express the problem you are addressing ▪ Identify problem Insight
Celine needs & wants because
____________ ____________ ____________
Persona - Celine Story Travel writer & well-known blogger in Paris. Just published her first book of travel photography. Has 30K+ followers on Twitter, 800K+ on Instagram
Celine needs
& wants
because
_to capture and share what she is doing now_ _the experience to be shared with friends_______ she values being able to take friends with her on trips___
Persona - Celine
● The Design Challenge ▪ How can a wearable computer be used to capture and share a user’s experience with remote people? ▪ Key Insight: A wearable computer allows a person to see and hear with the eyes and ears of another
● Creating ▪ Brain storming ▪ Rapid idea generation ▪ “How might we?” questions ▪ “Yes and..” responses
▪ Body storming ▪ Act out ideas
▪ Organize ideas ▪ Idea map
It is easier to tone down a wild idea than to think up a new one. Alex Osborn
#androidwear
● Focus and Flare Design is a convergent and divergent process
● Delivering ▪ Prototyping ▪ Create physical form of idea ▪ Use for empathy, exploration, testing, inspiration
▪ Testing ▪ Evaluate prototype ▪ Validate idea ▪ Learn about user experience
● Interaction Design Process
● Design is All About the User
▪ Users should be involved throughout the Design Process ▪ Consider all the needs of the user
Task #1 1. Create or Identify a Persona 2. Complete a Needs Statement 3. Define the problem being solved 4. Brainstorm possible solutions
▪ 4: Prototyping
Why Prototype? ▪ Quick visual design ▪ Capture key interactions ▪ Focus on user experience ▪ Communicate design ideas ▪ “Learn by doing/experiencing”
● Google Glass Prototyping
How can we quickly prototype Wearable
experiences with little or no coding?
● Design/Prototyping Tools
● Typical Development Steps ▪ Sketching ▪ Storyboards ▪ UI Mockups ▪ Interaction Flows
▪ Video Prototypes ▪ Interactive Prototypes
▪ Final Native Application
Increased Fidelity & Interactivity
● Prototyping Tools ▪ Static/Low fidelity ▪ Sketching ▪ User interface templates ▪ Storyboards/Application flows
▪ Interactive/High fidelity ▪ Wireframing tools ▪ Mobile prototyping ▪ Native Coding
▪ 5: Low Fidelity Prototyping
● Storyboarding
Application Storyboard
▪ http://dsky9.com/glassfaq/google-glass-storyboard-template-download/
Sketched Interfaces
▪ Sketch + Powerpoint/Photoshop/Illustrator
GlassSim – http://glasssim.com/
▪ Simulate the view through Google Glass ▪ Multiple card templates
Glass UI Templates
▪ Google Glass Photoshop Templates ▪ http://glass-ui.com/ ▪ http://dsky9.com/glassfaq/the-google-glass-psd-template/
Sample Slides From Templates
ToolKit for Designers
▪ Vectoform Google Glass Toolkit for Designers ▪ http://blog.vectorform.com/2013/09/16/google-glass-
toolkit-for-designers-2/
▪ Sample cards, app flows, icons, etc
Application Flow
● Glassware Flow Designer
▪ Visual tool for designing application flows ▪ Drag and drop interface developers.google.com/glass/tools-downloads/glassware-flow-designer
Limitations ▪ Positives ▪ Good for documenting screens ▪ Can show application flow
▪ Negatives ▪ No interactivity/transitions ▪ Can’t be used for testing ▪ Can’t deploy on wearable ▪ Can be time consuming to create
● Task #2 1. Sketch out a storyboard showing how the user interacts with the application 2. Sketch out some sample user interfaces and application screens 3. Create an interaction flow for the application using Google Glassware Designer
▪ 5: Wearable UX Design
● Consider Your User ▪ Wearable User ▪ Probably Mobile ▪ One/no hand interaction ▪ Short application use ▪ Need to be able to multitask ▪ Use in outdoor or indoor environment ▪ Want to enhance interaction with real world
● Design For Device
▪ Simple, relevant information ▪ Complement existing devices
● It's like a rear view mirror
Don't overload the user. S7ck to the absolutely essen7al, avoid long
interac7ons. Be explicit.
● Typical Usage Times
Micro Interac7ons
The posi*on of the display and limited input ability makes longer interac*ons less comfortable.
Using it shouldn’t take longer than taking out your phone.
Micro-Interactions
▪ On mobiles people split attention between display and real world
● Time Looking at Screen
Oulasvirta, A. (2005). The fragmentation of attention in mobile interaction, and what to do with it. interactions, 12(6), 16-18.
Design for MicroInteractions ▪ Design interaction less than a few seconds ▪ Tiny bursts of interaction ▪ One task per interaction ▪ One input per interaction
▪ Benefits ▪ Use limited input ▪ Minimize interruptions ▪ Reduce attention fragmentation
● Make Interface Glanceable
▪ Seek to rigorously reduce information density. ▪ Design for recognition, not reading.
Bad Good
● Reduce the number of info chunks
Reducing the total # of information chunks will increase the glanceability of your design.
1
2
3
1
2
3
4
5 (6)
● Single Interactions Faster than 4 s
1
2
3
1
2
3
4
5 (6)
Eye Movements: ~ 2 secs
Eye Movements: ~ 0.9 sec
Test the glanceability of your design ✓
● Don’t Get in the Way
▪ Enhance, not replace, real world interaction
● Design for Interruptions
▪ Gradually increase engagement and attention load ▪ Respond to user engagement
Receiving SMS on Glass
“Bing”
Tap Swipe
Glass
Show Message Start Reply User
Look Up
Say Reply
● Keep it Relevant
▪ Information at the right time and place
✓ Test your design indoors + outdoors
Interface Guidelines ▪ Design for device ▪ Use Micro Interaction ▪ Make it glanceable ▪ Do one thing at a time ▪ Reduce number of information chunks ▪ Design for indoor and outdoor use
6: Sample Use Cases
● Ideal Applications ▪ Use cases that require: ▪ Hands-free interaction ▪ Mobile information access ▪ Constant access to information ▪ Access to computing/communication ▪ Supporting activity in real world ▪ Low likelihood of social issues ▪ Enhanced view of reality
● Wearable Use Cases
▪ https://glass.google.com/glassware
Glassware Applications
Social ac7on First-‐person journalist Tim Pool broadcasts an in*mate view of Istanbul protests. 'I want to show you what it's like to be there as best I can, even if that ends with me running full-‐speed into a cafe and rubbing lemons all over my face a<er being tear-‐gassed'
● CityViewAR
▪ Using AR to visualize Christchurch city buildings ▪ 3D models of buildings, 2D images, text, panoramas ▪ AR View, Map view, List view ▪ Available on Android market
● CityViewAR on Glass
▪ AR overlay of virtual buildings in Christchurch
● Virtual Exercise Companion
▪ GlassFitGames ▪ http://www.glassfitgames.com
● Example –Telemedicine
▪ Vipaar + UAB - http://www.vipaar.com ▪ Endoscopic view streamed remotely ▪ Remote expert adds hands – viewed in Glass
● Example: Virgin Atlantic
▪ Virgin Atlantic trialing Glass for customer check in ▪ Features ▪ Agent greets customer curb-side, start check-in process ▪ Provide customer details, personalized service ▪ Document verification – camera scanning of boarding pass
▪ Advantages ▪ Focus attention on customer ▪ Moves agent to customer
- Earlier engagement ▪ Reduces technology barrier between agent and customer
- Hide behind computer/desk ▪ Provides personalized service
- Name, flight details, weather, diet, translation services, etc
“The trial was a huge success with positive feedback from both our staff and customers on the usage of
wearable technology” ▪ Key findings ▪ Google Glass permitted the agent to maintain eye contact showing they were engaged and interesting in helping. ▪ Some passengers were taken aback initially by Glass wearing concierges, but, passengers responded well. ▪ Some technical challenges to overcome
- Short battery life, camera resolution, wifi issues
▪ 7: High Fidelity Prototyping
● Transitions
▪ Series of still photos in a movie format. ▪ Demonstrates the experience of the product ▪ Discover where concept needs fleshing out. ▪ Communicate experience and interface ▪ You can use many tools, from Flash to iMovie.
● Video Sketching
See https://vine.co/v/bgIaLHIpFTB
● Example: Video Sketch of Vine UI
● UI Concept Movies
● Pop - https://popapp.in/
▪ Combining sketching and interactivity on mobiles ▪ Take pictures of sketches ▪ Link pictures together
● Using Pop
Interactive Wireframing ▪ Developing interactive interfaces/wireframes ▪ Transitions, user feedback, interface design
▪ Web based tools ▪ UXpin - http://www.uxpin.com/ ▪ proto.io - http://www.proto.io/
▪ Native tools ▪ Justinmind - http://www.justinmind.com/ ▪ Axure - http://www.axure.com/
Proto.io - http://www.proto.io/ ▪ Web based mobile prototyping tool ▪ Features ▪ Prototype for multiple devices ▪ Gesture input, touch events, animations ▪ Share with collaborators ▪ Test on device
Proto.io - Interface
Demo: Building a Simple Flow
Gesture Flow
Scr1
Scr2 Scr3
Scr4 Scr5 Scr6
Tap Swipe
Start Transitions
● Android Design Preview
▪ Mirror portion of desktop to Android devices ▪ Works with Google Glass and other Android wearables ▪ Using any desktop application for prototyping
https://github.com/romannurik/AndroidDesignPreview
Wireframe Limitations ▪ Can’t deploy on Device ▪ No access to sensor data ▪ Camera, orientation sensor
▪ No multimedia playback ▪ Audio, video
▪ Simple transitions ▪ No conditional logic
▪ No networking
● Task #3 1. Create a Pop interactive sketch project showing the interface transitions 2. Develop a proto.io project showing the interface interactivity
Processing ▪ Programming tool for Artists/Designers ▪ http://processing.org ▪ Easy to code, Free, Open source, Java based ▪ 2D, 3D, audio/video support
▪ Processing For Android ▪ http://wiki.processing.org/w/Android ▪ Strong Android support ▪ Generates Android .apk file
Processing - Motivation ▪ Language of Interaction ▪ Sketching with code ▪ Support for rich interaction
▪ Large developer community ▪ Active help forums ▪ Dozens of plug-in libraries
▪ Strong Android support ▪ Easy to run on wearables
http://processing.org/
http://openprocessing.org/
Basic Parts of a Processing Sketch
/* Notes comment */ //set up global variables float moveX = 50; //Initialize the Sketch void setup (){ } //draw every frame void draw(){ }
Processing and Glass ▪ One of the easiest ways to build rich
interactive wearable applications ▪ focus on interactivity, not coding
▪ Collects all sensor input ▪ camera, accelerometer, touch
▪ Can build native Android .apk files ▪ Side load onto Glass
Example: Hello World //called initially at the start of the Processing sketch void setup() { size(640, 360); background(0); } //called every frame to draw output void draw() { background(0); //draw a white text string showing Hello World fill(255); text("Hello World", 50, 50); }
Demo
Hello World Image PImage img; // Create an image variable void setup() { size(640, 360); //load the ok glass home screen image img = loadImage("okGlass.jpg"); // Load the image into the program } void draw() { // Displays the image at its actual size at point (0,0) image(img, 0, 0); }
Demo
Touch Pad Input ▪ Tap recognized as DPAD input
void keyPressed() { if (key == CODED){ if (keyCode == DPAD) {
// Do something ..
▪ Java code to capture rich motion events ▪ import android.view.MotionEvent;
Motion Event //Glass Touch Events - reads from touch pad public boolean dispatchGenericMotionEvent(MotionEvent event) { float x = event.getX(); // get x/y coords float y = event.getY(); int action = event.getActionMasked(); // get code for action switch (action) { // let us know which action code shows up case MotionEvent.ACTION_DOWN: touchEvent = "DOWN"; fingerTouch = 1; break; case MotionEvent.ACTION_MOVE: touchEvent = "MOVE"; xpos = myScreenWidth-x*touchPadScaleX; ypos = y*touchPadScaleY; break;
Demo
Sensors ▪ Ketai Library for Processing ▪ https://code.google.com/p/ketai/
▪ Support all phone sensors ▪ GPS, Compass, Light, Camera, etc
▪ Include Ketai Library ▪ import ketai.sensors.*; ▪ KetaiSensor sensor;
Using Sensors ▪ Setup in Setup( ) function
▪ sensor = new KetaiSensor(this); ▪ sensor.start(); ▪ sensor.list();
▪ Event based sensor reading void onAccelerometerEvent(…) { accelerometer.set(x, y, z); }
Sensor Demo
Using the Camera ▪ Import camera library
▪ import ketai.camera.*; ▪ KetaiCamera cam;
▪ Setup in Setup( ) function ▪ cam = new KetaiCamera(this, 640, 480, 15);
▪ Draw camera image void draw() { //draw the camera image image(cam, width/2, height/2); }
Camera Demo
WearScript
▪ JavaScript development for Glass ▪ http://www.wearscript.com/en/
▪ Script directory ▪ http://weariverse.com/
● WearScript Features ▪ Community of Developers ▪ Easy development of Glass Applications ▪ GDK card format ▪ Support for all sensor input
▪ Support for advanced features ▪ Augmented Reality ▪ Eye tracking ▪ Arduino input
● WearScript Playground
▪ Test code and run on Glass ▪ https://api.wearscript.com/
▪ 8: Design Presentations
▪ 9: Research Directions
Challenges for the Future (2001) ▪ Privacy ▪ Power use ▪ Networking ▪ Collaboration ▪ Heat dissipation ▪ Interface design ▪ Intellectual tools ▪ Augmented Reality systems
Starner, T. (2001). The challenges of wearable computing: Part 1. IEEE Micro,21(4), 44-52. Starner, T. (2001). The challenges of wearable computing: Part 2. IEEE Micro,21(4), 54-67.
Gesture Interaction With Glass ▪ 3 Gear Systems ▪ Hand tracking
▪ Hand data sent to glass ▪ Wifi networking ▪ Hand joint position ▪ AR application rendering ▪ Vuforia tracking
Performance
▪ Full 3d hand model input ▪ 10 - 15 fps tracking, 1 cm fingertip resolution
● Meta Gesture Interaction
▪ Depth sensor + Stereo see-through ▪ https://www.spaceglasses.com/
Current Collaboration
▪ First person remote conferencing/hangouts ▪ Limitations
- Single POV, no spatial cues, no annotations, etc
Sharing Space: Social Panoramas
▪ Capture and share social spaces in real time ▪ Enable remote people to feel like they’re with you
Context Sensing ▪ Using context to manage information ▪ progressive information display as user shows
interest ▪ Context from ▪ Speech ▪ Gaze ▪ Real world
▪ Wearable AR Display
Ajanki, A., Billinghurst, M., Gamper, H., Järvenpää, T., Kandemir, M., Kaski, S., ... & Tossavainen, T. (2011). An augmented reality interface to contextual information. Virtual reality, 15(2-3), 161-173.
Gaze Interaction
AR View
More Information Over Time
Social Perception
TAT Augmented ID
▪ 10: Conclusions
● Conclusions ▪ Wearable computing is a 4th generation of computing devices ▪ A range of wearables will appear in 2014 ▪ Ecosystem of devices
▪ There are many existing tools for prototyping ▪ Low fidelity, high fidelity
▪ Significant research opportunities exist ▪ User interaction, displays, social impact
More Information • Mark Billinghurst
– Email: [email protected]
– Twitter: @marknb00 • Daniela Busse
– Email: [email protected]
• Website – www.hitlabnz.org
▪ 11: Resources
Glass Developer Resources ▪ Main Developer Website ▪ https://developers.google.com/glass/
▪ Glass Apps Developer Site ▪ http://glass-apps.org/glass-developer
▪ Google Design Guidelines Site ▪ https://developers.google.com/glass/design/
index?utm_source=tuicool ▪ Google Glass Emulator ▪ http://glass-apps.org/google-glass-emulator
Other Resources ▪ AR for Glass Website ▪ http://www.arforglass.org/
▪ Vandrico Database of wearable devices ▪ http://vandrico.com/database
● Glass UI Design Guidelines
▪ More guidelines ▪ https://developers.google.com/glass/design/index
Books ▪ Programming Google Glass ▪ Eric Redmond
▪ Rapid Android Development: Build Rich, Sensor-Based Applications with Processing ▪ Daniel Sauter
▪ Microinteractions: Designing with Details ▪ Dan Saffer ▪ http://microinteractions.com/
▪ Beginning Google Glass Development ▪ by Jeff Tang
