Eyes-Free User Interaction

Eyes-Free User Interaction

T. V. RamanGoogle

http://emacspeak.sf.net/raman

Charles L. ChenGoogle

www.clcworld.net

May 21, 2009

OverviewAndroid Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 2 / 39

Android

Dialer

Contacts

TTS

Gestures

MusicBrowser

Overlays

Conclusion

Android Programming

Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Android: Engineer’s Dream!Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 4 / 39

■ Smart phones — computer in your pocket.■ Open platform — liberates innovation.■ New I/O — minimize user interaction.

Bend technology to your will!

Innovative User InterfacesAndroid Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 5 / 39

Create innovative end-user solutions that:

■ Fit into the user’s mode of working,■ Avoid one-size fits all solutions,■ Design context-aware solutions,■ Tailor solutions to user’s needs.

Eyes-Free InteractionAndroid Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 6 / 39

Not just for blind users!

■ Eyes-busy environments.■ One-handed use.■ Minimize explicit user interaction.

Rethink all assumptions about the user.

Devices That SenseAndroid Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 7 / 39

Mobile phones make UI design exciting!

■ Can capture user intent — keyboard,. . ..■ Can see — camera.■ Can hear and speak — speech I/O.■ Can sense and stimulate touch — touch screen and vibration.■ Can sense motion — accelerometer, compass.■ Can sense position — GPS, cell towers.

■....

Our Android has many eyes and ears!

Touch Screen Interaction, Eyes-Free!

Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Using A Touch ScreenAndroid Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 9 / 39

These areatomic tasks:

■ Identifying the item to touch.■ Touching item to activate it.■ Eyes-free use requires help with the former.■ Activation should produce appropriate feedback.

On-Screen TouchpadsAndroid Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 10 / 39

Eyes-Free use of an on-screen touchpad:

■ Need tosee the buttons to know where to touch.■ Because: buttons useabsolute positioning.■ Conclusion: (wrong) — you need to see to use a touch screen!■ Solution: Relax constraint of absolute positioning.

What if we usedrelative positioning instead?

Phone KeypadAndroid Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 11 / 39

Eyes-free phone dialer:

■ Keypad centered where you touch — relative positioning.■ Users already know layout of a phone keypad.■ Canstroke from the center for a desired digit.

Enables eyes-free one-handed dialing!

Dialer FeedbackAndroid Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 12 / 39

Enables one-handed, eyes-free use:

■ Buttonsvibrate when you move over them.■ Augmented by auditory feedback.■ Pressing buttons produces spoken and tactile feedback.■ Shake to erase input.

Synchronized auditory and haptic feedback is key.

Text Input Via Touch Screen

Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Contact ManagerAndroid Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 14 / 39

No one dials phone numbers any more!

■ Browsing contacts using touch screen.■ Finding contacts.■ Managing contacts.

Entering Text InputAndroid Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 15 / 39

Extending stroke dialer for textual input:

■ Arrange letters in4 concentric circles.■ Strokesaccess distinct circles.■ Choose desired letter bytracing chosen circle.

Any letter is at most 3 steps away!

Keypad LayoutAndroid Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 16 / 39

Gesture: Pick circle, and trace to desired character.

A I Q

⊙ Y

U M E

Letter Gesture

A Stroke up to the leftB Stroke A, trace rightE Stroke down to the right

Managing ContactsAndroid Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 17 / 39

Contacts live in the cloud.

■ Contacts managed over the network.■ Updated automatically on the device.■ No more editing contacts with two fingers!

Text To Speech On Android

Android Dialer ContactsTTS Gestures MusicBrowser Overlays Conclusion

TTS LibraryAndroid Dialer ContactsTTS Gestures MusicBrowser Overlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 19 / 39

■ Create speech-enabled apps.■ Developer compiles against the TTS library stub.■ Text To Speech is an Android service.■ ClassTTS manages communication with the service.■ Enables multiple apps to use TTS without code duplication.■ TTS library can be updated without a need to recompile clients.

TTS Method SummaryAndroid Dialer ContactsTTS Gestures MusicBrowser Overlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 20 / 39

■ speak(String text, int queueMode, String[] params)■ stop()■ isSpeaking()■ Also has methods for synthesizing to a file, specifying the language, etc.

TTS Object PropertiesAndroid Dialer ContactsTTS Gestures MusicBrowser Overlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 21 / 39

■ Can automatically prompt the user to install the TTS Service.■ When absent, fail silently (methods become no-ops)

See: http://eyes-free.googlecode.com/

Gesture Library For Stroke Input

Android Dialer Contacts TTSGesturesMusicBrowser Overlays Conclusion

Gesture OverlayAndroid Dialer Contacts TTSGesturesMusicBrowser Overlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 23 / 39

■ Overlay watches for touch events■ Returns identified gesture■ Exposes same UI as the stroke dialer■ Implemented as a custom transparentView with callbacks

Gesture Overlay ImplementationAndroid Dialer Contacts TTSGesturesMusicBrowser Overlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 24 / 39

■ Implement InterfaceGestureListener ,■ Create a new TouchGestureControlOverlay with it,■ Set aFrameLayout as the baseView,■ Add theView with the content as a child to the FrameLayout,■ Add the TouchGestureControlOverlay as a child

Can easily enable/disable theTouchGestureControlOverlay

A Simple Music File Browser

Android Dialer Contacts TTS GesturesMusicBrowser Overlays Conclusion

Music File BrowserAndroid Dialer Contacts TTS GesturesMusicBrowser Overlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 26 / 39

Barebones version:

■ Uses a simpleListView■ Based on a FileBrowser tutorial on Anddev.org■ Browse directories and mp3 files on SDCard.■ Play music by clicking on the file.■ Directory path is the first entry.■ Click this to cycle through files in current level.■ Tactile feedback when scrolling through list items.

Adding Spoken OutputAndroid Dialer Contacts TTS GesturesMusicBrowser Overlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 27 / 39

■ Create objectTTS in methodonCreate.■ Result: Runs methodTTS.onInit as part of setup.■ Can add application-specific code to methodTTS.onInit.■ Result: Application comes up speaking.

Music Player TalksAndroid Dialer Contacts TTS GesturesMusicBrowser Overlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 28 / 39

private InitListener i = new InitListener() {public void onInit(int version) {tts.speak("Music File Browser started.", 0, null);

}};

Speaking List ItemsAndroid Dialer Contacts TTS GesturesMusicBrowser Overlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 29 / 39

■ Add TTS call to speak list items.■ Can use event handler that produces tactile feedback.

Speaking List ItemsAndroid Dialer Contacts TTS GesturesMusicBrowser Overlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 30 / 39

Augment event handler with a TTS call.

public boolean onTrackballEvent(MotionEvent event) {String filename = directoryEntries.get(selectedId);tts.speak(filename.substring(1), 0, null);

}

Self-voicing functionality added with just 13 additional lines of code!

Adding Gesture Input To MusicBrowser

Android Dialer Contacts TTS Gestures MusicBrowserOverlays Conclusion

Advantages Of An OverlayAndroid Dialer Contacts TTS Gestures MusicBrowserOverlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 32 / 39

■ GestureOverlay is a transparent overlay.■ Can be layered on an existingView.■ Does not interfere with visual appearance.

Gestures For Music BrowserAndroid Dialer Contacts TTS Gestures MusicBrowserOverlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 33 / 39

■ Add GestureListener to handle user input.■ Gesture Tap: play/pause.■ Gesture Right: Browse to next track.

Add application-specific code in methodonGestureFinish.

Steps In Adding Gesture InputAndroid Dialer Contacts TTS Gestures MusicBrowserOverlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 34 / 39

private GestureListener g = new GestureListener() {public void onGestureChange(Gesture arg0) {}public void onGestureFinish(Gesture arg0) {if (arg0 == Gesture.CENTER) {// play/pause music

} else if (arg0 == Gesture.RIGHT) {playNext();

}}public void onGestureStart(Gesture arg0) {}

};

Adding Overlay To Music PlayerAndroid Dialer Contacts TTS Gestures MusicBrowserOverlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 35 / 39

myFrame = new FrameLayout(this);myFrame.addView(myList);myG = new TouchGestureControlOverlay(this,gestureListener);overlayActive = false;setContentView(myFrame);

Toggling Gesture SupportAndroid Dialer Contacts TTS Gestures MusicBrowserOverlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 36 / 39

if (overlayActive) {myFrame.removeView(myG);overlayActive = false;tts.speak("Gestures disabled.", 0, null);

} else {myFrame.addView(myG);overlayActive = true;tts.speak("Gestures activated.", 0, null);

}

Gesture support enabled by adding about 40 lines of code.

Conclusion

Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

SummaryAndroid Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 38 / 39

■ Programmable smart phones are an engineer’s dream!■ I/O peripherals open up new UI avenues.■ Provide desired information with minimal user interaction.

Profound impact on how we work and play!

Watch Computing Take Off!Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 39 / 39