View
216
Download
0
Category
Preview:
Citation preview
Ubiquitous Computing Toolkits
Tara MatthewsAdvance User Interface Software Fall 2004
Goals of Ubicomp
Scalable interfaces multiple devices inter-device communication multiple users
Natural interaction getting away from the desktop augmenting surrounding environment provide context-appropriate services
Calmness more info w/o overwhelming users aware of user’s interruptiblity
Toolkits for Goals
Scalable interfaces multiple devices Ubicomp
Infrastructure inter-device communication multiple users
Natural interaction getting away from the desktop augmenting surrounding environment provide context-appropriate services
Calmness more info w/o overwhelming users aware of user’s interruptiblity
Toolkits for Goals
Scalable interfaces multiple devices inter-device communication multiple users CSCW (not covered)
Natural interaction getting away from the desktop augmenting surrounding environment provide context-appropriate services
Calmness more info w/o overwhelming users aware of user’s interruptiblity
Toolkits for Goals
Scalable interfaces multiple devices inter-device communication multiple users
Natural interaction getting away from the desktop Physical UIs /
Mobile augmenting surrounding environment (other lectures)
provide context-appropriate services Calmness
more info w/o overwhelming users aware of user’s interruptiblity
Toolkits for Goals
Scalable interfaces multiple devices inter-device communication multiple users
Natural interaction getting away from the desktop augmenting surrounding environment provide context-appropriate services Context-Aware
Calmness more info w/o overwhelming users aware of user’s interruptiblity
Toolkits for Goals
Scalable interfaces multiple devices inter-device communication multiple users
Natural interaction getting away from the desktop augmenting surrounding environment provide context-appropriate services
Calmness more info w/o overwhelming users Peripheral Displays aware of user’s interruptiblity
Toolkits for Goals
Scalable interfaces multiple devices inter-device communication multiple users
Natural interaction getting away from the desktop augmenting surrounding environment provide context-appropriate services
Calmness more info w/o overwhelming users aware of user’s interruptiblity Interruptibility
(not covered)
Toolkits for Goals
Scalable interfaces multiple devices inter-device communication multiple users
Natural interaction getting away from the desktop augmenting surrounding environment provide context-appropriate services
Calmness more info w/o overwhelming users aware of user’s interruptiblity
Goals accomplished? Evaluation
Toolkit Categories
Ubicomp infrastructure Context-aware Peripheral displays Evaluation
Not covering: CSCW (separate field) Interruptibility (no toolkits yet) Physical UIs (covered in Jack’s lecture) Mobile Devices (covered in an unclaimed lecture)
Ubicomp Infrastructure
iROS Stanford Interactive Room Operating System
iStuff Physical application toolkit build on iROS
Aura CMU “Distraction-Free Ubiquitous Computing” project Distributed services infrastructure
Gaia University Of Illinois at Urbana-Champaign Infrastructure for Active Spaces
iROS
Interactive Room (iRoom) Operating System Definition: A ubiquitous computing
environment where groups come together to collaborate on solving problems (not a toolkit)
Contains: Embedded devices: large display screens, table-
top display, & other output devices Mobile devices: mobile devices brought into
space can be used with embedded facilities
iROS Goals Facilitate common Interactive Workspace usages observed
in iRoom prototype workspace: Move data between devices Control devices & apps from other devices Coordinate apps, including ones not written to work together
Additional goals: Provide for heterogeneity of devices in workspace,
& new devices being added over time Allow easy integration of legacy devices Robust to transient failures Portable across installations Actions appear instantaneous to users
iROS
3 sub-systems: Event Heap: stores and forwards events Data Heap: facilitates data movement in an
interactive workspace iCrafter: provides a system for service
advertisement and invocation, along with a user interface generator for services.
Event Heap Tuple space model
Associated memory: content is addressable using a pattern Distributed systems based on blackboard model are easily
created in tuple space Decouples apps in space & time
Added semantics: Apps not designed to work together, so need
Self-describing tuples, Flexible typing, Typed tuples Apps can snoop or interpose
Tuple sequencing Data shared & can build up
Tuple expiration Benefits
Anonymous coordination Interposability Snooping
Event Heap
Figure: Example operation showing placed events, and using template events for matching
Data Heap
Provides type & location independent storage
Machine independent: Don’t need explicit location
of data
Type independent: If a set of type converters
are available, system automatically transforms the data
Figure: Shows automatic retrieval of a Word document in PostScript form
iCrafter
Universal control system: control anything from anywhere
Services advertise how they can be controlled System returns appropriate interface per
device Interfaces can range from fully custom to
dynamically generated
iCrafter
Active Interface
UIRenderer
Generator Selector
Generator Processor
Environment Database
Generator Database
Appliance Description
Service Description
ServiceUser
InterfaceManager
Appliance
132
4
Remote Invocation
Discovery
Active Interface
UIRenderer
Generator Selector
Generator Processor
Environment Database
Generator Database
Appliance Description
Service Description
ServiceUser
InterfaceManager
Appliance
132
4
Remote Invocation
Discovery
1. USER REQUEST
The user requests an interface for a service. The appliance sends this request to the Interface Manager, along with appliance description information.
iCrafter
Active Interface
UIRenderer
Generator Selector
Generator Processor
Environment Database
Generator Database
Appliance Description
Service Description
ServiceUser
InterfaceManager
Appliance
132
4
Remote Invocation
Discovery
Active Interface
UIRenderer
Generator Selector
Generator Processor
Environment Database
Generator Database
Appliance Description
Service Description
ServiceUser
InterfaceManager
Appliance
132
4
Remote Invocation
Discovery
2. GENERATOR SELECTION
Upon receiving the request, the Generator Selector selects an appropriate generator based on the service(s) selected and the appliance description.
iCrafter
Active Interface
UIRenderer
Generator Selector
Generator Processor
Environment Database
Generator Database
Appliance Description
Service Description
ServiceUser
InterfaceManager
Appliance
132
4
Remote Invocation
Discovery
Active Interface
UIRenderer
Generator Selector
Generator Processor
Environment Database
Generator Database
Appliance Description
Service Description
ServiceUser
InterfaceManager
Appliance
132
4
Remote Invocation
Discovery
3. GENERATOR PROCESSING
The Generator Processor runs the selected generator with access to all context (appliance, service, environment), which outputs a UI description.
iCrafter
Active Interface
UIRenderer
Generator Selector
Generator Processor
Environment Database
Generator Database
Appliance Description
Service Description
ServiceUser
InterfaceManager
Appliance
132
4
Remote Invocation
Discovery
Active Interface
UIRenderer
Generator Selector
Generator Processor
Environment Database
Generator Database
Appliance Description
Service Description
ServiceUser
InterfaceManager
Appliance
132
4
Remote Invocation
Discovery
4. USER INTERFACE RENDERING
The UI description is sent to the UI Renderer on the appliance, which renders the interface and handles user interactions, including service invocations over the EH.
iStuff
Toolkit to prototype physical UIs Flexible, lightweight components Protocol independence Platform independence with cross-platform capabilities Ease of integration with existing applications Support for multiple simultaneous users
Built on iROS iStuff wireless devices (e.g., iButton, iSlider, iLight)
send/receive events to/from Event Heap via a proxy PatchPanel translates events (e.g., iButton: light on | light off) Apps get events from devices via PatchPanel + Event Heap
iStuff Architecture
Event Heap
PatchPanel Proxy
iStuff Device
Transceiver
iStu
ff c
ompo
nent
Application
Wireless connection
TCP/IP connection
iStuff Devices
iSpeaker iLight
iBuzzer
Anoto PeniMouse
iMike
iDog
X10
iStylusiSlider
RF
iButtons
InputInput
OutputOutput
Aura
CMU “Distraction-Free Ubiquitous Computing” theme: Mobility: allow users to move computational tasks
from one place to another Maximize use of available resources Minimize user distraction & drains on attention
Personal information aura: spans wearable, handheld, desktop, & infrastructure devices
Requires new system design: hardware, network, OS, middleware, & UI support
Aura Research Framework
Users
Tasks
Services
OS / Network
Physical Devices
Research Examples
UI / Agents / Speech
Task-driven computing Rapid service configuration
Energy-aware adaptation Multi-fidelity computation Nomadic data access
Intelligent networking Power-aware devices Wearable computers
Aura Example Scenario
Fred prepares presentation & is late Fred gets PDA & Aura transfers slides to it Finishes slides on PDA walking to meeting Aura finds location of meeting from calendar
& uploads slides to projection machine Aura recognizes unfamiliar meeting
attendees faces & skips budget slides
Aura Architecture
User tasks are represented as set of distributed services Database query interface to access distributed services Easily search mixed environments for needed services A key service is People Location Service
Environments self-monitor & re-negotiate task support given runtime variation of capabilities & resources Can detect when task requirements (e.g., min response
time) aren’t met, & search & deploy alternative configurations to support task
Uses software architecture models (graph of components & connectors) for runtime adaptability
Aura Architecture
Task Manager: tracks user context, environment, & task changes Context Observer: provides info on physical context & reports
relevant events Task & Environment Managers Environment Manager: handles access to distributed services Suppliers: provide services for tasks (text editing, video playing,...)
Privacy in Aura
Aura services depend on user location info Introduces privacy concerns
Location policies: define who gets location info, where, when, and at what granularity e.g., “Bob is allowed to know Alice’s location
when she is in NSH” Similar to Bell Labs’ Houdini system’s “rules”
Not based on how user’s actually disclose location
GAIA
Infrastructure for Active Spaces, UIUC Like Aura:
Distributed services architecture accessed via structured queries
Main difference: Focus on user customization of apps based on
resources in current space
Toolkit Categories
Ubicomp infrastructure Context-aware Peripheral displays Evaluation
Context
Context is key to ubicomp environment that reacts correctly to user’s current situation
Definition: Info used to characterize the situation of a person, place or object relevant to the interaction between a human & a computational service
Primary context types: Identity (who), Location (where), Time (when), Activity (what)
Secondary context types: e.g., for identity: email address, phone number, birthdate, etc
Context-Aware Applications
Definition: Uses context to provide relevant info and/or services to the user, where relevancy depends on the user’s task
3 categories: Present information & services to users
e.g., nearby wireless networks, directions Automatically execute services
e.g., phone vibrates when in meeting Tag info with context for later retrieval
e.g., class lectures and notes
Toolkits for Context 3 main approaches:
1. Widget model Context Toolkit Dey, Abowd, & Salber, 2001 Based on architecture of GUIs
2. Distributed services model Context Fabric Hong & Landay, 2001 Based on client-server dialog
3. Blackboard model iRoom Winograd, 2001 Based on artificial intelligence apps (Engelmore, 1988)
Widget Model: Context Toolkit
Operating system view Abstraction to hardware (sensors);
data is secondary Similar to GUI input handling: uses widget
abstractions for handling context input Uses 3 abstractions:
widgets, interpreters & aggregators
Benefits
Separates semantics from low-level input sensor handling
Apps can focus on context, not sensors or analysis of sensor input
Allows re-use of context widgets, interpreters & aggregators
Drawbacks
Tight coupling – less robust to component failure
Single-manager (Discoverer) control – less flexible
Context Toolkit Architecture
Widgets
Encapsulate info about a single piece of context (e.g., location or activity)
Hide details of underlying sensors from apps Provide customizable & reusable building
blocks of context sensing Provide historical context info
Widgets
Include optional services, or actuation of output in environment Like Interactors: responsible for input & output e.g., light widget could sense light level & adjust
lights accordingly Get context by subscription or polling
Callback methods used to notify subscribers
Aggregators
Widgets + ability to aggregate context info Collect sensory info about an entity (person,
place, thing, etc.) from multiple sources into one widget
Hide more complexity about context-sensing mechanisms by combining multiple sensors
Enable maintainability and efficiency
Interpreters
Abstract sensor info Use lower level sensory information to
deduce higher level info e.g., identity + location + sound sensors
“is there a meeting?”
Putting an App Together
Discoverer enables apps to locate and subscribe to context components
Distributed infrastructure uses p2p comm.
Distributed Services Model
Database view (vs. OS view of CTK) Data is primary, hardware is secondary Focus on how data is modeled, distributed, protected, & used
Middleware technologies that can be accessed through a network Example: infrastructure=Internet; service=DNS
Any kind of device or application can use context gathering & processing services by adhering to predefined data formats & network protocols
Benefits
Interoperable: independent of hardware platform, OS, & programming language
Decoupling: Sensors, services, & apps can be changed w/o affecting each other
Simpler devices: sensors, processing power, data, & services w/in infrastructure shared
Drawbacks
Applications must poll for data (proactively) Servers are specialized per sensor
Context Fabric
Includes 3 pieces1. Distributed data model of people, places, things
each device has a space: private data goes to local space multiple views of context (mine, yours, theirs)
2. Context Specification Language (like SQL) apps specify what context info they need in uniform way e.g., "What are the nearby movie theaters?”
3. Context service as API (per device) interpret CSL queries and provide context info
Privacy in Context Fabric
Decentralized architecture allows it to capture, store, process, & share in privacy-sensitive manner Capture, store, & process data on user’s device
Provides mechanisms for end-user control of sharing
Plausible deniability built-in e.g., if request for context info denied, system can
reply “unknown”
Aura
Also follows distributed services model Uses SQL-like interface to access a set of
distributed contextual services Handles privacy through user-set policies
Blackboard Model Communication goes through a centralized server Components
post messages shared blackboard subscribe to get posted messages matching specified pattern
Route by matching message content to subscriber's pattern Pattern matching uses AI (often apply sophisticated inference
procedures to logical representations) Direct communication between components simulated by including
an identifier for path endpoints as a field in message Announce-listen style
Components that provide services periodically post events Component that uses the data listens to these events, & if one does
not appear within the expected time can initiate restart operations
Benefits
Ease of configuration Robustness to component failure
When component fails, only communication link broken is from it to blackboard
Dependent components can detect failure and call for restart
Simplicity provided by a uniform communications path (to the blackboard) No complex protocol for finding ports or
resources, establishing connections
Drawbacks
Communication less efficient Each communication requires 2 hops General message structure not optimized for
particular data or interaction protocol
iRoom & iStuff
Event Heap: the blackboard & central communication server
Review of Context Approaches
Widget Model – CTK Widget architecture where Discoverer acts as a manager of
context widgets
Distributed Infrastructure Model – ConFab, Aura, Gaia Client-server architecture where client apps communicate
directly w/ services using structured language
Blackboard Model – iRoom, iStuff Central server architecture where apps communicate only w/
server & events are routed using subscriber pattern matching
Toolkit Categories
Ubicomp infrastructure Context-aware Peripheral displays Evaluation
Peripheral Displays
Provide awareness with min attention Separate from primary task Important to ubicomp vision of many devices to 1 user
Non-focal display not used unless providing peripheral info
Enable you to monitor many info sources while maintaining a calm environment But, only calm if designed to manage attention they attract
Why is creating PDs hard?
Need to abstract info to be glance-able Need mechanisms for dynamically
managing attention PDs attract: Deciding attention levels to attract
(notification levels) Displaying info appropriately (transitions)
Peripheral Display Toolkit (PTK) Supports these 3 key issues in PD creation
Example PTK Applications Remote Activity
Social Guitar Audio Monitor Motion Monitor Remote Awareness Display
Bus Displays Bus Mobile Bus LED
Instant Messenger Status
IM Picture FrameSocial Guitar Bus LED BusMobile
Orb showing remote activity
PTK Architecture
1. Support for managing impact on human attention using abstraction, notification levels, and transitions
2. Simplified code design and code re-use
3. Library of common PD components
Output
TransitionInput
NotificationMap
Abstractor
Architecture DiagramInput-Side Discovery
ServerOutput-Side
Peripheral Display 1
(Abstractors) (NotificationMaps)
(Output Widgets w/ optional Transition)
Peripheral Display 2
Input 1
Input 2
Input N
Abs. 1
Abs. NPTK
DiscoveryServer
Abs. 2
Abs. N
Abs. 1 N.M. 1
N.M. 2
N.M. N
Out 1Trans
Out 2
Out N
.
.
. …
Abs. 2
Abs. N
Abs. 1 N.M. 1
N.M. 2
N.M. N
Out 2Trans
Out 1
Out N
…
Input-Side DiscoveryServer
Output-Side
Peripheral Display 1
(Abstractors) (NotificationMaps)
(Output Widgets w/ optional Transition)
Peripheral Display 2
Input 1
Input 2
Input N
Abs. 1
Abs. NPTK
DiscoveryServer
Abs. 2
Abs. N
Abs. 1 N.M. 1
N.M. 2
N.M. N
Out 1Trans Out 1Trans
Out 2
Out N
.
.
. …
Abs. 2
Abs. N
Abs. 1 N.M. 1
N.M. 2
N.M. N
Out 2Trans Out 2Trans
Out 1
Out N
…
Library Components Input
audio, camera, Phidgets, Context Toolkit, online calendars, news, stocks, weather, Web page parser, serial port
Input audio, camera, Phidgets, Context Toolkit, online calendars,
news, stocks, weather, Web page parser, serial port
Output ticker text, Ambient Orb, Phidgets
Library Components
Input audio, camera, Phidgets, Context Toolkit, online calendars,
news, stocks, weather, Web page parser, serial port
Output ticker text, Ambient Orb, Phidgets
Abstractors motion, people counting, voices, phone ringing
Library Components
Input audio, camera, Phidgets, Context Toolkit, online calendars,
news, stocks, weather, Web page parser, serial port
Output ticker text, Ambient Orb, Phidgets
Abstractors motion, people counting, voices, phone ringing
Notification exact match, threshold, contains, degree of change
Library Components
info urgency user attention
Toolkit Categories
Ubicomp infrastructure Context-aware Peripheral displays Evaluation
Evaluation
3 tools for evaluating people in natural settings from Intille et al.:
1. Context-aware experience sampling based on GPS or other sensors
e.g., ask questions when user goes to store
2. Ubiquitous sensing system that detects environmental changes
e.g., tape-on touch, proximity, & light sensors
3. Image-based experience sampling system i.e., take a picture & ask user about it later
Summary: Ubicomp Goals
Scalable interfaces multiple devices inter-device communication multiple users
Natural interaction getting away from the desktop augmenting surrounding environment provide context-appropriate services
Calmness more info w/o overwhelming users aware of user’s interruptiblity
Summary: Ubicomp Toolkits
Ubicomp infrastructure iROS, iStuff, Aura, Gaia
Context-aware Context Toolkit, Context Fabric, iRoom
Peripheral displays Peripheral Display Toolkit
Evaluation Intille’s toolkit
Thanks!
CSCW
Groupware Architectures Phillips, W.G., 1999. Architectures for Synchronous
Groupware, Tech. Rep.. http://phillips.rmc.ca/greg/pub/ Greenberg, S. and Roseman, M., 1999. Groupware
Toolkits for Synchronous Work. In: Beaudouin-Lafon, M. (Ed.), Trends In CSCW'99, No. 7 in Trends in Software, John Wiley & Sons, New York, NY, USA, ch. 6, pp. 135–168.
Roseman, M. and Greenberg, S., 1992. GROUPKIT: a groupware toolkit for building real-time conferencing applications. In: Proceedings of the conference on Computer-supported cooperative work, ACM Press, pp. 43–50. http://doi.acm.org/10.1145/143457.143460
Recommended