Adaptive Social Assistants: Using Mobile Computing

Devices to Assist Individuals with Cognitive Disabilities

Project Update

Cathy Bodine

University of ColoradoJune 14, 2001

Project Objectives• Develop power-aware mobile computing devices

that adapt to their users based upon observed and predicted behavior.

• Configure these devices as adaptive social assistants that simplify daily living and job-related tasks for persons with cognitive disabilities.

• The role of the social assistant is to replace many of the functions of a personal attendant, while enhancing the independence and functioning of the user as he or she engages in daily life activities as a contributing member of society.

Project Team• John Bennett - CU Dept. of Comp. Science• Dirk Grunwald - CU Dept. of Comp. Science• Clayton Lewis - CU Dept. of Comp. Science• Michael Mozer - CU Dept. of Comp. Science• Tamara Sumner - CU Dept. of Comp. Science• Timothy Brown - CU Dept. of ECE and ITP• Cathy Bodine - UCHSC Depts. Of Peds. & Rehab. Med.;

Director, Assistive Technology Partners• Linda Crnic - UCHSC Depts. of Pediatrics and Psychiatry• Deborah Fidler - CSU Dept. of Human Development and

Family Studies• Lori Ramig - CU Dept. of Speech, Lang., and Hearing Sci.• Sally Rogers - UCHSC Dept. of Psychiatry• David Patterson - UCHSC Depts. of Biochemistry and

Molecular Genetics and Medicine

Project Expertise by Area• John Bennett (distributed, parallel, and mobile computing)• Dirk Grunwald (computer systems and networking)• Clayton Lewis (human-computer interaction)• Michael Mozer (machine learning)• Tamara Sumner (design and cognition, HCI)• Timothy Brown (wireless computing)• Cathy Bodine (assistive technology)• Linda Crnic (mental retardation) • Deborah Fidler (developmental disabilities, esp. outcomes)• Lori Ramig (speech pathology and remediation)• Sally Rogers (developmental disabilities, esp. autism)• David Patterson (genetics of developmental disorders)

Project Status• Preproposal ($5M budget) submitted Fall ’00

(1 of 661 submitted)

• Favorable action on preproposal by NSF(1 of 258 (39%) approved for full proposal)

• Full proposal submitted April ’01(Sept. ’01 start date if funded)

(about $90M allocated => ~ 50 proposals will be funded)

• $1M matching support pledged from Coleman Institute

Core Technology

• Machine Learning

(the means by which the social assistants learn from and adapt to user behavior)

• Mobility and Data Management

(power management, how we support both user and device mobility, and how we represent, access, update, and protect information)

• Human-Computer Interaction

(how the user interacts with the assistant)

Target Tasks• Route Navigation

• Assistant provides auditory / verbal instructions and observes trajectories to build up expectations. Significant violations of these expectations results in warning or other remedial action.

• Communication• Assistant provides augmentative / alternative

communication. “Keypad” adapts to user (not the other way around). We intend to build on work of Enkidu and Saltillo.

• Memory Prosthesis• Assistant provides memory cues to help user stay on and

complete tasks, as well improving ability to complete tasks more successfully. We intend to build upon work of AbleLink, integrating adaptive behavior into “Pocket Coach” like devices.

Design Issues• Safety, Welfare & Privacy

• Reliability, Durability, Ease of Use & Wearability

• High Abandonment Rate of Assistive Devices

• Training

• The Role of Adaptation

Participant Selection• Working or job-seeking adult aged ~16 - ~40• Diagnosed with developmental disability such

as one of Down, Williams, fragile X, or Prader-Willi syndromes, or autism

• Ambulatory and within gross normal visual and hearing acuity limits

• Able to follow single step instructions• Must have 24/7 emergency support• For now, exclude those with clinical range

aberrant behaviors, serious communicable diseases, severe motor impairments

Syndrome prevalence• Down syndrome (the most common genetic

mental retardation syndrome)• 1 in 700-1000 live births

• Williams syndrome• 1 in 20,000 live births

• Prader-Willi syndrome• 1 in 10,000 live births

• fragile X syndrome• 1 in 2,000- 4,000 live births

Targeting Syndrome Differences• Persons with Down Syndrome

• Tend to require visual support of auditory information• Show strength in sequential over simultaneous processing

=> social assistant should provide step-by-step instructions

• Persons with Williams Syndrome• Tend to require auditory support of visual information

• Persons with Prader-Willi or fragile X syndrome• Show strength in simultaneous over sequential processing

=> social assistant should provide instructions in larger, integrated context

Broader Impact of Project• Increasing independence and improving quality of

life of people with cognitive disabilities• Extension to general population, including:

• the elderly• persons who cannot read• persons who are blind• persons with traumatic brain injury (impaired memory)• anyone who needs help with daily tasks, including busy

executives and college professors

• Education and outreach• Improving our understanding of individuals with

cognitive disabilities• Technology transfer

Five Year Research Plan• Year 1 – [Fact Gathering, Exploration, and

Participatory Design]

• Year 2 – [Deliver First (Basic) Prototypes]

• Year 3 – [Evaluation; Additional Deployment and Testing]

• Year 4 – [Design Refinement]

• Year 5 - [Evaluation, Analysis, and Technology Transfer]

Year 1 – [Fact Gathering, Exploration, and Participatory Design]

• Establish dialogue with users, family members, caregivers, clinicians and manufacturers of assistive technology devices in order to develop an understanding of users needs and capabilities related to the target tasks. Refine target tasks as necessary.

• Design and implement data encryption, transmission and transaction mechanisms.

• Establish project web site.• Begin development of laboratory prototypes of social assistants. • Begin exploration of different user interface mechanisms and sensor

technologies.• Develop appropriate machine learning objective functions for device

adaptation.• Experiment with cellular, GPS, and other location sensing technologies. • Complete Institutional Review Board approval process prior to device

deployment in Year 2.• Develop and offer joint course in assistive technology for engineering and

health science graduate students.• Establish project Advisory Board

Year 2 – [Deliver First (Basic) Prototypes]

• Complete high-level design of system architecture.• Establish initial server infrastructure for both development and

field use.• Develop basic data and mobility management support

mechanisms.• Develop initial task modeling interface for use by clinicians and

caregivers.• Perform initial testing of machine learning objective functions

for device adaptation.• Develop initial user interface design. • Deploy 40 field units (20 at midyear, 20 more by year end.)

Release new software to field at 6-month intervals.• Begin to collect and analyze user data.

Year 3 – [Evaluation; Additional Deployment and Testing]

• Expand user population and task repertoire.• Expand server infrastructure as needed.• Refine and extend task modeling interface for use by clinicians

and caregivers.• Evaluate device adaptation.• Extend basic data and mobility management support

mechanisms.• Complete initial user interface design. • Deploy an additional 40 field units (20 every 6 months).

Release new software to field at 6-month intervals.

Year 4 – [Design Refinement]

• Expand user population and task repertoire.• Expand server infrastructure as needed.• Develop final task modeling interface for use by clinicians and

caregivers.• Continue to evaluate device adaptation and refine ML objective

functions based on user data.• Evaluate and refine user interface design based upon field

data. • Deploy an additional 40 field units (20 every 6 months).

Release new software to field at 6-month intervals.• Initiate technology transfer.

Year 5 - [Evaluation, Analysis, and Technology Transfer]

• Expand user population to full size and complete task repertoire.

• Expand server infrastructure as needed.• Finalize task modeling interface for use by clinicians and

caregivers.• Finalize user interface design.• Evaluate device adaptation based on analysis of user data.• Deploy 20 additional units and final software release at mid

year. • Perform comprehensive evaluation of system architecture and

design. • Complete transfer of project technology to commercial

developer.