EIA9806822: “Automated Understanding of Captured Experience”Georgia Institute of Technology, College of ComputingInvestigators: Irfan Essa, G. Abowd, C. Atkeson, & K. Ramachandran

Project Goals: To reduce substantially the human input necessary for creating and accessing large collections of multimedia, particularly multimedia created by capturing what is happening in an environment.

Outcomes: (provide each where applicable) Ideas Extended use of automated capture in everyday environments (classroom/office/home) over longer periods of time and the investigation of providing automated speech and visual transcription, with accompanying search, retrieval and visualization techniques.Tools Tools that support the inclusion of automatically sensed context as meta-data to tagcaptured information. This architectural approach has been instantiated as a toolkit (the Context Toolkit) and used to produce prototype capture applications. Other TOOLs include INfrastructure for Capture and Acess (INCA) and Stampede.People The Classroom 2000/eClass system has been presented on a variety of local, national and international science and education programs as a demonstration of how ubiquitous computing technology can impact the future of education. Many Post-Docs (2), PhD & MS Students (>10), and Undergraduates (>20) have helped develop and have used the technology developed in part by this funding are now working in academic, research, and industrial positions.

Cap

ture

d E

xper

ienc

e D

urin

g a

Lec

ture

, di

spla

yed

as a

WW

W

page

EIA9806822: “Automated Understanding of Captured Experience”Georgia Institute of Technology, College of ComputingInvestigators: Irfan Essa, G. Abowd, C. Atkeson, & K. Ramachandran

Interactive Web pages generated by the Classroom 2000 / Eclass System after a Lecture.

Vision-based Tracking of Activities

Recognition ofFlipping Pages using HMMs

Annotation ofObjects Interacted with

EIA9806822: “Automated Understanding of Captured Experience”Georgia Institute of Technology, College of ComputingInvestigators: Irfan Essa, G. Abowd, C. Atkeson, & K. Ramachandran

Impact:We developed methodologies that supported extended use of captured experiences classrooms

(primarily, with a few experiments in the home and office scenarios) over longer periods of time. To support this work, we researched and made significant contributions by

1. Taking the existing Classroom 2000 software systems and extended it to use automatic speech transcription, with accompanying search, retrieval and visualization techniques.

2. Software engineering principle were extensively used to provide infrastructure to support such large scale capture and access scenario and allow for easy interface between capture, sensing, and usabilty requirements (with the goal of reducing the load on the user). Resulted in Context Toolkit and the INfrastructure for Capture and Access (INCA) Toolkit

3. Developed a distributed Programming Structure (Stampede) to support such high-end multimedia processing.

4. Developed computer vision-based methods to track people and recognize their activities within a specific context.

5. System in use for educational transcription. Used by many students and educators.

Barriers & Opportunities: Developed systems and infrastructures allow for extensions to many domains. Aware environments will serve towards providing us with ability to sort through and find relevant information as we go about our daily activities. Complex environments like homes and offices provide very interesting opportunities. Further opportunities lie in the support of challenges associated with cognitive and/or age-related impairments.

Website: http://www.cc.gatech.edu/cpl/projects/AUCE