Fabio Pianesi Massimo Zancanaro

Fabio PianesiMassimo Zancanaro

FBK-irstAlessandro Cappelletti, Bruno

Lepri, Nadia Mana

Research questions

Recognition of is happening at a given time slice (mainly) from audio-visual signals

Fabio Pianesi & Massimo Zancanaro FBK

Data sharing requirements Indipendent modules for

Visual recognition robust for lighting for Detecting parts of the body Detecting and recognizing objects

Attentional module to focus cameras, mics and other sensors where “the action is” Avoids continuous streams of data from the environment

Standards Annotation of activities Meta-data descriptions

Type of sensors, their relative position in the environment Environment description (relative to sensors: riverberation, …)

Standard for data storing


Gregory D. AbowdGeorgia Tech

Research questions

How to address questions of health and (to a lesser degree) sustainability through instrumentation and augmentation of the home. How to enable others to collect data in real homes.

Chronic care management Early detection and monitoring of

interventions for autism Video data and sensor data

Sensing for the masses Infrastructure mediated sensing data

from real homes Energy awareness, location-tracking

Gregory D. Abowd, Georgia Tech

MachineLearningSystem

Room #1 Room #2BusMonitoringSensors

InferredHumanActivity

Room #n ElectricalOutlets andAppliancesAir Ducts

PlumbingFixtures

Data sharing requirements

I want annotated home movies of young child behavior, or at least movies that I can annotate and make available as a shared data set for the vision community to work on.

I want to provide (through commercial efforts) the ability to collect low-level sensor data of home activity so that you can collect data in real homes.


Aaron Crandall

Washington State University

D.J. Cook, M. Schmitter-Edgecombe, Chad Sanders, Brian

Thomas

Collecting and Disseminating Smart Home Sensor Data in the CASAS Project

D.J Cook, M. Schmitter-Edgecombe, Aaron Crandall, Chad Sanders and Brian Thomas

[email protected]

CASAS Testbed

• Comprehensive

• Agent-oriented

• Both office and living spaces

• Scripted and unscripted data

• Focused on ADL detection

The Space & Sensors

• Describing the physical space:

• Implications on resident behavior

• Issues with changes

• The sensors:

• Location

• Relationships

• Implications

• Configurations

• Versions

The Data Fields and Format When collected, the CASAS data is very simple:

Annotation & ADLs

• Correct annotation is still a limiting factor

• Detail of annotation drives cost of effort and accuracy

• Proper notation of both correct activity completion and activity errors

Final Core Issues

• Ensuring clean data

• Annotation accuracy & length

• Generating sufficiently varied data

• Properly describing test bed configurations

WSU Smart Home Dataset Available Now

Thank you

Contact info:

Aaron S. Crandall

[email protected]

Diane J. Cook

[email protected]

Shared Datasets:

http://www.ailab.wsu.edu/casas/datasets.html

mailto:[email protected]

mailto:[email protected]

Lorcan [email protected]

Lero – The Irish Software Engineering Research Centre

University of Limerick

Juan Ye, Susan McKeever, Stephen Knox, Matthew Stabeler, Simon

Dobson, and Paddy NixonUniversity College Dublin

Research questions

we are interested in activity recognition Bayesian networks & lattice theory,

Dempster Shafer evidence theory, case-based reasoning

more realistic and/or more crisp datasets for evaluations

we are also gathering our own datasets based on best principles? - CASL (also we have some “toy datasets”)

McKeever et al., Pervasive LBR 2008

Stabeler et al., Pervasive LBR 2008

Knox et al., RIA 2008

Ye et al., RIA 2008

Ye et al., ICPS 2008

Ye et al., Percom 2009

Lorcan Coyle, Lero@UL


there should be a web-based repository like the UCI ML repository

we need a common language for datasets and parsers to allow interoperability

algorithms should be released! like Weka or in Weka?

results need to be published beyond the paper! put results up with the datasets tag datasets with 3rd party opinions and cite the paper

where the results are presented

ultimately we need to make it transparent for reviewers/scientists to understand a “good result”


Fernando De la TorreJessica HodginsJavier MontanoSergio Valverde

Carnegie Mellon University

http://kitchen.cs.cmu.edu/

Fernando, Carnegie Mellon University

Research questions

How to build good computation models to characterize subtle human motion?

Develop machine learning algorithms for activity recognition and temporal segmentation (supervised/unsupervised) of human motion

Judgments about the quality of motion

How to select or fuse multimodal data for activity recognition?

What should be a good protocol for multimodal data capturing?



Shared datasets: 45 people cooking 5 different recipes (brownies, salad, pizza,

sandwich, eggs) Each recipe is about 22 minutes and 5 synchronized modalities

are recorded (audio, video, motion capture, inertial measurement units)

Anomalous situations (falling, fire, mistaken putting soap rather than salt, …)

Camera calibration parameters, time stamps for each modality Shared labels for object recognition, temporal segmentation

and activity recognition

Shared code: Multimodal data visualization toolbox (Matlab). Baseline experiments for activity recognition and temporal

segmentation. Aligned Cluster Analysis: Clustering of time series.

James FogartyAssistant Professor

Computer Science & Engineering

Research questions

Attacking human-computer interaction problems using statistical machine learning

Previously with a significant focus on sensing Sensor-based human interruptibility models Privacy-sensitive approach to collecting

sensed data in location-based applications Unobtrusive home activity sensing

(collaborations pulling me back into this)

More recently focused on domains where it is actually possible to attack the entire problem End-user interactive concept learning

(with application in Web image search) Mixed-initiative information extraction

(with application to semantifying Wikipedia)

James Fogarty, University of Washington


Convincingly answering compelling HCI questions typically requires some custom data collection (either formative or summative data)

Those datasets are expensive and difficult to collect

We therefore look for the minimal collection to answer our question

Rendering the collected data largely useless for other questions

Data sharing can have important value, but we also need to examine other approaches to achieving the same intended benefits

Work on different problems (like the Web, where there’s lots of data!)

Improved coordination of collection (work with others to reduce costs)

Improved standardization of collection (agree what’s important to collect)

Improved collection tools (lower barrier to getting it in the first place)

Improved annotation tools (lower barrier to coding it later)


Stephen Intille

Massachusetts Institute of Technology

Research questions

How can just-in-time information presented by context-aware technology in the home and worn on the body help people stay healthy as they age?

How do we make activity detection algorithms that work for non-techies in real life in complex situations using practical and affordable sensor infrastructures?

End-user concerns/challenges that have not been adequately addressed…* Practical sensor installation* Maintaining sensors* Fixing mistakes * Adding activities

Toothbrushing

Stephen Intille (MIT)



What shared datasets or tools, if any, would best advance your work (on automatic detection of activity for health systems) ?

Datasets of 10-100 families in their homes doing everyday activities for months with accurate labeling of activity, postures, and audio transcription and synchronized with data from 3-axis accelerometers on each limb, object usage data on as many objects as possible, current flow sensing on electrical devices, and indoor position information on each occupant (1m accuracy).

Datasets of 10-100 people doing everyday activities in natural settings for weeks or months with accurate labeling of type and intensity (energy expenditure) of physical activity while wearing 3-axis accelerometers on each limb.

Taketoshi MORI

Mechano-Informatics, The University of Tokyo

Masamichi Shimosaka,Akinori Fujii,

Kana Oshima,Ryo Urushibata,Tomomasa Sato,

Hajime Kubo,Hiroshi Noguchi

CHI 2009 WorkshopDeveloping Shared Home Behavior Datasets to Advance HCI and Ubiquitous Computing Research

Sensing Room and Its Resident Behavior Mining

Research questions

We have constructed several room-type human behavior sensing environments. These used many distributed sensors. The key was location sense. The problems were A long-term recording is difficult,

Time synchronization is difficult,

Annotating is such a bother!

Based on the collected behavior data, we have been constructing services such as action anticipation, beat-one information display and robotic support.

Taketoshi Mori, the University of Tokyo

We introduced for these problems, Multi-layered network system,

Distributed object software scheme,

RDF/OWL knowledge representations.



Developing algorithms to detect unusual behavioral phenomenon or to foresee stereotyped frequently occurring behaviors for supporting human, it is necessary to obtain human’s position in the space with timestamp. Distributed sensors should supply sufficient information to estimated the human position. It may help if the timestamp is marked both at the sensed time by sensors and the recorded time by the home server.



The datasets with many additional data, such as the resident’s profile, 3D room models, the wall and floor textures, the weather and temperature help to construct an appropriate behavior estimation method.

The datasets should be constructed based on some tagged formats as XML or YAML, and preferably the tags are added following RDF.

Tim van Kasteren

Intelligent Systems Lab Amsterdam

University of AmsterdamCo-author: Ben Kröse

Research questions

Which probabilistic model is best for modeling human behavior?

How to deal with unsegmented data?

How to capture long term dependencies?

How to deal with the large number of ways in which activities can be performed?

How can we apply these models on a large scale, without the necessity of training data from each house they are applied?

How to deal with different layout of houses?

How to deal with different behavior of people?

Tim van Kasteren (University of Amsterdam)


To validate the effectiveness of our models, we need:

Datasets consisting of several days (weeks) of data recorded in a real world setting.

We have mainly used wireless sensor networks, but we are interested in validating our models on other sensing modalities as well.

To validate the application of our models on a large scale, we need:

Datasets from multiple houses.

Ideally consisting of a fixed set of sensors and labeled activities.

We offer:

Several real world datasets consisting of at least two weeks of fully labeled data each.


Sumi HelalUniversity of Florida, Andres

Mendes-Vazquez, Diane Cook and Shantonu Hussein

www.icta.ufl.edu

Research questions

How can we synthesize sensory datasets either from scratch or by “stem-celling” existing actual datasets?

Synthesis is necessary to enable researchers with limited resources but with great ideas and algorithms that need to be thoroughly tested.

Synthesis could also be needed by the owner of an actual dataset, to enable/him/her to go back in time and explore additional concerns/goals not thought of during data collection.

What are the synthesis strategies/algorithms?

How good are synthesized datasets? How can we assess our success or failure in this direction.

What does “Sensory Dataset Description Language” standard has to do with data synthesis?

Sumi Helal, University of Florida


Simply, access to a database of well documented datasets will advance our research and tool development in sensory data synthesis. What is of great importance to us is documentation of the

“protocol” used to collect the data, not just the data itself.

To be able to utilize other people datasets, and to foster a greater level of interoperability and cross use of data sets, we have been working on defining a standard to propose to the community. We call the standard: “Sensory Dataset Description Language” or SDDL. The SDDL specification proposal can be downloaded from:

http://www.icta.ufl.edu/persim/sddl/

We have utilized 4 datasets in defining this standard proposal. We wish to consider many more datasets in refining this proposal. Your comments AND contributions to SDDL are sought.

Sumi Helal, University of Florida

Allen Yangwith Phil Kuryloski and Ruzena

Bajcsy

UC Berkeley

DexterNet: A Wearable Body Sensor System

Primary Goals1. Real-time control & sampling of

heterogeneous body sensors

2. Secured surveillance in indoors and outdoors

3. Provides geographical and social data

System Architecture1. Body Sensor Layer (BSL)

2. Personal Network Layer (PNL)

3. Global Network Layer (GNL)

Prototype Systems1. Human action recognition

2. State-of-the-art security features

3. Real-time communication between Berkeley and Vanderbilt Hospital tested

Allen Yang, UC Berkeley

Reference: BSN Workshop, 2009.

Wearable Action Recognition Database (WARD), version 1

Free for noncommercial users

5 motion sensors, each carries an accelerometer and gyroscope sampled at 30 Hz

20 test subjects (13 male & 7 female) ages 19-75

13 action categories collected in an indoor lab

Data processed in Matlab. Visualization tool is included


standing sitting sleeping

walking running jumping

turning upstairs/downstairs

pushing objects

9:00 Overview and goals

9:15 Introductions by attendees

10:30 Break

10:45 Targeted questions and answers

12:00 State-of-the-art in data collection

12:30 Lunch

14:00 Discussion: What's possible?

14:20 Group exercise

15:20 Group presentations

16:00 Break

16:15 Next steps

17:30 End of workshop

Workshop schedule

Gregory D. AbowdGeorgia Tech

Question & Answers #1

Why do you want family home movies? Sufficient retrospective research in the autism domain has

shown that there is evidence of developmental delay in home movies. This has value for early detection and early intervention.

We have shown that you can encourage the collection of relevant developmental milestone behavior from parents, but not of rich evidence like video.

We are working on filtering techniques to pull out the relevant snippets of social interaction.

Ultimately, I envision a way to upload home movies to a secured service that can then extract relevant portions to share with a pediatrician or other professional for screening purposes.



What is the value of infrastructure mediated sensing to other researchers?

This is a way to gather low-level sensing data from real homes.

There is both commercial and research opportunities here and I think the commercial opportunities in demand-side energy management may be able to drive the ability to provide valuable resources for researchers to leverage.


Fabio PianesiMassimo Zancanaro

FBK-irstAlessandro Cappelletti, Bruno

Lepri, Nadia Mana


How would low-bandwidth sensing (e.g. passive infrared motion detection, object movement sensors, RFID) complement the methods used in the NETCARITY project

Attention mechanism to that activates/deactivates camera/mikes when someone enters a room

Fusion of multiple modalities Recognition of objects (manipulation) Information about body (and body segments) activity levels,

posture changes, etc.



How could the data on target behaviors in NETCARITY be used to improve segmentation of activities in recordings of ongoing natural behavior?

Segmentation is a ill-posed problem because it confuses two level: the description of an activity and the intention of the performer

Example: While I cook spaghetti, I go to the restroom. A friend call and I say “I’m

cooking” (still in the restroom!) I grab an hammer and my wife asks me about what I’m doing: “I’m

hanging the painting” but I have not yet started (or not?)

Telic events have a clear end but still lack a clear start If the apple is finished, you have ate an apple But it’s hard to agree on the start (or if you leave the apple on the

table)



How might recordings from the high density microphone arrays used in this project provide value to other researchers? Would this justify the cost?

For what concerns event detection, we had disappointing results from microphone

They can be useful to monitor verbal and para-verbal activities to estimates: personality traits (Pianesi et al. 2008; Lepri et al. 2009) mood


Aaron Crandall

Washington State University

D.J. Cook, M. Schmitter-Edgecombe, Chad Sanders, Brian

Thomas

Lorcan [email protected]

Lero – The Irish Software Engineering Research Centre

University of Limerick

Juan Ye, Susan McKeever, Stephen Knox, Matthew Stabeler, Simon

Dobson, and Paddy NixonUniversity College Dublin

Combining Redundant Data

“In the CASL Dataset, how might overlapping data from Ubisense locator, pressure mats, and Bluetooth spotters be used to good advantage?”

tells us a lot about the data quality reveals when sensors are not operating optimally

allows us to make more educated guesses

we can test with subsets

the sensors aren’t where the cost is (imho)

without redundant data streams there are certain algorithms we cannot test voting/weighting strategies? it’s easier to play with DS evidence theory


Bootstrapping Users to a Dataset

“Describe the concept of “bootstrapping” datasets for new users and discuss how this might be done efficiently for large long-term datasets”

release parsers/interfaces to deal with your dataset

how about sample experiments?

really simple worked-through tutorial examples subsets of the dataset

e.g., using only RFID and object sensors: 10:12pm: Prof. Plum enters kitchen 10:14pm: candlestick sensor active 10:16pm: Prof Plum enters hallway

reducing the learning curve


Fernando De la TorreJessica HodginsJavier MontanoSergio Valverde

Carnegie Mellon University


How might body motion capture be most practically implemented in a natural home environment?

Wearable: Small wireless Inertial Measurement Units distributed

through the body.

Instrumented environment: Sparse information: motion sensors around the house. Rich information: multiple cameras (at least 3)




Given the choice between high resolution (1024x768, 30fps) or high frame rate (640x480, 60fps) video, which do you think would be more beneficial to the greatest number of researchers?

It depends on the task Subtle activity recognition (e.g. grasping a fork) or object

recognition probably is better higher resolution Egomotion computation from wearable camera or fast

activities such as cutting a cucumber, probably better higher frame-rate

James FogartyAssistant Professor

Computer Science & Engineering


Discuss why you believe it is or is not possible to collect general-purpose shared datasets on home behavior.

Very simple to collect shared datasets on home behavior(see the website for this workshop, we’ve already succeeded!)

The notion that a dataset is general purpose is what makes it difficult

Data collection is hard enough and expensive enough when focused on answering your own research questions

Asking the question also implies that recognition is the goal

Is that what we’re doing here in the HCI community?

Maybe we should be looking for the HCI contributions we can make without solving the hard general activity recognition problem



Describe one way researchers might solicit and distill community input before undertaking a data collection project.

Focus on researcher awareness of the benefits they personally might obtain from collecting data, then encourage them to share

Tag existing datasets by the types of data they contain

When designing a new data collection, a researcher can easily see what kinds of data other people have previously collected in tandem with what you are already planning to collect

Can also see why they collected it, imagine what additional benefit its collection would have to your current research

Try to identify a way to solicit wishlists or shortcomings of datasets

Make it really easy to search, link to related tools, etc.


Stephen Intille

Massachusetts Institute of Technology


Do you think low cost, off-body sensors be used to detect postural transitions or physical activity with any useful degree of accuracy?

Best hope: computer vision (technically and socially tough)

Interesting question: how close can you get when other sensors are ubiquitous?



How are you addressing the logistical challenges associated with mobile computing research (e.g., user compliance, comfort, battery life)?

- One night, one recharge- Same thing every day - Phone prompting if no compliance- Looking for apps to inspire compliance - Leave stuff by the door (retrain user) - Sensors: small enough to wear under clothing

- Tricky IRB/social issue: what to do outside home


Taketoshi MORI

Mechano-Informatics, The University of Tokyo

Masamichi Shimosaka,Akinori Fujii,

Kana Oshima,Ryo Urushibata,Tomomasa Sato,

Hajime Kubo,Hiroshi Noguchi

CHI 2009 WorkshopDeveloping Shared Home Behavior Datasets to Advance HCI and Ubiquitous Computing Research



What challenges would you anticipate for installing magnetic motion capture in natural homes? What alternative strategies for capturing bodily motion would you consider?



2D/3D stationary laser range sensors may be used to measure human position and pose. Appliances usage tells a lot about home behaviors. Some people may wear wrist-watch type accelerometer with gyros.

Magnetic motion capture systems work poorly when there exist many metallic things. Also, there may be troublesome cables between magnetic sensors distributed on human body and the controller. We do not expect the magnetic capture systems as the usual behavior collecting way, but it may be used to prepare ground truth motion/posi-tion since other mo-caps such as optical/super-sonic-based are greatly influenced by occlusions.


Describe your schema for annotating behaviors in Sensing Room. What were strengths and weaknesses of the annotation procedure?

Sensing Room accumulates the resident’s place by its floor distributed pressure sensors, object carrying actions by RFID readers, and many other acts by electric switches or electric current sensors. All the data are put together and can be displayed as 3D-CG images. Watching the CG video, several researchers write down the behavior annotation by hand.



Our procedure has advantage that it can be done offline. No camera surveillance is required. But, of course, it has the weakness that the correctness depends both on the lucidness of the graphics and the interpretation of the annotators.

Tim van Kasteren

Intelligent Systems Lab Amsterdam

University of AmsterdamCo-author: Ben Kröse


Q: How might you change the real-time voice-activated annotation procedure to reduce the burden on the user?

A: The current system requires the user to constantly be aware of the activity he/she is involved in and report this.

If the system would ask the user what activity is being performed this would reduce the burden.

The system could ask the user at times when sensor patterns are most ambiguous with respect to the activities annotated.

However, the question remains how this effects the reliability of the annotation method.



Q: Describe how the activities to be annotated were selected. How and why would you change this list in future data collection?

A: Activities were selected based on previous work, literature on activities of daily living (ADLs) and based on what would seem challenging yet feasible using the sensor platform used.

In future data collection more detailed activities would be annotated. For example, annotating getting tea and getting juice, instead of getting a drink.

More detailed activities can always be grouped into a collective activity afterwards, but extend the lifetime of a dataset.

However, there is always a trade-off between cost and gain.


Allen Yangwith Phil Kuryloski and Ruzena

Bajcsy

UC Berkeley


Advantages of wearable sensors over environmental sensors?1. Cost less to instrument, especially in outdoors

2. Richer interaction with subjects (physiological sensors, message feedback)

Integration of wearable and environmental sensors?1. Certain environmental sensors are not portable (size and battery)

2. Wearable sensors can provide localization services, which then correlate with environmental information.


Airborne Particulate Matter Concentrations


Plan for future databases?1. Human Motion Interaction: WARD version 2.

2. Integrating Geographic Data (with Oakland Children’s Hospital): Long-term monitoring of 160 obese patients correlated with environmental factors.

3. Integrating Social Interaction: Port DexterNet platform to consumer-ready smart phones (iPhone and gPhone).


Documents

Fabio Pianesi Massimo Zancanaro