❖

PEM-ID: Identifying People by Gait-Matching using

Cameras and Wearable Accelerometers

Thiago Teixeira, Deokwoo Jung, Gershon Dublon, Andreas Savvides

Yale ENALAB

Thiago Teixeira Yale ENALAB - http://www.eng.yale.edu/enalab 2

Introduction

Can we uniquely identify people in camera networks?(in cooperative enviroments)

Motivation: Assisted Living

identify people in a home Security

locate personnel Corporate environments

track facility usage

Plus, obtaining data traces for research: Yale BehaviorScope project

Thiago Teixeira Yale ENALAB - http://www.eng.yale.edu/enalab 3

Main Idea

Equip each person of interest with a wearable accelerometer node (with known ID)

Extract “motion signature” from: each accelerometer unique ID each track Position

Find pairs of matching signatures to obtain ID+Position

Thiago Teixeira Yale ENALAB - http://www.eng.yale.edu/enalab 4

Problem Statement

Given: a set {SAi} of accelerometer signals and a set

{SCj} of tracks extracted from a camera network

Find: the match matrix Λ which globally maximizes the similarity between pairs of signals SA

i and SCj

Main assumptions: Tracker: provides correct tracks in segments ≳ 4 steps Camera placement: oblique from top (typical CCTV) Occlusions: short-lived

0 1

1 0

0 0

Λ =

tracks

accelerometers

Thiago Teixeira Yale ENALAB - http://www.eng.yale.edu/enalab 5

Challenge: motion signature

Motion paths can be subdivided into two types: Transition motion

Starting, stopping, turning, changing speed Large changes in tangential and centripetal acceleration

Cruising motion Approximately same-speed linear motion Only small-scale changes in acceleration Gait Comprises majority of time

Intuition: to ID people most of the time, use gait Challenge: Nodes are not time-synchronized, have

limited processors and low bandwith

Thiago Teixeira Yale ENALAB - http://www.eng.yale.edu/enalab 6

Correlating Gait Signals from Asynchronous Sources

Sample-oriented methods are unsuitable for WSNs:(eg. Pearson's corr. coefficient, mutual information) Fail given time synchronization offsets (or must slide

one of the signals and recalculate) Require a large number of samples to converge Requires resampling/interpolation if signals have

different sampling frequencies and/or phases

We can do better, using gait frequency and phase…

Thiago Teixeira Yale ENALAB - http://www.eng.yale.edu/enalab 7

Timestamps of Gait Landmarks

Idea: Compare timestamps of heel-strike and midswing moments of gait: H = (tH

0, tH1, … )

M = (tM0, tM

1, … ) From accels., and cameras:

SAi = {HA

i, MAi}

SCj = {HC

j, MCj}

Next step: define time-noise independent metric (offset and jitter)

Thiago Teixeira Yale ENALAB - http://www.eng.yale.edu/enalab 8

Distance metric

Define distance from timestamp to sequence:

Then from sequence to sequence:

Then two metrics describing time offset and jitter:

Thiago Teixeira Yale ENALAB - http://www.eng.yale.edu/enalab 9

Global Optimization

Invariance to time offset, timestamp noise

Global Optimization

Thiago Teixeira Yale ENALAB - http://www.eng.yale.edu/enalab 10

Multiple-Person Simulations

We recorded 24 one-person traces: 12× walking straight in different directions 12× walking and turning in different directions

We overlapped multiple single-person traces with random time offsets (up to 1s) to simulate multiple-person scenarios:

Thiago Teixeira Yale ENALAB - http://www.eng.yale.edu/enalab 11

Three-Person Experiments

Three people walking through FOV One person wearing an accelerometer

Average recognition rate: 87.5%

http://enaweb.eng.yale.edu/drupal/PEM-ID-videos

Thiago Teixeira Yale ENALAB - http://www.eng.yale.edu/enalab 12

Conclusion

Presented a method to ID people in videos using accelerometers

Accuracy > 83%, for up to 10 people + 10 accels

Currently adapting for indoor use Much smaller FOV multiple cameras Occlusions use additional features

❖

Thank you.

Questions?

BehaviorScope:http://www.eng.yale.edu/enalab/behaviorscope.htm

Videos:http://enaweb.eng.yale.edu/drupal/PEM-ID-videos