ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

ISR – Institute of Systems and RoboticsUniversity of Coimbra - Portugal

WP5: Behavior Learning And Recognition

Structure For Multi-modal Fusion

Part I


Relationship of the WP3,4 and 5

WP3(Sensor modeling and multi-sensor fusion techniques )

Task 3.3Bayesian network structures

for multi-modal fusion

WP4(Localization and tracking techniques as applied to humans )

Task 4.3Online adaptation and learning

WP5Behavior learning and recognition

• Trackers results, • Events detected , • ids on re-identification situations

•Levels of the fusion(pixel, feature or decision level) ,•Bayesian structures for the implementation of the scenarios of WP2


Proposal

Applying a multi-modal Occupancy Grid and Multi-layer Homography to reconstruct human silhouette with stationary sensors.

(Indeed using Silhouette (Multi-layer Homography), Texture and Range information to build an Occupancy Grid)

Stationary Sensors (Structure):

• Image Data

• Range Data

• Sound Source Data


Inertial Compensated Homography

Projecting a world point on a reference plane in two phases

iI

xiW I

iC

X

Y

Z

iIW

iIW H

wiI

Hrefπ

xiI

xw

XHX

XHX

iIW

wi

refref

i

i

iIW

Iππ

II

W

Real

camera

Virtual camera

'iCi

i

C

C R

[Luiz2007]

Gravity

xrefπ

First step: Projecting real world point on the virtual image plane

Second step: Projecting form virtual image plane on a common

plane

refπ


Inertial Compensated Homography

iI

xiW I

iC

X

Y

Z

iIW

iIW H

wiI

Hrefπ

xiI

xw

333231

232221

131211

1

hhh

hhh

hhh

and

wi

ref

i

i

i

Iπ

CC

IW

H

KRKHReal camera

Virtual camera

'iCi

i

C

C R

[Luiz2007]

Gravity

xrefπ

Infinite homography

Homography between two planes

Camera calibration

matrix

Rotation between virtual and real

camera (given by IMU)

00

).1(10

).1(01

y

x

v

v

wi

ref

Iπ H

refπxrefπ

v

xiW I

iIW

refπ


Image Registration

iI iC

X

Y

Z

i

refI

π H

xiI

xrefπ

xw

[Luiz2007]

Gravity

XHHX i

iwi

refref II

WI

ππ

jC

kC

kI

jIxjI xkI

j

refI

π H k

refI

π H

refref Gπ

refπ


Extending A Virtual Plane To More

Plane to image homography:

Vanishing points for X,Y and Z directions

Vanishing line of reference plane

normalized

[Khan2007]

Vanishing point of reference direction

Scale factor

Scale value encapsulating and z

iI iC

xiI

jC

kC

kI

jIxjI xkI

]|[

]ˆ[

23 refrefref

iW

l

iW

πππI

ZYXπI

v0H

lvvvH

z

refπlπ

X

YZ

Nπ


Relationship Between Different Planes In The Structure

iI

kC

xiI

Homography between views i and j, induced by a plane i parallel to ref

having the homography of reference plane:

])|[1

1])(|[( 33 refπrefπ

πI

IπI

I

ref

refπ

ref

ref

iW

jW

l

iW

jW

v0Iv0HH

[Khan2007]

iC

jC

kI

jIxjI xkI

refπlπ

X

YZ

Nπ

Vanishing point of reference direction

Scale value

Homography of reference plane


Image & Laser Geometric Registration

iI iC

X

Y

Z

xiI

xrefπ

xw

[Luiz2007-Hadi2009]

Gravity

XTX i

i

refref LL

πLπ

jC

kC

kI

jIxjI xkI

jL

Lπ ref x

refref Gπ

iL

refπ


Registering LRF Data In a Multi-Camera Scenario

4..0j| jII

xTHu i

i

LL

W

[Hadi2009]

Reprojection of LRF data on the image (blue points)

Image planes

Projection of points observed by LRF

Transformation matrix between camera and LRF obtained by calibration

Projection of points observed by LRFon the image plane

+Result

A set of cameras and laser range finder

cN...0j| uu jC

cN...0j| jII

1310

tRT ii

i

LW

LW

LW

cN...0j| jHH

Camera projection matrix

Ima

ge

Ra

ng

e d

ata


Image & Laser & Sound Geometric Registration

refref Gπ

iI iC

X

Y

Z

xiI

xrefπ

xw

Gravity

jC

kC

kI

jIxjI xkI

jL

Lπ ref x

1M 2M

PA()

[JFC2008, 2009]

iL

refπ


Bayesian Binaural System for 3D Localisation

–Binaural sensing

•For sources within 2 meters range, binaural cues alone ( interaural time and level interaural time and level differences – ITD differences – ITD , quasi frequency-independent, and ILDs , quasi frequency-independent, and ILDs L(fL(fcc

kk)))) can be used to fully localise the source in 3D space (i.e. volume confined in azimuth , elevation and distance ).

• If the source is more than 2 meters away the source can only be localised to a volume (cone of confusion) in azimuth.

1m 2m

Binaural cue information +

+

Distance

Elevation

Azimuth θ Azimuth θ only

Z



Subset of [JFC2008, 2009]

CSCL(fck)

Z


Direct Auditory Sensor Model: (Direct Auditory Sensor Model: (DASM))(Bayesian learning through HRTF calibration using (Bayesian learning through HRTF calibration using

ITDs ITDs and ILDs and ILDs L)L)

AzimuthAzimuth

ElevationDistance

Binary variable denoting “Cell C occupied by sound-source”

Binary variable denoting “Cell C occupied by sound-source”

Inverse Auditory Sensor Model: (Inverse Auditory Sensor Model: (IASM))


Bayes Rule

Auditory Saliency MapAuditory Saliency Map

Solution: cluster local saliency Solution: cluster local saliency maxima points (i.e. cells with maxima points (i.e. cells with

maximum probability of occupancy, maximum probability of occupancy, 1 per sound-source)1 per sound-source)(front -to-back confusion effect avoided by (front -to-back confusion effect avoided by

considering only frontal hemisphere estimates)considering only frontal hemisphere estimates)


Bayesian Binaural System for Localisation in Azimuth Planes of Arrival

Direct Auditory Sensor Model: (Direct Auditory Sensor Model: (DASM))(Bayesian learning through HRTF calibration of (Bayesian learning through HRTF calibration of

interaural time differences – ITDs interaural time differences – ITDs ))

Inverse Auditory Sensor Model: (Inverse Auditory Sensor Model: (IASM))

-90º 90º

Bayes RuleBayes Rule

Auditory Saliency MapAuditory Saliency Map

Solution: cluster local saliency Solution: cluster local saliency maxima planes of arrival (PA) maxima planes of arrival (PA)

per sound-sourceper sound-source

PA()(front -to-back confusion effect avoided by (front -to-back confusion effect avoided by

considering only frontal hemisphere estimates)considering only frontal hemisphere estimates)


Demos on Bayesian Binaural System

(Arrival Direction of Sound Source)

Two Talking persons A walking person


Image & Laser & Sound Occupancy Grid

refref Gπ Image, Range and Sound

Occupancy Grid

& Fusion

2π

Nπ

X

refπ1π

ZY

2G

NG

X

refG1G

Discretization

YZ


Bibliography


Bibliography

•Franco, J. & Boyer, E. Fusion of Multi-View Silhouette Cues Using a Space Occupancy Grid Proceedings of the Tenth IEEE International Conference on Computer Vision (ICCV’05), 2005

•Christophe Braillon, Kane Usher, C. P. J. L. C. & Laugier, C. Fusion of stereo and optical flow data using occupancy grids Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, 2006.

•Saad M. Khan, P. Y. & Shah, M. A Homographic Framework for the Fusion of Multi-view Silhouettes Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on, 2007

•R. Eshel, Y. M. Homography Based Multiple Camera Detection and Tracking of People in a Dense Crowd Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, 2008

•Conference (Arsic2008) D. Arsic, E. Hristov, N. L. Applying multi layer homography for multi camera person tracking Distributed Smart Cameras, 2008. ICDSC 2008. Second ACM/IEEE International Conference on, 2008

•Francois Fleuret, Jerome Berclaz, R. L. & Fua, P. Multi-Camera People Tracking with a Probabilistic Occupancy Map IEEE transactions on Pattern analysis and Machine Intelligence, 2008


Bibliography•Sangho Park, M. M. T. Understanding human interactions with track and body synergies (TBS) captured from multiple views Computer Vision and Image Understanding, 2008

•Yuxin Jin, Linmi Tao, H. D. R. N. & Xu, G. Background modeling from a free-moving camera by Multi-Layer Homography Algorithm Image Processing, 2008. ICIP 2008. 15th IEEE International Conference on, 2008

•Luiz G. B. Mirisola, Jorge Dias, A. T. d. A. Trajectory Recovery and 3D Mapping from Rotation-Compensated Imagery for an Airship Proceedings of the 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems San Diego, CA, USA, Oct 29 - Nov 2, 2007, 2007

•Mirisola, L. G. B. & Dias, J. Tracking from a Moving Camera with Attitude Estimates ICR08, 2008

•Batista, J. P. Tracking Pedestrians Under Occlusion Using Multiple Cameras Image Analysis and Recognition, Springer Berlin-Heidelberg., 2004, 3212/2004, 552-562

•Joao Filipe Ferreira, Pierre Bessière, K. M. C. P. J. L. C. L. & Dias, J. Bayesian Models for Multimodal Perception of 3D Structure and Motion

•C. Chen, C. Tay, K. M. & C. Laugier (INRIA, F. Dynamic environment modeling with gridmap: a multiple-object tracking application 9th International Conference on Control, Automation, Robotics and Vision, 2006. ICARCV '06., 2006


Bibliography

•J. F. Ferreira, P. Bessière, K. Mekhnacha, J. Lobo, J. Dias, and C. Laugier, “Bayesian Models for Multimodal Perception of 3D Structure and Motion,” in International Conference on Cognitive Systems (CogSys 2008), pp. 103-108, University of Karlsruhe, Karlsruhe, Germany, April 2008.

•C. Pinho, J. F. Ferreira, P. Bessière, and J. Dias, “A Bayesian Binaural System for 3D Sound-Source Localisation,” in International Conference on Cognitive Systems (CogSys 2008), pp 109-114, University of Karlsruhe, Karlsruhe, Germany, April 2008.

•Ferreira, J. F., Pinho, C., and Dias, J., “Implementation and Calibration of a Bayesian Binaural System for 3D Localisation”, in 2008 IEEE International Conference on Robotics and Biomimetics (ROBIO 2008), Bangkok, Tailand, 2009.

•Hadi Aliakbarpour, Pedro Nunez, Jose Prado, Kamrad Khoshhal and Jorge Dias. An Efficient Algorithm for Extrinsic Calibration between a 3D Laser Range Finder and a Stereo Camera for Surveillance, ICAR2009.


Institute of Systems and Robitcs

http://paloma.isr.uc.pt

Documents

ISR – Institute of Systems and Robotics University of Coimbra - Portugal