Upload
luciano-oliveira
View
837
Download
1
Tags:
Embed Size (px)
Citation preview
Semantically Integrating Laser and Vision in Pedestrian Detection
Luciano Oliveira
Advisors:Prof. Urbano NunesProf. Paulo Peixoto
Motivation
SegmentationRecognition
TrackingSearching
Clustering methods
Kalman Filter
Efficient sub-window searching
(image)
Where is the pedestrianin the scene?
Goals
Object detectionusing laser/vision
Proof-of-concept: pedestrian detection, but can be applied to several other objects
Recover objectlocalization
DO NOT entirelyrely on laser, as previous methods do
Perform the fusion in a context-aware mode
Overview of the proposed method
Coarsesegmentation
Finesegmentation
Sensor registration
Parts-based ensemble detector
Semantic/contextualinterpretation
3D sliding windowsearching
Inference and decision outputs
Procrustesanalysis
Referenceshapes
Laser points
For each2D window
(label, confidence) for each fm
Images
For each3D window
(object, confidence)
Templating matching
Laser-image registration
Laser segmentation and labeling
MLN
HLSM-FINT
Ground MRF
{ }Nnnc 1=
{ }Mmmf 1=
for each cn
Experimental setupPointgrey camera
Sick LMS200 laser
Odometry
Sensor-driven detectors
Coarsesegmentation
Finesegmentation
Sensor registration
Parts-based ensemble detector
Semantic/contextualinterpretation
3D sliding windowsearching
Inference and decision outputs
Procrustesanalysis
Referenceshapes
Laser points
For each2D window
(label, confidence) for each fm
Images
For each3D window
(object, confidence)
Templating matching
Laser-image registration
Laser segmentation and labeling
MLN
HLSM-FINT
Ground MRF
{ }Nnnc 1=
{ }Mmmf 1=
for each cn
Ensemble of classifiers HFI
Perimeter rate
Distance / max(w,w´)
Fuzzy System
C1 scaled score
C2 scaled score
FuzzySystem
Join
t co
nfid
ence
Inte
rsec
tion
rate
Fina
l con
fiden
ceFuzzy
System
Hierarchical Fuzzy IntegrationFuzzy inputs
C2
C1
It suffers from exponential growing of rules and low overall performance over challenging situations
Drawbacks
Initially evaluated on Haar-like features / Adaboost and HOG / SVM classificationsystems
Ensemble of classifiers HLSM-FINT
HLSM-FINT – Rationale
• CNN – expert in background (BG) (60% of hit rate in NiSIS competition)• HOG/SVM – expert in objects (OB) (70% of hit rate in NiSIS competition)• Fuzzy integral (Sugeno) – providesa comprehensive framework andgreat synergism• 95.67% of hit rate in NiSIScompetition over 6125 croppedimages (ped + non-ped), usingHeuristic Majority Vote method• 96.4% of hit rate over fullDaimlerChrysler datasets : ~15.000 images
BG
BG BG
BG OB
OB
Parts-based HLSM-FINT
Upper HLSM-FINT (shoulder + header)
Lower HLSM-FINT (waist)
Laser detector
Laser detector
arm armtorso
armtorso
torsoarm armpartial
segment
• Featureless approach
• Coarse-to-fine segmentation
• Relative Neighboorhood Graph (RNG) clustering + clustering index
• Procrustes Analysis (PA) labeling procedure
Laser detector
Laser detector
Occlusion problem:
• z-buffer analysis +
• angle between start and endpoint (proportional to laser angle resolution)
Laser-image registration
Coarsesegmentation
Finesegmentation
Sensor registration
Parts-based ensemble detector
Semantic/contextualinterpretation
3D sliding windowsearching
Inference and decision outputs
Procrustesanalysis
Referenceshapes
Laser points
For each2D window
(label, confidence) for each fm
Images
For each3D window
(object, confidence)
Templating matching
Laser-image registration
Laser segmentation and labeling
MLN
HLSM-FINT
Ground MRF
{ }Nnnc 1=
{ }Mmmf 1=
for each cn
Laser-image registration
Zhang and Pless’ calibration method(with an error of 6 mm in the calibration)
Semantic Fusion
Coarsesegmentation
Finesegmentation
Sensor registration
Parts-based ensemble detector
Semantic/contextualinterpretation
3D sliding windowsearching
Inference and decision outputs
Procrustesanalysis
Referenceshapes
Laser points
For each2D window
(label, confidence) for each fm
Images
For each3D window
(object, confidence)
Templating matching
Laser-image registration
Laser segmentation and labeling
MLN
HLSM-FINT
Ground MRF
{ }Nnnc 1=
{ }Mmmf 1=
for each cn
Semantic fusion
Semantic fusion
MRF
Wi
• MRFs given by FOL formulas• Weights given by the MRF training (gradient ascent method over theconditonal log-likelihood)
Semantic fusion – Examples
ConclusionsHFI has achieved better performance than its components, but failedto get the gist of the fusionHLSM-FINT has succeeded to capture the aimed synergism of thefusion, but has had difficulties on hard situations (e.g. occlusion). Parts-based occlusion has improved this issue.The introduction of the laser sensor has brought significantimprovementThe proposed fusion method offers two main advantages:
Contextual and spatial relationship among the parts of theobject, dropping the false alarm rateIt is able to detect the object in spite of laser failing
The whole system is not able to run on-the-fly, although there is no code optimization. Nevertheless, parallel hardware can provideinteresting plataform to make the system faster. It will be subject offuture research.
Publications and awardsJournals
OLIVEIRA, L.; NUNES, U.; PEIXOTO, P.; SILVA, M. and MOITA, F. SemanticFusion of Laser and Vision in Pedestrian Detection, Journal of PatternRecognition, Elsevier, accepted for publication (ISI impact factor: 3.279).OLIVEIRA, L.; NUNES, U. and PEIXOTO, P. On Exploration of ClassifierEnsemble Synergism in Pedestrian Detection, IEEE Transactions onIntelligent Transportation Systems, pp. 16-21, 2010 (ISI impact factor: 2.844).
Awards3rd place in Intel/GV Entrepreneurship and Venture Capital Competition(2008)1st place in NiSIS Competition - Best accuracy model over Daimler Chrysler image dataset. Scheme of Primate's Visual Cortex Cells for Pedestrian Recognition (2007)
5 international conferences