Vehicle Segmentation and Tracking From a Vehicle Segmentation and Tracking From a Low-Angle Off-Axis CameraLow-Angle Off-Axis Camera

Vehicle Segmentation and Tracking From a Vehicle Segmentation and Tracking From a Low-Angle Off-Axis CameraLow-Angle Off-Axis Camera

Neeraj K. KanhereNeeraj K. Kanhere

Committee members

Dr. Stanley BirchfieldDr. Robert SchalkoffDr. Wayne Sarasua

Clemson UniversityClemson University

July 14th 2005

Why detect and track vehicles ?Why detect and track vehicles ? Intelligent Transportation Systems (ITS)Intelligent Transportation Systems (ITS) Data collection for transportation engineering applicationsData collection for transportation engineering applications Because it's a challanging problem!Because it's a challanging problem!

Vehicle TrackingVehicle Tracking

Loop detectorsLoop detectors Tracking using VisionTracking using Vision

Low per unit cost Field experience

No traffic disruption Wide area detection Rich in information

No tracking Maintenance difficult

Computationally demanding

Expensive

Available Commercial SystemsAvailable Commercial Systems

AUTOSCOPE (Image Sensing Systems)AUTOSCOPE (Image Sensing Systems)

Has been around for more than a decadeHas been around for more than a decadeDedicated HardwareDedicated HardwareReliable operationReliable operationGood accuracy with favorable camera Good accuracy with favorable camera placementplacement

VANTAGE (Iteris)VANTAGE (Iteris)

New in marketNew in marketAccuracy has been found to be lower than AutoscopeAccuracy has been found to be lower than Autoscope

Related ResearchRelated Research

Region/Contour BasedRegion/Contour BasedComputationally efficientComputationally efficientGood results when vehicles are well separatedGood results when vehicles are well separated

3D Model Based3D Model BasedLarge number of models needed for different vehicle Large number of models needed for different vehicle typestypesLimited experimental resultsLimited experimental results

Markov Random FieldMarkov Random FieldGood results on low angle sequencesGood results on low angle sequencesAccuracy drops by 50% when sequence is processed Accuracy drops by 50% when sequence is processed in true orderin true order

Feature Tracking BasedFeature Tracking BasedHandles partial occlusions Handles partial occlusions Good accuracy for free flowing as well as Good accuracy for free flowing as well as congested traffic conditionscongested traffic conditions

Factors To be ConsideredFactors To be ConsideredHigh angleHigh angle Low angleLow angle

Planar motion assumption

Well-separated vehicles

Relatively easy

More depth variation

Occlusions

A difficult problem

Overview of the ApproachOverview of the Approach

Offline CalibrationBackground model

Frame-Block #1

Frame-Block #3

Frame-Block #2 Feature Tracking

Estimation of 3-D LocationEstimation of 3-D Location

GroupingGrouping

segmented #1 segmented #2 segmented #3

Counts,Counts,Speeds and Speeds and

ClassificationClassification

Counts,Counts,Speeds and Speeds and

ClassificationClassification

Block Correspondence Block Correspondence and Post Processingand Post Processing

Processing a Frame-BlockProcessing a Frame-Block

Multiple frames are needed for motion information Tradeoff between number of features and amount of motion Typically 5-15 frames yield good results

Block # nBlock # n+1

frames

#features in

block

#features in

block#frames in

block#frames in

block

Overlap

Background ModelBackground Model

Time Domain Median filteringTime Domain Median filtering

For each pixel, values observed over timeFor each pixel, values observed over time

Median value among observationsMedian value among observations

Simple and effective for the sequences consideredSimple and effective for the sequences considered

Adaptive algorithm required for long term modelingAdaptive algorithm required for long term modeling

Frame DifferencingFrame Differencing

Partially occluded vehicles appear as single blobPartially occluded vehicles appear as single blob

Effectively segments well-separated vehiclesEffectively segments well-separated vehicles

Goal is to get filled connected componentsGoal is to get filled connected components

`

Offline CalibrationOffline Calibration

Required for estimation of world coordinatesRequired for estimation of world coordinates

Provides geometric information about the sceneProvides geometric information about the scene

Involves estimating 11 unknown parameters Involves estimating 11 unknown parameters

Needs atleast six world-image correspondancesNeeds atleast six world-image correspondances

Control pointsControl points

Calibration ProcessCalibration Process

Using scene features to estimate correspondences Using scene features to estimate correspondences

Standard lane width (e.g. 12 feet on an Interstate)Standard lane width (e.g. 12 feet on an Interstate)

Vehicle class dimensions (truck length of 70 feet)Vehicle class dimensions (truck length of 70 feet)

Relies on human judgment and prone to errorsRelies on human judgment and prone to errors

Approximate calibration is good enoughApproximate calibration is good enough

Estimation using Single FrameEstimation using Single Frame

Box-model for vehiclesBox-model for vehicles

Road projection using Road projection using

foreground maskforeground mask

Works for orthogonal surfacesWorks for orthogonal surfaces

camera vehicle

Road plane

Selecting Stable FeaturesSelecting Stable Features

Shadows, partial occlusions will result into wrong estimatesShadows, partial occlusions will result into wrong estimates

Planar motion assumption is violated more for features higher upPlanar motion assumption is violated more for features higher up

Select stable features, which are closer to roadSelect stable features, which are closer to road

Use stable features to re-estimate world coordinates of other featuresUse stable features to re-estimate world coordinates of other features

Estimation Using MotionEstimation Using Motion

➢ Estimate coordinates with respect to each stable featureEstimate coordinates with respect to each stable feature

➢ Choose coordinates which minimized weighted sum of euclidean Choose coordinates which minimized weighted sum of euclidean

distance and trajectory errordistance and trajectory error

Rigid body under translationRigid body under translation

Estimate coordinates with respect to each stable featureEstimate coordinates with respect to each stable feature

Select the coordinates minimizing weighted sum of Euclidean Select the coordinates minimizing weighted sum of Euclidean distance and trajectory errordistance and trajectory error

• Coordinates of P are unknown• Coordinates of Q are known• R and H denote backprojections • 0 : first frame of the block• t : last frame of the block• Δ: Translation of corresponding point

Affinity MatrixAffinity Matrix

Each element represents the similarity between corresponding featuresEach element represents the similarity between corresponding features

Three quantities contribute to the affinity matrixThree quantities contribute to the affinity matrix

Euclidean distance (AEuclidean distance (ADD), Trajectory Error (A), Trajectory Error (AEE) and Background- ) and Background-

Content (AContent (ABB))

Normalized Cut is used for segmentation

Number of Cuts is not known

Incremental CutsIncremental Cuts

We apply normalized cut to initial We apply normalized cut to initial AA with increasing number of cuts with increasing number of cuts

For each successive cut, segmented groups are analyzed till valid For each successive cut, segmented groups are analyzed till valid

groups are foundgroups are found

Valid Group: meeting dimensional criteriaValid Group: meeting dimensional criteria

Elements corresponding to valid groups are removed from Elements corresponding to valid groups are removed from A A and and

process repeated starting from single cutprocess repeated starting from single cut

Avoids specifying a threshold for the number of cuts Avoids specifying a threshold for the number of cuts

Correspondence Over BlocksCorrespondence Over Blocks

Formulated as a problem of finding maximum wieght graphFormulated as a problem of finding maximum wieght graph

Nodes represent segmented groupsNodes represent segmented groups

Edge weights represent number features common over two blocksEdge weights represent number features common over two blocks

an: groups in block N

bn: groups in block N+1

ResultsResults

ResultsResults

ConclusionConclusion

A novel approach based on feature point trackingA novel approach based on feature point tracking

Key part of the technique is estimation of 3-D world Key part of the technique is estimation of 3-D world

coordinates coordinates

Results demonstrate the ability to correctly segment vehicles Results demonstrate the ability to correctly segment vehicles

under severe partial occlusionsunder severe partial occlusions

Handling shadows explicitly Handling shadows explicitly

Improving processing speedImproving processing speed

Robust block-correspondance Robust block-correspondance

Future Work

Questions ?Questions ?

Thank You ! Thank You !