Distributed decision-making Distributed decision-making and reasoning with uncertain and reasoning with uncertain image and sensor dataimage and sensor data

Pramod K. VarshneyKishan G. Mehrotra

C. Krishna MohanElectrical Engineering and Computer Science Dept.

Syracuse UniversitySyracuse, NY 13244

Phone: (315) 443-4013Email: varshney@syr.edu

Feedback from 2002 Review of MURI Project “Show a stronger connection with

military scenarios”

“Urban warfare is a good scenario to consider”

“Relate theoretical work to military applications”

Our Main Themes Develop theory and algorithms oriented

towards practical military applications

Decision-making with multiple decentralized information streams

Uncertainty computation

Reasoning with uncertain data

Exploitation of models and tools developed by partners

What is the agent’s current location ?

Recognize activities of other agents.

What is the likelihood of damage at various locations ?

What would be the safest paths to a goal/exit zone ?

Main Contributions1. Temporal Bayesian network for target

tracking

2. Outdoor video tracking using multiple cameras

3. Image query-based approach for location determination

4. Distributed personnel movement planning for urban combat zones

5. Automated event/scenario recognition from video sequences

1. Target Tracking1. Target Tracking

-An Application of Temporal Bayesian Networks

Previous Work Defined Temporal Bayesian Networks: time-dep

endent updating of beliefs at nodes based on information obtained at earlier instants.

Formulated linear + exponential temporal decay model.

Implemented the above model in MATLAB. Developed a graphical user interface.

Target Tracking Application Model: The field contains one or more target

s, and is divided into many rectangular cells. Data: Each sensor obtains information from

a distinct subset of cells in its neighborhood, updated over time based on the latest data.

Goal: The fusion center must determine if targets exist in any cells, based on various sensor reports at current and recent time instants.

Causal Network Structure

Procedure: Compute beliefs associated with each lower level node

Oi Temporal belief updating at T using timestamps associ

ated with information from lower level nodes. Decision rules to determine target presence.

E

T

O1

Locationof Target

Observationfrom

Sensor 1

O2 ONO3

Observationfrom

Sensor 2

Observationfrom

Sensor 3

Observationfrom

Sensor N

Decision Rules Any cell L satisfying the following two

conditions is considered as a target cell:

Belief (target in cell L) > Threshold? Belief (target in cell L) > Belief

(target in each cell neighboring L)?

Belief Updating Belief updates depend on the length of

the time interval since the observation.

If the target is believed to be in cell i at time t, then this increases the belief that the target is near cell i at time t+1.

30

1

30

1,, ),()|()(

n jjnjnii OPOTPTBelief

where

901

1,

,,

)|()(

)|()()|(

iijni

ijnijni

TOPTP

TOPTPOTP

Example Grid size: 30X30 cells

36 sensors are uniformly distributed through the grid, and each sensor covers 5X5 cells.

Three targets in the field.

Results: Low error rate in tracking quality upto 0dB SNR.

Error Rate vs. SNREr

ror

Rate

2. Video Surveillance2. Video Surveillance

- Outdoor Tracking Using Multiple Cameras

Goal and Approach Goal: Monitoring large outdoor areas

Problem: Difficult to deal with changing illumination and weather conditions.

Our approach: Heterogeneous sensors (e.g. optical and IR) Automatic camera selection Measurement fusion techniques

Key Steps

Object tracking within the field of view of the connected sensors.

Automatic camera selection based on appearance ratio.

Data assignment based on gating.

Measurement fusion using a Kalman filter.

Parkinglot

Road

Sensor

Sensor

Sensor Sensor Sensor

Experimental Results

Tracking two persons with a B/W and a Color sensor

ResultsData Fusion B/W Sensor

Color Sensor

  Mean s.d.

B/W Sensor 10.64 4.75

Color Sensor 8.94 3.62

Data Fusion 8.21 3.02

Mean and s.d. of estimation error (in pixels)

3. Location Determination3. Location Determination

- Image Query-Based Approach

Given a city model, find the location based on images of the surrounding area

The city model is composed of street scenes, such as the Berkeley model (A. Zakhor et al.)

Goal

Approach Segment the

street scene into reference images so that one building is in one reference image

Given a query image, determine its location in the city model

Does the query i mage contai nbui l di ng’ s boundary?

Segment the queryi mage i nto two

Fi nd the best match i neach reference i mage

Fi nd the best matchamong reference i mages

Yes

No

Post Processi ng

QueryI mage

Matching Process Stage I

Co-occurrence matrix used for initial culling

Stage II Co-occurrence matrix and Gabor filter followed by

feature-level fusion for matching

Output Locations of sub-images ( , ) in the city model1L 2L

Post-Processing Case I: =

The two sub-images belong to the same building (segmentation may have been wrong)

Case II: = -1 The expected result , provide the final location

Case III: otherwise Weighted summation of the distances

of the two sub-images to reference images in the feature space

The shortest one is the final location

• Query Images

• Matching Results

1L

1L

1L

1L

2L

2L

2L

4. Personnel movement 4. Personnel movement planning in urban combat planning in urban combat

zoneszones

- Path computation algorithms for risk minimization

Goal and Approach Goal: To find the safest routes for personnel to a targ

et or safe zone Model: Geographic information (models) represente

d using a graph whose nodes represent locations associated with uncertain risk estimates

Approach: Maximize probability of traversing a path without damage, viz.,

n

(1- risk(Li)) for the path (L1, …, Ln) i=1

Problems

Personnel on the battlefield need to use the best paths computed within a given time.

Risk estimates change with time, hence best paths cannot be precomputed and reused.

Tradeoff: Solution quality vs. Computation time.

Iterative Improvement Algorithm Shortest paths are computed from a given

source point to various goal nodes.

Repeat:

These paths are incrementally modified and evaluated;

The modified path replaces the previous path if it is of better quality;

Until further small modifications do not improve quality.

Stochastic Algorithm for Path Planning

Multiple candidate paths are explored in parallel, and successively mutated until computational limits are reached.

A mutation to a candidate path (parent) is accepted with a probability that depends on the quality of mutated path, its parent, and the amount of computation performed so far.

Simulations Geography: A 100X100 grid, with 15 goal points

on the periphery of the grid.

The risk estimates were generated from a beta distribution with shape parameters (1,5).

Results reported are averages over the paths obtained for 100 randomly chosen source points.

Mutation Example

. (Exit)

.

. . . . ... . .(source)

. (Exit)

.

. . . . . ... . .(source)

. (Exit)

.

. . . . . .. .. . .(Source)

. (Exit)

.

. . . . . .. .. . .

(Source)

Results

Algorithm Computation Effort(no. of fitness

evaluations; seconds)

Average Quality

Uniform Cost Search

1018000 evals; 457 sec.

0.9912

Iterative Improvement

100 evals; 3 sec.1000 evals; 21 sec.

0.94160.9591

Stochastic Algorithm

100 evals; 3 sec.1000 evals; 21 sec.

0.96180.9735

Solution quality vs. computational effort

Observations The number of possible paths is too large to

permit computation of optimal solutions in real time.

Iterative Improvement algorithms can get stuck in local optima.

Best results were obtained using the stochastic algorithm.

Computations can be terminated at any point, and fairly good (though suboptimal) solutions are obtained very quickly.

5. Scenario Recognition5. Scenario Recognition

Feature Extraction and Classification from Video Image Sequences

Motivation and Goals Tasks:

Understand what is happening around agent in battlefield or urban combat zone

Predict behavior of other agents Detect anomalies from expected behavior Modify behavior models with time

Current Focus: Recognition of the activities depicted in a sequence of images

A video sample for analysis and feature extraction

Main Steps in Recognition Detection and tracking of moving objects

Extraction of features relevant to recognition task

Classification of simple events over short time intervals

Recognition of sub-scenarios consisting of multiple events

Identification of scenarios consisting of multiple sub-scenarios

Object Detection and TrackingBackground model

Difference the object from Background

Object detectionObject tracking

Object template

Sub-Scenario Recognition

Input: image sequences

Features for sub-scenario recognition

Identifying the Location of a Person in a Sequence of Images

Frame 346

Frame 112

Frame 172

Frame 141

Illumination

Effects

Shadows

Central Theme Sequences of values for features extracted from

successive images are viewed as time series (control charts).

Statistical, A.I. and signal processing techniques are used to formulate decision-making criteria from these time series.

Subtasks: Establishing temporal associations between

features and (sub)scenarios. Detecting where each sub-scenario begins and

ends.

Methods to Obtain Decision Criteria

Partition ranges of feature values into applicable intervals for each category.

Develop fuzzy membership functions and corresponding rules.

Apply decision tree learning algo. (C5) Extract frequency information using short-ter

m Fourier Transform.

Illustrative example

Person drinking water

Walkingtowards fountain

Bending to drink

Walking away from fountain

Slowingdown

Standing still

Scenario Sub -scenarios Events

Classifier

Features from image sequence

Detection of Sub-scenarios Example Categories for Classification: walking, bending, running, standing, sitting

Example Features : Aspect Ratio (bounding box width/height) Area of bounding box Relative Upper Density(from 1-D Projection) Speed (measured over several frames)

1-D Projections Distribution of foreground pixels in the y-

direction varies with activity

Statistical characterizations of this distribution hence provide features useful for classification.

Control Chart Examples

Control Chart of Aspect Ratio

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1 11 21 31 41 51 61 71 81 91 101

111

121

131

141

151

161

171

181

Frame Number

Asp

ect

Ra

tio Upper control limit=0.54

Low er control limit=0.31

Control Chart of Area

0

500

1000

1500

2000

2500

3000

1 11 21 31 41 51 61 71 81 91 101

111

121

131

141

151

161

171

181

Frame Number

Are

a

Upper control limit = 2091

Low er control limit=614

Control Chart of Shape

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

1 22 43 64 85 106

127

148

169

190

211

232

253

274

295

316

337

358

379

400

Frame Number

Ra

tio

Upper control limit = 0.28

Low er control limit = 0.13

Control Charts for Rule Production Approach: Rules are obtained using percentile range

s of feature values from control charts Example rule:

If (0.29 AR 0.57) and (Height > Width) and

(Speed 0.5mph) and (Relative Upper Density 0.16) and ( Distance > 0.2)

then “walking class” Results : 79% accuracy for 3 class problem

Fuzzy Rules Approach: Fuzzy classification with

trapezoidal membership functions

Example Rule: If velocity is low and aspect_variance is

low and aspect ratio is high,then sitting class.

Results: 82% accuracy (5 classes)

Automated Learning

Approach: Apply C5 learning algorithm to obtain classification tree and rules.

Results: 92% correct classification (for 3 sub-scenarios, i.e., standing, bending, walking)

Example Decision Tree obtained using C5

RUD 0.38

RUD 0.28

RUD .28 RUD 0.25

AR 0.3

RUD 0.39 RUD 0.39

RUD 0.40 AR 0.11

AR 0.13

Walking

Standing

Walking

Bending

Bending

Walking

Standing

AR 0.3

AR 0.32

true

false

true

true

false

true

true

truefalse

false

false

false

RUD-Relative Upper Density

AR – aspect ratio

Frequency-based Analysis

Motivation: Useful information is available in the frequency domain, but not the time domain.

Principle: Each category of activity is characterized by a distinctive frequency spectrum signature.

Spectrum-based Classification

Goal: Identify and categorize frequency components in the spectrum.

Example rule:

If max(abs(coeffs(20-40)))>0.3,

then walking class Results: 90% accuracy for a 3-class problem, u

sing the spectrum of a single feature (aspect ratio)

Next Steps Recognizing complex scenarios with

multiple moving objects and overlapping sub-scenarios

Extract more features useful for classification

More complex events

Multi-pass system with feedback

Future Work Temporal Bayesian reasoning for dynamic

route planning Learning algorithms to determine

parameters of temporal Bayesian networks Generation of high level descriptions from

video image sequences Detection of unusual activities from video

image sequences Extraction of uncertain information from

temporally contiguous images

Appearance Ratio

• This is an effective metric that indicates how well the target is detected and how well blobs are extracted.

Difference Map

Blob

sk

skj

Byxskj

ThB

yxDBRatioAppearance

skj

,

,

,,

),()(

Normalization with respect to image threshold

Measurement Gating

The gating distance (GD) could be chosen considering the Mahalanobis distance between each arriving observation and the predicted target position given by:

)(~)()(~ 12 kykkyd S

Gating Distance

Trajectory

Measure at time t

Position at time t-1

where is the residual vector and S isthe innovation covariance matrix. GD should be taken so that

)(~ ky

GPGDd )(Pr 22

Measurement Fusion The Kalman filter is used to fuse sensor measurements

(object’s map coordinates).

Measurements are weighted according to the associated AR measure: The measurement error covariance matrix of the Kalman Filter is defined as

where c is an adjustable constant that can be obtained via experiments.

)()(

,

2

, skj

skjii BAR

GDcBR

Image Segmentation to Construct Reference Image Database

Transformation to HSV space

Metric based on Euclidian distance between histogram of either side of a potential boundary

Obtain left and right feature of each column as a form of vectors, where,

represent thenormalized histogram of the region j in the HSV image space

Compute the weighted Euclidian distance of to

The maximum distances are considered as the boundaries of building.

),...,,( 21 nvvvL

Col umn i

1v'

1v

2v'2v

3v '3v

),...,,( ''2

'1 nvvvR

),...,,( 21 nvvvL ),...,,( ''2

'1 nvvvR

jv

iD

Image Segmentation to Construct Reference Image Database

Edge Detection of the query image

Hough transform to detect vertical straight lines

If the line is long enough to be considered as the boundary of the building, segment the query image into two sub-images

Segmentation of Query Images

Stochastic Algorithmfor Path Planning

The mutation is accepted if, offspring quality/parent quality >c where c is a number chosen randomly

from a uniform distribution in the interval (a,1) where a is defined by,

a=(c+(1-c)*t/max – epsilon) c=0.7,epsilon=0.001 and max=number

of evaluations after which the algorithm is to be terminated.

Mutation A portion of the candidate path is

changed by mutation operator.

Two random points along a path are chosen .

The path between these points is changed by choosing a random neighbor of the starting point and choosing successive random neighbors till the end point is reached.

Uniform Cost Search Algorithm

Uniform cost search finds an optimal path by always expanding the lowest cost node .

Starting form the source node, the algorithm expands by exploring neighbors of each successive node in terms of quality .

The path found is guaranteed to be of optimal quality among all paths explored.