Visualization and FusionVisualization and Fusion

Craig Scott,

Aisha Page, Olusanya Soyannwo, Hamzat Kassim Paul Blackmon, Pierre

Knight, Francis Dada

Engineering Visualization Laboratory

Outline• Overview

• Image/Spatial Fusion

• Motion detection and object tracking

• Synthetic Battlespace

• Conclusion

Overview

• The objective of this research is to advance ideas that apply to how to enhance battlespace awareness and move tactical decision making closer to the field soldier by applying image and spatial data fusion concepts.

BORG Experimental Architecture & Platform

IMPACT / PASTA / P2P - Willy & Damian(VS, Hendler - UMD)

Java/C++ Sequal Server/Oracle UML Cognitive Models Sun JXTA

B3A

Imagery DB(Roy GeorgeClark-Atlanta)

Soldier

HighlightsCraig JCDB

GIS...Requirements,Cognitive

DeficienciesPamela,Leroy

Autonomous Fusionand NavigationDistinct Sources

Data Streams

Content Based MessagingSystem

Willy & Damian(Andrew Cowley PNNL)

RSSRSSRSS...

Advanced Interactions

Story(VS - UMD)

VisualizationsCraig

Commander/Analysts

Tactical OperationsCenter Workstations

Command and Control

Interactive Fusion andNavigationSummary of Source

Website (Hendler - UMD), Vignette, Requirements Leroy, Pamela

Plans(Monmouth)

WWWHeterogeneousData sources

KnowledgeEngineering,

Cognitive Models,HCI

Leroy,Pamela

Seed DemoC++

DirectShowWindowsAVI video

Issues/Approach

• Exploiting Level of Detail Perception– Multiple channel architectures

– Area-of of-interest filters– Subscription-based aggregation

• Exploiting Temporal Perception– Render the entity in an accurate

location as long as the local user does not interact,

• Distingish active and passive entities

Image/Spatial Data FusionImage/Spatial Data Fusion: combining complete

spatially filled sets if data in 2-D or 3D

RegistrationCombination

Reasoning

Viewing Volume or Plane

Terrain Layer

Meaning

Data Structures

2D optical photos & video, FLIR, SAR3D terrain data, buildings, vehicles, weather models, LADAR

Tracking Studies• Objective

– To be able to provide one solution to soldiers query about unrecognizable objects (tanks, airplanes, people etc.) through object tracking

• Problem– Finding the best tracking algorithm efficient

enough to display and track desired object– Implementing different algorithms into

frame work which will display best algorithm suitable for the current field operation

• Solution– Implement various tracking algorithms with

varying environmental conditions– Validate effectiveness through perceptual

studies

Motion and Object Tracking Algorithms

• Background subtraction

• Motion templates

• Optical flow

• Active contours

• Estimators

Background Subtraction (cont’d)

• μt = αy+ ( 1 –α)μt

– μ - updated image– t – time constant– α – learning rate constant

• Specifies how fast (responsive) background model is adapted to changes

• 0 <= α <= 1

– y – new observation at time t

Active Contour

• Active Contours– Curves defined within an image domain– Can move under influence of

• Internal forces coming from within the curve itself• External forces computed from the image data

• Allows the computer to generate curves that move within images to– locate object boundaries– find other desired features within an image

Active Contour (cont’d)

• Energy equation associated with snake

• E= Eint +Eext

– Eint - internal energy formed by the snake configuration

• E int= Econt + Ecurv

– Econt – Contour continuity energy

» Minimizing Econt over all the snake points, causes the snake points to become more equidistant

– Ecurv – Contour curvature energy

» The smoother the contour is, the less the curvature energy

Active Contour (cont’d)

– Eext is the external energy formed by external forces affecting the snake

• Eext= Eimg + Econ

– Eimg – Image energy

– Two variants of image energy are proposed:

» Eimg = -I, where I is image intensity

» Eimg = -||grad(I)||, Snake is attracted to image edges

– Econ – Energy of additional constraints

KLT: Kanade-Lucas-Tomasi Feature Tracker

• Problem– Complex changes occur between frames

• Good features are located by:– Examining the minimum eigenvalue of each 2x2 gradient matrix

• Key components to feature tracker– Accuracy: relates to local sub-pixel accuracy attached to tracking– Robustness: relates to sensitivity of tracking with respect to

changes of:• Lighting• Size of image motion

• Goal– To find location on second image – Such that, image one and two are similar

Computing Image Motion• Residual function:

• J – second image• I – first image• W – given feature• X – point within image• w(x) – weighting function• A = 1 + D• d – translation (uniform movement) of feature

window’s center; from one frame to another• D – deformation (change of shape) matrix

– Used to determine if the first and current frames match

W

dxxwxIdAxJ )()]()([ 2

Feature Selection

• Means to select point in image I• Selection maximizes quality of tracking• Central step of tracking – computation of the

optical flow– Optical flow: motion of brightness patterns in image– At this critical step the minimum eigenvalue must be

larger than a threshold

• This characterizes pixels that are “easy to track”

Test Scene Virtual Laboratory for Data Fusion Studies

Tracking Example

Background Subtraction

Battlespace Visualization

Battlespace visualization is the process whereby the commander—

– Develops a clear understanding of the current state with relation to the environment.

– Envisions a desired end state that represents mission accomplishment.

– Visualizes the sequence of activity that moves the commander’s force from its current state to the end state.

Army Geospatial Guide for Commanders and Planners, TC 5-230, November 2003.

Synthetic Battlespace Research Issues

• Perception/Trust • Agent Roles & Interaction • 3D Scene Reconstruction• Automated Feature

Extraction• Registration Accuracy• LOD/Bandwidth Tradeoff

–“As DoD looks to the future, increasing demands on the warfighter dictate the increased use of simulations in operational situations. Ideally, the simulation power is placed at the immediate disposal of the warfighter so that it can be accessed and employed when needed.”

Conclusion

• We have implemented rudimentary prototype object tracking algorithms within a seed demo application to illustrate image/spatial fusion concepts.

• The synthetic battlespace concepts require integrating present joint simulation technology with fusion research concepts to articulate a COP that is easy to understand and react to.

• The commercial game market supplies a significant amount of talent and resource$ (market) to fuel this area