Background Removal

David HarwinAdviser: Petros Faloutsos

The State of the Art

This field has been one of great interest in the past decade

There have been many recent papers on the topic with intended applications ranging from visual effects processing to ...

In order to determine how to proceed, I have spent the quarter reading and analyzing existing papers.

Identifying motion

pixel differencing vs optical flow differencing vulnerable to signal noise, illumination

changes, and BG motion optical flow is computationally expensive

most opted for hybrid approach only Criminisi (2006) avoids optical flow

altogether, and instead uses spatio-temporal continuity of labels

Segmentation

2-pass approach

low-level then high-level (Calderara, Tsai, C alic) Calderara - 2nd pass checks for consistency of changes in color and intensity over

time. Objects with insufficiently coherent motion are ignored as BG

Tsai – 1st pass initial labelling (constantly vs never changing), 2nd only works on uncertain pixels

Calic - creates low-resolution representation, corrects BG for camera motion through matrix transformations, determines most ”salient” frames, then calculation of regions of interest occurs only on these keyframes

high-level then low-level (Park) Park - object tracking, followed by pixel differencing inside the object

window

Single pass Criminisi -

Key FeaturesPark

BG differencing

Runtime real-time

none

no no no

channels all colors unclear color, intensity, orientationdate of publication 2006 2006 (?) 2004 2005 2006

no no no

surveillance, distant objects

storage required

moving BG/ FG yes / yes yes / yes yes / yes yes / yes

Criminisi Calderara Tsai Calic

Feature detection method

motion/non-motion classifier

2D spatial scene graph, pixel difference-based

extract corner points with Gandalf library

multi-scale center-surround differences, as in human visual system

unknown, also performs other calculations

requires all data stored beforehand (not causal)

about ½ real-time without BG completion (assuming 24 fps)

real-time layout, keyframe extraction requires preprocessing

optical flow modeling/pixel velocity calculation

no optical flow, calculates spatio-temporal derivatives

determined to be foreground, compares spatio-temporal derivative to surroundings to find coherence

tracks “worms” of uncertain pixels between frames

moves tracking window around each frame to track single object

yes, used to determine camera work

required initial information/ user interaction

none, has “burn-in” period and learns BG features

none, bootstraps from “dirty” frames by classifying regions as stable or unstable (stable -> BG, unstable -> FG)

gets user input on pixels with the Uncertain label

user initializes tracking window

Has background model

color histogram for FG and BG averages (constant over time)

yes, constantly-updated BG model is median of past k pixel values and BG model at last iteration

segmentation discriminant

hidden Markov model – likelihood of motion, color (BG vs FG likelihoods), temporal and spatial continuity of labels

gray-level intensity difference of image compared to BG model, filter out low and high thresholds to get FG. Has object-validating second pass

Maximum a Posteriori (MAP) method

binary difference image from last frame if in tracking window of moving object inside a defined search window

is FG candidate if it has motion vector significantly different from camera-work induced transformations

yellow (spatio-temporal derivative), YUV color data

grayscale intensity, color and intensity

a priori knowledge/ assumptions

Gaussian BG color distribution, distinct BG/FG color difference, BG set with same tilt angle is static, narrowly defined content

intelligent cameraman following interesting motion

intended subject/ use

teleconferencing, small studios or offices

object movies (constant BG, single object in foreground rotated regularly)

film post-processing and VFX, user-directed removal of single object

viewing videos summaries on a small-screen device

FG/BG color histograms, past 2 frames + current frame

past k values for each pixel, BG model of size = size of image

2D spatial graph, per-pixel labeling,

all BG and FG features, 2 comparisons of object windows keyframes only

no / yes, but strictly controlled