Natural Video Matting with Depth

Jonathan FingerOliver Wang

University of California, Santa Cruz

{jfinger, owang}@soe.ucsc.edu

Motivation Given a video, replace the background with

something different

Isolate the find foreground in each frame

Image courtesy of Yung-Yu Chuang, Brian Curless, David Salesin, Richard Szeliski

Our Method

Use a depth camera to automate foreground extraction

Use Bayesian matting

Improve the matting algorithm to get more realistic video

The Matting Problem

Separation of a foreground image from a background image

Image obtained from Corel Knockout's tutorial.

The Easy Direction

Background(known)

Foreground(known) Composite

(unknown)

2 knowns, 1 unknown

The Hard Direction

Background(unknown)

Foreground(unknown)

Composite(known)

1 known, 2 unknowns

The Matting Problem

Actually there is another unknown

Represents areas that are a combination of

foreground and background

0 1

transparent opaque

:

The Matting Problem

=1

=0.5=0

Foreground

The Matting Problem

How do we isolate the foreground?

Use an alpha mask

Alpha Mask

An image who's color represents foreground and background

The Matting Problem

original alpha mask

The Masking Problem

Basic pipeline

Original composite

Alpha mask

Isolated foreground

New background

New composite

The Masking Problem

But, how do you get an alpha mask?

Previous Work

Blue Screen Matting

Petro Vlahos (1964)

Hollywood Special Effects pioneer

Can isolate the foreground if the background is a constant color

Previous Work

Background is known so it is easy to make a mask

Image courtesy of A. Smith and J. Blinn

The Matting Problem

How can this be done with an unknown background?

Use a general matting algorithm input: original composite + trimap

output: alpha mask

Trimaps

A three color image (usually drawn by hand)

Black = 100% background

White = 100% foreground

Gray = unknown

Trimaps

The matting algorithm fills in the gray area with estimated alpha values

Natrual Matting Algorithms

The matting equation

For each 2D location in the image,

there is a given composite pixel C

We are to find F, B, and at each pixel where

C = F + (1 - )B

Natural Matting

Original composite Trimap

Foreground estimation Background estimation Alpha mask

Natural Matting Algorithms

alpha mask background removed close up

Knockout

Ruzon and Tomasi

Bayesian

Image courtesy of Yung-Yu Chuang, Brian Curless, David Salesin1, Richard Szeliski

Problem with Natural Matting

These all require a manual trimap

Our goal is to do this with video

We do not want to make trimaps by hand

Previous Work

Defocus Video Matting

(McGuire)

Two cameras

one focused on the background

one focused on the foreground

Previous Work A trimap can be generated from the defocused

foreground

However, apertures have to be very specific and can be thrown off by lighting

Also requires texture in the sceneImage courtesy M. McGuire, W. Matusik,H. Pfister, J. Hughes, and F. Durand.

Previous Work

Bayesian Matting Using Learned Image Priors

(Apostoloff, Fitzgibbon)

Sequences of frames can be compared in order to find movement

Image courtesy N. Apostoloff and A. Fitzgibbon

Previous Work

assumptions

foreground is moving

nothing else is moving

Image courtesy N. Apostoloff and A. Fitzgibbon

Previous Work

The Z-Cam is able to separate a video scene into depth plains, but does not calculate alpha values.

Our Contribution

Automatically generated trimaps

Does not depend on lighting, texture or movement

Improved Bayesian Matting using depth information

Hella trimaps

OverviewLow res depth Original composite

High res depth Trimap

Alpha mask

Supersample Bayesian matting

Our Method Canesta depth camera

Uses infrared lasers to detect distances from the camera

Our Method

Optical image Depth image

Canesta takes 64x64 resolution image

Optical images are 640x640 or more

Trimap Overview

To get a trimap

1) Upsample depth image to resolution of optical image

2) Threshold to separate into two colors

3) Erode/dilate to create a gray border around the foreground

Upsampling

Use Qing's supersampled depth method

Use edge cues from high resolution color image

Can increase the depth resolution to up to 100X

Thresholding Assumption

Foreground is in front of background

Threshold on a distance plane Done once for entire animation

Erode/Dilate

Grow unknown area around edges

Improved Bayesian Matting

Bayesian matting is ill defined when the foreground and background are similar colors

Original image Alpha mask

Improved Bayesian Matting

Use depth information in Bayesian Matting optimization step

Original image Bayesian matting Depth map

Improved Bayesian Matting

Improved Bayesian Matting

Bayesian Matting Improved Bayesian Method

Results

video

Conclusion

Video matting can be done without the user having to manually tweak any individual frames

We were able to improve Bayesian Matting using depth information