Video Matting from Depth MapsJonathan FingerOliver Wang

University of California, Santa Cruz

{jfinger, owang}@soe.ucsc.edu

MotivationGiven a video, replace the background with something differentIsolate the find foreground in each frameGiven a video, replace the background with something differentIsolate the find foreground in each frame

Our MethodUse a depth camera to automate foreground extractionUse Bayesian mattingImprove the matting algorithm to get more realistic video

The Matting ProblemSeparation of a foreground image from a background image

The Easy DirectionBackground(known)Foreground(known)Composite(unknown)2 knowns, 1 unknown

The Hard DirectionBackground(unknown)Foreground(unknown)Composite(known)1 known, 2 unknowns

The Matting ProblemActually there is another unknownRepresents areas that are a combination of foreground and background

The Matting Problem=1=0.5=0Foreground

The Matting ProblemHow do we isolate the foreground?Use an alpha maskAlpha MaskAn image who's color represents foreground and background

The Matting Problemoriginalalpha mask

The Masking ProblemBasic pipelineOriginal compositeAlpha maskIsolated foregroundNew backgroundNew composite

The Masking ProblemBut, how do you get an alpha mask?

Previous WorkBlue Screen MattingPetro Vlahos (1964)Hollywood Special Effects pioneer

Can isolate the foreground if the background is a constant color

Previous WorkBackground is known so it is easy to make a mask

The Matting ProblemHow can this be done with an unknown background?

Use a general matting algorithminput: original composite + trimapoutput: alpha mask

TrimapsA three color image (usually drawn by hand)Black = 100% backgroundWhite = 100% foregroundGray = unknown

TrimapsThe matting algorithm fills in the gray area with estimated alpha values

Matting AlgorithmsThe matting equationFor each 2D location in the image,there is a given composite pixel C

We are to find F, B, and at each pixel whereC = F + (1 - )B

Bayesian MattingOriginal compositeTrimapForeground estimationBackground estimationAlpha mask

Matting Algorithmsalpha maskbackground removedclose upKnockoutRuzon and TomasiBayesian

Problem with Bayesian MattingThese all require a manual trimapOur goal is to do this with videoWe do not want to make trimaps by hand

Previous WorkDefocus Video Matting(McGuire)Two camerasone focused on the backgroundone focused on the foreground

Previous WorkA trimap can be generated from the defocused foregroundHowever, apertures have to be very specific and can be thrown off by lightingAlso requires texture in the scene

Previous WorkBayesian Matting Using Learned Image Priors(Apostoloff, Fitzgibbon)Sequences of frames can be compared in order to find movement

Previous Workassumptionsforeground is movingnothing else is moving

Our ContributionAutomatically generated trimapsDoes not depend on lighting, texture or movementImproved Bayesian Matting using depth informationHella Trimaps

OverviewLow res depthOriginal compositeHigh res depthTrimapNew compositeNew backgroundAlpha maskSupersampleBayesian mattingCompose

Our MethodCanesta depth cameraUses infrared lasers to detect distances from the camera

Our MethodCanesta takes 64x64 resolution imageOptical images are 640x640 or more

Trimap OverviewTo get a trimap Upsample depth image to resolution of optical image Threshold to separate into two colors Erode/dilate to create a gray border around the foreground

UpsamplingUse Qing's supersampled depth methodUse edge cues from high resolution color imageCan increase the depth resolution to up to 100X

ThresholdingAssumptionForeground is in front of backgroundThreshold on a distance planeDone once for entire animation

Erode/DilateGrow unknown area around edges

Improved Bayesian MattingBayesian matting is ill defined when the foreground and background are similar colorsOriginal imageAlpha mask

Improved Bayesian MattingUse depth information in Bayesian Matting optimization stepOriginal imageBayesian mattingDepth map

Improved Bayesian MattingBayesian MattingImproved Bayesian Method

Resultsvideo

ConclusionVideo matting can be done without the user having to manually tweak any individual framesWe were able to improve Bayesian Matting using depth information