Digital Photography with Flash and No-Flash Image Pairs

By: Georg Petschnigg Maneesh AgrawalaHugues Hoppe Richard Szeliski

Michael Cohen Kentaro Toyama, Microsoft Corporation

Presented by: Yael AmsterdamerAdvanced Topics in Visual Computing, Spring 2012

2Digital Photography with Flash and No-Flash Image Pairs

The Dilemma: to Flash or not to flash?

Introduction

• Natural lighting• Low signal-to-noise ratio

(SNR)• Loss of details• Longer exposure –

motion blur

• Harsh, unnatural lighting• High SNR• More details• May cause unwanted

artifacts (red eye, shadows, specularities)

3Digital Photography with Flash and No-Flash Image Pairs

Why not both?

• The idea: use the good features of each photo to create a better image

Introduction

4Digital Photography with Flash and No-Flash Image Pairs

Other works on combining images

Introduction

Different exposures

Panoramic view

Relighting

Current work • handles 2 images with specific

characteristics,• Envisioned for camera firmware

5Digital Photography with Flash and No-Flash Image Pairs

Different Applications

• Denoising• Detail transfer– Side requirement: handling flash induced

shadows and specularities• White Balancing• Continuous Flash Adjustment• Red Eye Removal

Introduction

6Digital Photography with Flash and No-Flash Image Pairs

Camera Adjustment to light• A certain amount of light is needed for a photo

• Taking pictures of a dark scene

– Increasing exposure time

– Opening the aperture

– Increasing ISO (sensor sensitivity)

• The problem – high ISO noise affects high-frequency details

Background

7Digital Photography with Flash and No-Flash Image Pairs

The Input

• F – the photo with flash• A – the no-flash (ambient) photo– Composed of 3 RGB intensity values for each pixel– Ap – one pixel in A

• Same camera position, focus, aperture– Otherwise, registration is required

• Adjusted ISO and exposure time• Undergo non-linear tone-mapping operations

Background

8Digital Photography with Flash and No-Flash Image Pairs

1st Application: Denoising

• A classic problem in image processing– The target: analytically reduce SNR

• One type of solution: smoothing filters• Bilateral Filter [Tomasi and Manduchi 1998]

– Fast, non-iterative– Averaging close-by pixels– Low-pass filter, combined with– Edge-stopping function

Denoising

9Digital Photography with Flash and No-Flash Image Pairs

Bilateral filter

• However, results still have noise or blur (or both)

Denoising

ambient

flashBilateral

filter

10Digital Photography with Flash and No-Flash Image Pairs

Improvement: Joint Bilateral Filter

• In the flash image there are much more details• Why not use F to find edges?

Denoising

Bilateral filter

Joint Bilateral filter

The difference

11Digital Photography with Flash and No-Flash Image Pairs

Not all edges are real …

• May cause over- or under-blur in joint bilateral filter• We need to eliminate their effect

Denoising

12Digital Photography with Flash and No-Flash Image Pairs

Detecting shadows• Observation: the pixels in the flash shadow should be similar to

the ambient image

• Not identical:

– Noise

– Inter-reflected flash

• Compute a shadow mask

• Take pixel p if

• is manually adjusted

• Mask is smoothed and dilated

Denoising

Pixels for which we will not use F

13Digital Photography with Flash and No-Flash Image Pairs

Detecting specularities• Take pixels where sensor input is close to maximum

(very bright)

– Over fixed threshold

• Create a specularity mask

• Also smoothed and dilated

• M – the combination of shadow and specularity masks

– Where Mp=1, we use ABase. For other pixels we use ANR.

Denoising

14Digital Photography with Flash and No-Flash Image Pairs

There is so much denoising can do• It cannot add details missing in the ambient image• Exist in flash image because of high SNR• We use a quotient image:

• Multiply with ANR to add the details• Masked in the same way

Detail Transfer

Bilateral filtered

Reduces the effect of

noise in F

15Digital Photography with Flash and No-Flash Image Pairs

Unified process• Denoising + detail transfer + masking shadows and

specularities

Detail Transfer

16Digital Photography with Flash and No-Flash Image Pairs

More examples

Detail Transfer

17Digital Photography with Flash and No-Flash Image Pairs

The opposite of preserving the atmosphere• The goal: simulate white light in the scene

– What color is the light in the scene?

• Flash image can help us:The difference between F and A is only the flash light

• Assume it is proportional to the surface albedo– Inherent property of the surface

– Assume the surface reflects light in the same color (will not work for plastic, glass…)

White Balancing

18Digital Photography with Flash and No-Flash Image Pairs

Computing the ambient light• We cancel the color of the surface

• The average c of Cp(col) for pixels with certain properties is the ambient light color

• Then we scale A to cancel the ambient light

White Balancing

19Digital Photography with Flash and No-Flash Image Pairs

Example

White Balancing

Estimated ambient light

Computed color

Ambient original

White-balanced

20Digital Photography with Flash and No-Flash Image Pairs

Combining flash and no-flash images• Providing complementary details about the scene

• Use for denoising and detail transfer

• Use for white balancing

• Future work

– More robustness

– Automatization of configuration

Summary

Thank you!