High Dynamic Range from Multiple Images: Which Exposures to Combine? Michael Grossberg and Shree...

High Dynamic Range from Multiple High Dynamic Range from Multiple Images: Images: Which Exposures to Combine?Which Exposures to Combine?

Michael Grossberg and Shree NayarMichael Grossberg and Shree Nayar

CAVE Lab, Columbia UniversityCAVE Lab, Columbia University

ICCV Workshop on CPMCV ICCV Workshop on CPMCV October, 2003, Nice, FranceOctober, 2003, Nice, France

Partially funded by NSF ITR Award, DARPA/ONR MURIPartially funded by NSF ITR Award, DARPA/ONR MURI

Combining Different ExposuresCombining Different Exposures

[Ginosar and Zeevi, 88, Madden, 93,Mann and Picard, 95, Debevec and Malik, 97,Mitsunaga and Nayar, 99]

Low

Dyn

amic

Ran

ge E

xpos

ures

+

Combination Yields High Dynamic Range

The Camera The Camera ResponseResponse

Scene Radiance

Linear Function(Optical Attenuation)

Image Irradiance

L s E

0 255

Camera Response

ImageIntensit

y

Bf

From Response To Measured Irradiance From Response To Measured Irradiance LevelsLevels

Measured Irradiance Levels

Irradiance E

Bri

ghtn

ess

Levels

B

Response Function f

Where do you want your bits?Where do you want your bits?

High Dynamic Range

Coarse quantization

Low Dynamic Range

Fine quantization

Bri

ghtn

ess

BB

rightn

ess

B

Irradiance E

Response Function f

Irradiance E

Response Function f

Effective Camera from Multiple Effective Camera from Multiple ExposuresExposures

CaptureHighDynamic Range

CaptureIrradianceUniformly +

+

Acquired Images Image from Effective CameraGoal

Flexible Dynamic Range Imaging:Flexible Dynamic Range Imaging:

• Can we create an effective camera with a Can we create an effective camera with a desired response?desired response?

– How many exposures are needed?How many exposures are needed?

– Which exposures to acquire? Which exposures to acquire?

– How to combine the acquired images?How to combine the acquired images?

Irradiance Levels From Multiple Irradiance Levels From Multiple ExposuresExposures

f(E)

Irradiance E

E10 E2 E3

21

3

0

2

01

3

6

45

h(E)

f(e2E)

E3 / e3 E2 / e2 E1 / e1 0

21

3

0

Sum

E1

^E6

^E5

^E4

^E3

^E2

^0

Exposure e1 = 1

Effective Camera

Exposure e2

Response of the Effective CameraResponse of the Effective Camera

1

)()(j

EfEh

Number of exposures

Exposures

Camera Response

Irradiance

Effective Response

Theorem: The sum of a set of images of a scene taken at different exposures includes all the information in the individual exposures.

jen

Camera Response EmulationCamera Response Emulation

• How can we tell if How can we tell if hh emulates emulates gg well? well?

Naïve answer:Naïve answer: | h – g | < | h – g | <

gf , ee = ( = (ee11, … ,, … ,eenn))

Level spacing characterizes similarity

Bri

ghtn

ess

Levels

B

Irradiance E

h

Desired Response

Emulated Responsedepends on:

h~

How Response Determines Level How Response Determines Level SpacingSpacing

Measured Irradiance Levels

Irradiance E

Bri

ghtn

ess

Levels

B

h

Observation: The derivative determines the distances between levels.

Dense Spacing

LargerDerivative

Sparse Spacing

SmallerDerivatives

The Objective FunctionThe Objective Function

MAX

MIN

E

E

dEwhg 2||),(

Spacing Based Comparison:

Weight prevents penalizing success:

Desired Response Effective Response

Weight

Number of Exposures Exposure Values

en

)(Ew { )()( EhEg 0,

1, otherwise

B

rightn

ess

B

g

Irradiance E

h

w=0 w=1

Which Exposures and How Many?Which Exposures and How Many?

• For fixed For fixed nn, find minimizing exposure , find minimizing exposure values values ee

• Choose min Choose min nn such that error within such that error within tolerance tolerance

• Method: Exhaustive searchMethod: Exhaustive search

– Objective function not continuousObjective function not continuous– Only need to search actual settings Only need to search actual settings – Offline build table of exposuresOffline build table of exposures

Flexible Dynamic Range ImagingFlexible Dynamic Range Imaging

Linear

Gamma =1/2

Constantcontrast(log)

1111111111111111

21.0031.0031.0151.0031.0033.0941.0031.0031.0031.00320.249.914.6894.8313.979

3

2.9851.0191.0071.007

5.1463.0191.0061.006

88.3837.2329.1816.01

4

3.6721.1661.031

11.235.0492.866

280144.964.22

5

4.9751.078

18.568.564

763.5305.4

6

5.636

33.7

1130

EffectiveCamera

Number ofExposures

Exposure Values

234562345623456

Desired Response

Constant Contrast

Gamma 1/2

Linear

High 1:16,000

Low 1:256

Dynamic Range

Medium 1:1,000

Desired Response

Constant Contrast

Gamma 1/2

Linear

High 1:16,000

Low 1:256

Dynamic Range

Linear

Gamma =1/2

Constantcontrast(log)

1111111111111111

21.0031.0031.0151.0031.0033.0941.0031.0031.0031.00320.249.91

4.6894.8313.979

3

2.9851.0191.0071.007

5.1463.0191.0061.006

88.3837.2329.1816.01

4

3.6721.1661.031

11.235.0492.866

280144.964.22

5

4.9751.078

18.568.564

763.5305.4

6

5.636

33.7

1130

EffectiveCamera

Number ofExposures

Exposure Values

2345623456

23456

Flexible Dynamic Range ImagingFlexible Dynamic Range Imaging

Baseline Exposure ValuesBaseline Exposure Values

• Typically exposures are doubledTypically exposures are doubled

• Baseline: Baseline: Combine the exposures Combine the exposures e e =(1,2,4)=(1,2,4)

[Ginosar and Zeevi, 88, Madden, 93,Mann and Picard, 95, Debevec and Malik, 97,Mitsunaga and Nayar, 99]

Increased Dynamic Range Linear Increased Dynamic Range Linear CameraCamera

• Real Camera: Real Camera: f f linear linear • Desired Camera: Desired Camera: gg linear (greater dynamic linear (greater dynamic

range) range)

Baseline Response (1,2,4)

Desired Response

0.8

0.6

0.4

0.2

0.0

1.0

0.80.60.40.20.0 1.0

Irradiance E

Bri

ghtn

ess

B

4-bit real camera

Baseline Response (1,2,4)

0.8

0.6

0.4

0.2

0.0

1.0

0.80.60.40.20.0 1.0

Desired Response

Irradiance E

Bri

ghtn

ess

B

8-bit real camera

Computed Response(1, 1.003, 2.985)

Computed Response(1, 1.05, 1.11)

Linear Camera: Synthetic Ramp Linear Camera: Synthetic Ramp ImageImage

From computed exposures (1,1.05,1.11)From baseline exposures (1,2,4)

High Dynamic Range Linear Camera

Linear Camera: Image of ClothLinear Camera: Image of Cloth

Computed Exposures (1,1.05,1.11)

Ground Truth (HDR image)

Baseline Exposures (1,2,4)

Constant Contrast from Linear Constant Contrast from Linear CamerasCameras

• Real Camera: Real Camera: f f linear linear • Desired Camera: Desired Camera: gg log response (constant log response (constant

contrast)contrast)

Baseline Response (1,2,4)

0.8

0.6

0.4

0.2

0.0

1.0

0.80.60.40.20.0 1.0

Irradiance E

Brig

htne

ss B

Desired Response

3 Exposures

Baseline Response

Irradiance E

1.0

0.8

0.6

0.4

0.2

0.00.80.60.40.20.0 1.0

Brig

htne

ss B

Desired Response

5 Exposures

Computed Response (1,9.91,88.38)

ComputedResponse

Constant Contrast : Image of TilesConstant Contrast : Image of Tiles

Computed Exposures

Baseline Exposures

Linear Camera from Non-linear CameraLinear Camera from Non-linear Camera

Baseline Input Exposures

Iso-brightnessCombined Iso-brightness

Computed Input Exposures

Combined

• Real Camera: Real Camera: f f non-linear (Nikon 990) non-linear (Nikon 990)• Desired Camera: Desired Camera: gg linear linear

SummarySummary

• Combine images using Combine images using summationsummation

• Method finds Method finds number of exposuresnumber of exposures and and exposure valuesexposure values to use to use

• Emulation of a variety of cameras:Emulation of a variety of cameras:

Flexible Dynamic Range ImagingFlexible Dynamic Range Imaging