Embed Size (px)
Citation preview
FELIX HEIDE1,2 MARKUS STEINBERGER3 YUN-TA TSAI1 NASA ROUF1,2 DAWID PAJĄK1 DIKPAL REDDY1 ORAZIO GALLO1 JING LIU4 WOLFGANG HEIDRICH5 KAREN EGIAZARIAN1,6 JAN KAUTZ1 KARI PULLI1
FlexISP: A Flexible Camera Image Processing Framework
1 NVIDIA 2 UBC 3 TU GRAZ 4 UCSC 5 KAUST 6 TUT
CAMERA IMAGE PROCESSING
4
Image Signal Processor (ISP)
IMAGE PROCESSING PIPELINE
Denoise Demosaic
Bad Pixel
Correction
Image
Enhancing
Tone
Mapping
Lens
Correction
Black Level
Metering
Exposure / Focus Adjustment
IMAGE PROCESSING PIPELINE
Denoise Demosaic
Bad Pixel
Correction
Image
Enhancing
Tone
Mapping
Lens
Correction
Black Level
Metering
Exposure / Focus Adjustment
IMAGE PROCESSING PIPELINE
Denoise Demosaic
Bad Pixel
Correction
Image
Enhancing
Tone
Mapping
Lens
Correction
Black Level
Metering
Exposure / Focus Adjustment
IMAGE PROCESSING PIPELINE
Denoise Demosaic
Bad Pixel
Correction
Image
Enhancing
Tone
Mapping
Lens
Correction
Black Level
Metering
Exposure / Focus Adjustment
IMAGE PROCESSING PIPELINE
Denoise Demosaic
Bad Pixel
Correction
Image
Enhancing
Tone
Mapping
Lens
Correction
Black Level
Metering
Exposure / Focus Adjustment
IMAGE PROCESSING PIPELINE
Denoise Demosaic
Bad Pixel
Correction
Image
Enhancing
Tone
Mapping
Lens
Correction
Black Level
Metering
Exposure / Focus Adjustment
IMAGE PROCESSING PIPELINE
Denoise Demosaic
Bad Pixel
Correction
Image
Enhancing
Tone
Mapping
Lens
Correction
Black Level
Metering
Exposure / Focus Adjustment
IMAGE PROCESSING PIPELINE
Denoise Demosaic
Bad Pixel
Correction
Image
Enhancing
Tone
Mapping
Lens
Correction
Black Level
Metering
Exposure / Focus Adjustment
IMAGE PROCESSING PIPELINE
Denoise Demosaic
Bad Pixel
Correction
Image
Enhancing
Tone
Mapping
Lens
Correction
Black Level
Metering
Exposure / Focus Adjustment
IMAGE PROCESSING PIPELINE
Demosaic Denoise
Bad Pixel
Correction
Image
Enhancing
Tone
Mapping
Lens
Correction
Black Level
Metering
Exposure / Focus Adjustment
IMAGE PROCESSING PIPELINE
Demosaic Denoise
Bad Pixel
Correction
Image
Enhancing
Tone
Mapping
Lens
Correction
Black Level
Metering
Exposure / Focus Adjustment
IMAGE PROCESSING PIPELINE
Demosaic Denoise
Bad Pixel
Correction
Image
Enhancing
Tone
Mapping
Lens
Correction
Black Level
Metering
Exposure / Focus Adjustment
Long Exposure
Short Exposure
Long Exposure
Short Exposure
NOVEL FEATURES
Single Exposure Interleaved Exposure
NOVEL FEATURES
NOVEL FEATURES
Refocusing Depth
Kepler GPU
OBJECTIVES
NO PIPELINE
FLEXIBILITY
PERFORMANCE
PROPOSED SOLUTION
FLEXISP
Bayer Burst iHDR Others
…
x B D z
IMAGE FORMATION
x B D z
z = DBx
IMAGE FORMATION
z = A x
IMAGE FORMATION
B D
LINEAR SYSTEM
𝑎𝑟𝑔𝑚𝑖𝑛𝑥 𝑧 − 𝐴𝑥 22
LINEAR SYSTEM
errors w.r.t. observation
x
A
𝑎𝑟𝑔𝑚𝑖𝑛𝑥 𝑧 − 𝐴𝑥 22
LINEAR SYSTEM
errors w.r.t. observation
x
A
𝑎𝑟𝑔𝑚𝑖𝑛𝑥 𝑧 − 𝐴𝑥 22
LINEAR SYSTEM
regularization
𝑎𝑟𝑔𝑚𝑖𝑛𝑥 𝑧 − 𝐴𝑥 22 + λ(𝑥)
TOTAL VARIATION (Rudin et al. ‘92)
CROSS-CHANNEL (Heide et al. ‘13)
COLLABORATIVE DENOISING
COLLABORATIVE DENOISING
LINEAR SYSTEM
Total variation Cross-channel
+ +
Collaborative denoising
𝑎𝑟𝑔𝑚𝑖𝑛𝑥 𝑧 − 𝐴𝑥 22 + λ(𝑥)
IMPORTANCE OF PRIORS
MORE ON PRIORS…
TV [Rudin et al. ‘92]
EPLL [Zoran and Weiss ‘11]
Curvelets [Candès and Donoho ‘99]
MORE ON PRIORS…
LINEAR SYSTEM
Convex Optimization (Mostly)
𝑎𝑟𝑔𝑚𝑖𝑛𝑥 𝑧 − 𝐴𝑥 22 + λ(𝑥)
CONVERGENCE
RESULTS
Bayer Burst iHDR
CAMERA SYSTEMS
Bayer Burst iHDR
CAMERA SYSTEMS
Bayer Burst iHDR
CAMERA SYSTEMS
A
Bayer Burst iHDR
CAMERA SYSTEMS
Bayer Burst iHDR
CAMERA SYSTEMS
Bayer Burst iHDR
CAMERA SYSTEMS
A
Bayer Burst iHDR
CAMERA SYSTEMS
Bayer Burst iHDR
CAMERA SYSTEMS
Bayer Burst iHDR
CAMERA SYSTEMS
A
PERFORMANCE
0.82
7.45
2.13
Image burst (0.4MPx16)
iHDR (13MP)
Demosaic (5MP)
TIME IN SECONDS
GTX Titan
PERFORMANCE
16.7
174.4
40.5
Image burst (0.4MPx16)
iHDR (13MP)
Demosaic (5MP)
TIME IN SECONDS
Tegra K1
MORE ON PRIORS…
PERFORMANCE (BM3D)
16.7
174.4
40.5
Image burst (0.4MPx16)
iHDR (13MP)
Demosaic (5MP)
TIME IN SECONDS
Tegra K1
PERFORMANCE (AVERAGING NLM)
6.6
33.2
6.7
Image burst (0.4MPx16)
iHDR (13MP)
Demosaic (5MP)
TIME IN SECONDS
Tegra K1
CONCLUSION
Bayer Burst iHDR Color Camera Array
+ others such as:
deconvolution, super-resolution, JPEG compression and de-blocking.
Bayer Burst AOHDR Color Camera Array
FLEXISP A
THANK YOU!
