Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Super-Resolution
Many slides from Miki Elad – TechnionYosi Rubner – RTC and more
1
Example - Video
53 images, ratio 1:42
40 images ratio 1:4
Example – Surveillance
3
Example – Enhance Mosaics
4
5
Super-Resolution - Agenda
• The basic idea• Image formation process• Formulation and solution• Special cases and related problems• SR in time• SR from Examples
6
D
For a given band-limited image, the Nyquistsampling theorem states that if a uniform sampling is fine enough (≥D), perfect reconstruction is possible.
D
Intuition
7
Due to our limited camera resolution, we sample using an insufficient 2D grid
2D
2D
8
Intuition
However, if we take a second picture, shifting the camera ‘slightly to the right’we obtain:
2D
2D
9
Intuition
Similarly, by shifting down we get a third image:
2D
2D
10
Intuition
And finally, by shifting down and to the right we get the fourth image:
2D
2D
11
Intuition
It is trivial to see that interlacing the four images, we get that the desired resolution is obtained, and thus perfect reconstruction is guaranteed.
Intuition
12
What if the camera displacement is Arbitrary ? What if the camera rotates? Gets closer to the object (zoom)?
Rotation/Scale/Disp.
13
There is no sampling theorem covering this case
14
Rotation/Scale/Disp.
15
3:1 scale-up in each axis using 9 images, with pure global translation between them
A Small Example
Further Complications
• Complicated motion– perspective, local motion, …
• Blur– sampling is not a point operation– Spatially variant blur– Temporally variant blur
• Noise• Changes in the scene
16
Super-Resolution - Agenda
• The basic idea• Image formation process• Formulation and solution• Special cases and related problems• SR in time• SR from Examples
17
Image Formation
Scene Image
HR LRk
Can we write these steps as linear operators?
Geometrictransformation
kF
OpticalBlur
kH
Sampling
kD
HRLRk ⋅= kkk FHD 18
Geometric Transformation
• Any appropriate motion model• Every frame has different transformation • Usually found by a separate registration algorithm
Scene Geometrictransformation
kF
19
Geometric Transformation
Can be modeled as a linear operation XkF
kF
X
=
kF XXkF
20
Optical Blur
• Due to the lens PSF and pixel integration• Usually
Geometrictransformation
OpticalBlur
kH
HH =k
21
H
PSF PIXEL H* =
22
Optical Blur
Can be modeled as a linear operation XH
H
X
=
XHXH
23
Sampling
• Pixel operation consists of area integration followed by decimation• D is the decimation only• Usually
Optical Blur Sampling
kD
DD =k
24
Decimation
Can be modeled as a linear operation XD
D
X
=
X
0101
...01
0101 o
oD
1
XD
25
Image Formation
Scene Image
HR LR
Geometrictransformation
kF
OpticalBlur
kH
Sampling
kD
HRLR ⋅= kkk FHD26
Image Formation
Scene Image
HR LRk
N HRLRk +⋅= kkk FHD27
+Noise
N
Super-Resolution - Agenda
• The basic idea• Image formation process• Formulation and solution• Special cases and related problems• SR in time• SR from Examples
28
Super-Resolution - Model
{ }N
knkkkkkk VVXY
1
2,0~ , =⎭
⎬⎫
⎩⎨⎧
+= σNFHD
X
High-Resolution
ImageH
H
Blur
1
N
F =I1
FN
Geometric warp
D
D1
N
Decimation
V 1
V N
Additive Noise
Y1
YN
Low-ResolutionExposures
29
Simplified Model
{ }N
knkkkk VVXY
1
2,0~ , =⎭
⎬⎫
⎩⎨⎧
+= σNDHF
X
High-Resolution
ImageH
H
Blur
F =I1
FN
Geometric warp
D
D
Decimation
V 1
V N
Additive Noise
Y1
YN
Low-ResolutionExposures
30
The Super-Resolution Problem
• GivenYk – The measured images (noisy, blurry, down-sampled ..)H – The blur can be extracted from the camera characteristicsD – The decimation is dictated by the required resolution ratioFk – The warp can be estimated using motion estimationσn – The noise can be extracted from the camera / image
• RecoverX – HR image
{ }2,0~ , nkkkk VVXY σNDHF +=
31
VX
V
VV
X
Y
YY
Y
NNNNN
+=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
+
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
= G
FHD
FHDFHD
MMM2
1
222
111
2
1
The Model as One Equation
[ ][ ]
[ ]1 size of , size of
1 size of
22
222
2
×
×
×
MrVXMrNM
NMYG
r = resolution factorMXM = size of the framesN = number of frames
r = resolution factor = 4MXM = size of the frames = 1000X1000N = number of frames = 10
=[10M×1]=[10M×16M]=[16M×1] Linear algebra notation is
intended only to develop algorithm 32
SR - Solutions
• Maximum Likelihood (ML):
∑=
−=N
kkkX
YXX1
2minarg DHF
Smoothness constraintregularization
Often ill posed problem!
{ }XAYXXN
kkkX
λ+−= ∑=1
2 minarg DHF
• Maximum Aposteriori Probability (MAP)
33
ML Reconstruction (LS)
( ) ∑=
−=N
kkkML YXX
1
22 DHFεMinimize:
( ) ( ) 0ˆ21
2
=−=∂
∂ ∑=
N
kkk
TTTk
ML YXX
X DHFDHFεThus, require:
k
N
k
TTTk
N
kk
TTTk YX ∑∑
==
=⋅11
ˆ DHFDHFDHF
A B
BA =X̂34
LS - Iterative Solution
• Steepest descent
( )∑=
+ −−=N
kknk
TTTknn YXXX
11
ˆˆˆ DHFDHFβ
Simulated error
Back projection
All the above operations can be interpreted as operations performed on images.
There is no actual need to use the Matrix-Vector notations as shown here.
35
LS - Iterative Solution
• Steepest descent ( )∑=
+ −−=N
kknk
TTTknn YXXX
11
ˆˆˆ DHFDHFβ
nX̂
1ˆ
+nX
geometrywrap
convolvewith H
downsample
upsample
convolvewith HT
inversegeometry
wrap
kY
-β
-
kF H D TD TH TkF
For k=1..N
36
Example
Simulated example from Farisu at al.IEEE trans. On Image Processing, 04
HR image Least squaresLR + noiseX4
37
Robust Reconstruction
• Cases of measurement outlier:– Some of the images are irrelevant – Error in motion estimation– Error in the blur function– General model mismatch
38
Robust Reconstruction
( ) ∑=
−=N
kkk YXX
1
2 1
DHFεMinimize:
( )∑=
+ −−=N
kknk
TTTknn YXXX
11
ˆsignˆˆ DHFDHFβ
39
Robust Reconstruction
• Steepest descent
( )∑=
+ −−=N
kknk
TTTknn YXXX
11
ˆsignˆˆ DHFDHFβ
sign
For k=1..N
nX̂
1ˆ
+nX
geometrywrap
convolvewith H
downsample
upsample
convolvewith HT
inversegeometry
wrap
kY
-β
-
kF H D TD TH TkF
40
Example - Outliers
Simulated example from Farisu at al.IEEE trans. On Image Processing, 04
HR image LR + noiseX4
Least squares
Robust Reconstruction41
20 images, ratio 1:4
L2 norm based
Example – Registration Error
L1 norm based
42
MAP Reconstruction
• Regularization term:
– Tikhonov cost function
– Total variation
– Bilateral filter
( ) { }XAYXXN
kkkMAP λε +−=∑
=1
22 DHF
{ } 2XXAT Γ=
{ }1
XXATV ∇=
{ } ∑ ∑−= −=
+ −=P
Pl
P
Pm
my
lx
mlB XSSXXA
1α
43
Robust Estimation + Regularization
( ) ∑ ∑∑−= −=
+
=
−+−=P
Pl
P
Pm
my
lx
mlN
kkk XSSXYXX
11
12 αλε DHF Minimize:
( )
[ ] ( )⎭⎬⎫
−−+
⎩⎨⎧
−−=
∑ ∑
∑
−= −=
−−+
=+
P
Pl
P
Pmn
my
lxn
my
lx
ml
N
kknk
TTTknn
XSSXSSI
YXXX
ˆˆsign
ˆsignˆˆ1
1
αλ
β DHFDHF
44
Example
• 8 frames• Resolution factor of 4
From Farisu at al. IEEE trans. On Image Processing, 0445
Example
Images from Vigilant Ltd.46
Handling Color in SR
( ) { }XAYXXN
kkkMAP λε +−=∑
=1
22 DHF
Handling color: the classic approach is to convert the measurements to YCbCr, apply the SR on the Y and use trivial interpolation on the Cb and Cr.
Better treatment can be obtained if the statistical dependencies between the color layers are taken into account (i.e. forming a prior for color images).
In case of mosaiced measurements, demosaicing followed by SR is sub-optimal. An algorithm that directly fuse the mosaic information to the SR is better.
47
20 images, ratio 1:4
SR for Full Color
48
20 images, ratio 1:4
Mosaiced input
Mosaicing and then SR Combined treatment
SR+Demosaicing
49
Super-Resolution - Agenda
• The basic idea• Image formation process• Formulation and solution• Special cases and related
problems• SR in time• SR from Examples
50
Special Case – Translational Motion
• In this case H and F commute (block circulant):
• SR is decomposed into 2 steps1. Find blur HR image from LR images non-iterative2. Deconvolve the result using H iterative
XZVZVXVXY
kk
kk
kkk
HDFHDF
DHF
=+=+=+=
TTk
Tk
Tkk HFFHHFHF ==
51
Intuition
XZVZY kkk HDF =+=
X Z=PSF*X
• Using the samples can, at most, reconstruct Z• To recover X, need to deconvolve Z
52
Step I – Find Blurred HR
• L2 For all frames, copy registered pixels to HR grid and average [Elad & Hel-Or, 01]
• L1 For all frames, copy registered pixels to HR grid and use median [Farisu, 04]
( ) ∑=
−=N
kkkML YZZ
1
22 DF εMinimize:
53
Solution for L2
( ) ∑=
−=N
kkkML YZZ
1
22 DF εMinimize:
⎪⎪⎭
⎪⎪⎬
⎫
⎪⎪⎩
⎪⎪⎨
⎧
=
=
∑
∑
=
=N
kk
TTk
N
kk
TTk
YP1
1
DF
DFDFR
PZ =ˆR
( ) 02
=∂
∂Z
ZMLεThus, require:
Sum of HR grid
Diagonal, number Of occurrences
per HR grid
54
Step II - Deblur
( ) { }XAZXX λε +−=2
2 HMinimize:
( ) { }⎭⎬⎫
⎩⎨⎧
∂∂
+−−=+ nn
knT
nn XAX
ZXXX ˆˆ
ˆˆˆ1 λβ HH
55
Example
From Pham at al. Proc. Of SPIE-IS&T, 05. Simulated.
64X64 LR 256X256Before deblur
256X256After deblur
56
Super-Resolution - Agenda
• The basic idea• Image formation process• Formulation and solution• Special cases and related problems• SR in time• SR from Examples
57
“Classical” Image Super-Resolution
Low-resolution images:
High-resolution image:
Scene:
58
time
time
time
Space-Time Super-Resolution
Super-resolution in space and in time.
time
High space-time resolution sequence: time
Low-resolution imagesvideo sequences:
59
What is Super-Resolution in Time?
Observing events “faster” than frame-rate.
• Handles:(1) Motion aliasing (2) Motion blur
• Application areas: - sports scenes- scientific imaging- etc...
60
(1) Motion Aliasing
The “Wagon wheel” effect: Slow-motion:
time
Continuous signal
time
Sub-sampled in time
time
“Slow motion”61
(2) Motion Blur
62
(2) Motion BlurCamera 1: longexposure-time
Camera 2: shortexposure-time
Motion-blur –A Spatial artifact caused by Temporal blurring.
exposure time
time
Operationof camera:
time
1/frame-rate 1/frame-rate 63
lnSlS1
Sh(xh,yh,th)
Space-Time Super-Resolution
x
y t
yx
t
Blur kernel:
PSF
Exposure time
T
64
Input 1 Input 2
Input 3 Input 4
Example: Motion-Aliasing
25 [frames/sec]65
Input sequence in slow-motion (x3):
75 [frames/sec]
SuperSuper--resolutionresolution in time (x3):
75 [frames/sec]
Example: Motion-Aliasing
66
Output trajectory:
Without estimating motion of the ball!
Output sequence:
(x15 frame-rate)
Deblurring:
Input:
Output:
3 out of 18 low-resolution input sequences (frame overlays; trajectories):
67
Frames at collision:
4 input sequences:
Output frame at collision:
Video 1
Video 3
Video 2
Video 4
Example: Motion-Blur (real)
68
Super-Resolution - Agenda
• The basic idea• Image formation process• Formulation and solution• Special cases and related problems• SR in time• SR from Examples
69
Example Based Super Resolution
• Image is complex for a generative model to model
• Non-parametric methods– Texture synthesis [Efros99],[Feng04]– Motion synthesis [Li02]– Resolution synthesis
Example Based Super Resolution
Generate a high-resolution image from a single low-resolutionimage, with the help of a set of training images.
Slides from Gu, Yu, Tian, Belokrylov 2006
Training Set
Training Set(Low-High resolution image pairs)
Example Based Super ResolutionOutput HR
Input LR
ExtensionsSuper-resolution Through Neighbor Embedding(Manifold learning)
1. For each patch x in image LR image:• (a) Find K nearest neighbors in the training set t1…tk.• (b) Compute the reconstruction weights w1…wk of the
neighbors that minimize the error of reconstructing x.
• (c) Compute HR y as a weighted linear combination of the corresponding high-resolution patches Ti using the weights w1…wk.
2
...1
min ∑−i
iiwwtwx
k
∑=i
iiTwy
H. Chang, D.Y. Yeung, Y. Xiong, “Super-resolution through neighbor embedding”, 2004
Super-resolution Through Neighbor Embedding(Manifold learning ,LLE)
w1 w2 w3 w4 w5
Training Set
3x3
9x9
* * * * *+ + + +
≈
w1 w2 w3 w4 w5* * * * *
+ + + + ≈
Super-resolution Through Neighbor Embedding(Manifold learning ,LLE)
True HR image Neighborhood Embedding
Interpolation
Input LR image
Super-resolution Using High – Low Frequencymatching pairs
Basic idea– Divide the image into frequencies (L,M,H)– Construct database to store the relations of mid-
frequency and corresponding high frequency– Interpolation for initial magnification– Add high frequency details by searching in the
training database
Freeman, Jones, Pasztor “Example Based Super Resolution”, 2002
Training phase
Blur downsample interpolation
filter
High-frequency
Mid-frequency
0.0751 0.1238 0.07510.1238 0.2042 0.12380.0751 0.1238 0.0751
⎡ ⎤⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦
1/ 9 1/ 9 1/ 91/ 9 8 / 9 1/ 91/ 9 1/ 9 1/ 9
− − −⎡ ⎤⎢ ⎥− −⎢ ⎥⎢ ⎥− − −⎣ ⎦
Data Set
Low resolution patches
High resolution patches
Retrieval phase
interpolation filter
Mid-frequency
High-frequency
1/ 9 1/ 9 1/ 91/ 9 8 / 9 1/ 91/ 9 1/ 9 1/ 9
− − −⎡ ⎤⎢ ⎥− −⎢ ⎥⎢ ⎥− − −⎣ ⎦
+
Input LR
Output HR Training Set
Super-resolution Using High – Low Frequencymatching pairs
True HR image High-Low Frequency
Interpolation
Input LR image
Super-resolution Using High – Low Frequencymatching pairs
Interpolation Output HR image
• Weighted Manifold restores high res patch by computing weighted combination of high res patches in training set.
• High-Low Freq Example Based Algorithm finds the best high res patch in Training Set whose low res is most similar to the patch.
Low-High Freq Weighted Manifold
Super-resolution Comparison
Overlapping of PatchesFor each patch embed the computed high res patch. Use overlapping patches.
1. For each pixel in overlap, compute average over all high res pixels.
2. Use Optimal Cut to combine overlapping patches.
B1 B2
Overlap Average
B1 B2
Minimal errorboundary cut
Overlapping of Patches3. “Grow” image and apply neighboring high res constraints.
a. Choose k best matches for patch and select that which best fits the neighboring constraints.
b. Select k best matches and use MRF to select patches so that neighboring patches are smooth.
SR by Examples –Training set limitation
• It might seem that SR of an image of one feature (eg, a cat) requires a training set that contains these features (images of other cats). However, this isn’t the case.
SR by Examples –Training set limitation
Although the training set doesn’t have to be very similarto the image to be enlarged, it should be in the same imageclass—such as text or color image.
SR from a Single Image• SR based on multiple LR images• SR based on Database of Images.
• Can SR be obtained from a single image?
?
SR from a Single Image
Use repetitions in single image.
SR from a Single ImagePatches in image are found at different locations and scale.Use different scale to predict what high res of patch should look like.
Glasner, Bigon & Irani “Super resolution from a Single Image”, 2010
SR from a Single ImageHR from many LR images – LR patches induce constraints on HR image.
Multi patch LR in single image induces same kind of constraints.
Multi scale patch in single image form an “Example”.
SR from a Single ImageMulti Resolution approach combines 2 approaches:
1) Multi LR Images to High res2) SR based on Examples
SR from a Single Image
Glasner, Bigon & Irani “Super resolution from a Single Image”, 2010
SR from a Single Image
Glasner, Bigon & Irani “Super resolution from a Single Image”, 2010
Bicubic x3 SR from Single Image x3