Super-Resolutioncs.haifa.ac.il/hagit/courses/CP/Lectures/CP08_SuperResolution.pdf · Special Case...

Super-Resolution

Many slides from Miki Elad – TechnionYosi Rubner – RTC and more

1

Example - Video

53 images, ratio 1:42

40 images ratio 1:4

Example – Surveillance

3

Example – Enhance Mosaics

4

5

Super-Resolution - Agenda

• The basic idea• Image formation process• Formulation and solution• Special cases and related problems• SR in time• SR from Examples

6

D

For a given band-limited image, the Nyquistsampling theorem states that if a uniform sampling is fine enough (≥D), perfect reconstruction is possible.

D

Intuition

7

Due to our limited camera resolution, we sample using an insufficient 2D grid

2D

2D

8

Intuition

However, if we take a second picture, shifting the camera ‘slightly to the right’we obtain:

2D

2D

9

Intuition

Similarly, by shifting down we get a third image:

2D

2D

10

Intuition

And finally, by shifting down and to the right we get the fourth image:

2D

2D

11

Intuition

It is trivial to see that interlacing the four images, we get that the desired resolution is obtained, and thus perfect reconstruction is guaranteed.

Intuition

12

What if the camera displacement is Arbitrary ? What if the camera rotates? Gets closer to the object (zoom)?

Rotation/Scale/Disp.

13

There is no sampling theorem covering this case

14

Rotation/Scale/Disp.

15

3:1 scale-up in each axis using 9 images, with pure global translation between them

A Small Example

Further Complications

• Complicated motion– perspective, local motion, …

• Blur– sampling is not a point operation– Spatially variant blur– Temporally variant blur

• Noise• Changes in the scene

16

Super-Resolution - Agenda

• The basic idea• Image formation process• Formulation and solution• Special cases and related problems• SR in time• SR from Examples

17

Image Formation

Scene Image

HR LRk

Can we write these steps as linear operators?

Geometrictransformation

kF

OpticalBlur

kH

Sampling

kD

HRLRk ⋅= kkk FHD 18

Geometric Transformation

• Any appropriate motion model• Every frame has different transformation • Usually found by a separate registration algorithm

Scene Geometrictransformation

kF

19

Geometric Transformation

Can be modeled as a linear operation XkF

kF

X

=

kF XXkF

20

Optical Blur

• Due to the lens PSF and pixel integration• Usually

Geometrictransformation

OpticalBlur

kH

HH =k

21

H

PSF PIXEL H* =

22

Optical Blur

Can be modeled as a linear operation XH

H

X

=

XHXH

23

Sampling

• Pixel operation consists of area integration followed by decimation• D is the decimation only• Usually

Optical Blur Sampling

kD

DD =k

24

Decimation

Can be modeled as a linear operation XD

D

X

=

X

0101

...01

0101 o

oD

1

XD

25

Image Formation

Scene Image

HR LR

Geometrictransformation

kF

OpticalBlur

kH

Sampling

kD

HRLR ⋅= kkk FHD26

Image Formation

Scene Image

HR LRk

N HRLRk +⋅= kkk FHD27

+Noise

N

Super-Resolution - Agenda

• The basic idea• Image formation process• Formulation and solution• Special cases and related problems• SR in time• SR from Examples

28

Super-Resolution - Model

{ }N

knkkkkkk VVXY

1

2,0~ , =⎭

⎬⎫

⎩⎨⎧

+= σNFHD

X

High-Resolution

ImageH

H

Blur

1

N

F =I1

FN

Geometric warp

D

D1

N

Decimation

V 1

V N

Additive Noise

Y1

YN

Low-ResolutionExposures

29

Simplified Model

{ }N

knkkkk VVXY

1

2,0~ , =⎭

⎬⎫

⎩⎨⎧

+= σNDHF

X

High-Resolution

ImageH

H

Blur

F =I1

FN

Geometric warp

D

D

Decimation

V 1

V N

Additive Noise

Y1

YN

Low-ResolutionExposures

30

The Super-Resolution Problem

• GivenYk – The measured images (noisy, blurry, down-sampled ..)H – The blur can be extracted from the camera characteristicsD – The decimation is dictated by the required resolution ratioFk – The warp can be estimated using motion estimationσn – The noise can be extracted from the camera / image

• RecoverX – HR image

{ }2,0~ , nkkkk VVXY σNDHF +=

31

VX

V

VV

X

Y

YY

Y

NNNNN

+=

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

+

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

=

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

= G

FHD

FHDFHD

MMM2

1

222

111

2

1

The Model as One Equation

[ ][ ]

[ ]1 size of , size of

1 size of

22

222

2

×

×

×

MrVXMrNM

NMYG

r = resolution factorMXM = size of the framesN = number of frames

r = resolution factor = 4MXM = size of the frames = 1000X1000N = number of frames = 10

=[10M×1]=[10M×16M]=[16M×1] Linear algebra notation is

intended only to develop algorithm 32

SR - Solutions

• Maximum Likelihood (ML):

∑=

−=N

kkkX

YXX1

2minarg DHF

Smoothness constraintregularization

Often ill posed problem!

{ }XAYXXN

kkkX

λ+−= ∑=1

2 minarg DHF

• Maximum Aposteriori Probability (MAP)

33

ML Reconstruction (LS)

( ) ∑=

−=N

kkkML YXX

1

22 DHFεMinimize:

( ) ( ) 0ˆ21

2

=−=∂

∂ ∑=

N

kkk

TTTk

ML YXX

X DHFDHFεThus, require:

k

N

k

TTTk

N

kk

TTTk YX ∑∑

==

=⋅11

ˆ DHFDHFDHF

A B

BA =X̂34

LS - Iterative Solution

• Steepest descent

( )∑=

+ −−=N

kknk

TTTknn YXXX

11

ˆˆˆ DHFDHFβ

Simulated error

Back projection

All the above operations can be interpreted as operations performed on images.

There is no actual need to use the Matrix-Vector notations as shown here.

35

LS - Iterative Solution

• Steepest descent ( )∑=

+ −−=N

kknk

TTTknn YXXX

11

ˆˆˆ DHFDHFβ

nX̂

1ˆ

+nX

geometrywrap

convolvewith H

downsample

upsample

convolvewith HT

inversegeometry

wrap

kY

-β

-

kF H D TD TH TkF

For k=1..N

36

Example

Simulated example from Farisu at al.IEEE trans. On Image Processing, 04

HR image Least squaresLR + noiseX4

37

Robust Reconstruction

• Cases of measurement outlier:– Some of the images are irrelevant – Error in motion estimation– Error in the blur function– General model mismatch

38

Robust Reconstruction

( ) ∑=

−=N

kkk YXX

1

2 1

DHFεMinimize:

( )∑=

+ −−=N

kknk

TTTknn YXXX

11

ˆsignˆˆ DHFDHFβ

39

Robust Reconstruction

• Steepest descent

( )∑=

+ −−=N

kknk

TTTknn YXXX

11

ˆsignˆˆ DHFDHFβ

sign

For k=1..N

nX̂

1ˆ

+nX

geometrywrap

convolvewith H

downsample

upsample

convolvewith HT

inversegeometry

wrap

kY

-β

-

kF H D TD TH TkF

40

Example - Outliers

Simulated example from Farisu at al.IEEE trans. On Image Processing, 04

HR image LR + noiseX4

Least squares

Robust Reconstruction41

20 images, ratio 1:4

L2 norm based

Example – Registration Error

L1 norm based

42

MAP Reconstruction

• Regularization term:

– Tikhonov cost function

– Total variation

– Bilateral filter

( ) { }XAYXXN

kkkMAP λε +−=∑

=1

22 DHF

{ } 2XXAT Γ=

{ }1

XXATV ∇=

{ } ∑ ∑−= −=

+ −=P

Pl

P

Pm

my

lx

mlB XSSXXA

1α

43

Robust Estimation + Regularization

( ) ∑ ∑∑−= −=

+

=

−+−=P

Pl

P

Pm

my

lx

mlN

kkk XSSXYXX

11

12 αλε DHF Minimize:

( )

[ ] ( )⎭⎬⎫

−−+

⎩⎨⎧

−−=

∑ ∑

∑

−= −=

−−+

=+

P

Pl

P

Pmn

my

lxn

my

lx

ml

N

kknk

TTTknn

XSSXSSI

YXXX

ˆˆsign

ˆsignˆˆ1

1

αλ

β DHFDHF

44

Example

• 8 frames• Resolution factor of 4

From Farisu at al. IEEE trans. On Image Processing, 0445

Example

Images from Vigilant Ltd.46

Handling Color in SR

( ) { }XAYXXN

kkkMAP λε +−=∑

=1

22 DHF

Handling color: the classic approach is to convert the measurements to YCbCr, apply the SR on the Y and use trivial interpolation on the Cb and Cr.

Better treatment can be obtained if the statistical dependencies between the color layers are taken into account (i.e. forming a prior for color images).

In case of mosaiced measurements, demosaicing followed by SR is sub-optimal. An algorithm that directly fuse the mosaic information to the SR is better.

47

20 images, ratio 1:4

SR for Full Color

48

20 images, ratio 1:4

Mosaiced input

Mosaicing and then SR Combined treatment

SR+Demosaicing

49

Super-Resolution - Agenda

• The basic idea• Image formation process• Formulation and solution• Special cases and related

problems• SR in time• SR from Examples

50

Special Case – Translational Motion

• In this case H and F commute (block circulant):

• SR is decomposed into 2 steps1. Find blur HR image from LR images non-iterative2. Deconvolve the result using H iterative

XZVZVXVXY

kk

kk

kkk

HDFHDF

DHF

=+=+=+=

TTk

Tk

Tkk HFFHHFHF ==

51

Intuition

XZVZY kkk HDF =+=

X Z=PSF*X

• Using the samples can, at most, reconstruct Z• To recover X, need to deconvolve Z

52

Step I – Find Blurred HR

• L2 For all frames, copy registered pixels to HR grid and average [Elad & Hel-Or, 01]

• L1 For all frames, copy registered pixels to HR grid and use median [Farisu, 04]

( ) ∑=

−=N

kkkML YZZ

1

22 DF εMinimize:

53

Solution for L2

( ) ∑=

−=N

kkkML YZZ

1

22 DF εMinimize:

⎪⎪⎭

⎪⎪⎬

⎫

⎪⎪⎩

⎪⎪⎨

⎧

=

=

∑

∑

=

=N

kk

TTk

N

kk

TTk

YP1

1

DF

DFDFR

PZ =ˆR

( ) 02

=∂

∂Z

ZMLεThus, require:

Sum of HR grid

Diagonal, number Of occurrences

per HR grid

54

Step II - Deblur

( ) { }XAZXX λε +−=2

2 HMinimize:

( ) { }⎭⎬⎫

⎩⎨⎧

∂∂

+−−=+ nn

knT

nn XAX

ZXXX ˆˆ

ˆˆˆ1 λβ HH

55

Example

From Pham at al. Proc. Of SPIE-IS&T, 05. Simulated.

64X64 LR 256X256Before deblur

256X256After deblur

56

Super-Resolution - Agenda

• The basic idea• Image formation process• Formulation and solution• Special cases and related problems• SR in time• SR from Examples

57

“Classical” Image Super-Resolution

Low-resolution images:

High-resolution image:

Scene:

58

time

time

time

Space-Time Super-Resolution

Super-resolution in space and in time.

time

High space-time resolution sequence: time

Low-resolution imagesvideo sequences:

59

What is Super-Resolution in Time?

Observing events “faster” than frame-rate.

• Handles:(1) Motion aliasing (2) Motion blur

• Application areas: - sports scenes- scientific imaging- etc...

60

(1) Motion Aliasing

The “Wagon wheel” effect: Slow-motion:

time

Continuous signal

time

Sub-sampled in time

time

“Slow motion”61

(2) Motion Blur

62

(2) Motion BlurCamera 1: longexposure-time

Camera 2: shortexposure-time

Motion-blur –A Spatial artifact caused by Temporal blurring.

exposure time

time

Operationof camera:

time

1/frame-rate 1/frame-rate 63

lnSlS1

Sh(xh,yh,th)

Space-Time Super-Resolution

x

y t

yx

t

Blur kernel:

PSF

Exposure time

T

64

Input 1 Input 2

Input 3 Input 4

Example: Motion-Aliasing

25 [frames/sec]65

Input sequence in slow-motion (x3):

75 [frames/sec]

SuperSuper--resolutionresolution in time (x3):

75 [frames/sec]

Example: Motion-Aliasing

66

Output trajectory:

Without estimating motion of the ball!

Output sequence:

(x15 frame-rate)

Deblurring:

Input:

Output:

3 out of 18 low-resolution input sequences (frame overlays; trajectories):

67

Frames at collision:

4 input sequences:

Output frame at collision:

Video 1

Video 3

Video 2

Video 4

Example: Motion-Blur (real)

68

Super-Resolution - Agenda

• The basic idea• Image formation process• Formulation and solution• Special cases and related problems• SR in time• SR from Examples

69

Example Based Super Resolution

• Image is complex for a generative model to model

• Non-parametric methods– Texture synthesis [Efros99],[Feng04]– Motion synthesis [Li02]– Resolution synthesis

Example Based Super Resolution

Generate a high-resolution image from a single low-resolutionimage, with the help of a set of training images.

Slides from Gu, Yu, Tian, Belokrylov 2006

Training Set

Training Set(Low-High resolution image pairs)

Example Based Super ResolutionOutput HR

Input LR

ExtensionsSuper-resolution Through Neighbor Embedding(Manifold learning)

1. For each patch x in image LR image:• (a) Find K nearest neighbors in the training set t1…tk.• (b) Compute the reconstruction weights w1…wk of the

neighbors that minimize the error of reconstructing x.

• (c) Compute HR y as a weighted linear combination of the corresponding high-resolution patches Ti using the weights w1…wk.

2

...1

min ∑−i

iiwwtwx

k

∑=i

iiTwy

H. Chang, D.Y. Yeung, Y. Xiong, “Super-resolution through neighbor embedding”, 2004

Super-resolution Through Neighbor Embedding(Manifold learning ,LLE)

w1 w2 w3 w4 w5

Training Set

3x3

9x9

* * * * *+ + + +

≈

w1 w2 w3 w4 w5* * * * *

+ + + + ≈

Super-resolution Through Neighbor Embedding(Manifold learning ,LLE)

True HR image Neighborhood Embedding

Interpolation

Input LR image

Super-resolution Using High – Low Frequencymatching pairs

Basic idea– Divide the image into frequencies (L,M,H)– Construct database to store the relations of mid-

frequency and corresponding high frequency– Interpolation for initial magnification– Add high frequency details by searching in the

training database

Freeman, Jones, Pasztor “Example Based Super Resolution”, 2002

Training phase

Blur downsample interpolation

filter

High-frequency

Mid-frequency

0.0751 0.1238 0.07510.1238 0.2042 0.12380.0751 0.1238 0.0751

⎡ ⎤⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦

1/ 9 1/ 9 1/ 91/ 9 8 / 9 1/ 91/ 9 1/ 9 1/ 9

− − −⎡ ⎤⎢ ⎥− −⎢ ⎥⎢ ⎥− − −⎣ ⎦

Data Set

Low resolution patches

High resolution patches

Retrieval phase

interpolation filter

Mid-frequency

High-frequency

1/ 9 1/ 9 1/ 91/ 9 8 / 9 1/ 91/ 9 1/ 9 1/ 9

− − −⎡ ⎤⎢ ⎥− −⎢ ⎥⎢ ⎥− − −⎣ ⎦

+

Input LR

Output HR Training Set

Super-resolution Using High – Low Frequencymatching pairs

True HR image High-Low Frequency

Interpolation

Input LR image

Super-resolution Using High – Low Frequencymatching pairs

Interpolation Output HR image

• Weighted Manifold restores high res patch by computing weighted combination of high res patches in training set.

• High-Low Freq Example Based Algorithm finds the best high res patch in Training Set whose low res is most similar to the patch.

Low-High Freq Weighted Manifold

Super-resolution Comparison

Overlapping of PatchesFor each patch embed the computed high res patch. Use overlapping patches.

1. For each pixel in overlap, compute average over all high res pixels.

2. Use Optimal Cut to combine overlapping patches.

B1 B2

Overlap Average

B1 B2

Minimal errorboundary cut

Overlapping of Patches3. “Grow” image and apply neighboring high res constraints.

a. Choose k best matches for patch and select that which best fits the neighboring constraints.

b. Select k best matches and use MRF to select patches so that neighboring patches are smooth.

SR by Examples –Training set limitation

• It might seem that SR of an image of one feature (eg, a cat) requires a training set that contains these features (images of other cats). However, this isn’t the case.

SR by Examples –Training set limitation

Although the training set doesn’t have to be very similarto the image to be enlarged, it should be in the same imageclass—such as text or color image.

SR from a Single Image• SR based on multiple LR images• SR based on Database of Images.

• Can SR be obtained from a single image?

?

SR from a Single Image

Use repetitions in single image.

SR from a Single ImagePatches in image are found at different locations and scale.Use different scale to predict what high res of patch should look like.

Glasner, Bigon & Irani “Super resolution from a Single Image”, 2010

SR from a Single ImageHR from many LR images – LR patches induce constraints on HR image.

Multi patch LR in single image induces same kind of constraints.

Multi scale patch in single image form an “Example”.

SR from a Single ImageMulti Resolution approach combines 2 approaches:

1) Multi LR Images to High res2) SR based on Examples

SR from a Single Image

Glasner, Bigon & Irani “Super resolution from a Single Image”, 2010

SR from a Single Image

Glasner, Bigon & Irani “Super resolution from a Single Image”, 2010

Bicubic x3 SR from Single Image x3