4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line

TECHNIONIsrael Institute of Technology

Department of Electrical EngineeringThe Vision Research and Image Science Laboratory

By

Elad Aharoni Guy Abramovich

Supervised by

Dr. Michael Elad

Summer 1998 - Winter 1999

Supported by the Ollendorff Research Center Fund

4/99 Super-Resolution 2

Super-resolution

Given a collection of measured images of thesame scene, it is possible to fuse them to a onehigher resolution image, called the "SuperResolution Image"

Definition

Project Goal

1. To reconstruct a super-resolution image, whilepreserving its edges

2. Develop fast algorithm for the pure translationmotion case


Out Line

Part One

1. Simulate the creation of measured images

2. The super-resolution as a quadratic problem

3. The basic Steepest-Descent algorithm

4. Improvements of the SD algorithm

5. The Normalized-Steepest-Descent algorithm

6. The Conjugate-Gradient algorithm

7. Regularization

8. Simulations and Results

Part Two

1. Motion estimation algorithm

2. Getting input images

3. Estimating the super-resolution image


Simulate Creation Process of

Target Images

Warp1F

1

Blur1

1H

Decimate1D

1

noise 1

1 measuredlow resolutionimage

st

1Y

Warp NF

N

Blur N

NH

Decimate ND

N

noise N

N measuredlow resolutionimage

th

NY

sourcehigh - resolutionimage X

Creation of Yk images

Images Yk1k

Nmr supposed to be taken by CCD

camera

We assume Yk kNmr1are different representations of a

single high resolution image X of size LL


Simulate Creation Process of Target Images (Cont.)

M o d e l i n g o f c r e a t i o n p r o c e s s : Yk

Dk

Hk

Fk

X Ek

Fk

L L 2 2 g e o m e t r i c w a r p i n g

Hk

L L 2 2 l i n e a r - s p a c e i n v a r i a n t b l u r

Dk

M L 2 2 d e c i m a t i o n o p e r a t o r

Ek

M 2 1 a d d i t i v e z e r o m e a n G a u s s i a n n o i s e

Yk

M 2 1 v e c t o r , C S r e p r e s e n t a t i o n o f t h e m e a s u r e d

i m a g e s

X L 2 1 v e c t o r , C S r e p r e s e n t a t i o n o f t h e

s u p e r - r e s o l u t i o n i m a g e

All matrices assumed to be known in advance


Simulate Creation Process of Target Images (Cont.)

Assumptions: Hk = H

Dk = D

Fk – Pure displacement.

G r o u p i n g N e q u a t i o n s i n t o o n eY

YN

D H F

DN

HN

FN

X

E

E N

C

CN

X E

1 1 1 1 1 1

L

N

MMMMM

O

Q

PPPPP

L

N

MMMMM

O

Q

PPPPP

L

N

MMMMM

O

Q

PPPPP

L

N

MMMMM

O

Q

PPPPP

w h e r e C D H Fk k k k

Final result YCXE


Creation of Yk Images from Source

Image

Source Image Yk Images


The Super-resolution Problem as Quadratic Problem

YCXE Classic model of restoration problem

Given a solution X the Error function:

E Y C Xrr

2 2

D if f e r e n t i a t i n g w i th r e s p e c t t o X a n d e q u a t i n g t o z e r o :

2 0 C Y C XT a fC Y C C XT T c h

t h e n :

X C C C YT T c h1

P C YT - ו R C CT

R X P

Well known classic pseudoinverse result XRP1

Practically impossible to invert

Requires alternative solutions – iterative algorithms

R - Size of L L2 2 2 2400 400 c h


Steepest-Descent Algorithm

Described by the following equations:

X X R X Pj j 1

R F H D D H F

P F H D Y

kT

k

NT T

k

kT T

k

NT

k

1

1

F1 H D DT HT FT1

XK

H D DT HTFN FNT

Y1

XK1

XK

YN

Xk

Xk

FjT

j

NHT DT D H F

jX

kY

j IKFH1

1

The basic Steepest-Descent algorithm

Error function (quality factor):

2 2

1

Y DHF Xk k

k

N

Relatively slow (H implemented as convolution)

constant chosen manually: =0.1 – slowconvergence

=0.5 – fast convergence


Restoration Results ofSteepest-Descent Algorithm


Enhanced Steepest-Descent Algorithm

Basic algorithm is very slow

Blurring operator is equal for all images

F i r s t e n h a n c e m e n t : F k a n d H a r e c o m m u t a t i v e

T h e n :

2 2

1 1

1

LNM OQP LNM OQP

Y D H F X R H F D D F H

P H F D Y

k kk

NT

kT T

kk

N

Tk

T

k

NT

k

A n d t h e n e w e q u a t i o n :

X X H F D D F H X Yk kT

kT T

j

N

j k j

11

a f


Enhanced Steepest-Descent Algorithm (Cont.)

Blurring operator performed twice instead of 2N in

every iteration

Save of 30% in run-time while retaining result

F1 D DT

HT

FT1

XK

D DTFN FNT

Y1

XK1

XK

YN

H

Commutative warping and blurring operators



S e c o n d e n h a n c e m e n t :

R H F D D F H H R HTk

T Tk

k

NT LNM OQP

10

R F D D FkT T

kk

N

01

P H F D Y H PTk

T Tk

k

NT LNM OQP

10

P F D YkT T

kk

N

01

T h e n e w e q u a t i o n : X X H R H X Pj jT

j 1 0 0

Matrices P0 and R0 calculated once only for the

whole algorithm

If warping is based on Nearest-Neighbor then:

R0 is diagonal

R0 implemented as mask



XK H R0

P0

HT XK1

The enhanced Steepest-Descent algorithm

Save of 80% in run-time while retaining result,compared to the original SD algorithm


The Normalized SD algorithm

SD algorithm - constant chosen by user

Normalized - SD: calculates optimal for every

iteration

X X Ej j j j 1

E R X Pj j

and jjT

j

jT

j

E E

E R E

1

Calculation of optimal adds 20% to run-time


Restoration Results of NSD Algorithm


Conjugate Gradient Algorithm

Complicated calculations

Comparison to NSD algorithm:

On synthetic square problem

On our current simulation

T h e C G a l g o r i t h m : G i v e n R , P , X k a n d V k - 1

1a f E R X Pk k

2 11

1 1

a f kkT

k

kT

k

E R VV R V

3 1 1a f V E Vk k k k

4a f kkT

k

kT

k

E VV R V

T h e f i n a l e q u a t i o n : X X Vk k k k 1


Conjugate Gradient Algorithm (Cont.)

E KK

K 1 V K

1

MUL

2 3

4

K KV XK1

XK

P

R

-

+

VK 1

The Conjugate-Gradient algorithm

CG iteration run-time is 50% longer than in the NSD


Comparison of CG vs. NSD

Comparison parameters: Error function 2 xafNorma X X true 2

Both algorithms converge to same result

The norma factor getting worse because of ringing effect

Xtrue calculation is explained later


Comparison of CG vs. NSD (Cont.)

Concentrating on the first few iterations

CG vs NSD in the first few interations


Comparison of CG vs. NSD (Cont.)

CG and NSD after 7 iterations

CG converges faster - damaged by stronger ringing effect


Regularization

Ringing effect damages images as we converge

Possible reasons: True R matrix is not inversible

Error function doesn't reflect realminimization

We're interested in - min X Xrestore true 2

Regularization component in the error function

2 2

1

2

Y D H F X D Xk k

k

N ~

Differentiating with respect to X and equating to zeroyields:

R F H D D H F D D

P F H D Y

kT T T

kk

NT

kT T

kT

k

N

1

1

~ ~

~~DDT- Laplasian operator

- determines the regularization scale. Constant chosen

manually


Regular Regularization

XK H R0

P0

HT XK1

~ ~D DXT

K

Regular regularization implementation in NSD

What is the optimal ?

Run for various values of :

Convergence factor for various values of

Optimal result for =0.6


Regularization Results for Various Values of


Comparison Between CG and NSD Including Regularization

Conclusion: CG and NSD converge to same resultCG converges faster in the first fewiterations


Adaptive Regularization

Is it possible to implement regularization while

preserving edges?

W e ig h te d re g u la riza tio n c o m p o n e n ts in th e e rro rfu n c tio n :

2 2

1

2

Y C X D Xk k

k

N

W

~

D iffe re n tia tin g w ith re sp ec t to X an d e q u a tin g to z e roy ie ld s :

R F H D D H F D W D

P F H D Y

kT T T

kk

NT

kT T

kT

k

N

1

1

~ ~

Implementation of adaptive regularization in NSDalgorithm:

XK H R0

P0

HT XK1

~ ~D WDXT

K


W Matrix

Prevents edges from being regularized

Detects edges by calculating derivatives

Built according to derivatives values

Implemented as mask

a

W

b ערךרת גז הנ

1

a

W

ערךרת גז הנ

1

a

W

b ערךרת גז הנ

1

Different threshold functions

Image of W mask


W Matrix (Cont.)

Low threshold High threshold

Adaptive regularization Regular


Motion Estimation

Relative translation is not known

M o tio nE stim atio n ן בי ת סי ח י זה ו תז

ת ונו מ ת שתי ה

I (x, y)1

I (x, y)2

(dx, dy)

Algorithm based on Taylor series

Accurate for small warping (0-2 pixels)

We deal with large warping (0-10 pixels)

Implement iterative algorithm


Motion Estimation Implementation as Iterative Algorithm

M o ti o nE stim ati o n

I (x, y)1 I (x, y)2

I nv erseW arp ing


I nv erseW arp ing


Tota lW arp ing(dx , dy)

(dx1 , dy1)

(dx2 , dy2)

(dxn , dyn)


Input Images

Take 60 images from video sequence

Create difference images

Choose best 15 with minimal rotation and scaling

Difference images including rotation and scaling

Difference images including minimal rotation and scaling


Final Results

Implemented in NSD algorithm

Specific blurring kernel of CCD camera -

Regularization parameter -

Optimal result for =1.8


Final Results (Cont.)

Optimal Result for =1.8 and =2


Final Results (Cont.)

Zoom on the final results

Good obtained results

Insufficient due to technical problems:

Strong anti-aliasing effect

Rotation and scaling


Aliasing Effect

The Original Image Sampled Image The Restored Image


The original images decimated in order to "add"aliasing effect

Very strong anti-aliasing effect caused by the camera


Aliasing Effect (Cont.)

No resolution improvement in the result image

compared to the original

Quality improvement in the result image compared to

the original



Summing

We developed a fast version algorithm forSuper-Resolution Reconstruction, assuming puredisplacement between the images

Simulations on synthetic data verified that the takenshortcuts are valid and produce 80% reduction nrun-time, while preserving output quality

Results on true video images show slightimprovement, due to too strong anti-aliasing effect

Documents

4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line