Joint Image Registration and Super-Resolution From Low-Resolution Images With Zooming Motion

Copyright (c) 2013 IEEE Personal use is permitted For any other purposes permission must be obtained from the IEEE by emailing pubs-permissionsieeeorg

This article has been accepted for publication in a future issue of this journal but has not been fully edited Content may change prior to final publication

Joint Image Registration and Super-resolution from Low-resolution Images with Zooming Motion

Yushuang Tian and Kim-Hui Yap

AbstractmdashThis paper proposes a new framework for joint image registration and high-resolution (HR) image reconstruction from multiple low-resolution (LR) observations with zooming motion Conventional super-resolution (SR) methods typically formulate the SR problem as a two-stage process namely image registration followed by HR reconstruction An important step in image SR is the effective estimation of motion parameters However the registration algorithms in these two-stage processes experience various degrees of errors This could degrade the quality of subsequent HR reconstruction In view of this this paper presents a new approach that performs joint image registration and SR reconstruction The proposed iterative SR framework enables the HR image and motion parameters to be estimated simultaneously and progressively This could increase the potential SR improvement as more accurate estimates of motion parameters could be obtained iteratively Experimental results show that the proposed method is effective in performing image registration and SR for simulated as well as real-life images and videos

Index Termsmdash Image super-resolution zooming image registration

I INTRODUCTION Image super-resolution (SR) is a process to reconstruct a

high resolution (HR) image by fusing a sequence of low resolution (LR) observations Typically subpixel motion exists amongst these LR images and the unique partial information captured in each LR observation can be combined to produce a HR image [1] Image SR has many applications which include medical imaging remote sensing and video surveillance among others

Various SR techniques and algorithms have been proposed in recent years Early SR algorithms are commonly performed in the frequency domain [2]-[4] These methods center on the shifting and aliasing properties of the Fourier transform Although they are computationally attractive the constraint of motion model could limit their applications Therefore recent research [5]-[12] tends to concentrate on spatial domain techniques as a more generic degradation model can be applied in the spatial domain These methods typically consist of two disjoint processes namely 1) image registration based

1

Copyright (c) 2012 IEEE Personal use of this material is permitted

However permission to use this material for any other purposes must be obtained from the IEEE by sending an email to pubs-permissionsieeeorg

Yushuang Tian is with the School of Electrical and Electronic Engineering Nanyang Technological University Nanyang Avenue Singapore 639798 (phone (65) 6790 6547 fax (65) 6793 3318 e-mail tian0042 entuedusg)

Kim-Hui Yap is with the School of Electrical and Electronic Engineering Nanyang Technological University Nanyang Avenue Singapore 639798 (phone (65) 6790 4339 fax (65) 6793 3318 e-mail ekhyapntuedusg)

on a specific motion model to calculate the motion parameters followed by 2) HR reconstruction that incorporates the estimated motion parameters into an inverse estimation Generally accurate registration is a challenging task for image SR reconstruction as motion parameters are often calculated from the captured aliased LR images The aliasing effect among LR images may reduce the accuracy of registration Therefore various SR algorithms have been proposed to alleviate the effect of registration error on the final estimated HR image

A number of SR methods [13]-[16] employ various techniques to improve HR estimation with the existing registration error In [14] Farsiu et al proposed a L1-norm SR algorithm which is able to suppress the artifacts due to the registration error He et al [16] proposed an algorithm that takes into account the inaccurate estimates of registration parameters and models them in the HR reconstruction Other works try to tackle the problem by improving the estimation of motion parameters [17] [18] [19] In [17] Hardie et al proposed an iterative scheme based on alternating minimization (AM) to estimate the HR image and motion parameters alternatingly A maximum a posteriori (MAP) based cost function is projected onto the HR image domain and motion parameter domain one a time and minimized iteratively In [19] Robinson et al proposed a variable projection (VP) based SR method and further developed an efficient Fourier domain implementation of the motion estimation for global translational motion model Typically these SR approaches consider registration and HR reconstruction as two disjoint processes Moreover they are designed under the assumption that all LR images have the identical spatial resolution This may reduce the potential SR improvement for such cases when there is relative zooming motion between LR images as illustrated in Fig 1 It is difficult to estimate the accurate registration parameters of captured LR images by using a two-dimensional (2D) in-plane motion model Hence this motivates the study of techniques that can be used to deal with zooming motion between the captured images including optical flow techniques [11] and parametric motion models [20]-[23]

Amongst the existing zooming SR literatures Li proposed a SR approach for synthetic zooming [20] In his work all LR images are related to each other by employing a line-geometry model and the zooming factors are estimated based on this model Joshi et al [21] presented a zooming SR algorithm using both the Markov random field (MRF) as well as a simultaneous autoregressive (SAR) model under the assumption that the SR field is homogenous Similar to [22] the HR image is estimated from multiple LR images of the same scene with continuous zooming Recently a zooming-



based SR approach with total variation (TV) prior has been proposed in [23] Generally these methods can be considered as the two-stage disjoint SR methods as discussed previously

In view of this this paper proposes an iterative method for joint image registration and SR reconstruction from LR observations with relative zooming motion The proposed method integrates image registration and zooming SR reconstruction into a single estimation process An iterative technique serves to ensure that progressively better estimates of the HR image and motion parameters can be obtained This is more promising when compared with other two-stage SR methods as the estimation of motion parameters could benefit from the information of reconstructed HR image Different from conventional methods such as [24] focus mainly on translation and possibly rotation the proposed method takes into account the relative zooming between the acquired LR images The adopted motion model enables the proposed method to handle more real-life applications involving motion containing translation rotation and zooming Further the proposed method also develops an adaptive weighting scheme and incorporates it into the HR reconstruction As the captured LR images with relative zooming motion provide different degrees of information to the HR reconstruction the adopted weighting strategy takes this into consideration

2

The rest of this paper is organized as follows The problem formulation of image SR is introduced in Section II The proposed cost function is presented in Section III An iterative algorithm using the regularized nonlinear least squares technique is developed in Section IV Experimental results on simulated and real-life images are presented and discussed in Section V A brief conclusion is given in Section VI

II PROBLEM FORMULATION First of all let us consider the generation of LR images

under the motion model consisting of zooming translation and rotation The kth (1 le k le N ) LR image ( )kg u v of size

g gM Ntimes

( )can be modeled by zooming the original HR image

f x y of size f fM Ntimes by a zooming factor rotating it by an angle shifting it by a translational vector [

kl

kθ ]xk yks s

( )n u v

blurring the zoomed rotated and shifted HR image by point-spread function (PSF) ( and down-sampling the result by a decimation factor of ρ Finally the LR image is degraded by additive white Gaussian noise (AWGN) The process can be expressed as

)( )k ch h x yotimes

[14] [25]-[27]

(( ) )

( )

( cos sin sin cos )

( ) ( )

k

k k k k xk k k k k y

k c

g u v

kf l x l y s l x l y s

h h x y n u vρ

θ θ θ θ= minus + +

otimes otimes darr +

+

+

(1)

where is the 2D-convolution operator denotes the down-sampling operator with a decimation factor where is employed to represent the resolution ratio between the desired HR image and acquired LR image In this work we consider the decimation factor in the vertical and horizontal

directions to be identical represents the camera lens blur in the kth LR image represents the effect of spatial integration of light intensity over a square surface region to simulate the LR image captured by sensor

otimes ρdarr

ρ ρ

kh

g D

ch

As in the work we assume that the lens condition ( ) is known and identical for all LR images The imaging process in

kh

(1) can be expressed in a matrix-vector form as

( )= HS α f n (2)

where represents the lexicographically ordered original image

f( )f x y [ ]T=g g g

(

1 TN

k

T T and are the vectors representing the discrete concatenated and lexicographically ordered

[ ]T TN=n n n1

)g u v

T T1

and respectively is the down-sampling operator which is identical for all LR

images and

( )kn u vD

[=H H ]TNH kH (1 le k le N) denotes the

corresponding matrix representing the blurring operator The matrix is formed by nonlinear

differentiable functions of an unknown motion parametric vector a where a = [a1

T a2T aN

T]T Hence S(a) = [S(a1)T S(a2)T S(aN)T]T and (1 le k le N) denotes the geometric motion operator for the kth LR image represents the unknown motion parametric vector of the kth LR image and In this work we consider the initial estimates of motion parameters α to contain certain degrees of error This is a more realistic assumption as accurate registration for SR is difficult to achieve particularly in the early stage The objective of image SR in this context is to reconstruct the HR image from N LR observations with the unknown motion parametric vector

As the first LR image is used as the reference we need to estimate 4 times (N minus 1) motion parameters and the desired HR image

k otimes chh

α

( )α

( kα

k k kl l=

S

S

os

)

sik

kα

k

[ c nθ θ ] Tk xk yks s

f

α

III PROPOSED COST FUNCTION The main ideas of the proposed joint image registration

and zooming SR include integration of registration and SR into an iterative process fusion of multiple LR images with different resolutions and incorporation of useful a prior information for the desired HR image and unknown motion parameters Joint image registration and zooming SR is an ill-posed inverse problem Thus regularizations for both the HR image as well as the motion parameters are considered in order to achieve a stable solution

Given the image formation model in (1) the estimated HR image and the unknown motion parametric vector α can be obtained by minimizing the following cost function

f

( ) ( ( ) ) ( ) ( )E λ β= minus + +α f V g DHS α f T f R a2 (3)

where

and The matrix represents the motion operator consisting of zooming rotation and

[ T TN=g g g1

[ ]T TN=H H H1

]T (S α) ( ) ( ) ( )TT T T

N⎡ ⎤= ⎢ ⎥⎣ ⎦S α S α S α1 2

T ( )kS α



translation In this equation the first term in (3) represents the data fidelity of the estimated f and with respect to their true values represents the channel weights and (1 le k le N) is the indicator of reliability for the kth channel (the kth LR image) The second and third terms in

α]V V [ Ndiag=V V1 2

k kV=V I

(3) are regularization functionals that introduce stability into the estimates of HR image and motion parameter α respectively We adopt the total variation (TV) regularization for the estimate of HR image

f

f

( ) dxdy

3

Ω= nablaintT f f (4)

It is noted that we need to solve the nonlinear partial equations (PDEs) during the minimization of (3) due to the incorporation of TV norm ( )T f To alleviate this problem an auxiliary variable similar to that in m nμ [5] is incorporated to formulate

( )

( ( ( ))m n x y m n

)f x m y f x ydxdyΩ

=minus =

nabla + + minus= =

nabla sumsumsumint

T f

ff

2 21 1

1 0

nμ

(5)

where is defined as m nμ

( ( ) ( ))y 2m n f m y n f x γ+ minus2 2 x+ + + Here γ s a small constant to ensure the term is non-zero

gt 0[5] The fixed-

point technique in [28] is adopted to calculate the auxiliary variable by using the estimated HR image f from the previous iteration Therefore the right-hand side of

m nμ(5) can be

expressed in a matrix-vector form as

)

m n

( ( ))m n x y

T Tm n m n m n

m n

T T

( f x m yμ=minus =

=minus =

+ +

=

= =

sumsumsum

sumsum f U W U

f L Lf Lf

21 1

1 0

1 1

1 0

2

c

n

f

fminus x y

(6)

where matrix and vector c represents the lexicographically ordered

m n diag=W

m nμ1 represents the first-order derivative operator on the HR image Therefore the TV norm can be constructed as

m nU

( )T f Lf 2 using the half-quadratic scheme similar to the work in [5] [29] [30] During the minimization process in Section IV-C we only need to construct TL L

n m nU W

which can be calculated

by m n=minus

UTm

m n=sum sum1 1

1 0

( )R a is the regularization term for motion parameter

and it is denoted asα minusα a 2 where α presents the initial estimate of motion parametric vector In many SR approaches [24] [31] the initial motion estimates α by current registration approaches [34] [35] are close reasonably to their true values Various experimental results show that they are also sufficient to produce satisfactory convergence

and results in our algorithm λ and β are regularization parameters that serve to control the relative contribution between the data fidelity term and the regularization terms The proposed method incorporates registration and zooming SR into a single optimization process considering the estimation of zooming rotation and translation parameters This could increase the SR improvement for such applications where relative zooming exists among the tured LR images cap

IV ITERATIVE JOINT REGISTRATION AND ZOOMING SUPER-RESOLUTION

A Development of the proposed nonlinear least square method for joint image registration and SR In this section we will develop a new framework for

simultaneous image registration and zooming SR Existing zooming SR methods typically perform image registration on LR images followed by HR reconstruction However the HR reconstruction in these methods relies on the initial estimates of motion parameters heavily To alleviate this dependency we propose an iterative method to perform joint image registration and SR progressively The minimization problem (3) can be rewritten as

(

minus

α f

Lf

α

)

min

( )

λ

βα f

Vr

a

2

(7)

where i denotes the L2-norm and is the fidelity residual vector

(

)

)Δ

f f

f f

(

) ( )= minusr α f g DHS α f

The residual vector is linear with respect to but nonlinear with respect to α In order to solve this minimization problem we extend the principle of the nonlinear parametric estimation to derive a linear approximation for r Let denote a small change in the HR image and denote a small change in the motion vector α The residual vector can be linearized with respect to Δ and Δ as follows

( )r α f

)fΔα

f

f

( α Δf

+nabla f

f

r

r

)

nabla

f( )r α f

α

( )( ) ( ) ( ( )

( ) ( ) (

O+Δ +Δ

= +nabla Δ Δ + Δ Δ

+nabla Δ +nablaα

α

r α α f fr α f r α f α α α f

r α f r α f α α

(8)

where and are the gradient of with respect to α and respectively O is the higher-order term that is ignored The approximation in

( )nablaαr α f ( nabla f r α ff

( )r α f)

Δf

( Δ Δα f

α

(8) is used to transform a nonlinear problem into a linear problem with the assumption that the estimates of Δ and are sufficiently small

The gradient and ( )nablaαr α f )f r α f can be expressed as



( )

( )

( )( )

( )

T

T

part minusnabla =

partpart

=minuspart

α

g DHS α fr α f

αS α f

DHα

(9)

( )

( )

( )( )

( )

( )

T

T

part minusnabla =

part

part=minus

part=minus

f

g DHS α fr α f

fS α f

DHf

DHS α

(10)

4

Combining with (9) and (10) (8) can be expressed as

( ) ( ) ( )+Δ +Δ = minus Δ minus Δr α α f f r α f DHG α DHS α f (11)

where ( )( ) T=part partG S α f α and it denotes the derivative of HR image with respect to α ( )S α f

Let and where is the increment for current motion vector estimate and is the increment for current HR image estimate

i i= +Δα α α i= +Δf f f i iΔα

i

iα iΔff The

minimization problem (7) can be written as (12a) Next based on the first-order approximation in (8) (12a) can then be simplified as (12b)

(

min ( )

( )i i

i i i i i

i i

i i

λ

βΔ Δ

+Δ +Δ

+Δ

Δ + minusα f

V r α α f f

L f f

α α a

2)

(12a)

( ) ( )

min

( )i i

i i i i i i i

ii

ii

λ λ

β βΔ Δ

⎛ ⎞ ⎛ ⎞minus⎟ ⎟⎜ ⎜⎟ ⎟⎜ ⎜⎛ ⎞⎟ ⎟Δ⎜ ⎜⎟⎟ ⎟⎜⎜ ⎜⎟⎟ ⎟+⎜⎜ ⎜⎟⎟ ⎟⎜⎜ ⎜⎟⎟ ⎟⎜Δ⎝ ⎠⎜ ⎟ ⎜ ⎟⎟ ⎟⎜ ⎜⎟ ⎟minus⎜ ⎜⎝ ⎠ ⎝ ⎠α f

V DHG V DHS α V r α fα

L Lff

α α

2

0

0

(12b)

Therefore the original problem for direct estimation of HR image and motion parametric vector by minimizing if iα

i

(7) has been transformed into the minimization problem for increment and in (12) iΔf iΔα

In order to calculate the increment Δ and we need to focus on the derivation of an explicit expression for

f iΔα

( )( ) Tpart S α f αpart Since is nonlinear with respect to α it cannot be expressed as Therefore a linear

approximation for

( )S α f( )S α

(=f

)Xα

( )i ipart S α fT

αpart is developed to handle this problem Under the assumption that ldquoclose enoughrdquo initial estimates of the motion parameters are in hand an iterative technique that performs reconstruction of the HR image and estimation of motion parameters is developed The overview of the proposed algorithm is given in Fig 2 We will discuss the derivation of linear approximation for ( )( )part S α f Tαpart in the following section

B Derivation of linear approximation for ( )( ) Tpart partS α f α

To describe the process clearly we will first introduce the formation of the kth zoomed rotated and shifted HR images

The relative position between the kth zoomed rotated and shifted HR image and the reference HR image is shown in

( )kS α f

f( )kS α f

Fig 3 We denote ( )kf i j

)to be the

pixel of and (( )kα fS ( ) ( )x i j

)

y i j to be its corresponding coordinates in the reference HR image Thus f

( ( ) ( )x i j y i j can be defined as

( ) cos sin( ) sin cos

k k k k

k k k k

xk

yk

x i j l i l j sy i j l i l j s

θ θθ θ

= minus +

= + + (13)

Using bilinear interpolation the pixel ( )kf i j can be expressed as a linear combination of surrounding pixels of the reference HR image f

( )( ) ( )( )

( )( ) ( ) ( ) ( )

( ) ( ) ( ) ( )

k

k k tl k k tr

k k bl k k br

f i jd i j e i j f d i j e i j f

d i j e i j f d i j e i j f

= minus minus + minus

+ minus +

1 1 1

1

(14)

where and denotes the operator

rounding the number to the nearest integer less than or equal to itself

( ) ( ) floor( ( ))kd i j x i j x i j= minusfloor( ( ))y i j= minus floor( )i( ) ( )ke i j y i j

The derivation of linear approximation for ( )( ) Tpart S α f αpart will be given based on the formation of

discussed above We assume that HR images are independent to each other Therefore

( )kS α f( )k kS α f

( )( )part S α f Tpartα can be expressed as

( ) ( ) ( )( ) ( ) ( ) N

T TN

diag T

⎡ ⎤part part part⎢ ⎥= = ⎢ ⎥part part part⎣ ⎦

S α f S α f S α fG

α α1

1 α (15)

where ( )( ) Tkpart S α f αkpart of size is the derivative

of the kth HR image with respect to Based on

the chain rule

g gM Nρ times2 4

( )kS α f

)kα

( ( ) Tk kpartpart S α f α can be expressed as

( )( )

( )( ) ( )( )( )( ) ( )

TT T

kkT TT T

k k

⎡ ⎤part ⎢ ⎥part ⎣ ⎦part=

⎡ ⎤part partpart ⎢ ⎥⎣ ⎦

x i j y i jS α fS α fα αx i j y i j

(16)

where and are ( )x i j ( )y i j g gM Nρ2 length column vectors representing the lexicographically ordered displacement of

( )x i j and respectively ( y i )jFirstly we will introduce the derivation of

( ) ( )( ) ( ) ( )Tk

T⎡ ⎤part part ⎢ ⎥⎣ ⎦S α f x i j y i j For simplicity we use

kE to represent it in the following derivation As the direct derivation of kE is difficult to obtain we will utilize the chain rule again and divide it into two parts



( )

( )( )

( )( )

( ) ( )

TT Tk k

kk T T T T

k k

a b

⎡ ⎤part ⎢ ⎥part ⎣ ⎦=

⎡ ⎤ ⎡part part⎢ ⎥ ⎢⎣ ⎦ ⎣

d eS α fE

d e x i j y i j

5

⎤⎥⎦

TT T⎡ ⎤⎥⎦

(17)

Hence we can utilize the dependency between and provided in ( )kS α f ( ) ( )⎢⎣x i j y i j (13) and (14) to

derive kE To simplify the demonstration the calculations of part (a) and (b) in (17) will be explained in the Appendix A Here we only show the final derivation result as follows

(18) [ ( ) ( ) ( )

( ) ( ) ( )]k k bl tl k br

k tr tl k br bl

diagdiag


minus minus + minus

E e f f e fd f f d f f

1

1trf

Next we will determine ( )( ) ( )TT T

k⎡ ⎤part part⎢ ⎥⎣ ⎦x i j y i j α T

As illustrated in Fig 3 ( )kf i j

)

is denoted as the pixel of the kth zoomed rotated and shifted HR image and ( )kS α f ( )i j

( ( ) ( )x i j y i j are denoted as the corresponding coordinates of pixel ( )kf i j in the HR image and reference HR image f Using the matrix-vector form we can rewrite

( )kα fS

(13) as

( )( ) k

⎡ ⎤⎢ ⎥ =⎢ ⎥⎣ ⎦

x i jCα

y i j (19)

where and

and are the

⎡ ⎤minus⎢ ⎥= ⎢ ⎥⎣ ⎦

i j 1 0C

j i 0 1 ] T

k xk yks s i[ cos sink k k kl lθ θ=α j

g gM Nρ2

kα

length column vectors representing the lexicographically ordered displacement of i and j respectively From (19) it is clear that ⎡ ⎤⎢ ⎥⎣ ⎦ is linear with

respect to Thus

( ) ( Tx i j y i

()

Tj T

)( ) ( )T T⎡part ⎢⎣x i j y i jT T⎤ part⎥⎦ kα can be

expressed as

( ) ( )( ) ( )TT T

kT

k k

⎡ ⎤part ⎢ ⎥ part⎣ ⎦= =

part part

x i j y i j CαC

α α T (20)

Substituting (20) and (18) into (16) we can obtain a simple expression for ( )( ) T

kpart S α f αkpart which is EkC Finally the derivative of HR image with respect to

can be expressed as follows ( )S α f

α

( )( )T

part= =

part

S α fG

αEC (21)

where

N

⎡ ⎤⎢ ⎥⎢ ⎥= ⎢ ⎥⎢ ⎥⎣ ⎦

EE

E

1 0

0

and ⎡ ⎤⎢ ⎥⎢ ⎥= ⎢ ⎥⎢ ⎥⎣ ⎦

CC

C

0

0

C Optimization procedure In this section we will calculate the solution for the

minimization problem in (12) Substituting (21) into (12) the optimization problem in (12) is equivalent to solving the following equation

i i i=A f b (22)

where ( ) ( ) ( ) ( )

( ) ( )

( )

( ) ( ) ( )

T T T TT i T T i i i T i T T i i i

iT Ti T T T i i i i T T T i i i T

β

λ

⎛ ⎞⎟+⎜ ⎟⎜ ⎟⎜ ⎟=⎜ ⎟⎜ ⎟⎜ ⎟+ ⎟⎜⎝ ⎠

C E H D V V DHE C I C E H D V V DHS αA

S α H D V V DHE C S α H D V V DHS α L L

( ) ( )( )

( ) ( )

( ) ( )

T TT i T T i i i i i

iTi T T T i i i i T i

β

λ

⎛ ⎞⎟minus minus⎜ ⎟⎜ ⎟⎜= ⎟⎜ ⎟⎜ ⎟⎜ ⎟minus ⎟⎜⎝ ⎠

C E H D V V r α f α αb

S α H D V V r α f L Lf and

ii

i

⎛ ⎞Δ ⎟⎜ ⎟=⎜ ⎟⎜ ⎟⎜Δ⎝ ⎠

αf

f

The calculation of the closed-form solution for (22) requires the inversion of matrix Ai which is computationally intensive To alleviate this difficulty a numerical approach using preconditioned conjugate gradient optimization (PCG) is adopted [32] We use Matlabrsquos generic linear optimization routine pcg to calculate the final solution of (22) The convergence criterion of PCG is set as where

is the relative residual defined as relativeR minuslt 1210

relativeR i i i iminusb A f b

Finally the estimated and iΔα iΔf will be utilized to update the motion parametric vector and the reconstructed HR image in the next iteration

i i+ = +Δα α1

iΔf

iαi i= +f f+1

The overall complexity of the proposed algorithm can be analyzed by considering the two steps that dominate the computation namely construction of gradient matrix ( )( ) T

part S α f αpart and joint registration and SR In this analysis the cost of a multiplication operation is assumed to be the same as an addition operation in line with the assumption adopted in [9] In one iteration the computational costs of the two steps above are O(Mg Ng) and O(J (Mf Nf + 4 N)) respectively where J is the number of iterations in PCG optimization As Mg Ng and 4N lt Mf Nf implying that O( Mg Ng ) lt O( J (Mf Nf + 4 N) ) = O( J Mf Nf ) the overall complexity of the proposed algorithm is O( I J Mf Nf ) where I is the number of iterations for the algorithm It is noted that the complexity cost of traditional zooming SR method [23] is approximate O( I J Mf Nf ) Therefore the computational cost of the proposed method is comparable to the traditional zooming SR methods

D Estimation of Adaptive Channel Weights The weight (1 lek le N ) captures the reliability

information for the kth LR image into the HR reconstruction Various algorithms have been used to estimate including JMAP

kV

kV[33] which uses weighting arising from the distance of

the current frame to the reference In this work we consider different degrees of information rendered by the zoomed LR images to the final reconstruction Hence both the residual



errors and the zooming factors of LR images are incorporated into the weighting scheme Therefore we employ the following

( )average

kk k k

VV

l=

minusg DHS α f (23)

where is the average residual value defined

as

averageV

( )

N

k k k k

N

l= minussum g DHS α f1

1

Based on (23) it is noted that the channel weight is inversely proportional to the zooming factor and the residual value

kV

kl( )k kminusg DHS α f In each iteration is

calculated based on the current estimates of HR image and motion parametric vector α The new weight is then incorporated into

kVf

(22)

6

V EXPERIMENTAL RESULTS

In this section we will demonstrate the performance of the proposed method and compare it with other methods Four images in Fig 4 are selected as the test images We conduct various experiments and compare the results obtained by the proposed method with four other methods namely the L1-norm method [14] two zooming-based SR methods of the Joshirsquos [21] and the Ngrsquos [23] methods and the AM method [17] Finally real-life experiments are also conducted to illustrate the effectiveness of the proposed method The algorithm will continue until the following criterion is satisfied

i i

i

minusminus

minus

minuslt

f f

f

1

6

110 (24)

To evaluate the performance of the methods we employ normalized mean square errors (NMSE) for the estimated motion parametric vector and peak signal-to-noise (PSNR) for the estimated HR image respectively which are defined as follows

αf

ˆˆ( )NMSE

minusequiv

α αα

α

2

2100 (25)

255ˆ( ) logˆ

M NPSNR equiv

minus

f fff f

2

10 210 (26)

A HR Reconstruction for Multiple Images with Low-level Zooming Motion

Various experiments have been conducted to demonstrate the effectiveness of the proposed method in handling motion model that consists of rotation translation and zooming Based on the LR image generation model (1) it is noted that the LR imagersquos resolution is inversely proportional to its zooming factor In this section we will examine image SR

under low-level zooming motion when the zooming ratio of the highest resolution LR image to the lowest resolution LR image is between 1 and 11 This describes that the captured scene has a gradual resolution change amongst the LR images As only small low-level zooming motion is considered in this section we have included the Farsiursquos shift-and-add method in [14] for comparison which is a fast alternative to the main method in [14] when the motion is translational This serves to demonstrate the importance of including the zooming motion into the problem formulation even when the amount of zooming involved is small

The ldquoBridgerdquo image in Fig 4 (a) was selected as the test image To generate 9 LR images the HR image was zoomed by randomly selected zooming factors from a uniform distribution over [095 105] rotated by randomly selected angles from a uniform distribution over [-5deg 5deg] and shifted by randomly selected translations from a uniform distribution over [-3 3] pixels Without loss of generality we set the first LR image as the reference and its zooming factor as 1 The shifted HR image was then blurred by a 3times3 Gaussian blur to simulate the blur operator followed by a down-sampling operator with the decimation factor of Finally these LR images were degraded by additive white Gaussian noise (AWGN) to produce a signal-to-noise (SNR) ratio at 35dB

kh hotimes c

imate of

or clo

ρ = 2

The initial motion parameter estimation consists of rotation and zooming estimation by method [34] followed by translation estimation by method [35] Both methods [34] and [35] are frequency-domain based registration algorithms For further information please refer to references [34] and [35] We employ an effective approach to choose an order-of-magnitude est λ= and -810β = similar to the algorithm adopted in

minus410[5] The initial estimate of

HR image for our method can then be calculated by minimizing (3) For the Farsiursquos shift-and-add method we have conducted various experiments with regularization parameters ranging from 1times10-5 to 1 and find that the best parameter is λ=1times10-1 Therefore we have used the best regularization parameter to obtain the result for Farsiursquos shift-and-add method in Fig 5 4 samples of the LR images and the scaled-up version of the most zoomed in LR image are shown in Fig 5(a) and (b) respectively The result obtained using Farsiursquos shift-and-add method [14] is shown in Fig 5(c) It is noted that there are some artifacts near the edges This illustrates that relative zooming that exists among the LR images cannot be neglected even when it is small Next the Joshirsquos method [21] and our proposed algorithm are performed on the LR images and the results are given in Fig 5(d) and (e) respectively Comparison shows that the proposed method can achieve better HR reconstruction This is because the proposed method performs joint registration and zooming SR iteratively where more accurate registration parameters are incorporated into the HR reconstruction when compared with the Joshirsquos method [21] A selected region of the reconstructed HR images by the three methods is enlarged in Fig 5(f)-(h) f

ser examination The PSNRs of all methods above are given in TABLE I

From the table it is observed that the proposed method outperforms the other two methods We have also conducted a



7

motion model consisting of zooming rotation and translation

n for Multiple Images with Moderate

osed method is superior to the extended Hardiersquos AM method

C SR on LR Images with Various Motions

subjective test to evaluate the quality of reconstructed HR images The highest quality score was set 5 11 volunteers graded the reconstructed HR images by Farsiursquos shift-and-add method [14] Joshirsquos method [21] and our proposed method The average scores are given in TABLE II It is noted that the quality score of image obtained using the proposed method is higher than the quality scores of images obtained using Farsiursquos shift-and-add method and Joshirsquos method by 07 and 04 respectively The subjective test result is consistent with the objective measurement Both the objective performance measure and human visual evaluation further confirm the effectiveness of the proposed method in handling

B HR ReconstructioZooming Motion

In this section we focus on reconstruction of multiple LR images with moderate zooming motion This describes scenarios where there is a moderate change in the resolution of captured LR images eg a vehicle is moving towards the camera The moderate zooming motion is considered when the zooming ratio of the highest resolution LR image to the lowest resolution LR image is between 11 and 2 As the original Hardiersquos AM method [17] is designed based on the in-plane motion model Therefore in order to have a fair comparison we have extended the original Hardiersquos method by using our motion model consisting of zooming rotation and translation Various experiments have been conducted to verify the effectiveness of the proposed algorithm These experiments were performed on LR images with moderate zooming motion where the zooming factors were randomly selected from a uniform distribution over [075 125] The ldquoWindowrdquo image in Fig 4 (b) was selected as the test image The same experimental setup as in the former experiment was used 4 samples of the LR images are shown in Fig 6(a) The scaled-up version of the most zoomed in LR image is shown in Fig 6(b) The proposed algorithm is performed on the LR images and the result is given in Fig 6(e) It can be observed that the proposed method can restore significant amount of details The results obtained using the Joshirsquos two-stage zooming SR method [21] and the extended AM method [17] at the best regularization parameter of λ=1times10-3 are shown in Fig 6(c) and (d) respectively It is observed that both methods have lower visual quality (eg less details near the windowwall letters and some artifacts near the edges regions) when compared with the proposed method The comparison shows that the proposed method can offer superior image reconstruction We also compared our result with the reconstructed HR image using known exact motion parameters as shown in Fig 6(f) It can be observed that the HR image reconstructed using the proposed method is similar to that reconstructed using the exact motion parameters The NSME of the estimated motion parametric vector and the PSNR of the reconstructed image are given in Fig 7 The objective performance measures demonstrate that the prop

In this section we have conducted various Monte-Carlo simulations to demonstrate the effectiveness of the proposed method in handling SR reconstruction from LR images with different motions The number of Monte-Carlo simulations is set to 10 for each motion The proposed method takes into account the relative zooming rotational and translational motion among the captured LR images Existing SR methods are only designed for one or two motions above In this context the experiments are divided into three groups based on the motion among the LR images namely (i) zooming motion only (ii) zooming and translation and (iii) zooming translation and rotation The same experimental setup as in Section B was used to conduct the experiments For a fair comparison our proposed method is compared with only the respective method that can handle the motion model in each case TABLE III shows the NMSE of the estimated motion parametric vector and the PSNR of the reconstructed HR image obtained by the proposed method and other methods Comparison results show that the proposed method is more flexible and effective in handling SR reconstruction under different motions

D Experiments on Real-life Images We also conducted various real-life experiments to

evaluate the performance of the proposed method Ten LR images were captured using a hand-holding web camera Four samples of the LR image are shown in Fig 8(a) It can be seen that there exists relative translation rotation and zooming among the LR images The scaled-up version of the most zoomed in LR image is shown in Fig 8(b) The first LR image was set as reference and a decimation factor of 2 was selected in the following experiment The registration methods in [34] and [35] were again employed to estimate the initial zooming rotation and translation parameters among the LR images respectively Next the two-stage zooming SR method [21] the extend AM method [17] and the proposed method were performed on the LR images and the reconstructed results are given in Fig 8(c)-(e) respectively To provide a fair comparison an image captured with the resolution of the HR image is used as the ground truth in Fig 8(f) From Fig 8(c)-(e) it is observed that considerable detail information of image has been recovered by the proposed method when compared with other methods Further it can be seen that the result by our proposed method has less artifacts than the extended AM and two-stage methods

The experimental results on another real-life image are given in Fig 9 4 samples of the LR images are shown in Fig 9(a) It is noted that there exists obvious relative zooming between the top-left and bottom-right LR images The proposed method and two other comparative methods were performed on the LR images and the results are given in Fig 9(c)-(e) It can be seen that the proposed method produces the superior HR result

From various experiments it is observed that the performance of the proposed method will start to deteriorate quickly when the zooming ratio between the most zoom-in to the least zoom-in LR image exceeds 17 Next we will discuss the computational time of the proposed method The experiments are conducted using the following settings Intel



24GHz CPU 8GB RAM and MATLAB The average computational time is 175s for the proposed method as compared to 240s for the extended AM method [17] implemented in MATLAB It is noted that the computational time can be reduced significantly if the algorithm is implemented in a compiler language such as C

8

quences

E Experiments on Video Sequences Finally we conducted experiments on a popular real video

sequence namely the Mobile sequence Ten 80times80 size LR images from frame 27 to frame 36 were selected as the test materials Four samples of the LR images are shown in Fig 10(a) The scaled-up version of the most zoomed in LR image is shown in Fig 10(b) The two-stage zooming SR method [21] the extended AM method [17] and our proposed method are performed on the LR images and the results are given in Fig 10(c)-(e) From the comparison it can be observed that the proposed method is superior to the other two methods in handling SR reconstruction from video se

We have also conducted an experiment on a real-life uncompressed video sequence captured by a web camera Ten LR images of size 80times80 were selected as the testing video frames 4 samples of the LR images are shown in Fig 11(a) It can be seen that there exists obvious relative translation rotation and zooming among the LR images and the zooming ratio is about 13 The two-stage zooming SR method [21] the extended AM method [17] and our proposed method are performed on the LR images and the results are given in Fig 11(c)-(e) The comparison results show that the proposed method is superior to the other two methods

VI CONCLUSION

This paper presents a new technique to perform SR reconstruction from LR images with relative zooming motion Different from most existing two-stage zooming SR methods the proposed method combines image registration and HR reconstruction into a single process and adopts an iterative framework to improve the estimated HR image and registration parameters progressively In addition the proposed method adopts a motion model consisting of zooming rotation and translation and a linear approximation technique is developed to solve the arising nonlinear least square problem Experimental results show that the proposed method is effective in performing zooming SR reconstruction

APPENDIX A CALCULATION OF kE

In this appendix the calculation of Ek in (18) will be explained Using the chain rule Ek can be divided into two parts as follows

( )( )

( )( )

( ) ( ) (

TT Tk k

k kk T T T T

k k

a b


⎡ ⎤ ⎡part part⎢ ⎥ ⎢⎣ ⎦ ⎣

d eS α fE

d e x i j y i j) ⎤⎥⎦ (27)

For part (a) It can be calculated as follows [32]

( )( )

( ) ( )

( ) ( ) ( ) ( )

( ) ( ) ( )

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )

( )( ) ( ) ( )

k k kT TT T

k kk k

k k k

k k g g k k g g

k g gk

k k g g

d d M N e e M N

f M Nfdiagd d M N

ρ ρ ρ ρ

ρ ρ

ρ ρ

k

⎡ ⎤part part part⎢ ⎥= ⎢ ⎥⎡ ⎤ part partpart ⎣ ⎦⎢ ⎥⎣ ⎦⎡ ⎤part part part part⎢ ⎥= ⎢ ⎥part part part part⎢ ⎥⎣ ⎦

⎛ ⎞partpart ⎟⎜ ⎟⎜= ⎟⎜ ⎟⎜ ⎟part part⎜⎝ ⎠

S α f S α f S α fd ed e

S α f S α f S α f S α f1 1 1 1

1 11 1

( )( ) ( ) ( )

k g gk

k k g

f M Nfdiage e M N

ρ ρ

ρ ρ g

⎡ ⎤⎛ ⎞partpart ⎟⎜⎢ ⎥⎟⎜ ⎟⎢ ⎥⎜ ⎟⎜ ⎟part part⎜⎢ ⎥⎝ ⎠⎣ ⎦

1 11 1

(28) where

( )( ) ( )(( ) )

( )( ) ( )

( )( )

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )( )

( ) ( )

k

k

k k tl k k

k k bl k k br

k

k bl tl k br tr

f i jd i j


d i j e i j f d i j e i j fd i j

e i j f f e i j f f

partpart

part minus minus + minus

+ minus +=

part


1 1 1

1

1

tr

and

( )( ) ( )(( ) )

( )( ) ( )

( )( )

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )( )

( ) ( )

k

k

k k tl k k tr

k k bl k k br

k

k tr tl k br bl

f i je i j


d i j e i j f d i j e i j fe i j

d i j f f d i j f f

partpart


+ minus +=

part


1 1 1

1

1

Hence (28) can be expressed in a simple form as ( )

( )( )

[ ( ) ( ) ( )( ) ( ) ( )]

kT T

k k

k bl tl k br tr

k tr tl k br bl

diagdiag

part⎡ ⎤part ⎢ ⎥⎣ ⎦


minus minus + minus

S α fd e

e f f e f fd f f d f f

1

1

(29)

For part (b) We follow the idea in [36] to obtain

( )

( )( )( )

( ) ( )

( ) ( ) ( ) ( )

T TT T T Tk k

T T T T

⎡ ⎤ ⎡ ⎤part part⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦= =

⎡ ⎤ ⎡ ⎤part part⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦

d e x i j y i jI

x i j y i j x i j y i j (30)

where I is an identity matrix of size g g g gM N M Nρ ρtimes2 22 2 Combining (29) and (30) we can finally obtain the expression

of kE in (18)

REFERENCE

[1] R Y Tsai and T S Huang Multiframe image restoration and registration Advances in Computer Vision and Image Processing vol 1 pp 317-319 1984

[2] S P Kim N K Bose and H M Valenzuela Recursive reconstruction of high resolution image from noisy undersampled multiframes IEEE Transactions on Acoustics Speech and Signal Processing vol 38 pp 1013-1027 1990

[3] P Vandewalle L Sbaiz J Vandewalle and M Vetterli Super-Resolution From Unregistered and Totally Aliased Signals Using Subspace Methods IEEE Transactions on Signal Processing vol 55 pp 3687-3703 2007

[4] M D Robinson C A Toth J Y Lo and S Farsiu Efficient Fourier-Wavelet Super-Resolution IEEE Transactions on Image Processing vol 19 pp 2669-2681 2010

[5] F Sroubek and J Flusser Multichannel blind iterative image restoration IEEE Transactions on Image Processing vol 12 pp 1094-1106 2003



[6] M Protter M Elad H Takeda and P Milanfar Generalizing the Nonlocal-Means to Super-Resolution Reconstruction IEEE Transactions on Image Processing vol 18 pp 36-51 2009

[7] Y He K-H Yap L Chen and L-P Chau A soft MAP framework for blind super-resolution image reconstruction Image and Vision Computing vol 27 pp 364-373 2009

[8] J Tian and K-K Ma Stochastic super-resolution image reconstruction Journal of Visual Communication and Image Representation vol 21 pp 232-244 2010

[9] M V W Zibetti and J Mayer A Robust and Computationally Efficient Simultaneous Super-Resolution Scheme for Image Sequences IEEE Transactions on Circuits and Systems for Video Technology vol 17 pp 1288-1300 2007

[10] K-H Yap Y He Y Tian and L-P Chau A nonlinear L1-norm approach for joint image registration and super-resolution IEEE Signal Processing Letters vol 16 pp 981-984 2009

[11] M Shen and P Xue Super-resolution from observations with variable zooming ratios in Proceedings of IEEE International Symposium on Circuits and Systems (ISCAS) 2010 pp 2622-2625

[12] Z Lin and H-Y Shum Fundamental limits of reconstruction-based superresolution algorithms under local translation IEEE Transactions on Pattern Analysis and Machine Intelligence vol 26 pp 83-97 2004

9

[13] A Zomet A Rav-Acha and S Peleg Robust super-resolution in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2001 pp I-645-I-650 vol1

[14] S Farsiu M D Robinson M Elad and P Milanfar Fast and Robust Multiframe Super Resolution IEEE Trans Image Processing vol 13 pp 1327-1344 2004

[15] E S Lee and M G Kang Regularized adaptive high-resolution image reconstruction considering inaccurate subpixel registration IEEE Transactions on Image Processing vol 12 pp 826-837 2003

[16] H He and L P Kondi An image super-resolution algorithm for different error levels per frame IEEE Transactions on Image Processing vol 15 pp 592-603 2006

[17] R C Hardie K J Barnard and E E Armstrong Joint MAP registration and high-resolution image estimation using a sequence of undersampled images IEEE Transactions on Image Processing vol 6 pp 1621-1633 1997

[18] N A Woods N P Galatsanos and A K Katsaggelos Stochastic methods for joint registration restoration and interpolation of multiple undersampled images IEEE Transactions on Image Processing vol 15 pp 201-213 2006

[19] D Robinson S Farsiu P Milanfar ldquoOptimal Registration of Aliased Images Using Variable Projection with Applications to Superresolutionrdquo The Computer Journal vol 52 no1 pp31-42 Jan 2009

[20] X Li Super-Resolution for Synthetic Zooming EURASIP Journal on Applied Signal Processing vol 2006 pp 1-12 2006

[21] M V Joshi S Chaudhuri and R Panuganti A learning-based method for image super-resolution from zoomed observations IEEE Transactions on Systems Man and Cybernetics Part B Cybernetics vol 35 pp 527-537 2005

[22] M V Joshi S Chaudhuri and R Panuganti Super-resolution imaging use of zoom as a cue Image and Vision Computing vol 22 pp 1185-1196 2004

[23] M K Ng H Shen S Chaudhuri and A C Yau Zoom-based super-resolution reconstruction approach using prior total variation Opical Engineering vol 46 Dec 2007

[24] Y He K-H Yap L Chen and L-P Chau A Nonlinear Least Square Technique for Simultaneous Image Registration and Super-Resolution IEEE Transactions on Image Processing vol 16 pp 2830-2841 2007

[25] U Mudenagudi S Banerjee and P K Kalra Space-Time Super-Resolution Using Graph-Cut Optimization IEEE Transactions on Pattern Analysis and Machine Intelligence vol 33 pp 995-1008 2011

[26] S D Babacan R Molina and A K Katsaggelos Variational Bayesian Super Resolution IEEE Transactions on Image Processing vol 20 pp 984-999 2011

[27] R C Gonzalez and R E Woods Digital Image Processing (3rd) Prentice-Hall 2002

[28] T F Chan and C K Wong ldquoTotal variation blind deconvolutionrdquo IEEE Transactions on Image Processing vol 7 pp 370ndash375 1998

[29] A Chambolle and P Lions ldquoImage recovery via total variation minimization and related problemsrdquo Numer Math vol 76 no2 pp167ndash188 1997

[30] G Aubert and P Kornprobst Mathematical Problems in Image Processing New York Springer Verlag 2002

[31] H Fu and J Barlow A regularized structured total least squares algorithm for high-resolution image reconstruction Linear algebra and its applications vol 391 pp 75-98 2004

[32] G H Golub and C F Van loan Matrix Computations The Johns Hopkins University Press Baltimore 1996

[33] M V W Zibetti J Mayer and F S V Bazan Determining the parameters in regularized super-resolution reconstruction in IEEE International Conference on Acoustics Speech and Signal Processing 2008 pp 853-856

[34] B S Reddy and B N Chatterji An FFT-based technique for translation rotation and scale-invariant image registration IEEE Transactions on Image Processing vol 5 pp 1266-1271 1996

[35] H Foroosh J B Zerubia and M Berthod Extension of phase correlation to subpixel registration IEEE Transactions on Image Processing vol 11 pp 188-200 2002

[36] [online] httpfunctionswolframcom040120000101

Pixels of LR image 1

Pixels of LRimage 2

Fig 1 Illustration of the relative zooming motion between two LR images

Initialization f 0 a0 G0

Estimate and using (12)

Update

and

Construct Gi+1 based on f i+1

and ai+1

Termination condition satisfied

Linear approximation for

Simultaneous image registration and SR

iΔfiΔα

i i i+ = +Δf f f1

i i i+ = +Δα α α1

( )( ) Tpart partS α f α

Fig 2 Overview of the proposed method



x

y

lk

ftl ftr

fbl fbr

fk(i j)dk

ek

Fig 3 The relative position between the kth HR grid and the reference HR grid fk ( ik jk ) denotes the HR pixels and ftl ftr fbl and fbr are its surrounding pixels in the reference HR grid

(a) (b) (c) (d) Fig 4 Test images (a) ldquoBridgerdquo image (b) ldquoWindowrdquo image (c) ldquoHillrdquo image (d) ldquoLenardquo image

10

(a) (b)

(c) (d) (e)

(f) (g) (h) Fig 5 SR on the ldquoBridgerdquo image (a) 4 samples of the LR images (b) The scaled-up version of the most zoomed in LR image (c)-(e) HR results using Farsiursquos shift-and-add method [14] the Joshirsquos method [21] and our proposed method respectively (f)-(h) Selected enlarged region of (c)-(e) respe

(a) (b) (c)

(d) (e) (f) Fig 6 SR on the ldquoWindowrdquo image (a) 4 samples of the LR images (b) The scaled-up version of the most zoomed in LR image (c)-(e) HR results using the Joshirsquos method [21] the extended AM method [17] and our proposed method respectively (f) Reconstructed image using known exact motion parameters

(a)

(b) ctively Fig 7 Objective measurements (a) NMSE of the estimated motion

parameters (b) PSNR of the reconstructed image



(a) (b)

(c) (d) (e)

(a) (b) (c)

(d) (e) (f) Fig 8 SR on real-life images (a) 4 samples of the LR images (b) The scaled-up version of the most zoomed in LR image (c)-(e) HR results using two-stage zooming SR method [21] the extended AM method [17] and our proposed method respectively (f) Ground truth

Fig 10 SR on video sequence (a) 4 samples of the LR images (b) The scaled-up version of the most zoomed in LR image (c)-(e) HR results using two-stage zooming SR method [21] the extended AM method [17] and our proposed method respectively

(a) (b) (c)

(d) (e) (f)

(a) (b)

(c) (d) (e)

Fig 9 SR on real-life images (a) 4 samples of the LR images (b) The scaled-up version of the most zoomed in LR image (c)-(e) HR results using two-stage zooming SR method [21] the extended AM method [17] and our proposed method respectively (f) Ground truth Fig 11 SR on real-life video sequence (a) 4 samples of the LR

images (b) The scaled-up version of the most zoomed in LR image (c)-(e) HR results using two-stage zooming SR method [21] the extended AM method [17] and our proposed method respectively

TABLE I COMPARISON OF PSNR Farsiursquo shift-and-add method [14]

Joshirsquos method [21]

Our proposed method

2656 2775 3087

TABLE II SUBJECTIVE TEST FOR VARIOUS METHODS Farsiursquo shift-

and-add method [14]

Joshirsquos method

[21]

Our proposed method

AVERAGE SCORE 40 43 47

11



TABLE III PSNR AND NMSE FOR VARIOUS MOTIONS

Zooming Zooming and translation Zooming translation and rotation

Joshirsquo method [22] Proposed method Ngrsquo method [23] Proposed method Extended AM method [17] Proposed method

NMSE PSNR NMSE PSNR NMSE PSNR NMSE PSNR NMSE PSNR NMSE PSNR Bridge 151 2744 00045 2972 181 2721 00029 3005 00712 2868 00033 2974

Window 179 2507 00034 2878 202 2521 00021 2891 00502 2617 00023 2882

Hill 149 2788 00045 2969 198 2702 00046 2965 00971 2828 00035 2968 Lena 119 3004 00037 3339 289 2911 00031 3339 00864 3178 00026 3342

Yushuang Tian received the BEng degree in information engineering from Zhejiang University Hangzhou China in 2005 and the MSc degree in information and communication engineering from Zhejiang University Hangzhou China in 2007 After that he worked as a Research and Development Engineer in the Huawei Technologies Co Ltd for one year He is currently a PhD student at Nanyang Technological University Singapore His research

interests include imagevideo processing computer vision and imagevideo super-resolution

Kim-Hui Yap (Srsquo99-Mrsquo03)

received the B Eng and PhD degrees in electrical engineering from the University of Sydney Australia in 1998 and 2002 respectively Since then he has been a faculty member at Nanyang Technological University Singapore where he is currently an Associate Professor His main research interests include imagevideo processing media content analysis computer vision and computational intelligence He has served as an

Associate Editor for the IEEE Computational Intelligence Magazine and Journal of Signal Processing Systems He has also served as an Editorial Board Member for The Open Electrical and Electronic Engineering Journal and as a Guest Editor for the IEICE Transactions on Fundamentals Dr Yap is a Senior Member of IEEE He has served as the Treasurer of the IEEE Singapore Signal Processing Chapter and the Committee Member of the IEEE Singapore Computational Intelligence Chapter He has also served as the Finance Chair in 2010 IEEE International Conference on Multimedia amp Expo the Workshop Co-chair of the 2009 MDM International Workshop on Mobile Media Retrieval and others He served as the Group Leader in Content-based Analysis for the Center for Signal Processing Nanyang Technological University Singapore He has numerous publications in various international journals book chapters and conference proceedings He has authored a book entitled ldquoAdaptive Image Processing A Computational Intelligence Perspective Second Editionrdquo published by the CRC Press in 2009 and edited a book entitled ldquoIntelligent Multimedia Processing with Soft Computingrdquo published by Springer-Verlag in 2005

12



based SR approach with total variation (TV) prior has been proposed in [23] Generally these methods can be considered as the two-stage disjoint SR methods as discussed previously

In view of this this paper proposes an iterative method for joint image registration and SR reconstruction from LR observations with relative zooming motion The proposed method integrates image registration and zooming SR reconstruction into a single estimation process An iterative technique serves to ensure that progressively better estimates of the HR image and motion parameters can be obtained This is more promising when compared with other two-stage SR methods as the estimation of motion parameters could benefit from the information of reconstructed HR image Different from conventional methods such as [24] focus mainly on translation and possibly rotation the proposed method takes into account the relative zooming between the acquired LR images The adopted motion model enables the proposed method to handle more real-life applications involving motion containing translation rotation and zooming Further the proposed method also develops an adaptive weighting scheme and incorporates it into the HR reconstruction As the captured LR images with relative zooming motion provide different degrees of information to the HR reconstruction the adopted weighting strategy takes this into consideration

2

The rest of this paper is organized as follows The problem formulation of image SR is introduced in Section II The proposed cost function is presented in Section III An iterative algorithm using the regularized nonlinear least squares technique is developed in Section IV Experimental results on simulated and real-life images are presented and discussed in Section V A brief conclusion is given in Section VI

II PROBLEM FORMULATION First of all let us consider the generation of LR images

under the motion model consisting of zooming translation and rotation The kth (1 le k le N ) LR image ( )kg u v of size

g gM Ntimes

( )can be modeled by zooming the original HR image

f x y of size f fM Ntimes by a zooming factor rotating it by an angle shifting it by a translational vector [

kl

kθ ]xk yks s

( )n u v

blurring the zoomed rotated and shifted HR image by point-spread function (PSF) ( and down-sampling the result by a decimation factor of ρ Finally the LR image is degraded by additive white Gaussian noise (AWGN) The process can be expressed as

)( )k ch h x yotimes

[14] [25]-[27]

(( ) )

( )

( cos sin sin cos )

( ) ( )

k

k k k k xk k k k k y

k c

g u v

kf l x l y s l x l y s

h h x y n u vρ

θ θ θ θ= minus + +

otimes otimes darr +

+

+

(1)

where is the 2D-convolution operator denotes the down-sampling operator with a decimation factor where is employed to represent the resolution ratio between the desired HR image and acquired LR image In this work we consider the decimation factor in the vertical and horizontal

directions to be identical represents the camera lens blur in the kth LR image represents the effect of spatial integration of light intensity over a square surface region to simulate the LR image captured by sensor

otimes ρdarr

ρ ρ

kh

g D

ch

As in the work we assume that the lens condition ( ) is known and identical for all LR images The imaging process in

kh

(1) can be expressed in a matrix-vector form as

( )= HS α f n (2)

where represents the lexicographically ordered original image

f( )f x y [ ]T=g g g

(

1 TN

k

T T and are the vectors representing the discrete concatenated and lexicographically ordered

[ ]T TN=n n n1

)g u v

T T1

and respectively is the down-sampling operator which is identical for all LR

images and

( )kn u vD

[=H H ]TNH kH (1 le k le N) denotes the

corresponding matrix representing the blurring operator The matrix is formed by nonlinear

differentiable functions of an unknown motion parametric vector a where a = [a1

T a2T aN

T]T Hence S(a) = [S(a1)T S(a2)T S(aN)T]T and (1 le k le N) denotes the geometric motion operator for the kth LR image represents the unknown motion parametric vector of the kth LR image and In this work we consider the initial estimates of motion parameters α to contain certain degrees of error This is a more realistic assumption as accurate registration for SR is difficult to achieve particularly in the early stage The objective of image SR in this context is to reconstruct the HR image from N LR observations with the unknown motion parametric vector

As the first LR image is used as the reference we need to estimate 4 times (N minus 1) motion parameters and the desired HR image

k otimes chh

α

( )α

( kα

k k kl l=

S

S

os

)

sik

kα

k

[ c nθ θ ] Tk xk yks s

f

α

III PROPOSED COST FUNCTION The main ideas of the proposed joint image registration

and zooming SR include integration of registration and SR into an iterative process fusion of multiple LR images with different resolutions and incorporation of useful a prior information for the desired HR image and unknown motion parameters Joint image registration and zooming SR is an ill-posed inverse problem Thus regularizations for both the HR image as well as the motion parameters are considered in order to achieve a stable solution

Given the image formation model in (1) the estimated HR image and the unknown motion parametric vector α can be obtained by minimizing the following cost function

f

( ) ( ( ) ) ( ) ( )E λ β= minus + +α f V g DHS α f T f R a2 (3)

where

and The matrix represents the motion operator consisting of zooming rotation and

[ T TN=g g g1

[ ]T TN=H H H1

]T (S α) ( ) ( ) ( )TT T T

N⎡ ⎤= ⎢ ⎥⎣ ⎦S α S α S α1 2

T ( )kS α





k kV=V I


f

f

( ) dxdy

3



( )

( ( ( ))m n x y m n


=minus =

nabla + + minus= =

nabla sumsumsumint

T f

ff

2 21 1

1 0

nμ

(5)



gt 0[5] The fixed-


m nμ(5) can be


)

m n

( ( ))m n x y

T Tm n m n m n

m n

T T

( f x m yμ=minus =

=minus =

+ +

=

= =

sumsumsum

sumsum f U W U

f L Lf Lf

21 1

1 0

1 1

1 0

2

c

n

f

fminus x y

(6)


m n diag=W


m nU


n m nU W


by m n=minus

UTm

m n=sum sum1 1

1 0







(

minus

α f

Lf

α

)

min

( )

λ

βα f

Vr

a

2

(7)


(

)

)Δ

f f

f f

(



( )r α f

)fΔα

f

f

( α Δf

+nabla f

f

r

r

)

nabla

f( )r α f

α

( )( ) ( ) ( ( )

( ) ( ) (

O+Δ +Δ


+nabla Δ +nablaα

α


r α f r α f α α

(8)



( )r α f)

Δf

( Δ Δα f

α





( )

( )

( )( )

( )

T

T

part minusnabla =

partpart

=minuspart

α

g DHS α fr α f

αS α f

DHα

(9)

( )

( )

( )( )

( )

( )

T

T

part minusnabla =

part

part=minus

part=minus

f

g DHS α fr α f

fS α f

DHf

DHS α

(10)

4






i

iα iΔff The


(

min ( )

( )i i

i i i i i

i i

i i

λ

βΔ Δ

+Δ +Δ

+Δ

Δ + minusα f

V r α α f f

L f f

α α a

2)

(12a)

( ) ( )

min

( )i i

i i i i i i i

ii

ii

λ λ

β βΔ Δ

⎛ ⎞ ⎛ ⎞minus⎟ ⎟⎜ ⎜⎟ ⎟⎜ ⎜⎛ ⎞⎟ ⎟Δ⎜ ⎜⎟⎟ ⎟⎜⎜ ⎜⎟⎟ ⎟+⎜⎜ ⎜⎟⎟ ⎟⎜⎜ ⎜⎟⎟ ⎟⎜Δ⎝ ⎠⎜ ⎟ ⎜ ⎟⎟ ⎟⎜ ⎜⎟ ⎟minus⎜ ⎜⎝ ⎠ ⎝ ⎠α f


L Lff

α α

2

0

0

(12b)


i



f iΔα


approximation for

( )S α f( )S α

(=f

)Xα

( )i ipart S α fT





( )kS α f

f( )kS α f


)to be the


)




k k k k

k k k k

xk

yk


θ θθ θ

= minus +

= + + (13)


( )( ) ( )( )

( )( ) ( ) ( ) ( )

( ) ( ) ( ) ( )

k

k k tl k k tr

k k bl k k br




+ minus +

1 1 1

1

(14)








( ) ( ) ( )( ) ( ) ( ) N

T TN

diag T



α α1

1 α (15)



the chain rule

g gM Nρ times2 4

( )kS α f

)kα


( )( )

( )( ) ( )( )( )( ) ( )

TT T

kkT TT T

k k




(16)



( ) ( )( ) ( ) ( )Tk





( )

( )( )

( )( )

( ) ( )

TT Tk k

kk T T T T

k k

a b


⎡ ⎤ ⎡part part⎢ ⎥ ⎢⎣ ⎦ ⎣

d eS α fE

d e x i j y i j

5

⎤⎥⎦

TT T⎡ ⎤⎥⎦

(17)



(18) [ ( ) ( ) ( )

( ) ( ) ( )]k k bl tl k br

k tr tl k br bl

diagdiag


minus minus + minus


1

1trf




)



( )kα fS

(13) as

( )( ) k

⎡ ⎤⎢ ⎥ =⎢ ⎥⎣ ⎦

x i jCα

y i j (19)

where and

and are the

⎡ ⎤minus⎢ ⎥= ⎢ ⎥⎣ ⎦

i j 1 0C

j i 0 1 ] T


g gM Nρ2

kα


respect to Thus

( ) ( Tx i j y i

()

Tj T


expressed as

( ) ( )( ) ( )TT T

kT

k k

⎡ ⎤part ⎢ ⎥ part⎣ ⎦= =

part part

x i j y i j CαC

α α T (20)




α

( )( )T

part= =

part

S α fG

αEC (21)

where

N

⎡ ⎤⎢ ⎥⎢ ⎥= ⎢ ⎥⎢ ⎥⎣ ⎦

EE

E

1 0

0

and ⎡ ⎤⎢ ⎥⎢ ⎥= ⎢ ⎥⎢ ⎥⎣ ⎦

CC

C

0

0



i i i=A f b (22)

where ( ) ( ) ( ) ( )

( ) ( )

( )

( ) ( ) ( )



β

λ

⎛ ⎞⎟+⎜ ⎟⎜ ⎟⎜ ⎟=⎜ ⎟⎜ ⎟⎜ ⎟+ ⎟⎜⎝ ⎠



( ) ( )( )

( ) ( )

( ) ( )



β

λ




ii

i

⎛ ⎞Δ ⎟⎜ ⎟=⎜ ⎟⎜ ⎟⎜Δ⎝ ⎠

αf

f





i i+ = +Δα α1

iΔf

iαi i= +f f+1





kV






( )average

kk k k

VV

l=



as

averageV

( )

N

k k k k

N


1


kV



kVf

(22)

6



i i

i

minusminus

minus

minuslt

f f

f

1

6

110 (24)


αf

ˆˆ( )NMSE

minusequiv

α αα

α

2

2100 (25)

255ˆ( ) logˆ

M NPSNR equiv

minus

f fff f

2

10 210 (26)





kh hotimes c

imate of

or clo

ρ = 2








7
















8

quences




VI CONCLUSION




( )( )

( )( )

( ) ( ) (

TT Tk k

k kk T T T T

k k

a b


⎡ ⎤ ⎡part part⎢ ⎥ ⎢⎣ ⎦ ⎣

d eS α fE

d e x i j y i j) ⎤⎥⎦ (27)


( )( )

( ) ( )

( ) ( ) ( ) ( )

( ) ( ) ( )

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )

( )( ) ( ) ( )

k k kT TT T

k kk k

k k k

k k g g k k g g

k g gk

k k g g

d d M N e e M N

f M Nfdiagd d M N

ρ ρ ρ ρ

ρ ρ

ρ ρ

k





1 11 1

( )( ) ( ) ( )

k g gk

k k g

f M Nfdiage e M N

ρ ρ

ρ ρ g


1 11 1

(28) where

( )( ) ( )(( ) )

( )( ) ( )

( )( )

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )( )

( ) ( )

k

k

k k tl k k

k k bl k k br

k

k bl tl k br tr

f i jd i j



e i j f f e i j f f

partpart


+ minus +=

part


1 1 1

1

1

tr

and

( )( ) ( )(( ) )

( )( ) ( )

( )( )

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )( )

( ) ( )

k

k

k k tl k k tr

k k bl k k br

k

k tr tl k br bl

f i je i j



d i j f f d i j f f

partpart


+ minus +=

part


1 1 1

1

1


( )( )

[ ( ) ( ) ( )( ) ( ) ( )]

kT T

k k

k bl tl k br tr

k tr tl k br bl

diagdiag



minus minus + minus

S α fd e


1

1

(29)


( )

( )( )( )

( ) ( )

( ) ( ) ( ) ( )

T TT T T Tk k

T T T T

⎡ ⎤ ⎡ ⎤part part⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦= =

⎡ ⎤ ⎡ ⎤part part⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦

d e x i j y i jI



of kE in (18)

REFERENCE















9


























Pixels of LRimage 2




Update

and


and ai+1




iΔfiΔα

i i i+ = +Δf f f1






x

y

lk

ftl ftr

fbl fbr

fk(i j)dk

ek



10

(a) (b)

(c) (d) (e)


(a) (b) (c)


(a)





(a) (b)

(c) (d) (e)

(a) (b) (c)



(a) (b) (c)

(d) (e) (f)

(a) (b)

(c) (d) (e)





Our proposed method

2656 2775 3087


and-add method [14]

Joshirsquos method

[21]

Our proposed method


11







Window 179 2507 00034 2878 202 2521 00021 2891 00502 2617 00023 2882

Hill 149 2788 00045 2969 198 2702 00046 2965 00971 2828 00035 2968 Lena 119 3004 00037 3339 289 2911 00031 3339 00864 3178 00026 3342






12





k kV=V I


f

f

( ) dxdy

3



( )

( ( ( ))m n x y m n


=minus =

nabla + + minus= =

nabla sumsumsumint

T f

ff

2 21 1

1 0

nμ

(5)



gt 0[5] The fixed-


m nμ(5) can be


)

m n

( ( ))m n x y

T Tm n m n m n

m n

T T

( f x m yμ=minus =

=minus =

+ +

=

= =

sumsumsum

sumsum f U W U

f L Lf Lf

21 1

1 0

1 1

1 0

2

c

n

f

fminus x y

(6)


m n diag=W


m nU


n m nU W


by m n=minus

UTm

m n=sum sum1 1

1 0







(

minus

α f

Lf

α

)

min

( )

λ

βα f

Vr

a

2

(7)


(

)

)Δ

f f

f f

(



( )r α f

)fΔα

f

f

( α Δf

+nabla f

f

r

r

)

nabla

f( )r α f

α

( )( ) ( ) ( ( )

( ) ( ) (

O+Δ +Δ


+nabla Δ +nablaα

α


r α f r α f α α

(8)



( )r α f)

Δf

( Δ Δα f

α





( )

( )

( )( )

( )

T

T

part minusnabla =

partpart

=minuspart

α

g DHS α fr α f

αS α f

DHα

(9)

( )

( )

( )( )

( )

( )

T

T

part minusnabla =

part

part=minus

part=minus

f

g DHS α fr α f

fS α f

DHf

DHS α

(10)

4






i

iα iΔff The


(

min ( )

( )i i

i i i i i

i i

i i

λ

βΔ Δ

+Δ +Δ

+Δ

Δ + minusα f

V r α α f f

L f f

α α a

2)

(12a)

( ) ( )

min

( )i i

i i i i i i i

ii

ii

λ λ

β βΔ Δ

⎛ ⎞ ⎛ ⎞minus⎟ ⎟⎜ ⎜⎟ ⎟⎜ ⎜⎛ ⎞⎟ ⎟Δ⎜ ⎜⎟⎟ ⎟⎜⎜ ⎜⎟⎟ ⎟+⎜⎜ ⎜⎟⎟ ⎟⎜⎜ ⎜⎟⎟ ⎟⎜Δ⎝ ⎠⎜ ⎟ ⎜ ⎟⎟ ⎟⎜ ⎜⎟ ⎟minus⎜ ⎜⎝ ⎠ ⎝ ⎠α f


L Lff

α α

2

0

0

(12b)


i



f iΔα


approximation for

( )S α f( )S α

(=f

)Xα

( )i ipart S α fT





( )kS α f

f( )kS α f


)to be the


)




k k k k

k k k k

xk

yk


θ θθ θ

= minus +

= + + (13)


( )( ) ( )( )

( )( ) ( ) ( ) ( )

( ) ( ) ( ) ( )

k

k k tl k k tr

k k bl k k br




+ minus +

1 1 1

1

(14)








( ) ( ) ( )( ) ( ) ( ) N

T TN

diag T



α α1

1 α (15)



the chain rule

g gM Nρ times2 4

( )kS α f

)kα


( )( )

( )( ) ( )( )( )( ) ( )

TT T

kkT TT T

k k




(16)



( ) ( )( ) ( ) ( )Tk





( )

( )( )

( )( )

( ) ( )

TT Tk k

kk T T T T

k k

a b


⎡ ⎤ ⎡part part⎢ ⎥ ⎢⎣ ⎦ ⎣

d eS α fE

d e x i j y i j

5

⎤⎥⎦

TT T⎡ ⎤⎥⎦

(17)



(18) [ ( ) ( ) ( )

( ) ( ) ( )]k k bl tl k br

k tr tl k br bl

diagdiag


minus minus + minus


1

1trf




)



( )kα fS

(13) as

( )( ) k

⎡ ⎤⎢ ⎥ =⎢ ⎥⎣ ⎦

x i jCα

y i j (19)

where and

and are the

⎡ ⎤minus⎢ ⎥= ⎢ ⎥⎣ ⎦

i j 1 0C

j i 0 1 ] T


g gM Nρ2

kα


respect to Thus

( ) ( Tx i j y i

()

Tj T


expressed as

( ) ( )( ) ( )TT T

kT

k k

⎡ ⎤part ⎢ ⎥ part⎣ ⎦= =

part part

x i j y i j CαC

α α T (20)




α

( )( )T

part= =

part

S α fG

αEC (21)

where

N

⎡ ⎤⎢ ⎥⎢ ⎥= ⎢ ⎥⎢ ⎥⎣ ⎦

EE

E

1 0

0

and ⎡ ⎤⎢ ⎥⎢ ⎥= ⎢ ⎥⎢ ⎥⎣ ⎦

CC

C

0

0



i i i=A f b (22)

where ( ) ( ) ( ) ( )

( ) ( )

( )

( ) ( ) ( )



β

λ

⎛ ⎞⎟+⎜ ⎟⎜ ⎟⎜ ⎟=⎜ ⎟⎜ ⎟⎜ ⎟+ ⎟⎜⎝ ⎠



( ) ( )( )

( ) ( )

( ) ( )



β

λ




ii

i

⎛ ⎞Δ ⎟⎜ ⎟=⎜ ⎟⎜ ⎟⎜Δ⎝ ⎠

αf

f





i i+ = +Δα α1

iΔf

iαi i= +f f+1





kV






( )average

kk k k

VV

l=



as

averageV

( )

N

k k k k

N


1


kV



kVf

(22)

6



i i

i

minusminus

minus

minuslt

f f

f

1

6

110 (24)


αf

ˆˆ( )NMSE

minusequiv

α αα

α

2

2100 (25)

255ˆ( ) logˆ

M NPSNR equiv

minus

f fff f

2

10 210 (26)





kh hotimes c

imate of

or clo

ρ = 2








7
















8

quences




VI CONCLUSION




( )( )

( )( )

( ) ( ) (

TT Tk k

k kk T T T T

k k

a b


⎡ ⎤ ⎡part part⎢ ⎥ ⎢⎣ ⎦ ⎣

d eS α fE

d e x i j y i j) ⎤⎥⎦ (27)


( )( )

( ) ( )

( ) ( ) ( ) ( )

( ) ( ) ( )

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )

( )( ) ( ) ( )

k k kT TT T

k kk k

k k k

k k g g k k g g

k g gk

k k g g

d d M N e e M N

f M Nfdiagd d M N

ρ ρ ρ ρ

ρ ρ

ρ ρ

k





1 11 1

( )( ) ( ) ( )

k g gk

k k g

f M Nfdiage e M N

ρ ρ

ρ ρ g


1 11 1

(28) where

( )( ) ( )(( ) )

( )( ) ( )

( )( )

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )( )

( ) ( )

k

k

k k tl k k

k k bl k k br

k

k bl tl k br tr

f i jd i j



e i j f f e i j f f

partpart


+ minus +=

part


1 1 1

1

1

tr

and

( )( ) ( )(( ) )

( )( ) ( )

( )( )

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )( )

( ) ( )

k

k

k k tl k k tr

k k bl k k br

k

k tr tl k br bl

f i je i j



d i j f f d i j f f

partpart


+ minus +=

part


1 1 1

1

1


( )( )

[ ( ) ( ) ( )( ) ( ) ( )]

kT T

k k

k bl tl k br tr

k tr tl k br bl

diagdiag



minus minus + minus

S α fd e


1

1

(29)


( )

( )( )( )

( ) ( )

( ) ( ) ( ) ( )

T TT T T Tk k

T T T T

⎡ ⎤ ⎡ ⎤part part⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦= =

⎡ ⎤ ⎡ ⎤part part⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦

d e x i j y i jI



of kE in (18)

REFERENCE















9


























Pixels of LRimage 2




Update

and


and ai+1




iΔfiΔα

i i i+ = +Δf f f1






x

y

lk

ftl ftr

fbl fbr

fk(i j)dk

ek



10

(a) (b)

(c) (d) (e)


(a) (b) (c)


(a)





(a) (b)

(c) (d) (e)

(a) (b) (c)



(a) (b) (c)

(d) (e) (f)

(a) (b)

(c) (d) (e)





Our proposed method

2656 2775 3087


and-add method [14]

Joshirsquos method

[21]

Our proposed method


11







Window 179 2507 00034 2878 202 2521 00021 2891 00502 2617 00023 2882

Hill 149 2788 00045 2969 198 2702 00046 2965 00971 2828 00035 2968 Lena 119 3004 00037 3339 289 2911 00031 3339 00864 3178 00026 3342






12



( )

( )

( )( )

( )

T

T

part minusnabla =

partpart

=minuspart

α

g DHS α fr α f

αS α f

DHα

(9)

( )

( )

( )( )

( )

( )

T

T

part minusnabla =

part

part=minus

part=minus

f

g DHS α fr α f

fS α f

DHf

DHS α

(10)

4






i

iα iΔff The


(

min ( )

( )i i

i i i i i

i i

i i

λ

βΔ Δ

+Δ +Δ

+Δ

Δ + minusα f

V r α α f f

L f f

α α a

2)

(12a)

( ) ( )

min

( )i i

i i i i i i i

ii

ii

λ λ

β βΔ Δ

⎛ ⎞ ⎛ ⎞minus⎟ ⎟⎜ ⎜⎟ ⎟⎜ ⎜⎛ ⎞⎟ ⎟Δ⎜ ⎜⎟⎟ ⎟⎜⎜ ⎜⎟⎟ ⎟+⎜⎜ ⎜⎟⎟ ⎟⎜⎜ ⎜⎟⎟ ⎟⎜Δ⎝ ⎠⎜ ⎟ ⎜ ⎟⎟ ⎟⎜ ⎜⎟ ⎟minus⎜ ⎜⎝ ⎠ ⎝ ⎠α f


L Lff

α α

2

0

0

(12b)


i



f iΔα


approximation for

( )S α f( )S α

(=f

)Xα

( )i ipart S α fT





( )kS α f

f( )kS α f


)to be the


)




k k k k

k k k k

xk

yk


θ θθ θ

= minus +

= + + (13)


( )( ) ( )( )

( )( ) ( ) ( ) ( )

( ) ( ) ( ) ( )

k

k k tl k k tr

k k bl k k br




+ minus +

1 1 1

1

(14)








( ) ( ) ( )( ) ( ) ( ) N

T TN

diag T



α α1

1 α (15)



the chain rule

g gM Nρ times2 4

( )kS α f

)kα


( )( )

( )( ) ( )( )( )( ) ( )

TT T

kkT TT T

k k




(16)



( ) ( )( ) ( ) ( )Tk





( )

( )( )

( )( )

( ) ( )

TT Tk k

kk T T T T

k k

a b


⎡ ⎤ ⎡part part⎢ ⎥ ⎢⎣ ⎦ ⎣

d eS α fE

d e x i j y i j

5

⎤⎥⎦

TT T⎡ ⎤⎥⎦

(17)



(18) [ ( ) ( ) ( )

( ) ( ) ( )]k k bl tl k br

k tr tl k br bl

diagdiag


minus minus + minus


1

1trf




)



( )kα fS

(13) as

( )( ) k

⎡ ⎤⎢ ⎥ =⎢ ⎥⎣ ⎦

x i jCα

y i j (19)

where and

and are the

⎡ ⎤minus⎢ ⎥= ⎢ ⎥⎣ ⎦

i j 1 0C

j i 0 1 ] T


g gM Nρ2

kα


respect to Thus

( ) ( Tx i j y i

()

Tj T


expressed as

( ) ( )( ) ( )TT T

kT

k k

⎡ ⎤part ⎢ ⎥ part⎣ ⎦= =

part part

x i j y i j CαC

α α T (20)




α

( )( )T

part= =

part

S α fG

αEC (21)

where

N

⎡ ⎤⎢ ⎥⎢ ⎥= ⎢ ⎥⎢ ⎥⎣ ⎦

EE

E

1 0

0

and ⎡ ⎤⎢ ⎥⎢ ⎥= ⎢ ⎥⎢ ⎥⎣ ⎦

CC

C

0

0



i i i=A f b (22)

where ( ) ( ) ( ) ( )

( ) ( )

( )

( ) ( ) ( )



β

λ

⎛ ⎞⎟+⎜ ⎟⎜ ⎟⎜ ⎟=⎜ ⎟⎜ ⎟⎜ ⎟+ ⎟⎜⎝ ⎠



( ) ( )( )

( ) ( )

( ) ( )



β

λ




ii

i

⎛ ⎞Δ ⎟⎜ ⎟=⎜ ⎟⎜ ⎟⎜Δ⎝ ⎠

αf

f





i i+ = +Δα α1

iΔf

iαi i= +f f+1





kV






( )average

kk k k

VV

l=



as

averageV

( )

N

k k k k

N


1


kV



kVf

(22)

6



i i

i

minusminus

minus

minuslt

f f

f

1

6

110 (24)


αf

ˆˆ( )NMSE

minusequiv

α αα

α

2

2100 (25)

255ˆ( ) logˆ

M NPSNR equiv

minus

f fff f

2

10 210 (26)





kh hotimes c

imate of

or clo

ρ = 2








7
















8

quences




VI CONCLUSION




( )( )

( )( )

( ) ( ) (

TT Tk k

k kk T T T T

k k

a b


⎡ ⎤ ⎡part part⎢ ⎥ ⎢⎣ ⎦ ⎣

d eS α fE

d e x i j y i j) ⎤⎥⎦ (27)


( )( )

( ) ( )

( ) ( ) ( ) ( )

( ) ( ) ( )

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )

( )( ) ( ) ( )

k k kT TT T

k kk k

k k k

k k g g k k g g

k g gk

k k g g

d d M N e e M N

f M Nfdiagd d M N

ρ ρ ρ ρ

ρ ρ

ρ ρ

k





1 11 1

( )( ) ( ) ( )

k g gk

k k g

f M Nfdiage e M N

ρ ρ

ρ ρ g


1 11 1

(28) where

( )( ) ( )(( ) )

( )( ) ( )

( )( )

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )( )

( ) ( )

k

k

k k tl k k

k k bl k k br

k

k bl tl k br tr

f i jd i j



e i j f f e i j f f

partpart


+ minus +=

part


1 1 1

1

1

tr

and

( )( ) ( )(( ) )

( )( ) ( )

( )( )

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )( )

( ) ( )

k

k

k k tl k k tr

k k bl k k br

k

k tr tl k br bl

f i je i j



d i j f f d i j f f

partpart


+ minus +=

part


1 1 1

1

1


( )( )

[ ( ) ( ) ( )( ) ( ) ( )]

kT T

k k

k bl tl k br tr

k tr tl k br bl

diagdiag



minus minus + minus

S α fd e


1

1

(29)


( )

( )( )( )

( ) ( )

( ) ( ) ( ) ( )

T TT T T Tk k

T T T T

⎡ ⎤ ⎡ ⎤part part⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦= =

⎡ ⎤ ⎡ ⎤part part⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦

d e x i j y i jI



of kE in (18)

REFERENCE















9


























Pixels of LRimage 2




Update

and


and ai+1




iΔfiΔα

i i i+ = +Δf f f1






x

y

lk

ftl ftr

fbl fbr

fk(i j)dk

ek



10

(a) (b)

(c) (d) (e)


(a) (b) (c)


(a)





(a) (b)

(c) (d) (e)

(a) (b) (c)



(a) (b) (c)

(d) (e) (f)

(a) (b)

(c) (d) (e)





Our proposed method

2656 2775 3087


and-add method [14]

Joshirsquos method

[21]

Our proposed method


11







Window 179 2507 00034 2878 202 2521 00021 2891 00502 2617 00023 2882

Hill 149 2788 00045 2969 198 2702 00046 2965 00971 2828 00035 2968 Lena 119 3004 00037 3339 289 2911 00031 3339 00864 3178 00026 3342






12



( )

( )( )

( )( )

( ) ( )

TT Tk k

kk T T T T

k k

a b


⎡ ⎤ ⎡part part⎢ ⎥ ⎢⎣ ⎦ ⎣

d eS α fE

d e x i j y i j

5

⎤⎥⎦

TT T⎡ ⎤⎥⎦

(17)



(18) [ ( ) ( ) ( )

( ) ( ) ( )]k k bl tl k br

k tr tl k br bl

diagdiag


minus minus + minus


1

1trf




)



( )kα fS

(13) as

( )( ) k

⎡ ⎤⎢ ⎥ =⎢ ⎥⎣ ⎦

x i jCα

y i j (19)

where and

and are the

⎡ ⎤minus⎢ ⎥= ⎢ ⎥⎣ ⎦

i j 1 0C

j i 0 1 ] T


g gM Nρ2

kα


respect to Thus

( ) ( Tx i j y i

()

Tj T


expressed as

( ) ( )( ) ( )TT T

kT

k k

⎡ ⎤part ⎢ ⎥ part⎣ ⎦= =

part part

x i j y i j CαC

α α T (20)




α

( )( )T

part= =

part

S α fG

αEC (21)

where

N

⎡ ⎤⎢ ⎥⎢ ⎥= ⎢ ⎥⎢ ⎥⎣ ⎦

EE

E

1 0

0

and ⎡ ⎤⎢ ⎥⎢ ⎥= ⎢ ⎥⎢ ⎥⎣ ⎦

CC

C

0

0



i i i=A f b (22)

where ( ) ( ) ( ) ( )

( ) ( )

( )

( ) ( ) ( )



β

λ

⎛ ⎞⎟+⎜ ⎟⎜ ⎟⎜ ⎟=⎜ ⎟⎜ ⎟⎜ ⎟+ ⎟⎜⎝ ⎠



( ) ( )( )

( ) ( )

( ) ( )



β

λ




ii

i

⎛ ⎞Δ ⎟⎜ ⎟=⎜ ⎟⎜ ⎟⎜Δ⎝ ⎠

αf

f





i i+ = +Δα α1

iΔf

iαi i= +f f+1





kV






( )average

kk k k

VV

l=



as

averageV

( )

N

k k k k

N


1


kV



kVf

(22)

6



i i

i

minusminus

minus

minuslt

f f

f

1

6

110 (24)


αf

ˆˆ( )NMSE

minusequiv

α αα

α

2

2100 (25)

255ˆ( ) logˆ

M NPSNR equiv

minus

f fff f

2

10 210 (26)





kh hotimes c

imate of

or clo

ρ = 2








7
















8

quences




VI CONCLUSION




( )( )

( )( )

( ) ( ) (

TT Tk k

k kk T T T T

k k

a b


⎡ ⎤ ⎡part part⎢ ⎥ ⎢⎣ ⎦ ⎣

d eS α fE

d e x i j y i j) ⎤⎥⎦ (27)


( )( )

( ) ( )

( ) ( ) ( ) ( )

( ) ( ) ( )

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )

( )( ) ( ) ( )

k k kT TT T

k kk k

k k k

k k g g k k g g

k g gk

k k g g

d d M N e e M N

f M Nfdiagd d M N

ρ ρ ρ ρ

ρ ρ

ρ ρ

k





1 11 1

( )( ) ( ) ( )

k g gk

k k g

f M Nfdiage e M N

ρ ρ

ρ ρ g


1 11 1

(28) where

( )( ) ( )(( ) )

( )( ) ( )

( )( )

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )( )

( ) ( )

k

k

k k tl k k

k k bl k k br

k

k bl tl k br tr

f i jd i j



e i j f f e i j f f

partpart


+ minus +=

part


1 1 1

1

1

tr

and

( )( ) ( )(( ) )

( )( ) ( )

( )( )

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )( )

( ) ( )

k

k

k k tl k k tr

k k bl k k br

k

k tr tl k br bl

f i je i j



d i j f f d i j f f

partpart


+ minus +=

part


1 1 1

1

1


( )( )

[ ( ) ( ) ( )( ) ( ) ( )]

kT T

k k

k bl tl k br tr

k tr tl k br bl

diagdiag



minus minus + minus

S α fd e


1

1

(29)


( )

( )( )( )

( ) ( )

( ) ( ) ( ) ( )

T TT T T Tk k

T T T T

⎡ ⎤ ⎡ ⎤part part⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦= =

⎡ ⎤ ⎡ ⎤part part⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦

d e x i j y i jI



of kE in (18)

REFERENCE















9


























Pixels of LRimage 2




Update

and


and ai+1




iΔfiΔα

i i i+ = +Δf f f1






x

y

lk

ftl ftr

fbl fbr

fk(i j)dk

ek



10

(a) (b)

(c) (d) (e)


(a) (b) (c)


(a)





(a) (b)

(c) (d) (e)

(a) (b) (c)



(a) (b) (c)

(d) (e) (f)

(a) (b)

(c) (d) (e)





Our proposed method

2656 2775 3087


and-add method [14]

Joshirsquos method

[21]

Our proposed method


11







Window 179 2507 00034 2878 202 2521 00021 2891 00502 2617 00023 2882

Hill 149 2788 00045 2969 198 2702 00046 2965 00971 2828 00035 2968 Lena 119 3004 00037 3339 289 2911 00031 3339 00864 3178 00026 3342






12




( )average

kk k k

VV

l=



as

averageV

( )

N

k k k k

N


1


kV



kVf

(22)

6



i i

i

minusminus

minus

minuslt

f f

f

1

6

110 (24)


αf

ˆˆ( )NMSE

minusequiv

α αα

α

2

2100 (25)

255ˆ( ) logˆ

M NPSNR equiv

minus

f fff f

2

10 210 (26)





kh hotimes c

imate of

or clo

ρ = 2








7
















8

quences




VI CONCLUSION




( )( )

( )( )

( ) ( ) (

TT Tk k

k kk T T T T

k k

a b


⎡ ⎤ ⎡part part⎢ ⎥ ⎢⎣ ⎦ ⎣

d eS α fE

d e x i j y i j) ⎤⎥⎦ (27)


( )( )

( ) ( )

( ) ( ) ( ) ( )

( ) ( ) ( )

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )

( )( ) ( ) ( )

k k kT TT T

k kk k

k k k

k k g g k k g g

k g gk

k k g g

d d M N e e M N

f M Nfdiagd d M N

ρ ρ ρ ρ

ρ ρ

ρ ρ

k





1 11 1

( )( ) ( ) ( )

k g gk

k k g

f M Nfdiage e M N

ρ ρ

ρ ρ g


1 11 1

(28) where

( )( ) ( )(( ) )

( )( ) ( )

( )( )

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )( )

( ) ( )

k

k

k k tl k k

k k bl k k br

k

k bl tl k br tr

f i jd i j



e i j f f e i j f f

partpart


+ minus +=

part


1 1 1

1

1

tr

and

( )( ) ( )(( ) )

( )( ) ( )

( )( )

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )( )

( ) ( )

k

k

k k tl k k tr

k k bl k k br

k

k tr tl k br bl

f i je i j



d i j f f d i j f f

partpart


+ minus +=

part


1 1 1

1

1


( )( )

[ ( ) ( ) ( )( ) ( ) ( )]

kT T

k k

k bl tl k br tr

k tr tl k br bl

diagdiag



minus minus + minus

S α fd e


1

1

(29)


( )

( )( )( )

( ) ( )

( ) ( ) ( ) ( )

T TT T T Tk k

T T T T

⎡ ⎤ ⎡ ⎤part part⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦= =

⎡ ⎤ ⎡ ⎤part part⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦

d e x i j y i jI



of kE in (18)

REFERENCE















9


























Pixels of LRimage 2




Update

and


and ai+1




iΔfiΔα

i i i+ = +Δf f f1






x

y

lk

ftl ftr

fbl fbr

fk(i j)dk

ek



10

(a) (b)

(c) (d) (e)


(a) (b) (c)


(a)





(a) (b)

(c) (d) (e)

(a) (b) (c)



(a) (b) (c)

(d) (e) (f)

(a) (b)

(c) (d) (e)





Our proposed method

2656 2775 3087


and-add method [14]

Joshirsquos method

[21]

Our proposed method


11







Window 179 2507 00034 2878 202 2521 00021 2891 00502 2617 00023 2882

Hill 149 2788 00045 2969 198 2702 00046 2965 00971 2828 00035 2968 Lena 119 3004 00037 3339 289 2911 00031 3339 00864 3178 00026 3342






12



7
















8

quences




VI CONCLUSION




( )( )

( )( )

( ) ( ) (

TT Tk k

k kk T T T T

k k

a b


⎡ ⎤ ⎡part part⎢ ⎥ ⎢⎣ ⎦ ⎣

d eS α fE

d e x i j y i j) ⎤⎥⎦ (27)


( )( )

( ) ( )

( ) ( ) ( ) ( )

( ) ( ) ( )

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )

( )( ) ( ) ( )

k k kT TT T

k kk k

k k k

k k g g k k g g

k g gk

k k g g

d d M N e e M N

f M Nfdiagd d M N

ρ ρ ρ ρ

ρ ρ

ρ ρ

k





1 11 1

( )( ) ( ) ( )

k g gk

k k g

f M Nfdiage e M N

ρ ρ

ρ ρ g


1 11 1

(28) where

( )( ) ( )(( ) )

( )( ) ( )

( )( )

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )( )

( ) ( )

k

k

k k tl k k

k k bl k k br

k

k bl tl k br tr

f i jd i j



e i j f f e i j f f

partpart


+ minus +=

part


1 1 1

1

1

tr

and

( )( ) ( )(( ) )

( )( ) ( )

( )( )

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )( )

( ) ( )

k

k

k k tl k k tr

k k bl k k br

k

k tr tl k br bl

f i je i j



d i j f f d i j f f

partpart


+ minus +=

part


1 1 1

1

1


( )( )

[ ( ) ( ) ( )( ) ( ) ( )]

kT T

k k

k bl tl k br tr

k tr tl k br bl

diagdiag



minus minus + minus

S α fd e


1

1

(29)


( )

( )( )( )

( ) ( )

( ) ( ) ( ) ( )

T TT T T Tk k

T T T T

⎡ ⎤ ⎡ ⎤part part⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦= =

⎡ ⎤ ⎡ ⎤part part⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦

d e x i j y i jI



of kE in (18)

REFERENCE















9


























Pixels of LRimage 2




Update

and


and ai+1




iΔfiΔα

i i i+ = +Δf f f1






x

y

lk

ftl ftr

fbl fbr

fk(i j)dk

ek



10

(a) (b)

(c) (d) (e)


(a) (b) (c)


(a)





(a) (b)

(c) (d) (e)

(a) (b) (c)



(a) (b) (c)

(d) (e) (f)

(a) (b)

(c) (d) (e)





Our proposed method

2656 2775 3087


and-add method [14]

Joshirsquos method

[21]

Our proposed method


11







Window 179 2507 00034 2878 202 2521 00021 2891 00502 2617 00023 2882

Hill 149 2788 00045 2969 198 2702 00046 2965 00971 2828 00035 2968 Lena 119 3004 00037 3339 289 2911 00031 3339 00864 3178 00026 3342






12




8

quences




VI CONCLUSION




( )( )

( )( )

( ) ( ) (

TT Tk k

k kk T T T T

k k

a b


⎡ ⎤ ⎡part part⎢ ⎥ ⎢⎣ ⎦ ⎣

d eS α fE

d e x i j y i j) ⎤⎥⎦ (27)


( )( )

( ) ( )

( ) ( ) ( ) ( )

( ) ( ) ( )

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )

( )( ) ( ) ( )

k k kT TT T

k kk k

k k k

k k g g k k g g

k g gk

k k g g

d d M N e e M N

f M Nfdiagd d M N

ρ ρ ρ ρ

ρ ρ

ρ ρ

k





1 11 1

( )( ) ( ) ( )

k g gk

k k g

f M Nfdiage e M N

ρ ρ

ρ ρ g


1 11 1

(28) where

( )( ) ( )(( ) )

( )( ) ( )

( )( )

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )( )

( ) ( )

k

k

k k tl k k

k k bl k k br

k

k bl tl k br tr

f i jd i j



e i j f f e i j f f

partpart


+ minus +=

part


1 1 1

1

1

tr

and

( )( ) ( )(( ) )

( )( ) ( )

( )( )

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )( )

( ) ( )

k

k

k k tl k k tr

k k bl k k br

k

k tr tl k br bl

f i je i j



d i j f f d i j f f

partpart


+ minus +=

part


1 1 1

1

1


( )( )

[ ( ) ( ) ( )( ) ( ) ( )]

kT T

k k

k bl tl k br tr

k tr tl k br bl

diagdiag



minus minus + minus

S α fd e


1

1

(29)


( )

( )( )( )

( ) ( )

( ) ( ) ( ) ( )

T TT T T Tk k

T T T T

⎡ ⎤ ⎡ ⎤part part⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦= =

⎡ ⎤ ⎡ ⎤part part⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦

d e x i j y i jI



of kE in (18)

REFERENCE















9


























Pixels of LRimage 2




Update

and


and ai+1




iΔfiΔα

i i i+ = +Δf f f1






x

y

lk

ftl ftr

fbl fbr

fk(i j)dk

ek



10

(a) (b)

(c) (d) (e)


(a) (b) (c)


(a)





(a) (b)

(c) (d) (e)

(a) (b) (c)



(a) (b) (c)

(d) (e) (f)

(a) (b)

(c) (d) (e)





Our proposed method

2656 2775 3087


and-add method [14]

Joshirsquos method

[21]

Our proposed method


11







Window 179 2507 00034 2878 202 2521 00021 2891 00502 2617 00023 2882

Hill 149 2788 00045 2969 198 2702 00046 2965 00971 2828 00035 2968 Lena 119 3004 00037 3339 289 2911 00031 3339 00864 3178 00026 3342






12










9


























Pixels of LRimage 2




Update

and


and ai+1




iΔfiΔα

i i i+ = +Δf f f1






x

y

lk

ftl ftr

fbl fbr

fk(i j)dk

ek



10

(a) (b)

(c) (d) (e)


(a) (b) (c)


(a)





(a) (b)

(c) (d) (e)

(a) (b) (c)



(a) (b) (c)

(d) (e) (f)

(a) (b)

(c) (d) (e)





Our proposed method

2656 2775 3087


and-add method [14]

Joshirsquos method

[21]

Our proposed method


11







Window 179 2507 00034 2878 202 2521 00021 2891 00502 2617 00023 2882

Hill 149 2788 00045 2969 198 2702 00046 2965 00971 2828 00035 2968 Lena 119 3004 00037 3339 289 2911 00031 3339 00864 3178 00026 3342






12



x

y

lk

ftl ftr

fbl fbr

fk(i j)dk

ek



10

(a) (b)

(c) (d) (e)


(a) (b) (c)


(a)





(a) (b)

(c) (d) (e)

(a) (b) (c)



(a) (b) (c)

(d) (e) (f)

(a) (b)

(c) (d) (e)





Our proposed method

2656 2775 3087


and-add method [14]

Joshirsquos method

[21]

Our proposed method


11







Window 179 2507 00034 2878 202 2521 00021 2891 00502 2617 00023 2882

Hill 149 2788 00045 2969 198 2702 00046 2965 00971 2828 00035 2968 Lena 119 3004 00037 3339 289 2911 00031 3339 00864 3178 00026 3342






12



(a) (b)

(c) (d) (e)

(a) (b) (c)



(a) (b) (c)

(d) (e) (f)

(a) (b)

(c) (d) (e)





Our proposed method

2656 2775 3087


and-add method [14]

Joshirsquos method

[21]

Our proposed method


11







Window 179 2507 00034 2878 202 2521 00021 2891 00502 2617 00023 2882

Hill 149 2788 00045 2969 198 2702 00046 2965 00971 2828 00035 2968 Lena 119 3004 00037 3339 289 2911 00031 3339 00864 3178 00026 3342






12







Window 179 2507 00034 2878 202 2521 00021 2891 00502 2617 00023 2882

Hill 149 2788 00045 2969 198 2702 00046 2965 00971 2828 00035 2968 Lena 119 3004 00037 3339 289 2911 00031 3339 00864 3178 00026 3342






12

Documents

Joint Image Registration and Super-Resolution From Low-Resolution Images With Zooming Motion