Wavelet Paper

8/3/2019 Wavelet Paper

1/18

CIRCUITSSYSTEMS SIGNALPROCESSVOL. 19, NO. 4,2000, PP. 321-338

A WAVELET-BASEDI N T E R P O L A T I O N - R E S T O R A T I O NM E T H O D F O RSUPERRESOLUTION (WAVELETSUPERRESOLUTION)*Nhat Nguyen 1 and Peyman Milanfar 2

A bs trac t. Superresolution produces high-quality, high-resolution im ages from a set ofdegraded, low-resolution images where relative frame-to-frame motions provide differ-ent look s at the scene. Sup erresolution translates data temp oral bandwidth into enhancedspatial resolution. If considered together on a reference grid, given low-resolution dataare nonuniformly sampled. However, data from each frame are sampled regularly on arectangular grid. This special typ e o f nonun iform sam pling is called interlaced sam pling.W e propose a new wavelet-based interpolation-restoration algorithm fo r superresolution.O ur efficient wavelet interpolation technique takes advantage o f the regularity and structureinherent in interlaced data, thereby significantly reducing the com putational burden. Wepresent one- and two-dimensional superresolution experiments to demonstrate the effec-tiveness o f ou r algorithm.K ey w ord s: Superresolution, wavelet interpolation, interlaced samp ling.

1 . I n t r o d u c t i o n

I m a ge supe r r e so lu t ion r e f e r s t o ima g e p r oc e s s ing a lgo r i thms tha t P r O duce h igh -qua l i t y , h igh - r e so lu t ion ( H R ) im a ge s f r om a s e t o f low - qua l i ty , l ow - r e so lu t ion( LR ) ima ge s . The r e i s a lw a ys a de ma nd f o r be t t e r - qua l i t y ima ge s . H ow e ve r , t hel e ve l o f ima ge de ta i l i s c r uc i a l f o r t he pe r f o r ma nc e o f s e ve r a l c ompu te r v i s iona lgo r i thms . Ta r ge t r e c ogn i t ion , de t e c t ion , a nd ide n t i f i c a t ion sys t e ms a r e someof the mi l i t a r y a pp l i c a t ions tha t r e qu i r e t he h ighe s t - qua l i t y a c h ie va b le ima ge s .L ic e nse p l a t e r e a de r s , su r ve i l l a nc e mon i to r s , a nd me d ic a l ima g ing a pp l i c a t ionsa r e e xa m ple s o f c iv i l ia n a pp l i ca t ions w i th the s a me r e qu i r e me n t . I n m a ny v i sua l

* Received December 6, 1999; revised April 15, 2000.t Scientific Computing and Computational Mathematics Program, Stanford University, Stanford,

California 94305-9025, USA. E-mail: [email protected]. This work was sup-ported in part by the National Science Foundation Grant CCR-9984246.

2 Department of Electrical Engineering, University of California, Santa Cruz, California 95064,USA. E-mall: [email protected]


2/18

322 NGUYENAND MtLANFAR

a p p l i c a t i o n s , t h e im a g i n g s e n so r s h a v e p o o r r e so l u t io n o u t p u t s . W h e n r e so l u t i o nc a n n o t b e i m p r o v e d b y r e p l a c in g s e n s o rs , e i th e r b e c a u s e o f c o s t o r h a r d w a r e p h y s -i c a l l i m i t s , we r e so r t t o u s i n g a su p e r r e so l u t i o n a l g o r i t h m s . E v e n wh e n su p e r i o re q u i p m e n t i s a v a i l a b le , su p e r r e so l u t i o n a l g o r i t h m s a r e a n i n e x p e n s i v e a l te r n a t iv e .

Su p e r r e so l u t i o n , a t i ts c o r e , ~ s a p r o c e s s b y wh i c h o n e g a i n s sp a t i a l r e so l u t i o ni n r e t u rn f o r t e m p o r a l b a n d wi d t h . T e m p o r a l b a n d w i d t h r e f e r s t o t h e a v a i l a b i li t yo f m u l t i p l e n o n r e d u n d a n t i m a g e s o f t h e s a m e sc e n e . L u k o sz [ 8 ], [ 9] wa s f i r stt o r e a l i z e t h is p o s s i b i li ty . Ho w e v e r , su p e r r e so l u t io n c a n n o t p e r f o r m m i r a c l e s . W ec a n n o t e x p e c t t o e x t r a c t s u b p ix e l i n f o r m a t i o n f r o m a s e q u e n c e o f i d e nt ic a l i m a g e s ;t h e r e m u s t b e n o n r e d u n d a n t i n f o r m a t i o n a m o n g t h e i m a g e s . W e m u s t b e a b l e t ot ra n s la t e d a t a te m p o r a l b a n d w i d t h i n t o s u b p i x e l i m a g e c o nt e n t. E a c h L R f l a m ep r o v i d e s a d i f f e r e n t " l o o k " a t t h e s a m e sc e n e . T h e o r e t i c a l ly , b y p r o v i d i n g d i f f e r e n tl i g h t i n g c o n d i t i o n s o r d i f f e r e n t s e n so r s , su p e r r e so l u t i o n c a n b e a c h i e v e d w i t h o u tr e l a t iv e s c e n e m o t i o n . T h i s i s th e m u l t i c h a n n e l d a t a f u s i o n su p e r r e so l u t i o n p r o b -l e m . I n t h i s p a p er , h o w e v e r , w e a s s u m e o n e i m a g i n g d e v i c e a n d t h e s a m e l i g h ti n gc o n d i t i o n s , a n d w e r e q u i r e th a t t h e r e b e so m e re l a ti v e m o t i o n f r o m f r a m e to f r a m e .F r a m e - t o - f r a m e m o t i o n c a n b e a c o m b i n a t i o n o f c a m e r a p l a t f o r m m o t i o n r e la t iv et o t h e s c e n e , m o v i n g o b j e c t s in t h e s c e n e , a n d c a m e r a j it te r s . F o r e x a m p l e , i ns a t el li te i m a g i n g , i m a g e s o f t h e g r o u n d b e l o w a r e c a p t u r e d a s t h e c a m e r a o r b i tst h e e a r t h , wh e r e a s i n su r v e i l l a n c e a n d m o n i t o r i n g a p p l i c a t i o n s , t h e c a m e r a i sp l a c e d o n a f ix e d p l a t f o r m , a n d o b se r v e d o b j e c t s m o v e w i t h in t h e s c e n e . M o t i o na n d n o n r e d u n d a n t i n f o r m a t i o n a r e w h a t m a k e s u p e r r es o l u ti o n p o s s i b le . W i th t hi si n f o r m a t i o n , w e a r e a b l e t o e x t r a c t su b p i x e l c o n t e n t a t a h i g h e r r e so l u t io n t h a n i ne a c h i n d i v i d u a l f r a m e .

F i g u r e 1 i ll u s t ra t e s th e p r o b l e m se t u p . T h e f i g u r e sh o w s t h r e e 4 x 4 p i x e l L Rf r a m e s o n a n 8 8 HR g r i d . E a c h sy m b o l ( sq u a r e , c ir c le , t r ia n g l e ) i n d i c a te s t h es a m p l i n g p o i n t s o f a f r a m e w i t h r e s p e c t t o t he H R g r id . W e p i c k a n a r b i tr a ry f r a m ea s a r e f e r e n c e f r a m e ; in t h is c a se , th e f r a m e m a r k e d b y t h e c i rc u l a r sy m b o l s . T h esa m p l i n g g r i d f o r t h e tr i a n g u l a r f r a m e is a s i m p l e tr a n s l a ti o n o f t h e r e f e r e n c ef r a m e g r id . T h e m o t i o n b e t w e e n t he s a m p l i n g g ri d f o r th e s q u a r e f r a m e a n d t her e f e r e n c e f r a m e g r i d i n c l u d e s t r a n s l a t i o n a l , r o t a t i o n a l , a n d m a g n i f i c a t i o n ( z o o m )c o m p o n e n t s .

T h e f o r w a r d r e l a t i o n s h i p b e t w e e n a d e g r a d e d , L R f l a m e a n d t h e i d e a l H Ri m a g e c a n b e d e sc r i b e d a s f o l l o ws [ 5 ] :

f k = D C E k x + I l k , 1 < k < p , ( i )wh e r e D i s t h e d o wn sa m p l i n g o p e r a t o r , C i s t h e b l u r t i n g / a v e r a g i n g o p e r a t o r ,E k ' S a r e t h e a f fi n e t r a n s fo r m s t h a t m a p t h e H R g r i d c o o r d i n a te s y s t e m t o t h eL R g r i d s y s t e m s , x i s t h e u n k n o w n i d e a l H R i m a g e , a n d n k ' s a r e t h e a d d i t i v en o i s e v e c t o r s . T h e L R f r a m e s f k a r e g i v e n , a n d t h e d e c i m a t i o n o p e r a t o r D i sk n o w n . B e c a u s e C h a r g e -C o u p l e d D e v i c e ( C C D ) s e n s o rs o n t h e s a m e a r r a y h a v eprac t i ca l ly iden t i ca l cha rac te r i s t i c s , C i s spa t i a l ly l i nea r sh i f t i nva f i an t . In th i sp a p e r , w e a s s u m e t h a t f r a m e - t o - f r a m e m o t i o n a n d b lu r r in g p a r a m e t e r s a r e k n o w n


3/18

" i AA

[ ]

m mmE

Am

A 9

n 9 [ ] A A[ ]m

W A V E L E T S U P E R R E S O L U T I O N 3239 Q o 9

4

9 9 9 9

9 9 9 9

9 Q # 9

i

A

Frame 1

A & A ~,

lk J, A &

~k dL ~, ~k

A A A A

z [] E m

I A A A A . 9 , .[ ]] [] B

a[ ]I m n m

F i g u r e 1 . L o w - r e s o l u t i o n d a ta o n a h i g h - r e s o lu t i o n g r id .

Frame 2

Frame 3

a p r io r i o r ha ve be e n e s t im a te d ( c f. [ 12 ], [ 13 ] ) f r om g ive n da ta a nd tha t f r a m e -to - f r a m e m ot ion i s pu r e ly t ra ns la tiona l o r ha s be e n c o r r e c te d to be so . F ina l ly ,w i th m ul t ip l e inde pe nde n t sou r c e s o f e r r o r , t he c e n t r a l l im i t t he o r e m a l lows usto a s sum e G a uss ia n n o r m a l d i s tr ibu tion f o r the a dd i t ive no i se ve c to r s nk wi thp o s s i b l y u n k n o w n v a r i a n ce .

The sh i f t i nva r i a nc e p r ope r ty a l lows the ope r a to r s C a nd Ek to c om m ute .He nc e , ( 1 ) c a n be r e w r i t te n a s

f k = D E k C x + nk, 1 < k < p . (2)Equa t ion ( 2 ) a nd F igur e 1 m ot iva te ou r two- s t e p a ppr oa c h to supe r r e so lu t ion .F i r s t , u s ing the LR da ta f r a m e sa m ple s f k , 1 < k < p , w e in t e r po la t e f o r C x ,the b lu r r e d ve r s ion o f the o r ig ina l HR im a ge . Ne x t , we de c o nvo lv e the b lu r C toob ta in a n e s t im a te f o r x . De c on vo lu t ion ha s be e n a tho r oug h ly s tud ie d p r ob le m ,a nd se ve r a l robus t t e c hn iqu e s a r e a va i la b le . Th e r e s t o f the pa pe r w i l l t he r e f o r em os t ly a ddr e s s the in t e r po la t ion s t e p o f the a lgo ri thm .

Th e r e i s i nhe r e n t s t r uc tu r e a nd r e gu la r ity in the g r id o f LR sa m pl ing po in t sf o r supe r r e so lu t ion . I n F igu r e 1 , i f p ixe l va lue s f r om a l l f r a m e s a r e c o ns ide r e dtoge the r , the da ta a r e i r r e gu la rly sa m ple d . Ho we ve r , f o r e a c h f r a m e , da ta po in t sa r e sa m ple d on a r e c ta ngu la r g r id . Th i s spe c ia l c a se o f i r r e gu la r s a m pl ing i sc a l l e d in t e r l a c e d sa m pl ing [ 15 ]. Ou r in t e r po la t ion m e th od t a ke s a dva n ta ge o f th i s


4/18

324 N G U Y E N A N D M I L A N F A Rsampl ing s t ruc ture . The method i s based on the mul t i resolu t iona l bas i s f i t t ingrecons t ruc t ion (MBFR) me thod , de sc r ibed in t he pape r by Ford and E t t e r [6 ] .In pa r t i cu l a r , w e ex t end the w ork o f Ford and E t t e r t o tw o-d imens iona l (2D )inte r laced sampl ing gr ids .

1 . I . P r e v i o u s w o r k

Superresolut ion recons t ruc t ion f rom mul t ip le f rames i s a re la t ive ly new c lassof res tora t ion problems. Most techniques proposed for superresolut ion fa l l in toone of three main ca tegor ies : f requency domain , i t e ra f ive spa t ia l domain , andinte rpola t ion-res tora t ion . In th i s paper, w e wi l l exam ine the las t c lass .

Sauer and A l lebach [14] w ere the f irs t to con s ider superresolution as an in te r-po la t i on p rob lem w i th nonun i fo rmly sampled da t a . They used a p ro jec t ion o n toconvex se ts (POCS) a lgor i thm to recons t ruc t the unknown va lues . Namely , theycons ide red

~ - ( / + 1 ) = P n " " P 2 P I f i " (l) , (3 )w h e r e . f i - ( l ) i s the l th appro xim ate of the idea l HR imag e .T, and P i a r e p r o j e c u o nopera tors tha t correc t for e r rors be tween 5 (l ) an d ~- and im pose band - l imi tednesscons t ra in ts . The solu t ion to the f ixed-point it e ra tion (3) i s the i r es t imate to .T.Aizawa e t a l . [1] a l so modeled superresolut ion as an in te rpola t ion problemw i th non un i fo rm sam pl ing and used a fo rm ula re la t ed to t he Shannon sampl ingtheore m to es t imate va lues on an H R gr id . The w ork of both [1] and [14] ignoredthe e ffec t of sensor b lurr ing . T eka lp e t a l. [18] la te r extended these a lgo r i thm s toinc lud e b lurr ing an d sensor noise and prop osed the add i t iona l res tora t ion s tep forthe in te rpola t ion a lgor i thm s. Ur and Gross [19] cons idered Papou l i s ' gen era l izedmul t ichanne l sampl ing theorem for in te rpola t ing va lues on a h igher-resolut iongr id . Beca use l ight de tec tors a re not idea l lowp ass f il te rs , so m e h igh-frequen cyinfor m at ion about the scene i s represented in the ima ge in a l iased form. Pap oul i s 'theorem recons t ruc ts th i s a l iased h igh-frequency content by taking a proper lyw e igh ted su m of t he spec tra l in fo rma t ion f rom the L R f rames . Sbeka r fo roush andChe l l appa [15] ex t ended Papou l i s' t heorem fo r me rg ing o f nonun i fo rm sam ple so f mul t ip le chann e ls in to h igh-resolut ion da ta .

P rev ious w ork d id no t cons ide r t he impl i ca t i ons o f t he 2D in t e r l a ced sam-p i ing s t ruc tu re on the com puta t iona l comp lex i ty o f t he re su l ti ng a lgor it hms . T hecomputa t iona l complex i ty fo r ex i s t i ng a lgor i t hms fo r 2D da t a i s squa red tha tfor 1D da ta . As we wi l l show, by exploi t ing sampl ing regula r i ty , the computa-t iona l burden for our a lgo r i thm does n ot dras t ica lly increase for 2D da ta . In fac t ,com puta t iona l com plex i ty o f t he a lgor i thm fo r 2D da t a i s on ly tw ice tha t fo r 1Dprob lems .

Th e out l ine of the res t of the paper i s as fo llows. W e br ie f ly review m ul t i resolu-t ion ana lys i s wi th or tho gon a l wave le ts in Sec t ion 2 . In Sec t ion 3 , we descr ibe our1D and 2 D wave le t in te rpola t ion meth od for in te r laced da ta . Sec t ion 4 d iscusses


5/18

WAVELET SUPERRESOLUTION 325imp lemen ta t i on and com plex i ty i s sue s fo r t he a lgor i thm. S ec t ion 5 show s in te r -po la t i on and supe r re so lu t ion expe r imen t s d emo ns t ra ti ng the e f fec ti veness o f o u rt echn iques . W e conc lude w i th som e comm ent s i n Sec t ion 6 .

2 . M u l t i r e s o l u t io n a n a l y s i s w i t h o r t h o n o r m a l w a v e l e tsThe fundamenta l concep t beh ind w ave le t t heory i s t he decompos i t i on o f s ig -na ls in to components a t d i f fe rent sca les or resolu t ions . The advantage of th i sdecomposi t ion i s tha t s igna l t rends a t d i f fe rent sca les can be i so la ted and s tud-ied . Globa l t rends can be examined a t coarse r sca les , whereas loca l var ia t ionsare be t te r ana lyzed a t f ine sca les . This sec t ion wi l l present a br ie f summary ofo r thonorm a l w ave le t mul t ir e so lu t ion ana lys i s o f 1D and 2D s igna l s. We w i ll on lyreview essent ia l ideas necessary for the mate r ia l in la te r sec t ions . For more de-ta i led t rea tments of wave le t s , the reader i s re fe rred to the exce l lent books byS t rang and N gu yen [17] and M a l l a t [10 ].

2 .1 . M u l t ir e s o l u t io n a n a l y s i s f o r 1 D s i g n a l s

L et L2 (R ) b e the vec tor space of square- in tegrable 1D s igna ls f ( t ) . There exis t sa sequence o f ne s t ed approx ima t ion subspaces V j , j ~ Z , and a sca l ing func t ionq~(t ) sa t i s fy in g the fo l lowin g requi rem ents [ 17]:

( i) V j C V j + I an d [ ' ~ j ~ z V j . ~ - {0 } an d [ . J j ~ z V j .= L 2 ( R )(ii) f ( t ) ~ V j r f ( 2 t ) ~ V j + l

(iii) f ( t ) ~ V0 4:~ f ( t - k ) ~ V 0(iv) V0 has an orth on orm al basis {~b(t - k)}Be cau se the se t {~b( t -k)} i s an or thon orm al basi s for V0, d i la tions and t rans la t ionsof ~b( t ) , { ~ b j , k ( t ) = 2 J / 2 q s ( 2 J t - - k)}k~Z, fo rm an o r thonorma l ba s i s fo r V j .Fur therm ore , becau se Vo C V1, the sca l ing func t ion ~b(t ) sa t is f ies the fo l low ingtwo -sca le d i la t ion equa t ion:

d p (t ) = ~ _ , c t q b ( 2 t - k ) , (4)kfo r som e se t o f exp ans ion coe f f i ci en ts c~ .

For a func t ion f ( t ) e La(R) , t he p ro j ec t ion f j ( t ) o f f ( t ) onto the subspaceV j rep re sen t s an approx ima t ion o f t ha t func t ion a t sca le j . The approx ima t ion be -com es m ore accura t e a s j i nc rea ses . The d i f fe rence i n succe ss ive approx ima t ionsg j ( t ) = f j + l ( t ) - f j (t ) i s a de ta i l s igna l tha t lives in a wav e le t subspace W j . Infac t , w e can decom pose the approx ima t ion space V j+ l a s

V j + I = V j (~ W j . ( 5)Equ a t ion (5 ) show s us one o f the fundam enta l r e a sons w h y w ave le ts have beenused so success fu l ly i n s igna l p roce ss ing . S igna l s c an be nea rly b roken d ow n in to


6/18

3 2 6 N G U Y E N AND MILANFARa c o a r s e a p p r o x i m a t i o n s i g n a l a n d a f i n e d e t a i l si g n a l . A n y f ( t ) ~ L 2 ( R ) c a n b ew r i t t e n a s a s u m o f i ts a p p r o x i m a t e a t s o m e s c a le J a l o n g w i t h t h e s u b s e q u e n td e t a i l c o m p o n e n t s a t s c a l e J a n d h ig h e r. H e n c e ,

L 2 ( R ) = V j ~ ~ W j . (6 )j___J

A s i n t h e a p p r o x i m a t i o n s p a c e s c as e , t h e w a v e l e t s p a c e s W j a r e s p a n n e d b y as e t o f o r t h o n o r m a l b a s i s f u n c t i o n s { T e L l(t ) = 2J/2~p(2Jt - k ) } k s Z , w h i c h a r ed i l a t i o n s a n d t r a n s l a t io n s o f a s i n g l e w a v e l e t f u n c t i o n 7 t ( t ) . F u r t h e r m o r e , t h ew a v e l e t f u n c t i o n s a t is f ie s t h e w a v e l e t e q u a t io n

~p t) = ~ E dkq~ 2 t - - k ) , ( 7 )kf o r s o m e s e t o f e x p a n s i o n c o e f f ic i e n t s d k .

B y e q u a t i o n ( 6 ) ' w e c a n e x p a n d a n y f u n c t io n f ( t ) ~ L Z ( R ) a s fo l l o w s :f ( t ) = E aL k(PL k( t ) + E E b j,k*[tJ k ( t ) ' (8 )k ~ Z j> _ J k ~ Z

w h e r eaJ,k = f f (t )dPJ, k ( t ) d tb j,k = f f ( t ) ~ j , k ( t ) d t

a r e t h e e x p a n s i o n c o e f f ic i e n t s f o r f ( t ) .2 . 2 . M u l ti re s o lu t io n a n a lys i s f o r 2 D im a g es

T h e w a v e l e t m o d e l i n t h e p r e v i o u s s e c ti o n f o r 1 D s i g n a ls c a n b e e x t e n d e d t o 2 Di m a g e s . W e d e s c r i b e i n t h i s s u b s e c t i o n a s e p a r a b l e m u l t i r e s o l u t i o n a n a l y s i s o fL 2 ( R 2 ) s tu d i e d b y M e y e r [ 1 1 ] a n d M a l l a t [ 10 ]. G i v e n a m u l t ir e s o l u ti o n a n a l y s i s( v ~ l ) ) j e Z o f L 2 ( R ) , a s e t o f n e s te d s u b s p a c e s ( V ~ 2 ) )j ~ z f o r m s a m u l t i re s o t u t io na p p r o x i m a t io n o f L Z ( R 2 ) w i t h e a c h v e c t o r s p a c e ~r b e i n g a t e n s o r p r o d u c t o fi d e n t i c a l 1 D a p p r o x i m a t i o n s p a c e s

V j(2) ~---v j ( l ) ~ v ) l ) , ( 9 )F u r t h e r m o r e , t h e s c a li n g f u n c t i o n O ( t , s ) f o r t h e 2 D m u l t ir e s o lu t i o n s u b s p a c e sc a n b e d e c o m p o s e d a s

9 ( t , s ) = 4 , ( t ) ~ ( s ) , ( 1 0 )w h e r e ~ b ( t) i s th e 1 D s c a l i n g f u n c t i o n o f t h e m u l t i re s o l u t io n a n a l y s i s ( V ( ~ )r j~Z"T h e s e t o f f u n c t io n s

9 j , k , l ( t , S ) = d p j , k ( t ) ~ j , l ( S ) , j , k , 1 E Z ( i 1 )


7/18

WAVELET SUPERRESOLUTION 327

(2)(2) T he 2D wav ele t subspaces W j a re gene ra teds an o r thono rma l ba s is fo r V j .by three wave le t s to capture de ta i l informat ion in the hor izonta l , ve r t ica l , andd iagon a l d i rect ions :

tph( t , s ) = ~lr( t )qb(s) (12)q J v ( t , s ) = q ~ ( t ) q / ( s ) (13)q j d ( t , s ) = a p ( t ) T t ( s ) . (14)

(2) is th e s ethe co r re spond ing o r thonorma l w ave le t ba s is fo r W jqjh,k, t ( t , s) = ~ t j , ~ ( t ) 4 ~ j , l ( s ) , (15)q J ~ , k ,t (t , s ) = d ? j , k ( t ) ~ p j j ( s ) , (16)W : k , l ( t , s ) = ~ j , k ( t ) ~ j , l ( S ) , j , k , l 6 Z. (17)

A n a logou s t o t he 1D ca se, any image f ( t , s ) ~ L 2(R 2 ) can be ex panded a s a su mof i t s approxim ate imag e a t som e sca le J in V (2) a long w i th subsequen t de ta ilcom pone n t s a t s ca le J and h ighe r.

f ( t , S ) = Z a J , k , l ~ J , k , l ( t , s ) +E E b h , , l * h , d ( t ' s )k,l~Z j> J k,l~Z+ E Z Z E (18)j > J k , 1 6 Z j > _ J , l ~ Z

w i thaj,~,z= f f f (t ,s ) q ~ j ,k , l( t,s ) d t d sbhj , k , ~ = f f ( t , s ) * h , k j ( t , s ) d t d s

f f, k , l = f ( t , s ) q J ~ , k , l ( t, s ) d t d sb d =J,k,l f f f ( t , s ) q J J , k , l ( t , s ) d t d sTh e f i rs t t e rm on the r ight -ha nd s ide of (18) represents the coarse -sca le approx-ima t ion to f ( t , s ) . T h e second te rm represents the de ta i l component in the hor-izonta l d i rec t ion , the th i rd and four th the de ta i l components in the ver t ica l anddiag ona l d i rec t ions , respec tive ly .

3 . W a v e l e t i n t e r p o l a t i o n o f i n t e r l a c e d d a t aTh is se c t ion wi l l descr ibe our in te rpola t ion techn ique for in te r laced da ta . We usethe exp ans ion form ulas (8) and (18) to f i rst es t ima te for the w ave le t coeffic ients .Usin g thes e es t im ates , we in te rpola te for the fun c t ion va lues a t the H R gr id points .


8/18

328 NGUYEN AND MILANFAR

3 .1 . I n t e r p o l a t io n f o r n o n u n i f o r m l y s a m p l e d 1 D s ig n a l sW e f i rs t c o n s i d e r t he c a s e o f n o n u n i f o r m l y s a m p l e d 1 D s ig n a ls . S u p p o s e t h a t w eh a v e a fu n c t i o n f ( t ) f o r w h i c h w e w o u l d l i k e t o c o m p u t e M u n i f o r m l y d i st r ib u t e dv a l u e s , s a y , a t t = 0 , 1 . . . , M - 1. W e a r e g i v e n P n o n u n i f o r m l y s a m p l e d d a t ap o in ts o f f ( t ) a t t = to , t l . . . . . t p - 1 , 0 < ti < M , wh e r e t y p i c al l y , P < M .

W e t a k e th e u n i t - ti m e sp a c i n g g r i d t o b e r e so l u t i o n l e v e l V0 . B y r e p e a t e d a p p l i 'c a t io n o f (5 ) , w e c a n d e c o m p o s e V 0 i n th e f o l l o w i n g f a s h io n :

- IV 0 = V ~ ~ ~ ) W j , J _< - 1 . ( 1 9 )

j = JH e n c e , w e c a n s e p a r a t e f ( t ) ~ V0 i n t o i t s a p p r o x i m a t i o n a n d d e t a i l c o m p o -n e n t s a n d f u r th e r e x p a n d t h es e c o m p o n e n t s i n t h e o rt h o n o r m a l b a s e s o f V j a n d{Wj }- 1>__j>__J:

- If ( t ) = f j ( t ) + Z g j ( t ) , f j ( t ) ~ V j , g j ( t ) ~ W j (20)

j = J- 1

= ~ a j , k c b j ,k ( t ) + Z Z b j , k ~ j , k ( t ) , ( 2 1 )k j = J k

a j , k = f f ( t ) ( p L k ( t ) d tb j ,k = [ f ( t ) q / j , k ( t ) d t .d

Su b s t i t u t i n g i n t h e v a l u e s o f t h e s a m p l e d d a t a , we h a v e a s e t o f P l i n e a r e q u a t i o n s-1

f ( t i ) = Z a j , k d p j , k ( t i ) i = 0 . . . . . P - l . (2 2)k j = JSu pp ose tha t [0 , N ] i s t he supp or t i n t e rva l fo r ~p( t) and l e t t rnax = m ax/ t~ and

train = m ini t i . I n t h e f ir s t su m m a t i o n o n t h e r i g h t -h a n d s i d e o f ( 2 2 ) , o n l y f i n i te l ym a n y t e r m s a r e n o n z e r o b e c a u s e d?j,k(t i) = cp(2Jti - k) i s n o n z e r o i f a n d o n l y i f2 J t i - k i s in the support in terval for ~b( t ) , i .e . ,

0 < 2 J t i - - k < N . (23)T h e r e f o r e ,

- N + [2Jtmin] < k < L2Jtmaxj . (24)S i m i l a r a r g u m e n t s c a n b e m a d e f o r t h e w a v e l e t b a si s f u n c ti o n s ~pj,~( t i ) . L e t S s ={ - N -+- [ 2 Jt m i n] . . . . . [ 2 Jt m a x J } b e t h e s e t o f sh if t s w i t h n o n z e r o c o n t r i b u t i o n o nt h e r i g h t - h a n d s i d e o f (2 2 ) . W e c a n n o w r e w r i t e (2 2 ) a s

-1f ( t i ) = ~ a J , k ~ J , k ( t i ) q - ~ ~ b j , k ~ j , k ( t i ) , i = 0 . . . . . P - - 1 , (25)

k ~ S j j = J k 6 S)


9/18


w h i c h , i n v e c t o r f o r m , b e c o m e s-1

f = G j a j + E H j b j , (26)j= J

w h e r ef = ( f ( t , ) ) i = 0 . . . p _ 1 , , , = b j =

. ' ,kESj kcSja j = (e p J , k ( to ) i= o . . .. . e - l , H j = ( v j ,k ( ti )) i= 0 . . .. . p _ l .T o c o n s t r u c t G j a n d H i , w e ne e d to know ba s is f unc t ion va lue s a t s a mp l ing po in t s{ti }. F o r m os t w a ve le t ba ses , t he r e a r e no c lo se d - f o r m e x p r e s s ions f o r ba s is f un c -t i ons {dp j , k ( t ) , a / s j ,k ( t ) } . H ow e ve r , ba s i s f unc t ion va lue s a t dya d ic po in t s c a n bec a l c u l a t e d e f f i c i e n t ly by r e c u r s ion . A t s c a l e K a nd f o r s ca l ing f unc t ion w i th sup -po r t on [0 , N ] , the se t o f dy adic po in ts is de f in ed to be 9 = {0, 1 / 2 K . . . . . N -1 /2K } . W e c hoo se K l a r ge e nou gh so tha t t he s e t o f s a mp l ing po in t s { 2 J t i -k , 2 J t i - k } c a n b e w e l l a p p r o x i m a t e d b y a s u b s e t o f 7 ) g .

F r o m E q u a t i o n ( 2 6) , w e c a n a p p r o x i m a t e t h e c o a r se - s ca l e a p p ro x i m a t i o n c o e f -f ic i en t s a j b y i g n o r in g t h e d e t ai l c o m p o n e n t s a n d c o n s i d e r in g j u s t

f ~ G j a j . (27)F o r d a nd E t t e r [6 ] r e c o m m e nd tha t J be c hose n so t ha t t he sy s t e m a bo ve c a nbe so lve d in a leas t - squares sense ; tha t i s , P > k2Jtmaxj - F2Jtrran] + N + 1.T h e sc a l e c hose n i s de pe n de n t on the t o t a l num be r o f s a mp le po in t s , the i n t e rva lspa nn e d by the se po in t s , a nd the suppo r t s i z e o f t he s c a l ing a nd w a ve le t f unc -t i ons . Be c a u se ( 27 ) i s a n a pp r ox ima t ion , w e so lve f o r a r e g u l a r i z e d l eas t - squarese s t im a t e i n t h e w a v e l e t d o m a i n

aJ = a rgm ina j I[f - G j a j I t~ + 3 . l l a s 1 1 2 , (28)o r e qu iva l e n t ly , a j = ( G T G j q - 3 . l ) - l G ~ f ( 29 )f o r som e r e gu la r i z a t ion p a r a m e te r 3 .. T h e r e gu la r i z a t ion pa r a m e te r 3. p l a ys a ba l -anc in g ro le in (28) and (29) . I f 3. i s too la rge , the so lu t ion ob ta in ed wi l l be too fa ra w a y f r om the o r ig ina l sy s t e m w e w i sh to so lve . I f 3. i s too sm a l l, no i s e e f f e c t sa r e e xa c e r ba t e d in t he so lu t ion o f t he unde r - r e gu la r i z e d sy s t e m. T h e l e a s t -squa r e se s t ima te ~ j o f t he c oe f f i c i e n t s y i e ld s a c oa r se - sc a l e e s t ima te o f f, d e no te d b y f j ,

b = a . , a j . ( 30 )T h e d i f f e r e n c e b e t w e e n f a n d } j c a n t h e n b e u s e d t o a p p r o x i m a t e t h e w a v e l e tc o e f f c i e n t s b j

g j = f - f j= f - G j f i j,~ H j b j . ( 31 )


10/18

330 NGUYENAND MILANFARB e c a u s e t h e n u m b e r o f n o n z e r o c o e f fi c ie n t s b j , n i s t h e s a m e a s t h e n u m b e r o fn o n z e r o c o e f f i c i e n t s a j , n , e q u a t i o n ( 3 1 ) c a n a l s o b e s o l v e d i n t h e l e a s t - s q u a r e ss e n s e f o r a r e g u l a r i z e d e s t i m a t e o f bj . I n g e n e ra l , t h e r e g u i a ri z a t io n p a r a m e t e r i sc h o s e n t o b e s m a l l fo r th e c o a r s e - s c a l e a p p r o x i m a t i o n a n d l a r g e f o r t h e fi n e - s c al ed e t a i l b e c a u s e t h e s i g n a l - t o - n o i s e r a t i o t e n d s t o b e s m a l l e r i n t h e f i n e s c a l e . I nf a c t , a n e s t i m a t e f o r X c a n b e c o m p u t e d u s i n g p r i o r i n f o r m a t i o n o r a s ta t is t ic a lm o d e l f o r t h e w a v e l e t c o e f f i c ie n t s ( c f . [ 1 6 ] , [3 ] ). T h e d e s i r e d v a l u e s o f f ( t ) a t t h eH R g r i d p o i n t s t = 0 , 1 . . . . M - 1 c a n t h e n b e c o m p u t e d u s in g th e e s t i m a t e dc o e f f i c i e n t s :

f (t ) ~ ~ gtj , kdPJ,k(t) + ~ b J , k ~ J , k ( t ) , t = 0 , t . . . . . M - 1 . ( 3 2 )k c S 1 k ~ S j

F o r w a v e l e t s u p e r r e s o l u t i o n , t h e d a t a i s s a m p l e d n o n u n i f o r m l y b u t i n a r e c u r -r i n g m a n n e r . T h i s t y p e o f s a m p l i n g i s c a l le d n o n u n i f o r m r e c u r r i n g s a m p l i n g o ri n t e r la c e d s a m p l i n g [ 1 5] . M o r e s p e c if i ca l ly , w e a r e g i v e n s a m p l e d d a t a o n a n L Rg r i d i n t e r m s o f " f r a m e s , " w h i c h a r e se t s o f d a t a p o i n t s s e p a r a t e d b y a u n i f o r ms h i ft . L e t r b e t h e r e s o l u t io n e n h a n c e m e n t f a c to r , m b e th e n u m b e r o f d a t a p o i n t sp e r f l a m e , a n d n b e t h e n u m b e r o f g i v e n fr a m e s . T h e a v a i la b l e s am p l e s a r e

{ f ( 6 i ) , f ( r + ~i) , f ( 2 r + Ei) . . . . . f ( ( m - 1 ) r + ~ i ) } ,0 < E l < r , i = l . . . . . n .

G i v e n t h e s e m n s a m p l e p o i n t s , w e w o u l d l i k e t o r e c o n s t r u c t v a l u e s o f f ( t ) f o r t h eH R g r i d p o i n t s t = 0 , 1 . . . . m r - 1 .

F o l l o w i n g ( 2 7 ) a n d p u t t i n g t h e d a ta i n v e c t o r f o r m , w e g e t th e f o l lo w i n g s e t o fe q u a t i o n s t o s o l v e f o r t h e c o a r s e - s c a l e c o e f f i c i e n ts a j :

f (i) ~ G ~ ) a j , i = 1 . . . . n , ( 3 3 )w h e r e

- . k ~ S lf( i) = ( f ( p r + E i ) ) p =0 , . .. , m _ ! , G ( j ) = ( ~ b j , k ( p r -q - E i ) ) p = O , . . . o m _ 1 ,W e e s t i m a t e t h e w a v e l e t c o e f f i c i e n t s by a n d t h e v a l u e s o f f ( t ) o n t h e H R g r i d i nt h e s a m e f a s h i o n d e s c r i b e d p r e v i o u s l y .

3 .2 . I n t e r p o l a ti o n f o r i n t e r la c e d 2 D i m a g e sI n i m a g e s u p e r r e s o l u t i o n , t h e d a t a f r a m e s a r e L R r e c t a n g u l a r g r i d s o f s a m p l ep o i n t s . L e t h , to d e n o t e t h e h e i g h t a n d w i d t h ( in u n it s o f p i x e ls ) o f a n L R f fa m e ~T h e s e t o f a v a i l a b l e d a ta i s th e n

[ f ( p r + 6it, q r + e i s )} , 0 < E i ; ~ Eis < r,p = 0 . . . . . h - l , q = 0 . . . . . t o - - l , i = 1 . . . . . n .

F r o m t h es e n h t o s a m p l e p o i n t s o n L R g r id s , w e w o u l d l ik e t o re c o n s t ru c t v a l u e s o ff ( t , s ) o n H R g r i d p o i n t s { t , s ) I t = 0 , . . . , h r - 1 , s = 0 , . . . . w r - 1 }. A n a l o g o u s


11/18

W A V E L E T S U P E R R E S O L U T IO N 3 3 1to the 1D case , we sub s t i tu te in samp le va lues of f ( t , s ) to obta in a se t of l inearequa t ion s an d solve a leas t -squares sys tem f or the coarse -sca le coeff ic ients ;

f ( p r + Ei ,, q r + El , ) ~ Z Z a j , k , l~y ,k , l (pr -t- E i , q r + Eis ) (34)k ~ S h I E S js

= ~ Z a l ,k ,l fbJ ,k(pr-]- e i ,)dpj ,l (qr + Eis) . (35)k ~ S j t l ~ S j s

In m a t r ix fo rm , the doub le sum above can be w r i t ten a s a K ronecke r p roduc t o f1D w ave le t tr ans fo rm m a t r ic e s

f ( i ) ,~ (G~ t) | G ~ ) ) a j , (36 )wh ere f (i ) i s the v ec tor wi th the p ixe l va lues of the i th f ram e reordered roww ise ,a j i s the vec tor of unk now n coarse -sca le coeff ic ients , and the ent r ies G~t),"G(ijs-a re ba s is fu nc t ion va lue s a t s ampl ing po in t s o f f l am e i a long the hor i zon ta l andver t ica l d i rec tions , respec tive ly . Proceed ing as in the 1D case , we solve (36) fo ra regu la r i zed l e a s t- squa re s e s t ima te f i j o f a j . The d i f fe rence be tw een f ( i) an d i t scoarse -sca le es t imate (G(j.) | G q )) ~ j can next be used to es t imate the hor izonta lde ta i l coeff ic ients b~:

g~) = f ( i ) _ ( G ~ ) | G ~ ? ) ' ~ j (37)(G~ ,) | Hj(1))bh. (38)

Co n t inu ing a s be fo re , t he re s idua l i s t hen used to c a l cu la t e b~ and b d . Th e cho iceo f sca le J m akes a c ruc ia l d i f fe rence in recons t ruc t ion qua l i ty . W e pick the f ines tsca l e J so t ha t t he numb er o f s ample value s i s mo re t han the num ber o f unknow ncoeff ic ients in (36) and (38) .

4. Im plem entation and com putational com plexityW e d i scuss an e f f ic i en t implem en ta t ion and the com puta t iona l complex i ty o f o u rin te rpola t ion approach for in te r laced da ta in th i s sec t ion . We f i rs t cons ider the1D in te r laced f ram es equa t ion (33). Th e regula r ized leas t -squares so lu t ion can be

f i~ = " G ~ l ) r ' " G ( f ) r ] G i ) + ) ~ ' [ G ( J ) r '" G ( j n ) r] "n T ~ - 1 n( i ) ( i )= ~ G j G j -t- )~I ~ G (j T f ( , ) (40)

i=1 J i=1Th e equ a t ion above can be so lved mos t e f fi c ien t ly by an i t e ra t ive me tho d such a sthe con juga t e g rad i en t m e thod (c f . [2 ]) . We o n ly need to com pute m a t r ix -vec to r

expre s sed a s


12/18

3 3 2 N G U Y E NAND MILANFAR

pr oduc t s i nvo lv ing E n = l G ~ ) T G J ) +) ~ I a nd n o t i ts e xp l i c i t i nve rse . F u r the r mor e ,be c a use o f t he f i n it e suppo r t o f q~( t), G ~ ) ha ve a ba nde d s t r uc tu r e tha t ma y be f u r -the r e xp lo i t e d . W e c a n de r ive s imi l a r e xp r e s s ions f o r w a ve le t det a il c oe f f ic i e n t s.F o r 2D in t e r l a c e d ima ge s , t he r e gu la r i ze d l e a s t -squa r e s e s t ima te f o r t he c oa r se -sc a l e w a ve le t c oe f fi c i e n t s f r om e qua t ion ( 36 ) i s

~ , = ( G ( t l , ) | G ( J ] ) T 9 9 9 ( G ( f ) | T - 9 + L IG (n) | G (n)

[ t G 0 ) o , G O ) ' , r G ( n ) r " . ( 4 1 )" L ' " ~ J ,J " " ( G ~ ) | . 1 , / ] f(~)jR e c a l l i n g h e f o l l o w i n g r o p er t i es f t h e K z o n e c k e r p r o d uc t :

( i) (A | B ) T = A T | B r .( ii) ( A B ) | ( C D ) = ( A | C ) ( B | O ) .

( ii i) ( A | B) r e sha p e ( V ) = r e s h a p e ( A V B T ) , w he r e r e sha pe ( . ) r e o r de r s t hee n t r i e s o f a ma t r ix i n r ow w ise o r de r i n to ve c to r fo r ma t ,

a nd a pp ly ing the se p r ope r t i e s t o e qua t ion ( 41 ), w e ha v en . T , [ G ( i ) T G ( i ) ~ ) T | a ( i ) T ~ f ( i) ( 42 )= ( c ? , , ,

i = 1n T | -~ - ,~ . I~ -1 n= { X - " t G (i) G q ) ~ [ G ( i ) r G ( i ) ' ~ " ~ /" ( ~ , . .& ,L . . A J , J , , , , & , ] ~ _ .. , .G . ,,T F f i , ~ ( i ) ' ~ (43)

\i=1"= i=1w h e r e F (i) de no te s t he i t h f r a me in ma t r ix f o r m. S imi l a r f o r ms f o r w a ve le t d i -r e c t iona l de t a i l c oe f fi c i e n t s e s t ima te s c a n be de r ive d in t he s a m e m a nne r . A na lo -gous t o t he 1D c a se, e qua t ion ( 43 ) c a n b e so lve d m os t e f f ic i e n t ly by a n i t e r a ti ve

v~ n [ r . , 2 ( i )T g , 2 ( i ) ' ~m etho d . Ma tr ix -v ec to r p rod uc ts involv ing the sys tem m atr ix z_ .i=l ~ '- ' j , '~ j , ) |(G ~ ) T G ~ ) ) + L I c a n t a k e a d v a n t a g e o f c o m p u t a t io n a l p r o p e rt i es o f t h e K r o n e c k e rp r oduc t . W e qua n t i f y t he c ompu ta t iona l c omple x i ty o f so lv ing ( 40 ) a nd ( 43 ) i nm or e de t a i l i n t he f o l low ing d i s c uss ion .

T h e c o m p u t a t i o n a l b u rd e n f o r th e m e t h o d i s co m p r i s e d o f t w o m a i n c o m p o -ne n t s . T h e f i r st c om po ne n t i s t he c os t o f c ons t r uc t ing the ma t r i c e s o f w a ve le t ba si sfunc t ions eva lua ted a t the sampl ing po in ts . In the 1D case , a t sca le K and w i thw a v e le t suppo r t o f s iz e N , t he c os t fo r ge ne r a ting the ma t r i c e s o f s a mp le d w a ve le tba s is f un c t ions is O ( N 2 2( K +I ) ) . O ur 2D w a ve le t ba s is i s a s e pa ra b le bas i s, so t hec ons t r uc t ion o f t he ma t r i c e s o f s a mp le d w a ve le t ba s is f unc t ions on ly c os ts tw ic ea s mu c h a s i n t he 1D c ase .

T he s e c ond c ompone n t o f c ompu ta t iona l bu r de n i s t he l e a s t - squa r e s so lu t iono f ( 40 ) a nd ( 43 ). W e use c on juga te g r a d i e n t f o r an i t er a ti ve so lu t ion . Co m ple x i ty


13/18


pe r i t e r a t ion o f c on juga te g r a d ie n t c a n be ba se d on the c os t o f a m a t r ix - ve c to rp r od uc t w i th the sys t e m m a t r ix . I n the I D c a se , t he sys t e m m a t r ix i s

n ( i ) T ( i )E G G j + L I,i=1

wi th G ~ ) a m a t r ix a ppr ox im a te ly m x ( 2Ym r + N + 1 ) in d im e ns ions , wh e r em i s t h e n u m b e r o f s am p l e s p e r f r a m e . T h e c o m p u t a ti o n a l c o m p l e x i t y f o r t h is i sO ( n m ( 2 Jm r + N ) ) . Ana log ous ly , t he sys t e m m a t r ix f o r in t er l a c e d in t e r po la t ioni n t w o d i m e n s i o n s i s

n [ ~ ( i ) T G ( i ) ~i - - - - I

w h e r e G(jt) an d G (i)s a r e m a t r i c e s a ppr ox im a te ly h x (2Jhr + N + 1) and w x( 2 2w r + N + 1 ) in d im e ns ion s , r e spe ct ive ly . Th e va ri a b le s h a nd w de n o te thehe igh t a nd w id th o f a n LR f r a m e , r e spe c tive ly , i n un i t s o f s a m ple s o r p ixe l s. Thec o m p u t a t i o n a l c o m p l e x i t y fo r a m a t ri x v e c t o r p r o d u c t w i th t h e s y s t e m m a t ri x i sO ( n h ( 2 J h r + N ) + n w ( 2 J w r + N ) ) . B y ta k ing a dva n ta ge o f the in t e r l a c ings t r uc tu r e a nd the K r one c ke r p r od uc t re p r e se n ta t ion , t he c om p uta t iona l c os t f o rour in t e r po la t ion a ppr oa c h on ly doub le s f o r the 2D c a se a s c om pa r e d to the 1Dc a se .

5. Numerical experimentsThis se c t ion p r e se n t s num e r ic a l r e su l t s f o r in t e r po la t ion e xpe r im e n t s w i th 1Ds igna l s a nd su pe r r e so lu t ion e xpe r im e n t s f o r 2D im a ge s .

5.1. W avelet interpolation experiments o r 1D signalsI n th i s f i r s t s e t o f e xpe r im e n t s , we wi l l u se the wa ve le t t e c hn ique s de sc r ibe da bove to in t e r po la t e va lue s o f a 1D s igna l . We s t a r t w i th a n o r ig ina l s igna l o fl e ng th 168 . The s igna l is t he n b lu r r e d wi th a Ga uss i a n p o in t sp r e a d f unc t ion w i thv a r i a n c e 1 an d d o w n s a m p l e d b y a f a c t o r o f 3 t o g e n e r a te t h r ee L R f r am e s , e a c hwi th 56 sa m p le po in t s . W e ke e p on ly one o f those LR f r a m e s , le a d ing to a se ve r e lyunde r de te r r n ine d in t e r po la t ion p r ob le m . The g ive n f r a m e ha s sa m ple va lue s o ff ( t ) a t t = 0 , 3 . . . . . 165 . Th e a lgo r i thm a t t em p ts to r e c ons t r uc t t he s igna l at t im et = 0 , 1 . . . . 167 . F igu r e 2 d i sp la ys the r e su l t o f wa v e le t r e c ons t r uc t ion us ing aDa u be c h ie s D B 6 f i lt e r [4 ].


14/18

334 N G U Y E N A N D M IL A N F A R

200100

40 60 80 100 120 140 160 t800 0 20200 f X X x x x x x x x x X X x x x x x x x x1 00 x x 2 1 5 x x x x x x X x ~

x x x

~ X x ~ X XX X X O0 20 ~ 80 IO0 120 140 160 0

1000 ~ .0 20 40 60 80 1 ~ 120 140 160 !8 0

10 0. _ , _ _ _ , i0

0 20 40 60 80 1~ t20 140 t60 180Fig ure 2. Top to bottom . Tile first plo t show s the o riginal signal. The seco nd contains she av~ilabJedata samp les. The third plot displays he result of a coarse-scaleapproximationat scale J = -2 . Thelast graph p lots the final result against he given samples.

5.2. W avelet superresolution experiments o r 2D imagesT h e s e t u p f o r th e f i rs t 2 D su p e r r e so l u t i o n e x p e r i m e n t is s i m i l a r to t h e 1 D e x p e r i -m e n t s . A 1 7 2 x 1 7 2 p ix e l H R i m a g e ( u p p e r l ef t c o m e r o f F i g u r e 3 ) is b l u r re d w i th aG a u s s i a n p o i n t s p r e a d fu n c t io n o f v a r i a n c e 1 an d d o w n s a m p l e d b y a f a c t o r o f 4 t os i m u l a t e 1 6 L R f r a m e s . W e r a n d o m l y c h o o s e 1 0 o f th o s e L R f r a m e s , e a c h o f s iz e4 3 x 4 3 p i x e l s , a g a i n l e a d i n g to a s e v e r e l y u n d e r sa m p l e d su p e r r e so l u t i o n p r o b l e m .T h e r e so l u t i o n e n h a n c e m e n t f a c t o r i s 4 . T h e wa v e l e t i n t e r p o l a t i o n - r e s t o r a t i o np r o c e s s f i r s t i n t e r p o l a t e s f o r b l u r r e d v a l u e s a t t h e HR g r i d p o i n t s . An e s t i m a t ef o r t h e o r i g i n a l H R i m a g e i s o b t a i n e d b y d e c o n v o l v i n g t h e i n t e r p o l a t e d v a l u e swi t h t h e k n o w n b l u r. F i g u r e 3 sh o w s t h e r e su l t o f w a v e l e t su p e r r e so l u t io n f o r o u rt e st 2 D s e q u e n c e u s i n g D a u b e c h i e s D B 4 f il te r in t e rp o l a ti o n i n c o m b i n a t i o n w i t hT i k h o n o v r e g u l a r i z e d r e s t o r a t io n ( c f. [ 1 2] ).

T h e L R f o r w a r d l o o k i n g i n f r a r e d ( F LI P ,) i m a g e s i n o u r s e c o n d s u p e r r e s o l u t io ne x p e r i m e n t a re p r o v i d e d c o u r t e s y o f B r ia n Y a s u d a a n d t h e F L I R r e s e a r c h g r o u p i nt h e S e n s o r s T e c h n o l o g y B r a n ch , W r i g h t L a b o r a t o r y , W P A F B , O h i o . E a c h i m a g ei s 6 4 x 6 4 p i x e l s , a n d a r e so l u ti o n e n h a n c e m e n t f a c t o r o f 5 i s so u g h t . T h e o b j e c t si n t h e s c e n e a r e s t a t i o n a r y , a n d 1 6 f r a m e s a r e a c q u i r e d b y c o n t r o l l e d m o v e m e n t s


15/18

WAVELET SUPERRESOLUTION 33 5

Figure 3. The first (upper left) display is the original Stanford HR image. The second (upper right)shows a samp le LR frame. Su bsequent images are the coarse-scale approximation plus variousincrem ental levels o f deta il refinements.


16/18

336 NGUYE N AND M I L ANF AR

Fig ure 4 . The f irst (upper left) d isplay is a sample FLiR LR frame. The subsequent images are thecoarse-scale approxim ation plus various incremental levels of detail refinements.


17/18

WAV ELET SUPERRESOLUTION 33 7

of an F L IR im ager desc r ibed i n [7] . F igu re 4 con t a ins t he r esu lt s o f wave l e t super -reso lu t i on fo r t he F L IR t es t s equence us ing Daubech i es D B 4 f il te r in t e rpo la t ion ,a long wi th r egu l a r ized res to ra ti on .

6 . S u m m a r yThi s paper p resen t s a new wave l e t i n t e rpo l a t i on - res to ra t i on method fo r imagesuperreso lu t ion . In con t ras t to previous in terpola t ion-res tora t ion approach es , ou rmethod exp lo i t s t he i n t e r l ac ing s t ruc tu re o f t he s ampl ing g r id i n super reso lu -t i on . Us ing a s eparab l e o r thonormal wave l e t bas i s fo r 2D images , we de r ive awa ve l e t decomp os i t ion us ing K ronecker p roduc ts . As a r esu l t, t he com pu ta t i ona lp roper t ie s o f t he Kro necker p roduc t s a l l ow e f f i c ien t ca l cu l a t ion o f t he wav e l e tcoef f i c ien t s . C om puta t i ona l comp lex i ty o f ou r m ethod app l i ed t o 2D in t e rl acedda t a i nc reases on ly by a f ac to r o f 2 com pared t o t ha t fo r 1D da t a .

R eferences[1] K. Aizawa, T . Kom atsu, and T. Saito, Acquisition of very high resolution images using stereocameras, in Proceedings SP IE Visual Comm unications an d Ima ge Processing "91, Boston , MA,

pp. 318-328, N ovem ber 1991.[2] O. Axelsson, Iterafive Solution Methods, Cambridge University Press, New York, 1994.[3] M . Crouse, R . Now ak, and R. Baraniuk, Wavelet-based statistical signal processing using hidden

Markov models, 1EEE Trans. Signa l Process., 46(4), 886-902, April 1998.[4] I. Daube chies, Ten Lectu res on Wavelets, SIAM, N ew Y ork, 1992.[5] M . Elad, Super-resolution reconstruction o f images, Ph .D . thesis, Th e Technion-lsrael Institute

of Technology, Decem ber 1996.[6] C. Ford and D. Etter, Wavelet basis reconstruction o f nonuniform ly sam pled data, tEE E Trans .

Circuits a nd Sy stems II , 45(8), 1165-1168, August 1998.[7] R. Hardie, K. Bamard, and E. Armstrong, Joint M AP registration and high-resolution ima geestimation using a sequence o f undersampled images, 1EEE Trans. hnage Process ., 6(12),

December 1997.[8] W. Lukosz, O ptical systems with resolving pow er exceeding the classical limit, J. Opt. Soc. Am.,56(1 i) , 1463-1472, 1966.[9] W. Lukosz, O ptical systems w ith resolving power exceeding the classical limit II, J. Opt. Soc.

Am. , 57(7), 932-941, 1967.[10] S. M allat, A theory for multiresolution in signal decom position: Th e wavelet representation,

IEEE Trans. Pattern Anal. Mach. lntellig., 11(7), 674-683, July 1989.[I 1] Y. M eyer, Principe d'incertitude, bases H ilbertiennes et algebres d'operateurs, in Bourbak i

Seminar, vol. 662, Paris, pp. 1985-1986.[12] N. Nguyen, N umerical techniques for image superresolution, P h.D . tbesis, Stanford University,M ay 2000.[13 ] N. Ngu yen, P. M ilanfar, and G. Golub, Blind superresohition w ith generalized cross-validation

using Gauss-type quadrature rules, in Proceedings o f the 33rd Asilo ma r Conference on Signals,Systems, and Computers, Pacific Grove, CA, October 1999.

[14 ] K. Saner and J. Allebach, Iterative reconstruction o f band-limited images from non-uniformlyspaced samples, IE EE Trans. Circuits and Systems, CAS-34, 1497-1505, 1987.


18/18

3 3 8 N G U Y E N A N D M I L A N F A R[15] H. Shekarforoush and R. Chellappa, Data-driven mu lti-channel super-resolution w ith applicatioi~

to video sequences, d. Opt. Soc. Am. A , 16(30), 481-492, March 1999.[16] E. $imo ncelli, Statistical models fo r images: Comp ression, restoration and synthesis, in Pro-ceedings o f the 31st Asilomar Conference on Signals, Systems, and Computers, Pacific G rove,CA , November 1997.[17 ] G. Strang and T. Nguyen, Wav elet and F ilter Banks, W ellesley-Camb ridge Press, Wellesley, MA ,

1997.[18] A. Tekalp, M. Ozkan, and M. Sezan, High-resolution image reconstruction from lower-resolution image sequences and space-varying image restoration, in Proceedings 1CASSP '92,vol. 3, San Francisco, CA, pp. 169-172, March 1992.[19] H. Ur and D. Gross, Improved resolution from subpixel shifted pictures, CVGIP: GraphicalModels and Im age Processing, 54(2) , I81-186, March 1992.

Documents

Wavelet Paper