National Library 1+1 of canada Bibliothèque nationale du Canada
Acquisitions and Acquisitions et Bibliographie Services services bibliographiques
395 Wellington Street 395. rue Wellington OttawaON K l A W OnawaON K l A W canada canada
The author has granted a non- exclusive licence allowing the National Library of Canada to reproduce, loan, distribute or sell copies of this thesis in microform, paper or electronic formats.
The author retains ownership of the copyright in this thesis. Neither the thesis nor substantial extracts fkom it may be printed or othenvise reproduced without the author's permission.
L'auteur a accordé une licence non exclusive permettant à la Bibliothèque nationale du Canada de reproduire, prêter, distribuer ou vendre des copies de cette thèse sous la fome de microfiche/fïim, de reproduction sur papier ou sur format électronique.
L'auteur conserve la propriété du droit d'auteur qui protège cette thèse. Ni la thèse ni des extraits substantiels de celle-ci ne doivent être imprimés ou autrement reproduits sans son autorisation.
Abstract
C'ornpressecl image and vicleo bit streams are very sensitive to channel errors and
nia!. be altered or lost during transmission. Error concealment by post-processing
interids to reconstruct lost visual information by esploiting the correlation Letiveen
the irnage/video data. The applications of concealment of errors in coded visual
information incIude visual communication over unreliable channels such as wireless
networks and the Internet.
For most types of encoders and input data. coded visual information consists
of a collection of coded texture (DCT coefficients). shape and motion information.
In ttiis tliesis ive present concealment methods for errors in texture and shape infor-
mation. and address the concealment of errors in motion data in conjiinct ion \vit h its
corresponding texture or shape information. The method developed for coricealment
of crrors in cocled testure in\+olves compensation of the effects of the missing da ta
on the rest of the texture information and then using a deterministic or a statistical
algori t h m for the restorat ion of missing information. The deterministic algori t hm
acliievcs a goocl performance level in t h e reconstruction of' edges. The stat istical
algorithm which is based on maximum n postefion (hIAP) estimation. employs an
adaptive llarliov random field (MRF) as the image a-priori model. The adaptation
enables the estimation procedure to incorporate more information without a dramatic
increase in computational complexity. AIAP estimation is also employed for the re-
construction of missingshape data. Although it uses an adaptive JIRF. theest imator
is different in the sense t h a t it is designed for binary shape information.
In the second part of t h e thesis. Ive evaluate the performance of the developed
concealment methods for three different types of coded visual data: baseline .JPEG
coded st il1 images. H.26:3 coded vicfeo and IIPEG-4 coded t-ideo. Our clcperimental
rcsults demonstrate t hat t h e met hods presenteci in t his thesis achieve consistently
good corn put at ion-performance t radeoffs. making theni very beneficial for real time
communication over error prone networks. In fact. the proposed error concealment
methocis can lead t o acceptable visual quality at loss rates as high as 20%.
Contents
Abstract
Contents
List of Tables
List of Figures
Acknowledgements
Dedication
iv
viii
ix
xiv
xvi
1 Introduction 1
. . . . . . . . . . . . . . . . . . . . . . . . . . . 1 . 1 OutlineoftheThesis 4
2 Background 7
- 2.1 Image and Video Compression . . . . . . . . . . . . . . . . . . . . . . 1
. . . . . . . . . . . . . . . . . . . . 2 The .JPEC: Compression Standard 9
. . . . . . . . . . . . . . . . . . . . . . . . 2.3 Block-Basecl Video Coding 1 L
. . . . . . . . . . . . . . . . . 3 Mot ion Compensated Prediction 1 1
List of Tables
. . . . C'ompti tat ional complexity of various reconstruction methods.
PSSR results (in d B ) for the 512 x 512 LEU and Pepper images. The
niinibers are for the images after removing the st ripes and reconst riict-
. . . . . . . . . . . . . . . . . . . . . . . . . . . ing t h e missing data.
PSSR results (in d B ) for the 512 x 51'3 LEYA image for varioiis recon-
st ruction met ho&. T h e numbers are for the image after ren~oving t h e
. . . . . . . . . . . . . . stripes and reconstriicting the missing data.
PSS R corn parison of different niet hocls for the video secjuence FORE>,i.-\S. 1 10
PSSR cornparison of differcnt mcthods for thc video seqiience A K I Y O . 116
List of Figures
The block diagram of a typical image encoder . . . . . . . . . . . . . .
The scanning of t hc AC' coefficients in JPEG . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . The block cligrarn of a typical video coder
A basic block diagram of a n 1IPEG-4 video coder . . . . . . . . . . .
.-\ \'OP enclosed in a rectangular bouncling box ancl di\-ided into mac-
roblocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . Binary alpha plane
. . . . . -4 simplifieci block diagram of a \-ide0 commiinication system
Error propagation in the frarnes of a video secluence . . . . . . . . . .
A site (,Y ) and its ( a ) first order neighboriiood systeni . ( b ) second order
. . . . . . . . . . . . . . . . . . . . . . . . . . . . neigh borhood system
3.1 Error concealment of intra coded texture information . . . . . . . . . .
1.2 Error concealn~ent of inter cocled texture information . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1 Estimation of DC' valiic
3.4 Post-processing for removing the stripes . . . . . . . . . . . . . . . . .
. . . 3 ..i Removing the stripes for blocks with different horizontal positions
. . . . . 3.6 Histogram of the difference value: (a ) Fighter: (b ) Mandrill.
. . . . . . 3.7 Tlie partitioning of the blocks adjacent to a missing block.
S .A pisel. its clique c and the eight directions. The complement of t h e
clique. c' is t h e dark area- . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . 9 -4 rnissing pixel in a vertical line.
1%. 10 (a) The four sub-hlocks. (b) their correspondirig sub-blocks in t h e prc-
. . . . . . . . . . . . . vioiis fraine ancl t h e blocks connected to t hem.
A pixel. its clique c and the eight directions. The complcment of the
. . . . . . . . . . . . . . . . . . . . . . . . clique. c' is the dark area.
.A :3 x 13 pisel u-indoiv. t h e border pixels in i t (shaded) and the best
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . line-fit.
. . . . . . . . . . . . . . . . . Original images: (a) Lena: (b ) Pepper.
Images: (a) Lena with 10% missing data: ( b ) Pepper u-ith 3% missing
data. The missing blocks are shown in white to show their positions
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . clearly.
Decoded images: (a) Lena witli 10% missing data: ( b ) Pepper with 3 %
niissing data. The value of each pisel of t h e n~issing blocks is repIaccd
. . . . . . . . . . . . . . . with the estimated DC value of t h block.
. . . . . . . Pictures after removing the stripes: ( a ) Lena: ( b ) Pepper.
Reconstructed images after removi ng t lie stripes and restoration of
niissing blocks using the method based on Equation (:3.4) (pieceirise
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . constant weights).
5.6 Reconstructed images after removing the stripes and restoration of
missing blocks iising t h e rnethod based on Equation (3.6) (linear ramp
weights). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - - 3 . Reconstructed image iising our proposed statistical rnethod. . . . . .
5.8 Reconstructed images using the method proposed in . . . . . . . . .
-5.9 Reconst riicted image iising the met hoc1 proposed in . . . . . . . . . .
5 . IO Reconst riicted image when cIiagonally adjacent blocks are used in ad-
dition to horizontalIy and vertically adjacent blocks. . . . . . . . . . .
5.1 1 LIagnified picttire of part of t lie image i.ES.4 shown in Figure 5.5- . .
5-12 liagnified pictiire of the same area shown in Figure 5-11 for the image
LEN=I of Figure 5.6. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
.3.1:3 llagnifiecl picture of the sanie area shown in Figure 5.11 for the image
. . . . . . . . . . . . . . . . . . . . . . . . . . . . LESA of Figure 5.7.
5.1-l llagnified picture of the same area shown in Figure 5.11 for the image
LESA of Figure 5.S. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.1.5 LIagnified pictiire of the same area shown in Figure 5.11 for the image
. . . . . . . . . . . . . . . . . . . . . . . . . . . . LESA of Figure 5.9.
.5.16 Magnified pictiire of the same area shown in Figure 5 . i 1 for the image
LESA of Figure 5-10. . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.1 T Ps.\iR \rilues for image sequence FOREM.4'; with 10% C;OB missing for
di fferent conceahent met hods. . . . . . . . . . . . . . . . . . . . . .
5 . iS A 1 1 inter coded frame of t h e secluence FOREM.6\X (irame '22) concealed
b- the (a) replacement. ( b ) median. and ( c ) proposecl met hods. . . .
A frame from the video sequence FOREMAS. (a) original. ( b ) miss-
ing blocks. reconstructed using (c) a CillRF model. (d) a suboptimal
HIIRF model. (e) the method proposed in . . . . . . . . . . . . . . .
.-\ frame from the video sequence FORESI AS (a j 1s-it h missing COBs. re-
constructed using ( b ) a GJ lRF rnodel. ( c ) a suboptirnal HMRF model.
( d ) the nlethod proposed in . . . . . . . . . . . . . . . . . . . . . . .
A frame from the video sequence A K I Y O ( a ) original ( b ) wit h miss-
ing hlocks. reconstructed using (c) a GSIRF model. (cl) a suboptimal
Hl[ RF model. (e) the method proposed in . . . . . . . . . . . . . . .
frarne from the video sequence AKIYO ( a ) !vit h missing GOBs. re-
constructed using ( b ) a GMRF model. (c ) a suboptimal HBIRF model.
( c i ) the met hod proposed in . . . . . . . . . . . . . . . . . . . . . . .
Vicleo packet structure of SiPEG-4 in (a) 1-LOPs and ( b ) P-VOPs. .
The shape of the first [-\,'OP of the X K I Y O video object. . . . . . . .
The shape of the A K I Y O video object niissing :IO% of the shape blocks.
The reconstr~icted shape of the first 1-VOP of the AKIYO video object.
The shape of the first 1-VOP of t h e BREAM \-ide0 object. . . . . . .
The shape of the B R W M video object missing '25% of the shape blocks.
The reconstructed shape of the first 1-VOP of the BREAM video object.
The sIiape of the first 1-VOP of t h e \VE.-\THER video object. . . . . .
The shape of the \\'EATHER ïideo object missing 13.5%~ of the shape blocks. 122
The reconstructed shape of the first 1-VOP of the \ V E A T ~ I E R video object . l22
A piscl. its clique (dark) and the complement of t h e clique ( in shade). 146
xii
A.? Simplified diagram of a parallel processing machine. . . . . . . . . . . 148
A.:3 The connection of adjacent PEs for error concealment based on the
)[RF models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
.A.-! The lookrip table used in the error concealment based on the GMRF
model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.30
A..? The connection of adjacent PEs in the error conceaiment based on
kI3IRF rnodel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-51
A.6 The bit pattern for a pixel value of 9s and = 2. . . - . . . . . . . . 1.52
A.7 The lookiip table for the Hiiber f~inction with y = 2. . . . . . . . . . 1.58
Acknowledgement s
There i.i no such thing as a self-made' man. Ib-e ffrE made up of thousnnds
of others. Eccryone ~cho ha.$ Ewr doné a kind decd for us . o r spoken
o n c trord of éncouragérrzenf to us. has en té red into t h e nzcrrl-é-up of our
chnrncter and of our thoughts. as u d as our success.
George ;\latlheu: Adams
First ancl foremost. 1 woiild l i k e to express my deep gratit iide and appreciation
to my supervisors. Professor Faouzi Kossen t ini and Professor Rabab \Varci. for t tieir
consistent and generous support. encouragement and guidance in both technical and
rioritechnical mat ters. 1 worild also l i ke to t hank mj- colleagues in the Signal Processing
ancl ~Iultimeclia Groiip of the Cniversity of British Columbia (L:BC) for creating
a fricndly and stimulating work environment. In particular. L a m grateful for mu
friencls and colleagiies Jaehari In. Giiy Cote. 'rlichael Gallant and Berna Er01 for
tlicir assistance wit h the JPEC;. H.26:3 and NPEG-4 standards. 1 am t hankful for my
friericis and officemates Adriana Durnitras and Da\*e Tompkins. for reading parts of my
t hesis ancl making constructive suggestions. 1 \rorild like to express ni-. appreciat ion
to Dr. -Alen Docef who was an invaluable source of support u-hile lie iras at UBC.
Also. 1 wodd Iike t o t h a n k my friend I iayvan S a j a r i a n for al1 t h e friiitful discussions
t liat ive tiad tliiring my y a r s nt UBC. I a m gratefui t o Dr. Ali derbi of the Xat ional
Research Council of C a n a d a where I spent an internship in 1998. for his s u p p o r t a n d
stimiilat ing envi ronment . M y gra t i tude also goes t o t h e Faculty of G r a d u a t e Stiidies
of I'BC' who helped make mu education possible bu awarding m e with a University
Graduate Fellowship.
SI!- deepest grat i tude. Iove a n d affection bclong of course t o my parents. t o
nlioni. as 1 begin t o realize more and more. 1 owe al1 t h a t I am and al1 the I have ever
acconiplished. And finail?. I detficate this n-ork to rny dear wiie. SIetirnoosh. whose
love. ca r t a n d pa t ience have supportecl me.
Th€ I - n i r c m i t g of British Columbia
.\fcrrch L'OU0
To my dear Pamnts
and
To Mehmoosh, rny beloved wife.
Chapter 1
Introduction
.-ln ediicntion k n ' f hou. much you hare cornrnitted to memory . or m e n hou
rnuch gou knouq. 1l-s being able to difler~ntinte bctlrecn chat yoii do bnow
and what yorr don't.
T h e fast growth of digital transmission sen-ices has created a great interest i i i digital
transmission of image and video signals. Digital image and video signals require very
higli bit rates. therefore compression is rised to reduce the amount of data needed
for represent at ion of soch signals. Compression is achieved by esploi t ing the spatial
ancl temporal redundancies of t h e signal arid the characteristics of t h e user. Several
conipression standards have emerged to facilitate the growth of new visual commiini-
cation applications. iricluding the Jo in t Pliotographic Experts Group (JPEC;) stan-
tlard for st il1 iniage compression. the International Telecornmunication Union ( ITU )
reconirnendation H.261 for video telephony conierencing and its subsequent revisions,
H.266 and H.26:3+. and the lloving Pictures Expert Group (AIYEC;) standards for
full rnotion video compression and coding in digital storage media and digital commu-
nication applications. namelx MPEG-1. 41PEG-2 and 4IPEG-4. The common feature
of t hese compression standards is the use of the discrete cosine transforrn ( DCT). due
to its effectiveness. efficienq- and sirnplicity.
Communication channels are not error free, and conseqriently. the encoded bit
streams are viilnerable to trarisrnission errors. usiially causing loss of blocks of data
ancl/or loss of sunchronization. The cornpresseci image and \-icleo bit streams are
very sensitive to channel errors due t o the variable Iength codewords and prerliction
techniques. which are t ~p i ca l l y used in image/\-ideo coding algorithms.
_\Ianx techniques have been proposed to combat the transmission error prob-
lem. In one group of technicliies. the source coding algori t hm and/or transport cont rol
mechanisms are designed either to minimize the effects of transmission errors wittiout
rcquiring any error concealment at the decoder or t o makc the error concealment
task at the decoder more effective. Esamples of such techniques are forwarcl error
correct ion ( FEC). joint source-channel coding anci la>-ered cocling. T hese techniques
~is i ia l l~* increase t h e overhead in terrns of the overaIl bitstream size: hence. part of the
coclirig efficiency achievcd 11y compression is lost. In another group of techniques. it
is assurnecl that a backward channel froni the decoder to the encoder is a\ailable and
the encocler and decoder work coopcratively to minimize the impact of the errors.
Esamples of siich technicliies are Aiitornatic Retransmission Recluest (ARQ) and se-
lect i vc preclictive coding hasccl on the feedback from the decoder. Retransmission
has generally been considered unacceptahle for real time video applications becaiise
of the associated delay. Retransmission has also been considered inappropriate for
miilt ipoint video transmission because the repeat request from a large number of de-
codcrs can overwhelm the encoder. Xnot her d i sad~~antage of retransmission is t hat i t
ma!- increase the loss rate by adding more craffic on the network. In another group of
tech niques. referred to as error concealment by post processing. the clecoder fulfills the
task of error concealment. The objective of post-processing is t o remove the visuaI1y
annoying ar t ifacts t hat degrade the picture quali ty without distorting t tie informa-
t iori content. In generai. error concealment bx post-processi ng a t tempts to reco~-er
or reconst riict or restore the lost informat ion by estimation or interpoIation wit hout
relying on additional informat ion from t lie encoder. Since the available information
cannot uniquely determine the original iniormat ion. prior knowledge abolit the im-
age/vicIeo signal is typicaIly iised in an effort t o characterize the lost information.
In t his t hesis. ive propose post-processing met hocls for concealment of errors
resiil t ing from t h e transmission of coded visual informat ion in error prone environ-
ments. Basecl on the tl-pe of the encoder and the input data. cocled visiial information
consists of a collection of coded testiire (DC'T coefficients). shape and motion infor-
mation. Object-based video cocling algorit lims generate coded texture. shape. and
motion information while bitstreanis of block-based coded video do not include the
shape information. In the still image coding case. t h e output bit stream consists of
only testurc information. In this work. Ive address concealment of errors in testiire.
shapc and motion information. The motion da ta is solely used for motion coinpen-
satecl prediction of tes ture and shape. Thus. we consider the concealment of errors
in mot ion data in conj unct ion wit h i ts corresponding texture o r shape informat ion.
&> a h cvaluate the performance of our proposed concealment methods for three
cliffercnt types of coded visiial data: baseline JPEC; coded still images. H.26:3 coded
vicleo and LIPEG-4 coded \-ideo. In each case. ive evaluate our concealment results
subjectively and objectively. Moreover. we compare our reconstruction results with
thosc of other rnethods proposed in the literature. We show t hat the rnethods pro-
poscd in this thesis achieve consistently Better performance-computation tradeoffs. In
particular. the proposed rnetliocls are suitable for real t ime video communication over
packet lossy and bit error prone networks. Employing the proposed error concealment
rnethods can leacl to acceptable video qiiality at loss rates as liigh as 20%.
1.1 Outline of the Thesis
In cliapter 2. the image and video compression standards ancl various methods that
have been proposed t o combat channel errors in transmission of compresseci t-isual
information are re\.iewecl. IIoreover. \-arious error conceaiment met liods are discussed
in detail. Finally. l[asirnurn a Posteriori (II.AP) estimation a n d Llarkov Ranclom
Fielclç ( I IRFs) are re~.icwed.
In chapter 3. we propose noive1 error concealment methods for the texture
coniponent of codecl visiial clata. Proper error concealment methods are suggested
for int ra coded ancl inter cocleci testure information. Obviously. conceaiment met h-
ocls dcveloped for iritra coded texture is applicable to compressed still images. The
method introduced for concealrnent of errors in intra coded testtire, iinlike previoiisly
piiblished algorithnis. consiclers the clependency between codecl blocks in a frame.
It uses a deterministic or a statistical rnethod for the restoration of missing testure
informat ion. The deterministic method achieves a good performance level in the re-
construction of edges. The statistical method uses an adaptive MRF as the image
a-priori model. The adaptation enables the estimation procedure t o incorporate more
information without increasing the order of MRF. The met hod proposed for conceal-
ment of errors in inter coded texture uses the information in t he adjacent €rame to
find an initial estimate of the missing texture. The initial estimate is then combined
u-ith the estimate of the prediction error of the rnissing block t o find the best ap-
prosi niat ion of the missing texture information. Since 1IRF-based error concealment
rnet Iiocls are iisiially comput at ionally cornples, we discuss fast implementations of
these nlethods in Appendix A.
In Chapter 4, we propose efficient concealment methods for errors in the shape
information in cocled vitleo. This subject has. so far. not been addressed in the
literature. Similar to C'hapter 13. ive propose error concealment r~iethods for intra
codccl and inter codecl shape data. For intra cocled shape. a 11XP estimator. which
uses an adaptive 1 IRF as the image a-priori model. is used to est imate the missing
shape information- For inter coded shape inforniation a tivo stage reconstruction
method. similar t o the one that is proposecl in Chapter 13. is devised.
Finally. in Chapter 5 . ive esamine the performance of the error concealment
rriettiods proposed in Chapters 3 and -1. \\:e consider the bitstreams generated by
~Warious coding standards: JPEC;. H . X I and 11PEG-f. First . we evaluate the perfor-
niance of the proposecl niethods for baseline JPEC; coded images. Kext. ive consider
the performance of the proposed error concealment methods for H.26:3 codecl video
secltiences. Finally. ive show the resiilts of the ~ r o p o s e d method for reconstruction of
missing shape information in MPEG-4 coded video data. In each case. w e compare
the performance of our proposed method with that of ot her met hods available in t he
literature.
Chapter 2
Background
Thc farther bnck you cnn look. t h e farther forvard you are l i k d y Io sec .
1 t,l nst o n Ch rr rch il1
In this chapter. first. the main image and video compression standards a re briefll-
re\-iewcd. T hen. t h e challengi ng issue of transmission of compressed visual informa-
tion in an error prone en\-ironment is introduced. and various methods that Iiave been
proposecl to combat channe1 errors are re\-iewed. Ses t . error concealment is clisciissect
arici various error concealment met hods a re surveyed in detail. FinaIl'-. ~ I a s i m i i m a
Posteriori ( h1.A P) estimation ancl llarkov Randorn Fields ( l l R Fs) are revieived.
2.1 Image and Video Compression
The importance of visrial communications has increased tremendously in the last few
ciecacles as a result of the progrcss in electronics and compiiter technology ancl the
- I
creat ion of networks operat ing wit h various capacities. Emerging applications such
as video conferencing. mobile terrninals. and Internet-based audio-visual communi-
cation have a great impact on everyday life. education and entertainment. Visual
information is one of the richest but also the most handwidth consuming components
in a commiinication sustem. To meet the requirements of new applications. powerfiil
data compression techniques are needecf t o reduce the bit rate dramatically. even in
t h e prescnce of growing communication channels offcring increased bandwidth.
.A nunzber of compression standards have been defined t o facilitate the g r o ~ h
of new visiial communication applications. The JPEC; still-image compression stan-
clard \vas proposed by the .Joint Photographic Experts Group [1. 2. 13. 4. 5 . 61. Rec-
ornmericlat ion M-261 (ancl its sti bsequent revisions. namely H.2613 and H..26:3+) was
introcluceci by the International Telecommunication CTnion (ITL-) for video conferenc-
ing at bit rates down to 64 Iib/s. The l[PEC; standards from the Stol-ing Pictures
Expert Group (SIPEG) address the compression of vide0 signals and the associatecl
audio information [ I I . The cornmon feature of these compression standards is the lise
of the discrete cosine transform ( DC'T). mainly because of its effectiveness and sim-
pliciry. Encocling algorit hms t hat are cornpliant wit h the abo\-e DC'T-basecl standards
have provided good i mage/video reproduction quali ty a t relati vely high compression
ratios (more than 20: 1 for JPEG. more than 100:l for H.263. and more than 100: 1
for AIPEC;l/lIPEG'L).
2.2 The JPEG Compression Standard
.JPEC; is the current DC'T-basecl stanclard for compression of stiil. cont inuoiis-tone.
monochrome and color images. JPEG has four distinct modes of operation: sequent ial
DC'T- baserl. progressiw DC'T- based. loss less. and hierarchical.
1 - - - - - - - - - - - - - T - - - - - - - - - -
Encropy Codine
I I 1
i I - Image 8x5 Block Quantizer Mode-specific
FDCT (Optional) -
Envopy C d e r ) (ticadcrs/
Tables/
Figure 2.1: The block diagram of a tj-pical image encoder.
Al1 DC'T-based modes of operation in JPEC; follow the same basic coding
procediirc stiown in Figure '2.1. First. an input image is partitioned into 8 x S blocks.
Each blocli is t lien transformed using the forwarcl discrete cosine transforni and.
optionallx. quantized. The first element of each S x S DCT block is called the DL
coefficient and the rest are called the AC' coefficients. The DC'T coefficients are
t licn prepared for entropy encoding. The J PEG standard allows two algori t hms for
cntropy coding. The first algorithm is Huffman-based and the seconcl one is based
on arithriirtic coding. Huffman-based coding is widely used hecause it is simple and
pu blicly availahle. ivhereas ari t limet ic coding (alt hough more compression efficient) is
niore comples and it is mostlx patentecl. Final15 necessary information for clecoding
sucli as headers (image dimension and sample precision ) and tables (quant izat ion and
Low Freqücncy
7 High Frequency
Low High Frequency Frequency
Figure 2.2: The scarining of the .AC coefficients in JPEG.
Hiiffman tables) are added t o the encoded da ta t o form the compressed bit stream.
Likewise. at the decoder. the received bit streams are entropy decoded t o pro-
vide 8 x 8 blocks of DCT coefficients. which are then deqiiantized and transformed
using inverse DCT t o yietd the reconstructed image.
in the sequential DCT-hased mode the 8 x S DC'T blocks in an image are
critropy encoded in a raster scari order. T h e DC coefficient of the 8 x 8 Glock is
always cocled first. The diKercnce between the DC value and its predicted valiie.
which is the DC coefFicient of the most recently coded block. is fecl to the entropy
coder. The :IC' coeEcients a re first scanned in t h e zigzag sequcnce order as shown in
Figure 2.2 and t tien entropy encoded t o form part of t fie corn presscd image data.
There are two types of sequential DCT-based .JPEG. T h e sirnpler form of the
sequent ial DCT-based mode of operat ion is called the %aseline .J P ECi- [ï]. Only
a n 8-bit input precision and two sets of Hiiffman tables (each set incliicles one DC
tahle and one AC tahle) for entropy coding are allowetl in baseline JPEC;. sequential
DCT-based .JPEG that bas capabilities beyond the baseline .JPEG is called -estendeci
seqriential .JPEG." which allows 1'2-bit input precision and -1 sets of Huffrnan tables.
111 the progressive JPEC; (PJPEC;) mode. S x 8 biocks are fornled in the same
order as in sequential JPEG. DCT transforrned and (optionally ) quantized to a spe-
cific niimher of bits. The qiiantized DC'T coefficients are ttien partially encoded in
rnrilt iplc scans. Each scan corresponds to a feu- bits of one or more DCT coefficients.
ancl represents a port ion of t h e image bei ng encoded/decoded. tVhiIe t lie seqiient ial
.J PEC; niode yields essent ially t h e same level of conipression performance for most
cncoder implementations. the performance of progressive JPEG depencls highly iipon
the structure of the clesigned encoder. This is due to t he flexibility the JPEC; standard
leaves open in designing progressive J PEG encoders [S. 91.
2.3 Block-Based Video Coding
AIost of block-based video compression algorithms (e-g.. H.263. LIPEG-2) rel': on two
basic techni~~iies: block-hased motion cornpensateci prediction for the reduction of
temporal recliinclancies and t rarisform domain ( DCT) based coding for the reduct ion
of spatial rediinclancies (10. I l ] . .A block diagram of a typical video encotler is sliown
in Figure 2-13 and is briefly esplaineci in the following.
2.3.1 Motion Compensated Predictiou
LIotiori compensated prediction techniques are used t,o reduce temporal redundan-
cies bet.wecri video franles. Most of these techniques assume tha t the pixels within
11
Figure 3.3: T h e block digram of a t ~ p i c a l video coder.
Coding Control
t tie ciirrent picture can be rnodeled as translations of t h e pixels within previoiis o r
* INTER/INTRA dccision flq
fiitiirc pictiires. Main\_\- d u e t o their simp1icit)- and efficiency. block-based motion
compensation algorithrns tha t are based on a translational moclel are popular. ln
L " T m m i t t e d o r noi" flag
Quantizrr indication t
such algorithrns, t h e motion \-ector is obta i~ied by finding a rnacroblock (a LG x 16
DCT .
arca of the picture) wit hin a pre-defined search area. such that a cost fiinction ~vhich
+ Quaniizer index for trimsform coefficients
nieasiircs the niisnlatch betiveen this rnacroblock and the currcnt rnacroblock is min-
iniizcd. T h c most \videlx usecl cost measure is the siim of absoliite differences (SAD).
Vidco in
Inverse Quantizer
lnvcrx DCr
1- - 1
P '
To fincl the hest matcliing macroblock. the S A D stioulcl L e compiited a t se\-eral loca-
t ions wit hin t Iie searcti area. The simplest, but the most conipiitationally intensive
? ! - - - hfoiion vcctors
--J
i-riethod. known as t he full or exhaustive search methoci. evaluates t h e S.AD for mac-
t
Picture ~ l e m o q
roblocks positioned a t evcry possible pixel location in t he search area. To reduce
the computational complesity. several algorithms with leiver search points have been
proposeci [12. 13. 14. 1.51. The Picture .\lemor- in Figure 2.3 stores one or more pre-
\-ioiisly reconst ructed frames and represents the mot ion est i mat ian component of the
video cocling algorit hm.
2.3.2 Transformation
Transforms decorrelate t he image content and compact energx into as few coefficients
as possihlc. The DCT ancl Inverse DC'T blocks showri in Figure 2.3 perforrn 8sS
trarisforriiations of t h e luminance and chrominance blocks of macroblocks.
2.3.3 Quantizer
The iiiinian viewer is more sensitive to reconstruction errors in the low spatial fre-
cluency regions of an image t han in the high frecluency ones [16. 1 ï]. Low frccli~enc~-
cliariges in intensity or color are easily distinguished by the el-e. Quick. high frequency
clianges are harder to see and may therefore he discardeci. In vicleo coding. quanti-
zation consists of applying iiniform mid-treacl quantizers t o the DCT coefficients.
2.3.4 Entropy Coding
Prior to ent ropj- coding. t he quant ized DC'T coefficients a r e zig-zag scannecl. in order
of increasing frequencj-. T h e re-arranged a r rq - is coded into a scquence of riin-length
coclcs. The riin is defined as the distance between two non-zero coefficients in the
array. The le\-.el is the non-zero value irnrnecliately foIlowing a secluence of zeros. The
riin-level pairs. and ot her relevant in format ion about the macroblock such as mot ion
vectors. and predict ion types. are then ent ropy codeci most ly using Huffman-li ke
cocling.
2.3.5 Prediction Types
Tlie coding mode in wtiich temporal prediction with reference to a previoiis pictiire
is applicd is called inter. and it is called intra i f no temporal prediction is ~iscd. The
intra coding mode can bc selected for a complete picture or macrobIocks in an inter
codecl pict ure. Int ra coclinp is performed periodically to eliminate error propagation
int rociucecl bj+ inter- frame predict ion. h t ra and inter coded pict ures are iisually calleci
I-pict ures and P-pictiires. respect ii-ely. There is anot her tl-pe of coded pict ure caIIcd
B-picture. in which each 16 x 16 macroblock can be forward predicted. backwarcl
prcclictecl or both. The motion information consists of one vector for a forward or
hacku-ard predicted macroblock. and two vectors for bidirectionallj- predictecl mac-
roblocks. The H.263 standard has an optional mode called PB-frame. A PB-franie
corisists of two pictures being coded as one unit: one P-picture tvhich is predicted
from t lie previously decodecl P-pict cire and one B-pict ure wIiich is predicted from
bot h t lie prcviously clecoded P-picture ancl the P-pict tire current 1'- being clecoded.
Thc switches in Figure 2.:3 differentiate between intra and inter prediction.
2.3.6 Coding Layers
St anclard DC'T-hased video cocting algorit hms typically ciecompost. the video sequence
into sis layers: Block Layer. Nacroblock La)-er. Slice Layer. Picture Layer. Group of
Pictiires Layer. and Scqtience Layer. Each layer supports a definite function: either a
signal processing function (e-p.. DC'T at the Block Layer. motion compensation a t the
SIacroblock Layer) or a logical function ( resynchronizat ion a t t he Slice Layer. random
access at the Grotip of Pictures or at the Sequence Layer). Frorn a coding standpoint.
the riiacroblock l q e r is t lie most important one. as most coding decisions are made
at t his l q e r . Coding decisions itsually incliide prediction type. motion vector choice.
and clriant izer select ion .
2.4 The MPEG-4 Compression Standard
The new generation of highly interactive multimectia applications recpire that the
iisers be able to access anci manipulate multimedia data. This has fueled recent
A I PEG--1 internat ional standardization activities for a content -baseci coded represen-
tation of multimedia data.
.\IPEG--1. l ike IIPEG-1 and SIPEG-2 [27. 281 and the [TI'-T H.263/H.26:3+
[29. 301 standards. offers Iiigh compression performance levels. rendering the storage
aitd t ransmission of audiovisiial data more efficiently. The ot lier key ohject ives of
.\IPEC;-4 are to provide 013 ject-hased access and provicie funct ionalities çuch as error
resiliencc. scalability and Iiybrict coding of synthetic and natiiral data [31. 32. 3 3 . 134.
3.3. 361.
2.4.1 Audiovisual Object Represeiitation
\ l PEG-4 acliieves content -basecl representation by defining audiovisiial objects and
coding them into separate bit streani segments [XI. 37. 381. .An aiidiovisual object
1 5
(.A\-O) consists of a visual object component. an audio object component. or a combi-
nation of tliese cornponents. Some examples of AVOs include a sound recorded with
a microphone. a speech synthesized [rom a tes t . a person recorded by a video camera.
and a :3D image with text overlay.
1IPEG--I supports the composition of a set of audiovisual objects into a scene.
also referred to as an audiovisual scene. In order t o allow interactivity with indivicliial
A\'Os [vit hin a scene. it is essential to transmit the information that describes each
.-\\**O.s spatial and temporal coordinates. This information which is referred to as the
scene description information is t ransmi t ted as a separate st ream and mi11 t iplesed
[vit b .-\\-O elementarj- bit streams so that the scene can be composed a t the iiser*s
end. This fiinctiona1it~- niakes it possible to change the composition of .-\VOS wit hoiit
having to change the content of AVOs.
2.4.2 The MPEG-4 Visual Coding Standard
The LIPECL! visiial coding standard provides standardized core processing elemcnts
that allow efficient storage. transmission and manipulation of visual data [:31]. Dif-
fcrcnt rcpresentat ions a n d compression algori t hms ni-- offer optimal soli1 t ions for
cliffcrent applications. bit rates. and formats. Thercfore. XI PEG-4 provides four
clifferent types of coding tools: Cfdeo object coding for coding of a natural ancl/or
synt het ically originatecl. rectangular or arbitrarilx shaped video object. m esh object
coding for coding of a visual object represented with a mesh striictiire. rnodel-bnsfd
coding for coding of a synthetic representation a n d animation of the hiiman face and
t~ot l~ . . and st il1 fcst u r r coding for waveIet coding of st il1 text iires.
Video Object Coding
.A \-ideo object (VO) is an arbitrarily shaped video segment that Ilas a semantic
meaning. .A 2 D snapshot of a VO a t a particular time instant is called a video object
plane (VOP). .A \:OP is defined by its texture (luminance and chrominance values)
and its shape. 1IPEC;-4 allows access to the video objects and also to temporal
instances of the t-idto objects, i.e-. LOPs. To cnable access to an arbitrarily sliaped
object. siich an object needs to be separateci from the t~ackgroiind and other objects.
This process is called segmentation. and it can be performcd during encoding (on-
linc). or prior to encocling (off-line). The segmentation process is not stanclardized in
JIPEG-4. There are a number of automatic and senii-aiitomat ic tools a\-ailable for
segment atiori [XI]. Xlso. it is possible to generate image secluences t hat are segmented
init ially by using techniques such as chroma lie)-ing [ - IO] . where a unique color is iised
to separate the background from a \-icleo object.
hIPEG-1 vicleo object coding consists of shape coding (for arbit rarily shaped
\*Os). motion conipensatcci prediction to recluce temporal redtindancies. and DCT-
haseci texture cotling of t h e motion compensated prediction error data to reduce
spatial rctlunclancies. \-ide0 coding in AIPEG-4 is performed a t t h e macroblock level.
\-OPs are divideci i r~to macroblocks. siich t hat they arc reprcsented witti t he min-
irrium number of macroblocks within a bouncling rectangle. \ m e n the VOP is a
rectangiilarly shapecl vicleo frarne. 3IPEC;-4 vicleo coding becomes cliiite sirnilar to
that specified in 1IPEG-l/SIPEG-2 [Yi. 281 and H.263 [29]. Siniilar to SIPEG-1
and .\IPEC;--2. 1IPEG-4 supports intra coded (1-). temporally predicted ( P-). and
hi-directionally preclictetl (B-) VOPs.
.. 3exture coder (Scaa/] i
Variable > + Length
\Coder J Vidw I M m - 1 inverse ] 1
I
4 Arbitrary t shaped VOP ?
C shape' / @der,
Figure 2.4: :\ basic block diagram of an l1PEC;-4 \-ide0 coder.
Figure 2.4 shows the basic VOP encoder structure. T h e encocler consists
riiainly of two parts: a hpbrid of a motion compensatecl preclictor and a DCT-based
coder. and a shape coder. In t h e first part. motion estimation and compensation is
prrfornietl (escept for 1-VOPs) on texture data. followed by DCT. quantization and
VLC' coding. Motion information is also encodetl using VLCs. Then . the VOP is re-
constructecl as in the decoder. that is. by appll-ing inverse VLC. inverse DCT. in\-erse
qiiaiitization. and adding the resulting da ta to the motion cornpensatecl prediction
data. The resulting VOP is t hen used for the prediction of future VOPs. The stiape
coder encodes the binary shape information of the object. The shape infornlation is a
t\vo dimensional binary mask used to represent the sliape of a vicieo object such that
18
t h e pisels t i iat a r e opaque a r e par t of t h e object whereas pixels t h a t a r e t ransparent
a r e not par t of t h e object. S ince t h e s h a p e of a VOP ma>. not change significantly
between consecutive VOPs. predictive coding is employed t o reduce tempora l redun-
dancies. Thus. mot ion est iniation a n d compensat ion a r e also performecl for t h e shape
of t h e object . Finally. motion. texture. a n d shape information a r e mult iplesed with
t h e headers t o forrri t he coded \'OP bit Stream. At t he decoder end . t h e \,-OP is
reconstructed bj- combining motion. texture. a n d shape d a t a dccodccl from t h e bit
s t reani .
2.4.3 Motion Vector Coding
)lotion vectors ( I I V s ) a r e predicted using a spat ia l neighborhood of th ree MVs- a n d
t h e prediction error is t hen variable lengt h codeci. l [ o t ion vectors a r e t ransmi t t ed
onlj- for P -VOPs ancl B-VOPs. Since t h e V O P s can be arbi t rar i ly shaped. t he re
ma'- not b e a corresponding pixel available for precliction of t h e cur ren t VOP. In
order t o giiarantee tha t e\-ery pixel of t h e ciirrent L'OP can be predicted. sortie o r
al1 of t h e boiinclary and outsicle blocks of t h e reference VOP need to be padded by
e s t rapolat ion. Tl ie boundary blocks a r e padded by first repeating t h e bounclarj- pixels
in t hc horizontal direction. a n d then repeat ing t h e boiindary pixels in t h e vertical
direct ion w h i l ~ averaging pixels whose values were already obtairied by horizontal
padding. \Vhen a reference pixel belongs t o a hlock tha t is completely outsicte t h e
1-OP. then t h e hlock is filied by extended padding. where t h e pixels a r e assigned
a\-erage vaiiies t ha t a r e de te rmined by t h e neighboring blocks.
Bounding
Outside block box
Boundary bloc ks
Inside bloc k
Figure 2..5: A \,-OP enclosed in a rectangiilar boiinding box and dividecl into mac- roblocks.
Intra codecl blocks. as ive11 as motion compensated prediction error blocks. are
test tire coded. As in 'clPEG-1. JlPEG-2 ancl H.L6:3. DCT basecl coding is emploj-ed
to redrice spatial redundancies. That is. each VOP is divided into niacrol>Iocks. as
illiistrated in Figure T5. and DCT cocling is applied to the four Sxd luminance and
t wo SsS chrominance blocks of the macroblocks. If a macroblock lies on the boundary
of an arbitrarily shaped VOP. then the pisels that are outside the VOP are padded
hcforc DC'T coding. For intra VOP boundary macroblocks. padding is perfornled
as describecl i n the previous section. whereas for residual blocks. t h e region that is
oiitsicle the VOP is paclded with zeros. llacroblocks that are completely iriside the
L'OP are DCT transformed as in MPEG- 1. 1IPEC;-2 and H.263. The bIocks t h a t do
not helong to the VOP are not coded. DCT transformation of the blocks is followed
by ~~riantization. zigzag scanning. and variable length coding.
- Boundary Blocks
Figure 2.6: Binary alpha plane.
2.4.5 Shape Coding
L[PEG--k supports coding of shape information to enable content-bascd access to
individual video objects in a scene [ 3 3 . 41. 42. 401. Because of its liigh compression
performance and low complesitj-. a bit map-based shape coder was adopted in .LI PEG-
-1. In hitmap-bascd shape coding. t he shape and transparency of a VOP are defined
1)'- tlieir hinary and grayscale (respectively) alpha planes. A hinary alpha plane
indicates whet her or not a pixel helongs to a \'OP. .A grayscale alpha plane indicates
t h e t rançparency of eacli piscl within a VOP. MPEG-4 provides tools for Imth lossless
ancl 10s~'- coding of bina- and gray scale alpha planes. Furt hermore. bot h intra and
inter shape coding are supportecl.
Binarj- alpha planes are clivicied into 16 x 16 blocks as illustrateci in Figure
2.6. The hlocks t h a t are inside the VOP arc signaled as opaque blocks and the blocks
t liat arc outside the VOP are signaled as transparent blocks. The pixels in boundary
hlocks (i-e.. blocks that contain pixels hoth inside and outsicle the VOP) a r e scanned
in a raçter scan order and coded using arithmetic coding.
In inter shape coding. the shape of the current block is first predicted frorn the
shape of the temporally previous VOP by performing motion estimation and corn-
pensat ion. The shape mot ion vector is t hen coded predictively. Ses t . the tlifference
Letween the current and the predicted shape block is arit hrnetically coded.
2.5 Transmission of Compressed Image and Video
in Error Prone Environments
Input vidbo Output vide0
8 Source Coder
1 $
8
Transporl Coder - Channei - Tranqmt dscoder
Figure 2.7: A siniplified block diagram of a video conirnunication system.
Figure 2.7 shows a sirnplified block diagram of an image/video communication
SJ-stem. The input iniage/video is compressed by the source coder to t h e desired
bit rate. The source coder iç us~ially composed of a waveform coder and an ent ropJ-
coder. The transport coder in Figure 5.7 refers t o an ensemble of devices performing
cliannel codi ng. packet izat ion and/or modu1at ion. and transport level cont rol tising a
part icular t rarisport protocoi.
If al1 t h e coded information of a n image o r video seqiience is correctly re-
ceiveci. t he receiver can perform t h e corresponding decoding. yielding t h e intended
reconstriicted image or video sequence. As communicat ion channels a r e not error free.
t h e encodecl bit s t reams a r e viilnerable t o transmission errors. In fact , cornpressed
image and video bit s t r eams a r e ver>- sensitive t o channel errors due t o t he variable
length codewords a n d prediction techniques. which a r e iisually used in image/vicleo
coding algorit hxns.
Transmission crrors c a n be roughlv classified i n to two categories: random bi t
errors and erasure errors. R a n d o m bit errors a r e caused bj- imperfections of t h e
phi-sical channel. which c a n result in bit inversions. bi t insertions. a n d bit deletions.
T h e impact of random bit errors depends mos t ly o n t h e coding methocl. \\:ben
fised-Iength coding is used, a randoni bit error will on ly affect o n e cotleword and t h e
rcsult ing damage is generally acceptable. \\;ben a VLC is iised. random bit errors c a n
dcsynchronize t h e cotled iriforrnation such tha t many f d o w i n g bits a r e undecodabIe
i l rit i l t h c nes t synchronizat ion code word appears. Erasure errors. o n t h e ot her hancl.
can bc caused by packet loss ( in packet networks). burs t errors. o r short- t ime s'stern
failiirc. Random bit errors i n VLC' can also cause effect i\*e erasure errors since a single
Ilit error can lead t o man'. following bits being undecodable a n d hence useless. Sincc
almost al1 t h e state-of-the-art image/video compression techniques employ L'LC's.
there is no need t o t reat randorn errors a n d crasure errors separately [-KI!. Thercfore.
t hc generic t e rm -transmission error" o r .*cIiannel e r rorc will b e used t hroughout t his
t hesis.
T h c visual effects of er rors rnay propagate tempora l iy and s p a t i a l l ~ For ex-
Transmission Error
Figure 2.8: Error propagation in t he f rames of a video secluence.
aniple. in H.268 loss of information in a n intra coded f rame can propagate t o t h e
follo\ving frames. T h e resulting artifacts remain visible for a long t i m e w-hich is an-
noying t o e n d user. Figure 2.8 illustrates t h e typical effect of loss in a frame. AS
indicatcd in t h e figure. t h e e r ror m a y also propagate f rom i t s original position d u e t o
mot ion corn pensatecl prediction [U].
I I a n y techniques have been proposed t o comba t transmission errors. These
technicliles can be divicled in to th ree groups: e r ror resilience encoding. interactive
crror compensation. and error concealnient by post processing. \.+-e briefly review
cach of these tecliniques.
2.5.1 Error Resilieiice Encoding
Error resilience encoding refers t o those techniques t ha t t h e source coding algorit hm
ancl/or t ransport coder shown in Figure 2.8 are designed e i t he r t o minimize t h e effects
of t~~ansrr-iission errors without requiring any error coi-icealment a t t h e decoder. or t o
rnakc the error concealment t ask at the decoder more effective. There are man?- w-s
to make the coded data more resilient t o errors. Essentially. tliey al1 add a controlled
amount of redundanq* in either the source coder or the transport coder. The most
important techniques in t his group are:
a Layered coding: The \-ide0 information iç partitioned into several groups or lay-
ers. To combat channet errors. tayered coding must be combined with transport
prioritization so that the base layer is delivered with a higher degrce of error
protection. Layerecl cotling can be implementecl in diffcrent tvays depending on
t h c way the video information is part itionecl. leaciing to temporal scalabi1it.j..
spatial scalabilitj- and SNR scalability [45. -161.
l lul t iple description coding: In multiple description coding. sevcral coded bit
streams (descript ions) of the samc source signal are generated and transmit ted
over separate channels. At the dest.ination. dependi ng on which descriptions are
received correct ly. diflerent reconstruction scliemes are emploj-ed [-4 7. -18. 49. 501.
Joint source and channel coding: These techniques invoke source-channel in-
teraction. In general. joint source channel coding is accomplishecl bj- designing
tlie quantizer anci entropy coder for the given channel error characteristics to
rninimize the effect of transmission errors.
0 Robust wavcforrn coding: In traditional source coder design. the goal is to
eliminate 130th the statistical and visual redundancy of the source signal to
achieve the best compression gain. This. however. niakes the error concealment
task as the decoder very difficult. One approach to solve this problem is to
intentionally keep sorne redundancy in the source coding stage such t liat better
error concealment can be performed a t the decoder tvhen transmission errors
occiir. An esample of this technique is intra coding to eliminate teniporal error
propagation in a predictive video coding systern [ j l . 52. %3].
O Robust entropy coding: In this group of techniclues. redundancy is addecl t o the
entropj--coding stage of the source coder to help detect bit errors anci/or prer-ent
error propagation. Esamples of these techniqiies are using tlie synchronization
code words [52. 531. error-resilient entropy codes [54. 55. 561. ancl reversi ble
\-ariahle-lerigt tl codes.
O Forward error correction (FEC) coding: To make the video codeci data more
resiiient to cliannel errors. FEC' codes such as Reed-Solornon codes and BCH
codes are typically employed by the encoder. At the decocler. t hese FEC' codes
are then used to correct errors in the bitstrearn. -4lthough FEC techniques
provc quite effective against random bit errors. their perforniance is usually in-
aclecluate against long-cliiration burst crrors. The FEC techniclues also increase
overilead in terrns of the overall bitstream size: hence. part of the cocling effi-
ciency achievecl t>y video compression is lost. FEC is u s u a l l ~ appliecl to provide
a certain let-el of protection to the conipressed bi tstrearn. and the resicliial errors
are handled bj- other tools [ ~ i ] .
0 Transport level control: Examples of t liese techniques are the packet izat ion
schemes in packet vicleo, and intcrleaving. In packet ization. the out put of tlie
source coder is assenibled irito transport packets in siich a way that when a
packet is lost. the other packets can still be useful. Because a fised amount
of data has to be accumulated to perform interlea~ing. the latter introduces
relat ively long delays.
2.5.2 Interactive Error Compensation
If a backward channel from the decoder to the encoder is available. better perfor-
mancc can be achieved i f the encoder and decocler cooperate in the process of corn-
bating channel errors. This cooperation can be realized a t either the source codirig or
transport Ievel. At t h e source coder. coding parameters can be aclapted bwed on the
feecl back in format ion from t hc decoder. .At the transport Ievel, the feedback informa-
tion can be employed t o change the percentage of the total bandwidth used for FEC'
or retransmission I.581. Retransmission has been succes s fu l~ iised for non-real time
clat a trarismission. but i t has been generally considered unacceptable (due to incurred
delay) for real tinie video applications. Retransmission has also been considered inap-
pro priate for mrilt i point \-ide0 transmission hecause the ret ransmission request froni
a large nuniber of decoders can overwhelm the encoder. Another corlcern about using
retransmission is that i t may increase the loss rate becaiise it will add niore traffic to
the network.
2.5.3 Error Concealment by Post-Processing
Error concealment b ~ - post processing includes techniques in which the decoder fulfills
t lie task of error concealment. In general. t hese met hods a t tempt to recover the lost
in format ion by est iniat i ng and interpolat ing wi t hout relying on addit ional informat ion
from the encocler. LVe will discuss the various method of error concealment by post
processing (which is henceforth called error concealment) in more detail in Section
2.7.
2.5.4 Error Detection and tocalization
Before the transmission errors can be concealed. they nlust be detected. FEC' tech-
niques can he usecl to detect errors and pass the location of the errors to the video
clecoder so that the video decoder can concea1 t h e errors. In addition to FEC'. syn-
tactic ancl semantic error detect ion techniques can aIso be applied a t the tlecoder for
dctcction of crrors. In a typical block based video compression techriiclue that uses
motion compensated prediction and DCT. the following rules are applied to cletect
hitstrearn errors: 1 ) motion vectors are out of range. 2) invalid VLC table entry is
fountl. :3) t h e DC'T coefficients is out of range. and -1) the number of DCT coefficients
in a hlock esceeds 64.
2.6 Error Resilience Tools in Current Video Com-
pression Standards
1-1.263 incliic1es error resilience tools which are clefined in the baseline syntas and in
four of its normatii-e Annexes. The visual part of the IIPEG-4 standard also provides
support for error resilience. \Ve here siimmarize the error resilience tools of the abovc
ment ioned standards.
2.6.1 Resynchronization
Diie t o t h e application of VLCs in the vicleo compression algorit hms. t he location
in the bit strearii where t h e tlecoder detects an error is some irndetermined distance
away froni n-here the crror actiially occiirred. Hence, wtien the clecoder detects a n
error it loses synchronization !vit h t he encoder. tliat is. it is iinablc t o iclentify t h e pre-
cise location in t lie image where the current d a t a betongs. Resynclironizat ion tools.
as t lie nanie iniplies. a t ternpt t o enablc resynchronizat ion bet weeri t h e ciecocler and
t h c bitstreani after a resiclual error or errors have heen cletectctl. Resynchronization
niarkcrs. \\.hich arc distinct froni al1 the valicl code worcls. arc insertcc1 into t h e bit-
strcani cliiring encoding. Cpori detection of a n crror. t h e clecoder seeks forwartl in
t h e bitstreani for t he next res~-nchronization marker. Once this niarker is foiind, the
decocler gains s'-nchronization with the encoder. Resynchronization rnarkers localize
t h e errors spat iall~.. Generally. t h e d a t a betu-een the synchronizat ion point prior t o
t h e crror and t h first point u-here s>mchronization is re-established. is rliscarded. If
t lie rcsynchroniaation approach is effective a t localizing the ariioiint of da ta cliscarded
bj- t lie clecoclcr. t lien the abilities of ot her tools t hat recol-er da ta and/or conceal t h e
c fkc t s of errors is greatly enhanced.
-l'lie rrsynclironization approacties adoptcd H.26~ and 11 P EG-4 are simi-
lar and are briefiy reviewed in ivhat follows. In the t1 .26~ stanclarcis. a -Groiip of
Blocks'. ( G O B ) is tfefineci as one o r more rows of niacroblocks. A t t Iie s tar t of a new
GOB. iriforniation called a GOB header is placed rvit hin the bitstrearn. This header
iriforriiation contains a GOB s ta r t code. wliich al1ows the decoder to locate t h e GOB.
Fu rt hcrniorc. t tie GO B header contains in format ion which al lows t h e clecoding pro-
ccss t o be restarted (i-e.. resynchronize the decoder t o t h e bi ts tream and reset al1
codecl clata tha t is predicted). .A similar procedure can b e applied t o a n arbi trary
n i m b e r of acljacent MBs called -slice--
T h e C;OB approach t o resynchronizat ion is based on spat ial resynchronizat ion.
Tha t is. once a part icular macroblock location is reached in t h e encoding process. a
resj-nchronization marker is inserted into the bitstream. X potential problem with this
approach is t hat the resynchronizat ion markers will most l i keli- bc rinevenly spacetl
throughout the bitstrcam hecause t h e encoding process is variable ra te in nature.
Tliercforc. certain portions of the scene. stich as high motion areas. \vil1 be more
siisceptible to errors. which will also be more difficult t o conceal.
The video packet approach adopteci by hIPEG-1 is based on t h e insertion of
perioclic resj-nchronization markers t hroughout t h e bitst ream. In o t her words. t h e
length of t h e video packets are not based on t h e number of macroblocks. but instead
on the number of bits contained in each packet. If t h e niirnber of bits contained in
tlic ciirrent vicieo packet esceeds a predetermined thresholcl. then a neiv video packet
is created at the s ta r t of the iiext macroblock.
Synchronization markers a re ver- helpful in IocaIizing spat ial errors. hoivever.
t hey iiicrease bit rate. In [52. S 3 ] . it is shown t hat location of synchronization markers
cari l x optimally chosen to - i e l c l best rate-clistortion ( R D ) tradeoffs.
2.6.2 Data Partitionhg
Typical video decoders discard al1 t h e d a t a containing errors between two rcsynchro-
nizat ion markers. O n e of t h e main reasons is t h a t hetween two resynchronization
markers. motion and DCT da ta for each of the MBs are coded together. Hence.
the decocier cannot ident i f - if the motion or the DCT data of any of the MBs in the
packet is not erroneous. In MPEG-1. the data partitioning mode separates the motion
and the niacroblock header information frorn the texture (DC'T) information. This
approach requires t hat a second resynchronizat ion marker be inserted between t h e
motion and the tes ture information. Data partitioning is sigrialed to the decoder.
If the test tire information is Lost. this approach utilizes the mot ion information to
conceal the errors. Tha t is. due to the errors the tes ture information is discarded.
n-hile the motion is used to motion compensate the previously decoded VOP. The
fort hconiing t hi rd version of H.2613 will likely include da ta part itioning as well.
2 -6.3 Reversible Variable Lengt h Codes (RVLCs)
AS mcntioned earlier. one of the problems $vit h transmit t ing compressed video over
error pronc channels is the use of variable lengtli codes. During the decoding process.
if the decocier detects an error while dccoding LLC data. it loses synchronization
ancl hence. it typically discards al1 the data up to the ne'rt resyncfironization point.
Ri'LC's alle\-iate this problem. RVLCs are special VLC's that have the prefis property
bot 11 in t lle forward and the reverse directions. I-ience. t hej. can be uniquely decoded
bot h in the forward and reverse directions. Thereiore. when the decoder detects an
crror wliile decoding t h e bitstream in the forward direction. it jumps to the nest
resynclironizat ion marker and decodes the bitst ream in the hackicard direct ion till i t
encoiinters an error. In this way. the decoder can recover some of the data that would
haive ot herwise been discarded [59. 601. Both H.263 and .\f PEG-4 support the use of
2.6.4 Reference Picture Selection
The H.26:3 standard defines a reference picture seiection mode. which provides error
resilience. In this mode. bot h the encoder and the decoder have multiple prediction
framr hiiffers. Besicles vide0 data. t he encoder and the decoder eschange messages
regarding what is correctly received and what is not. Based on this information. the
encodcr cletermines which frame buffers have heen damagcd at t h e decoder. Then
tlic cncocier will use an rindamagetl frame buffer for prediction. The information for
t hc selected prediction buffer is also incliided in the encodecl bit st reani so that the
decocler can use the same frame buffer for prediction [4:3].
2.6.5 Header Duplication
Some of the header information present at the heginning of a coded video frame is
cssential ir i orcler to decode the frame, If sorne of this information is not recei\-ed
1)'- t h e decoder. the whole franic mal have to he discarded. In order to increase the
probability of receiving this information. header diipiication allows the introduction
of driplicatc copies of important Iieader information in the vicko packet. This option
is supported in XIPEG-4 and a simiIar tool is included in H.26:3+.
2 -6.6 Independent Segment Decoding
\\:hm slices are used in H.26:3. al1 predictions within a pictrire are restricted t o the
hoiiridaries of the slices. This. however. does not pret-ent error propagation due to
motion compensation. Error propagation can be preverited using the independent
segment clecoding mode supported in H.263. which enforces the treatment of segment
boiinciaries as if they were picture boundaries. A segment is defined as a slice. a
GOB. or a niimber of consecut i~e GOBs. By employing this mode. error propagation
oiitsicle the segment boundaries due to motion compensation can be avoided.
2.7 Error Concealment
Error coricealment iniends to conceal the effccts of channel errors by esploiting the
recliiriclancies in the video signai and the liniitations of the hiirnan visual systcrn.
witliout recluiring additional information [-131. Error concealment cannot provide per-
fect recovery. however. it can oFten provide a subjectively acceptable picture quality
wit fioiit increasing the transmission bandw-idt h.
Error concealment in an image or cideo signal. that is. reconstruction of lost
pixels is a n ill-posed problem. and as sucti. it does not have a iiniqiie solution- Er-
ror concealment methods solve this ill-posed problem by introdiicing assiimptions to
rcst. sict t lie ot hcrwisc infinitc number of admissible solrit ions. Different conceal~iieiit
met tiods arc outcomes of different assiimptions about the image and video signais
and/or different interpretat ion of t hesc assumpt ions. Basecl on the particular as-
siinipt ions made. the spat ially ancl temporally adjacent aiailable data are used for
restoring t Iie rnissing information. For compressed st i l 1 inlages and int ra-frames ( 1
framcs) of vicleo sequences. the avaitahle data are the spatially adjacent c!ata. For
prcdictccl frames ( P frames). both spatially and the temporallÿ adjacent data can he
used in concealinent procedure. Error concealnient methods that use the spatially
a n d temporal11 adjacent d a t a a r e referred t o as spat ia l a n d tempora l e r ror conceal-
ment methods. respectively. I n what fo l~ows we review different a-priori assurnptions
niade about image a n d vicleo signals a n d t h e concealment me thods t h a t are based
on these assiimptions [61]. Following t hat. I v e divide t h e rnethods i n to spatial a n d
temporal error concealment.
2.7.1 Consistency
.-\ frinclamental a-priori knowledge abou t pisel values in images a n d video is consis-
tcncy witli known valiles. Tliis reqiiires t h a t correctly rcceived pixel values reniain
iinaltered by the restoration process. a n d t h a t t he restorcd values lie within a n ac-
ceptable range (e.g.. O t o '25.5).
2.7.2 Smoothuess
Smoothness is a basic a s sumpt ion which has beeri iised in mos t of t h e concealment
techniques, and it has b e n interpreted in many different ~vays . O n e implication of
t lie sniootliness property of image signals used in [G2] is t h a t t h e variation b e t t e e n
adjacent pisels wit hin t. h e darnaged block a n d t heir s p a t ially neighboring pixels in
adjacent blocks should be small . Tiius. t o restorc t h e niissing da t a . a measiire of
spat ia l ~ a r i a t i o n s between ad jacent pisels (e-g. gradient or Laplacian) is minimized
[62. 631. T h e main shortcomings of t his niethod a r e t l ia t it uses a smal l fraction of
t h e a:-ailable d a t a (one pixel wide arca around the missing block) in t h e concealment
procecliirc ancl i t makes the assumpt ion t h a t t he lost block is smoo th . .As t he re-
coiistriiction method proposed in [62. 631 uses only a one pisel wide a r ea arouncl t h e
niissing block. it does not adequatelj- exploit structures that exist in the image in the
reconstruction process. The smoot hness assiimption. on the ot her hand. Iimits the
capahili ty of the reconstruction procedure in restoring image details and edges. The
smoothness asstimption may also Iead to artifacts around reconstructetl blocks.
The smoothness assumption can be formulated for temporally adjacent pisels
in a vicleo sequence to restore missing data in P-frames. In [64]. it is proposect that
in calculating the smoothness measirre. temporal information shoiild be considcred.
Thus. in forniulating the smoothness meaçure in addition to the term reprcsenting the
spatial pixel differences for pixels located at the boundary of the missing block. a term
icpresent ing the differcnce between the reconstruction resiilts and the pixel values at
the sarne position in the previous frame is also incIuded. This tcrm provides temporal
smoot hness and en forces a smoot h transition between adjacent frames. Obviously.
for fast changing scenes. this technique generates unacceptable resdts.
In [6.5]. the damaged blocks are recovered iitilizing the smoothness property
of an image at the boundaries of t h e blocks. In contrast to the met hod proposed
in [@]. the method proposed in [65] cloes not minimize the variations between the
saniple values in the missing block. Instead. a cost function ivhich repreçents t h e
\-ariations hetween values of the pisels of the missing block at the boundary and their
corresponding pixels in the acljacent blocks is dcfinecl. In [G]. the cost function i s
written in the DC'T domain and its derivatives with respect to eacfi of the 64 nlissing
DC'T coefficients are calculatecl. The resulting 64 cquations are not independent.
as t h e proposed cost function imposes restriction o n k on the boundary pixels. It
is proved in [6J] that for S x S blocks. tliis method is able t o restore at most '28
coefficients. m a i n l - lom frequency coefficients. while assuming t h a t t h e rest of t h e
coefficients a r e zero. Conseqiiently. t his met hod is iinable t o reconst ruc t edges and
details of t h e lost block. In [65] . t h e methocl described above hns been used for t h e
restoration of lost d a t a in bo th 1 a n d P frames. neglecting ternporally ad jacen t d a t a
available for P frames.
Another implication of t h e smoothness proper ty is t ha t a pixel ( o r a DCT coef-
f icirnt) is likcly t o have a value which is close t o t h e values of pixels (DCT coefficients)
in spatially adjacent blocks. Csing t h i s assumption. in [66]. t h e erroneous block is
replaccd wi t h t h e average of i t s t h ree nearest neighboring blocks in t h e previous row.
This tnethocl is computat ional ly very s imp le but ol>viously l imited in t e r m s of perfor-
niance. In [ 6 i ] . each pixel in a d a m a g e d block is interpolated i rom t h e corresponding
pixels of i t s four neighboring blocks siich t h a t t h e tota l scluared border e r ro r is min-
iniized. B y reconstructing each lost pixel in a block as a linear combina t ion of t h e
corresponding pisels froni neighboring blocks. t h e inter block correlation is exploitecl
and t lie stri ict u r e of the reconstructed block is determincd iising t h e s t r u c t u r e of t h e
rieiglihoring blocks rat lier t han by a global smoothness assumption which m a y yield
tinn-anted artifacts. 3Ioreover. local image characterist ics a r e preserved a n d liigh
frequcncy cletails can be reproduced. T h i s me thod yields good reconstruction of hor-
izont al. vertical. near-horizontal a n d near-ver t ical eclges. lines a n d pa t te rns . S t rong
diagonal edges thoiigh. a r e no t ire11 reconstrtictect 13'. tliis technique. The poor re-
prodiiction qual i ty around diagonal edges is d u e to t h e fact t h a t on ly o n e weight is
assignecl to each of t h e top. below. right. a n d left blocks in t he linear interpolation of
t h inissing block.
Based on the s a m e assurnption. it has been proposed in [68] and 1691 tha t the
pixel raliies in a damaged block be interpolated as a weighted combination of the
pisels in four boundaries ( 1-pixel-wide neighbor of the missing block). Ohviously t his
rnethod is very simple. but t h e performance is limited because t h e structure of the
image is not exploited and thus edges cannot be recovered,
In [TOI. a one-pixel overlap block structure is proposed for the reco\-ery of
lost information. Each block has a size q u a 1 t o 9 x 9 pixels. and overlaps with
the below ancl the right blocks. Using this method. if a lost block is surrounded by
undamageci blocks. t h c botinciary pixels can be ret rieved from neighbori ng hlocks.
The lost coefficients are estimated so that a cost furiction consisting of the surn of the
sqiiarecl differences between t h e estimated and the act ual intensities of t h e boundary
pixels is rninimized. The proposecl 9 x 9 overlapping blocks are not compatible with
n-iost of the current image and video compression standards. which require 8 x 8
non-overlapping hlocks.
For P-franies. smoothness nlay be interpreted in a iashion which assumes that
an area in the current frame is the same as the corresponding area in t h e pre\*ious
franle- A simple and yet effective temporal reconstriiction met hod iising t his as-
siimpt ion replaces the corrupted/missing rnacroblocks ( hlBs) wit h t heir correspond-
i n g parts in the previous frame. Although this method generally works well in still
parts of the picture. such as the l ~ a c k ~ r o u n d . it cannot produce sat isfactor~- resiilts
when the vide0 sequence eshi bits fast moving objects or sudden scene changes [il].
2.7.3 Statistical Correlatiou
StatisticaI correlation is another a-priori assiirnption made about images. This im-
plies ttiat the pixel values in an image are realizations of an underlying statistical
model. In the method proposed in [6S]. the original image and its corrupted version
are moclelecl as 1Iarkov Random Fielcls. -4ssiiming a spccific n prion distribution for
t hr original image. a maximum a posteriori (lI.L\P) estimation of the missing data is
obtainecl conditioned on the values of the received data. This met hod is compiitation-
aIiy espensive and it has limited performance in the restoration of edges. To rediice
t h e coriipiitational complesity. a suboptimal solution of the lI.r\P estimation problcm
lias bcen proposecl in ( T I ] . bx selecting eacli pixel in the missing block as the median
of its neighboring pisels. The median filtering methocl. however. cloes not yield a
iiniclue solution and cannot reconstruct clamaged edges. T h e above mentioned works
in spatial crror concealment exploit ing the LIRF mode1 use a single pixel [vide region
aroiind the erroneous area t o achieve t h e reconstruction. This practically restricts
t tic amount of available information iised in t lie concealment proceclure to a small
region around the missing area. Incorpoïating more pixels u s u a I l ~ leads t o a higher
order mode1 and t his is computat ionally espensive as the computat ional complesi tv
gron-s esponentially with the order of the MRF nlodel. Csing the above mentionecl
approaches. the damaged area is recoristructed fairly u d l in very low tfeqiiency por-
t ions of the image. However. the reconstruction process yields blurry restilts \vit h
a significant Ioss of detail in high frec!uenq+. or edge portions of the image. In the
statistical approach proposecl in [ ï3]. the image is modeled as an Auto Regressive
( .AR) proccss and the missing information arc estimated using the model. In [TA] .
a video seqiience has been niodeled as a 3-D AR process and the missing data is
ses tored using t his model. Because of the non-stat ionary characterist ics of the image
and video data. the methods proposed in [73] and [T - l ] are applied locallj..
In (751. error concealmeiit in the spatial doniain is aciiieved using a General-
ized ~ l a x i m u m Likelihood Ordered Statistics (GMLOS) which is a robust. nonlinear.
recursive filter with a variable kernel. The reconstruction starts frorn the boundaries
of t h e missing blocks and the valucs of the missing pisels are estimated recursii-ely
towards the center of the missing blocks. The filter kernels and the processing win-
clows arc chosen t o esploit the structural correlation of the rnissing 1IBs with their
surrounding neighbors. At the same tirne. the GAILOS filter exploits the statistical
correlation of the neighboring AlBs.
2.7.4 Edge Continuity
Edge contiriuity or more generally continuity of linear patterns in a n image requircs
t h a t edges in a scene be continuous. This assumption implies that if an edge is
present in the neighboring blocks of a missing block and its direction implies that
t h e eclgc passes through the niissing area. then the edge should pass through the
rnissing block. In [TG] this assumption is ~isecl in a Projection Onto Conves Sets
(POC'S) based retrieval met tiocl. T h e met hod proposed in [76] assumes a simple edge
structure (i.e.. onlj- a single dominant edge) and. therefore. does not work d l for
comples structures such as streaks. corners. etc. In [ T i ] . a heuristic approach. whicli
is cornputationally less espensive. replaces the POC5 stage in the above rnentioned
met hod. Alt hough yielding more accurate rcsults t han the approaches based on
snioothness assumptions. edge-based metliods a re cornpritationally demanding. The.
reqiiire t tie estimation of edge directions and for t h e POCS rnethod. many iterations
are recliiired before convergence. The rnetliod proposed in [ I S ] is also based on the
iclea of edge continuitu. To detect linear patterns (edges) in blocks adjacent t o a
missing block. iiowever. a tomographie projection based met hod has been proposed.
Alt horigh t lie edge detection stage of this algorit h m is different irom other edge based
error concealment nlethods. the reconstruction phase is almost the same. Therefore.
t his method has basically the same performance level as t hose of other eclge based
rcconst riict ion met hods.
In ['ïS] and [SOI. the local geometrical st riict ure of the image is estracted using
the available pixels siirroiinding a missing area. and t hen the missing pixels are inter-
polated using that structure. X two pixel wicle frame surroiinding the missing area is
iisecl to extract the geornetrical information. T h e frame is first thresholded t o form
a hinarj- pattern. Csing the location and number of white to black transitions in a
franiti. t h e directions and the number of edges passing through the missing area are
estirriatccl. Based on the estimatcd direction and number of edges. the rnissing area
is partitionecl into a nuniber of regions. In each region a bilinear interpolation of t h e
rnissing pixels is performed.
In [SI]. a mcthod which combines the spatial interpolation method iising edge
contiriiiity ancl temporal replacement techniques. has been suggestecl, -The objec-
ti\-c of t h e proposcd method is t o maintain tlic edge continuity between the niissing
macroblock and its surroiinding neighbors. while trying to minimize t h e difference be-
twer i the missing macrohlock and its temporal replacement. r! pre-processing stage
is performed to determine the similarity between the missing 41B ancl i t s correspond-
i ng temporal replacement. T h e similarit- rneasiire is based on com parison among
prominent feature (e.g.. the edge direction. motion vector direction and magnitude
ancl prediction error). I f the similarity measure is high. temporal reconstruction is
usecl. That is. first the motion \-ector of the missing LI B is synthesized as the average
of mot ion vectors of .\I Bs above and below i t . Then the missing area is replaced \vit h
the motion cornpensated area in the anchor frame. If t h e similarity meastire is low.
spatial restoration based on the edge continuity metliod is employed. For medium
values of the siniilarity measure a conibination of spatial and temporal restoration
met hods is proposecl.
In [S'L] spatial and temporal estimations of missing da ta are combined using
the following relation
u-hcre E(-Y 1 r) is the (optimal) estimate of rnissing information. L'(.Y 1 u;. x ) is
each of the spatial or temporal estimates of missing information and p(t1; 1 x ) is the
likelihood of that cstimatc. rt-hich is calcii1atecI by modeling the video sequeiice as
rnist lire of 1Iarkov processes.
2.7.5 F'ractal Behavior
The rnet hod proposed in [Y31 assunies blockwise similarity in an iniage (i.e.. a Cractal-
l i kc assumption). The wfiole image is searclied to find the best match of an enlarged
niissi~ig N B composed of a missing MB and a two pixel-wide frame around it. The hest
niatch is the one tliat niinimizes a cost fiinction which is a function of the clifference
between t h e tn-O pisel-wide frame around the missing block a n d two pisel-wicte frames
correspondin; t o o ther 3IBs of t h e image.
In [S-l]. an image is modeled a s t h e density of particles of a material wliose
behavior is expressed by the diffusion ecluation. To restore the missing pixel values.
t h e diffusion equation is solvecl given t h e houndary conditions which a re t h e \-altxes
of t h e pixels in the neighboring blocks a t t h e border with t h e missing block. Below.
UT provc that this method is esac t l j - t h e same w the one proposed in [62] and.
consequent [y. t heir performances levels and compiitat ional complexi tics a r e the sanie.
The mctliod proposed in [G?] finds the pixel vaiues f ( r . y ) such tha t t h e siim of t h e
scltiare of clifferences betwcen adjacent pixels over t h e niissing area is minimized. In
o t lier words t h e function f minirnizes t h e following integral
where c(.r. y ) is t h e weight at position (s. y) . According t o t h e Euler t heorem. if L is
a given fiinction. then the function t~'(x. y ) tha t minimizes t h e integral
is t lie solution of the follo~ving different ial equation
where cc, and uqY are t he partial derivative of rc with respect t o s and y respectivelj-.
Lysing t h e Euler theorem for Eqiiation (2.1 ) ive have
Thus. f is t he solution of t h e following equation
which can be wri t ten as
which is the diffusion eqriation.
2.7.6 Temporal and Spatial Error Concealment
Error coricealment niet hods use one or more of t h e a-priori assumpt ions discussed
abm-e along with t h e available d a t a to reconstruct t h e niissing information. Based o n
t lie avaiIahle d a t a wtiich is eniployed in the reconstruction process. error concealment
met hods are cli viclecl into t tvo groups: temporal a n d spatial.
In temporal error concealment methods. which are usualiy rised for error con-
ccaInient in inter coded frames. t h e missing blocks are restorecl using t h e information
in t h e temporallj- adjacent frames. Data of a typical motion compensated inter coded
block is cornposeci of t h e mot ion vector and the DCT coefficients of t h e precliction er-
ror. .A simple temporal concealment met hod involves replacing the corriipted/missing
I~locks \vit h their corresponding parts i r i t he prex-ious frame [ T l ] . This niethod. whicti
works well in still par ts of t h e picture (e.g.. background). cannot procluce satisfactory
rcsults whcn ttic vicleo sequence eshibi ts fast moving objects. lighting changes. or
sudclen sccne changes. If t h e motion vectors are adeqiiately protected so tha t they
are received wit hout errors. t he information in t h e previous frame is motion conipen-
sated and serves t o conceal the lost information in t h e current frarne. However. in
man>- of the vidco compression aigorit hms. the loss of dat.a of a block results in the
loss of bot h t h e motion vectors and the DCT coefficients. This is hecause the codecl
motion vectors and DCT coefficients a r e iisually interleaved. Therefore. most of the
proposed temporal concealment met hods first estirnate the motion vectors associateci
with a missing block. The estimatecl motion vector is then useci to find a block in
the prel-ious frame rvhich yields the restoration of the missing block. To est imate
t h e niissing motion vectors. methods such as averaging [TL]. median [TZ] and AL--\P
estimation [T2] have been suggested. T h e main draivback of methods tliat use the
motion vectors of blocks adjacent to t h e missing block as basis for estimating the
niissing motion vector is tliat. if adjacent blocks are coded in non-predictive mode
(intra mode). there rvouid not be an'- da ta available for the estimation procedure.
Thiis. ad-hoc assiimptions for estimating the missing motion vectors shouId be made.
ancl the results m+- be unreliable. 1Ioreover. if the missing block lies at the bouridary
of two objects moving in opposite directions. such methods perforrn quite poorlj-. Last
bii t not t hc Icast . mot ion vector estimation can gencrally be inaccurate. cspecially
wlien large block sizes (e-g.. 16 x 16 pisels) are used.
Spatial error concealrnent methods restore the missing blocks usirig the infor-
mation in t lie ciirrent frame. Spatial concealment metiiocfs proposed earlier incliide
averagi ng or linear interpolation [67. 681. const rai ned linear interpolation [68]. mult i -
direct ional edge- basecl met hods [76. 77. Y.3.I. and stat istical met hods [Y-. 7-11 .
2.8 MAP Estimation of Missing Data
Let .Y be an ,\:, x , \ f rame a n d let 1'- b e t h e received version of ,Y a t t h e o u t p u t of
t h e channel. where 1' ma? have nlissing da t a . Each transmit ted pictiire consists of .\/
blocks. each having -1' x .Y pixels. Let xi be t h e lexicographie orclering of t h e ith block
of ,Y. T h e vector X is then defined t o be t h e concatenation of t h e xl.xz. - - - .xJ,.
tha t is X = [x1.x2. - - - .x-\I].
Siniilarly. t h e vector Y is defined as t h e concatenation of t h e blocks of l'*.
I f t h e ;th block is n-iissing then Y c a n h e espresscd a s Y = TX. n-here 7 is a n
- x m a t r i s t h a t corisists of t h e identity m a t r i s excluding t h e r o w
O r u - \ - O O ( + 1 - 1- If n of t h e J I blocks a r e missing. t hen 7 will
l>c a n - n,\") x ;\-1-V2 niatr is . Vsing XIAP estimation. t h e e s t i m a t e of t h e
clcconlpressed image is given bj.
x = nrg m a s log P r ( X 1 Y). X
L-sing Bayes rule i v e ob ta in .
\rllcre t!ir term log P r ( Y ) has been dropped because it is independent of X. Lsing
t hc relation Y = TX. t h e conditional probabili t j- of the corrupted image can then be
Siibst ituting Equation ('2.3) in Equation (2.2). the Mr\P estimation of the decorn-
~ r e s s c d image becomes
x = « r y min(- log P r ( X ) } . XES
where S = {Z : Y = 72).
2.9 MRF Modeling
Iniage mocleling refers to the development of analytical representat ions for the inten-
sity ciistributions in an imagc. There are different types of image models. each being
appropriate for one or more classes of applications [97. 981. The most important
reasori for iising image models is the abstraction thcy provide of the large arnoiint of
da ta contained in images. L-sing analytical representations for images. one can de-
\-clop s>*sternatic algorit hms for accornplishing a particular image-related task. C" 1 I ven
tha t the pixels in a local neighborhood are correlated. researchers have proposed 2-D
extensions of time-series moclels for images. The Markovian notion has also been cx-
tendcd to 2-D and l I R F has been introdiiced and estensively iised for imagc mocleling
[99]. The at t ractii-e features of an I IRF model are its ability t o represent different
iniage sources and the local naturc of the resiilting estimates. To enable the rnoclel t o
acc i i ra te i~ characterize the image data. adjustable parameters are iisually considered
in t h e model. Here. we hriefly review h'lKF mociels.
T h e concept of an )[RF is a generatization of tha t of l la rkov processes (XIP)
n-hic11 a r e u-idely used in sequence analysis. An MP is a secluence (cliain) of random
variables . - - . - - . .Y,. - - - defined on t h e t ime indices { - - - . 1. - . - . rn. - - -1. . ln n-th
order unilateral I\IP satisfies
.A i~i iaterai or non-causal l [ P depends not only on the past but aIso on the future.
An n-th order bilateral MP satisfies
1IP is gcneralization into 1 I R F s when t h e t ime indices are replaced with spatial
indices. To this end. t h e concept of neighborhood should be introduced.
Let S represent a discrete set of $ sites
in wliich 1.. - - . .V a re indices. X s i te often represents a point or a region in the
Eiiclidcan space such as an image pixel o r a n image feature such as a corner point.
a lirie segment or a surface patch. The sites in S a re relateci t o one anotlier via a
ncighborhood system. A neighborhood system for S is clefined as
where .\; is the set of sites neighhoring S. T h e neighboring relationship lias t he
foIlowing propert ies:
:\ si te is not neighhoring to itself: s ,\i;. and
47
F i r e 2-91 .A site (X) and its ( a ) first order neighborhood system. (b) second order ncighborhood system.
0 Tlie neighhoring relationship is mutual: s E .\: o r E .\:
I n t h e first order neigh borhood system. also called the 4-neighborhood system. every
site has four neighbors ris shoirn in Figure 2.9a. In the second order neighborhood
systeni. also callecl the Y-neighborhood srstem. there are eight neighbors for each site.
as stiorvn in Figure 2.9b. T e pair A forms a graph. A siibset c 2 S is a clique
i f cvcry pair of distinct sites in c are neighbors. C denotes the set of cliques.
Let .\- = .ysI. .si E S he a famil)- of random variables defined on the set S. in
wliicti each randon1 variable .Ys, takes a value r,,. For simplicit; we can assume a
cornnion state space. SV .\ = 0. 1.2. - . L - 1. Tlie faniily .Y is called a random
field. X joiii event of (S,, = x,, . -Ys, = x,,. - - - . .Y,, = r , , ) is abbreviated as .Y = x.
irherc s = ( .r , , . - - - .+,,) is a configuration of .V-. corresponding to the realization of
thc fielcl. Let O be the set of al1 possible configurations:
S is sait1 to be a Markov random field on S with respect to a neighborhood system
-\- i f a n d only if t h e foliowing two conditions a r c satisfied:
P(-Y = cc) > O . and
.-Iccording t o t h e Hammerstey-Clifforci theorem. everq- 3IRF h a a Gibbs dis-
t r ibut ion j99]. A n hIRF with a Gibbs distribution is given by
wliere Z is a normaIizing constant . G(.) is callecl a potent ia l function ancl is a functiori
of a locaI group of pisels c called clique. a n d C denotes t h e set of al1 cliqucs t hrotighout
t h e image. T h e potential function characterizes t he relationsliip hetween a group of
pixels hy assigning larger costs t o configurations of pisels wIiich a r e Iess likelj- to
occiir. The choice of t h e potent ial function inipacts subs t an t iaIlq- the performance of
tlic image niodel. T h e function xcSC C;(x) niiist be conves in order t o have a n casil)--
ob ta inahlc gIobal minimuni. If th is function is not conves . local min ima a r e present.
tvhicli can he avoiclecl \.ia con ip i i t a t i ona l l~ espensive techniques such as simulated
annealing. C'ornmonlj-. t h e poteritial fiinctions a r e selected t o be in t h e forni
where d, is a coefficient vector a n d x is t h e vector of pixels in t h e clique c and p( . ) is
the cost function. T h e convexit,? propertg of t h e cost function insiires that t h e srini
of poterit ial lunctions (CcEC C . ' ~ ( J - ) ) remains conves. T h e select ion of t h e coefficients
d, is based on the a priori assumptions about the image. The coefficients are iisually
selectcd so that d:x provides an approximation of the first or second derivatives of
the image at each pixel. For the special case of p f r ) = x'. the mode1 is called a
Gauss-Jlarkov random field (GJIRF). To reduce the snioothing effect of the GhIRF
model. other forms of cost functions have been introduced such as the Huber function
~vliere 7 is a threstiold. The image niodel using this cost fiinction is knowri as the
Huber SLarko\- Random Field (HXIRF) model. The Huber fiiiiction is a n esarnpie of
robiist error meastires. There are other CV0 (e-g.. Lorentzian. Ckman-McLiire) and
non-Co (e-g.. truncated qiiadratic) measures. in general cailed redescending estinia-
tors.
2.10 Summary
In t liis chaptcr. IVC first rcviewed image and video compression stanciaïtIs. Tlien. we
surveyed different error concealment methods suggested in t h e literatiire for coded
image a n d vicleo aric1 compared t heir n-pr ior i assiimpt ions. performance and com-
plesity. C;enerally. error concealment met hods t hat exploi t the st riictures t hat esist
wi t hin t lie image in the reconstruction process have bet ter performance le\-els. al-
t liougli t tiey are more compiitat ionally expensi \-e. as extra computat ions are required
to est ract the image striicture. Tliere is an inherent, t rade-off between computational
com plêsi ty and cliiality of the restoration resiilts.
Chapter 3
Error Concealment: Texture
Informat ion
-4 man who hn.5 comrnif td a mistake and doesn-t corr-ect i f is cornrnilting
a n o t h ~ r nz is tnk .
Con fucirrs
3.1 Introduction
In this chapter. we propose error concealment met hods for the test ure component
of coded \+isual data. I t is assumed that the missing tes ture information lielongs
to olie or more blocks. W e consider intra coded and inter codecl testure separately
and propose error concealment methods for each of theni. Obviously. concealment of
errors in cornpressed still images is identical to t hat developed for intra coded testure.
5 1
Position of missing blocks. Data of intra cadeci frame
Compensate the effects of missing blocû on the rest of the
blodrs [if applicable)
Reconstmct the rnissing data (deteministic or statistical
methods)
* Reconstruction resuit
Figure :3.1: Error concedment of int ra coded test ure information.
Concealmcnt oierrors in intra coded testure information is perforrned by invok-
ing spatial error concealnient metliods. That is. the correlation hetween the missing
t~lock and its adjacent blocks in the same frame is used for error concealment. The
metliod introduced in this chapter for concealment of errors in intra coded textiire
consists of two stages as shown in Figure 3.1. First. if the loss of coded data of a
block !las any effect on the rest of the blocks of the frame. this effect is compensated.
For esample. in man- of the image and video compression schemes ( c g . . JPEG and
AIPEC2) the DC coefficient of the DC'T transform is differentially codecl. Therefore.
the decoder is unable t o decode the DC coefficients of the blocks which follow the
missing one. Second. the missing texture corresponding to the missing blocks is re-
constructed. k k present both a deterministic technique and a statistical one for the
restoration of the missing testure inforniat ion.
In most of the error concealment methods proposeci for testure (e-g.. those
proposed for JPEG codecl images). the dependency between the coded data of blocks
has been ignored. In other words. it h a s been assiimed that the DC coefficients of
blocks are coded non-differentially [67. 621. This constrains the impact of a block
loss to onl~. the siibject block and therefore simplifies the error concealment task
but is not conipliant w i t h many image and i-ideo compression standards (e-g.. J PEG
coded images) [ 3 ] . Here. ive propose an error concealment met hod for t h e intra codcd
texture t tiat acldresses the two main problems. tha t is. t h e loss of DC track and the
reconstruction of data of missing blocks. These two probIems are independent in
t h e sense that any methocl for the recovery of the lost DC track can in principle
t x coinbined with an!- method for the recovery of the data of a missirig block ancl
vice-versa.
It shoulcl be noted that the cfependency between the codecl blocks of a frame
can he remot-cd by employing tools supported in compression standards (e.g. rest art
nlakers in J P E G ) . Also. some of t h e \-ide0 coding schemes (e.g.. H.26:3) do not use
di fferential encoding of DU coefficients. In t hese cases. the error concealmcnt algo-
rithm skips the first stage shown in Figure :3.1 and proceeds to the nest stage which
is t lie reconstruction of the missing information of the block.
In concealment of inter coded test ure informat ion. the post-processing tech-
nique makes use of the correlation between the missing block and its neighboring
Position of missing Mocks. Data of inter coded frame
Spatial reconstruction of the prediction e m of the missing
block
Data of adjacent ftames
Initial estirnate of the missing b M s
Figure : 3 2 : Error concealment of inter coded testuce information.
I~locks in t h e same frame and/or the adjacent frames to accomplish error conceaI-
nient. A block ciiagranl of the method proposed for concealment of errors in inter
coclecl testiire information is depicted in Figure 3.2. First. the information in the
previous frame is used to fintl a n initial estimate of the missirig testiire information.
If t h e mot ion vectors of the missing hlocks are avaitable. motion conipensat ion is usecl
to provide the estimates. Othertvise. a n algorithni which preserves image continuity
is empIoyed to compute the initial estimates. Ses t . the spatial error concealment
nietliod is applied to the prediction error of the blocks acljacent to the missing blocks
in orcler to find an estimate of the prediction error of the missing block. The re-
sul t is then combined with t h e initial estimate of the missing block to find its bcst
approsimat ion.
In the rest of this chapter. we rvill describe each of the stages shown in Figures
3.1 and 3.2 in more detail. In Section 3.2. a post-processing method is explaineci
t hat cornpensates the effects of the loss of a block on the rest of the hlocks in the
sarne frame. Sections 3.:3 and 13.4 present the deterministic and stat istical met hods
proposecl for reconstruction of missing texture information. respectivelu. Section 3..5
presents the niethod used to find the best initial estimate of a missing inter codecl
hlock in an adjacent frame.
3.2 Removing the Effects of a Missing Block on
the Rest of the Blocks
In man'. of the compression schemcs. the DC' coefficients of the blocks of the image
are codecl clifferentially. That is. the input source image is partitionecl into (8 x 8 )
blocks dong the raster scan order (i.e.. left to right / top t o bottom) and DCT
transfornied. Each of the resulting DCT coefficients is quantizecl iising a different
ilni form quant izer. The DC coefficient wliich is first coefficient is different ially coded.
T tiat is. t Iic clifference coefficient DIFF given hy
D I F F = di - di-1.
irhere d, and d Z e l are the current block and the pre\-ious block DC coefficient \.cllues
(respect i\-ely ). is coded.
The recovery of the lost DC track. which is necessary for the decoder t o decode
t h e DC' coefficients of the blocks which follorv t h e missing one. has not b e n much
addressed in the literature. In [66]. different.ia1 encoding of t he DC coefficient in a
DCT-based coder has been considered. but it was assumed that there is a synchro-
nization code word a t t he end of eacli tine of blocks. The synchronization code word
iimits the propagation of errors due to loss of the DC' coefficients. making the error
concealnient task easier.
To enable the decoder t o decode the DC value of hlocks following a missing
t~Iock. t h e DC 1-alue of the missing block shou1d bc estinlateci.
3.2.1 Estimation of DC values
Figure 13-13: Estimation of DC value.
I t is assiimeci ttiat the clecoder knows the location of missing blocks. This can
l>e achieved. for esaniple. by transmit t ing the blocks of the image/frame in a predcter-
miilcd ordcr and assigning secluence numbers to packets in packet bascd transmission
[ f > i ] . The dccoder s tar ts decoding the data. and when i t reaches a n arca representing
a missing block. it estimates t he DC value of the missing block using the DC' value of
adjacent hlocks. The DC value of the missing block is estirnateci hl- a iinear, causal
estimator ci. which is given by
wlierc d l . d2 . & and d4 are t he DC values of the adjacent blocks shown in Figure 3.13.
Ttic coefficients of the estimator are ohtained iising well-k~iown Linear Least Square
Error (LLSE) techniques. To obtain a set of fised values for t h e coefficients that
can be used for naturai images. four training images. hi,\';DRILL. FICHTER. L-AKE
and C O U P L E were used. The obtained values for the coefficients are u?l = -0.35.
rr- = 0.54. 1 ~ 3 = 0.1:3 and Z P ~ = 0.68. If for a specific missing block any of the
neigliboring blocks does not exist. (e-g.. for the b!ocks in the top-rnost row or in the
Ieft-rnost coliimn). the corresponding DC value is set to zero. In partictilar. i f the
block at the top left ( the first blockj of the image is rnissing. the estimated DC value
of t lie niissing block is set t o zero. This causes a constant level shift in al1 the pixels
of t hc image which can be corrected by adjusting the bright ness of the image.
3.2.2 Removal of stripes
'in crror in estirnating the DC value of a missing block causes a shift in the gray
level of al1 the pixels between tha t block and the ncxt missing block. This level shift
appears a s a stripe across the image. To remove the stripe. the amount of the level
shift should be deterniined. Adjacent pixels in an image are highly correlated and
t heir valries are close to each other. Therefore. the difference betu-een the average of
pisel values in the st ripe and t hat of t heir spatially neighboring pisels outside (above)
the stripe seems to be an appropriate estimate of the level shift. Lysing the a-priori
information about the position of missing blocks. we conipute
where li.i+l is t lie average value of t he pixels d o n g the iipper line segment connecting
tlic missing hlocks i and i + 1. and is the average value of the pixels in the
corresponding line segment in the previous row (see Figure 3.4). li,i+i can he wri tten
wherc r; is t h e column niimber of the right coliimri of block i. li+l is the colurnri
nuriifm- of left colrirnn of block i + 1. p is t he row nuniber of thc uppcr pisels in t h e
t~locks i and i + 1 and .\;v;+i is t he number of pixels in the ron. p between columns
r; ancl i;,,. Sitiiilar definition applies t o m;.;+l- If (Ail > T . where T is a threshold.
a s tr ipc is detected ancl removed bu subtract ing 1; from al1 the pisels bctweeri t h e
missing hlocks i ancl i+1. The proceclure is repeated for a11 the 11 niissing blocks.
Figure 3.4: Post-processing for reniot-ing the stripes.
T'lie nlissing btocks i a n d i + 1 ma>- have differcnt horizontal positions. but
t Iiis docs not affect t h e aho\-e procecliire. AS discusse~l ahove. t h e average of t h e
pisrl t-allies d o n g t h e uppcr l i ne segment inside t h e s tr ipc is corriputeci. and it is thcn
coniparecl to t h e average of t h e correspo~~ciiiig piscls in t tic previous line segment.
'The liric scgnicnts woulcl. of course. be broken lines in t liis case. Figure 13.5 shows
ttic 1-arious cases tha t can occur when t h e missing blocks have different horizontal
positions. Obviously. if two missing blocks a r e inimecliatelj- adjacent t o cach otlier.
Figure 3.5: Rernoving t h e s t r ipes for blocks wit h different horizontal positions.
rio s t r ipe is forrned between them. and t h e a lgor i thm proceecls t o t h e nes t pair of
iiiissing hlocks. For t h e inissing blocks appear ing in t h e upper-niost par t of t h e
image. there is not a previous row. Therefore. t h e average of pixel valiies in a block
I d o r e a n d a block a f t e r t h e missing block a r e compared t o e s t ima te the level stiift.
T o determine t h e t-altie of t h e threshold T. t h e dis t r ibut ion of t h e differences
I ~ c t tveeii t h c nt-erage pixel valiies of consecut ive rows in a n image was obtained for a
set of images. Figure 3.6 shows t h e histogram of t h e difference values for t h e images
l l x s ~ n i t ~ and FIGHTER. -4s it can be seen. t h e his tograms have a peak a round
zero. indicating tliat mos t of t h e rows have a lmos t t h e saine average pixel values.
Scvertheless. wit h a sma l l t hreshold value (i-c.. a r o u n d zero). t here is t he possibili ty
of esceecling t h e threshold level because of a n edge a t in Figure 3.4. despi te
tliat there are no visible stripes. Taking this fact i n t o account a n d considering t h e
diffcrence value at which s t r ipes becorne visible t o t he hiiman e o (especially in t h e
sniootli regions of t h e image) . and t o maintain the probabi l i ty of false a l a rm (i-e..
t licrc is not a s t r ipe bu t t h e algorithm de tec t s one) to less t han 1%. a tlireshold value
Figure 13.6: Histogram of t h e difference value: ( a ) Fighter: ( b ) 1Iandrill.
of 4 [ras selected [LOO].
3.3 A Deterministic Method for Reconstruction of
Missing Texture Informat ion
The estimation of the DC' values of the
t h e blocks foilowing the missing block
data shoulcl be reconst ructetf.
, niissing blocks enables the decoder to decode
. Xfter the stripes are removed. the missing
Our determiriistic method for estimating t h e missing data of a I~lock is based
on the one siiggested in [65] . The method proposed in [67] estimates the data of a
missing biock as a linear combination of its top. hottom. right and left blocks. i.e..
t lie estimated matr is pz is given by
d i e r e PT- PB. PL. ancl PR are 8 x 8 matrices of pixel values of the top. bottom. left
and right blocks (respectivelj-) of t h e rnissing block '. The weights IL '~ .WB.UL.WR a r e
selectecl so t hat t h e to ta l sqiiared border error
is minirriized. Each border error is given b ~ .
whcre the ma t r i s is generated using Equation ( 1 3 . 1 ) . The vectors pZt. pZb.
and p z r are those elements of pz t ha t corresponci t o the top. bottom. Ieft and right
pixels in the missing bIock. respectively. Similar ly p ~ a consists of t h e bot tom line
pixels of the top block. p~~ consists of the top line pixels of t he bot tom block. p ~ ,
consists of the right line pisels of t he Ieft block and p ~ , consists of t he left pixels of
the right block. By reconstriicting each pixel in a block as a linear combination of
the corresponcling pisels from neighboring blocks. t h e s tructure of t h e reconstructecl
block is determined by t h e s tructure of the neighboring blocks rather than hy a
global smoothness assumption which may J-ielcl iinwanted artifacts. Moreover. local
image characteristics a r e preserved and high frequcncy details can be reproduced.
This met hod can reconstriict very weIl horizontal. vert icd. near-Iiorizontal and near-
vertical edges. S t rong diagonal edges. however. are not reconstructed well bu t his
' \Vithout loss of generality, in t his section. WC assume that the size of a block is 8 x 8.
tectiiiiqiie. What J-ields a poor reprodiiction qualitj- arountl diagonal edges is the fact
ttiat only one weight for eacli of the top. belou*. riglit. and left blocks is used. In our
proposecl method. two weights are assigned to each of these blocks. First. we consider
the case tliat each of these blocks is partitioned into two parts. and a constant rveight
is assigned to each part. Thiis. t h e estimation eqiiation ivill be modified t o
where Li,.l is the 4 x 4 identity rnatrix. O is the I x -1 zero matrix and WTl. i q - 2 .
~ L ' B I . (1'82. ICLI . LL'LZ. I C R ~ . and are weighting coefficients. Values Cor the latter
coefficients are selected minimizing the cost function described in (3.2) and (3.3).
Equation (:3.-1) can also be ivritten as follow.
wlicrc p7-l is p~ half replacecl wi th zeros and PT, is t lie o t lier half of p ~ . .A similar def-
. . *
~riition applies to PLI . PL* PRI and PR*. Clonsidering Eqtiations (3.1) and (3.5).
tlie reconstriiction procedure can hc interpreted as tlie estimation of a 2-D function
~ising a set of basis fiinctions. The niimber of basis fiinctions is four in Eqiiation (3 .1 )
a n d eiglit in Equation ( 3 . 5 ) . Equation (3.1) is a special case of Equation ( 3 .5 ) . where
l C ~ l = ( ~ 7 ~ . U'BI = tL'82. 1 ~ ~ 1 = uy~2 : aiid (CRI = Therefore. al1 the satisfactory
fcatiircs of the method proposed in [6ï] are preserved in our method. Increasing t he
nuniber of independent b a i s fiinctions is espec ted t o ciecrease t h e estimation error.
T h e manner in which adjacent blocks a re parti t ioned affects t h e basis functions a n d
thus t h e est imation results. .Ml t h e adjacent blocks should not be parti t ioned in
jiist one direct ion (horizontally or vert ical1~-). as siich a part i t ioning wotild m a k e Our
niethod alrnost t h e s a m e as the one proposed in [6C] with a block size of 8 x 4 o r -1 x S.
1Ioreoi-er. two blocks t ha t lie in t lie s a m e side of a diagonal edge (e-g.. left and bot tom
blocks) shouid not b e partitioried in t h e s a m e direction. becairse this cariscs four of
t lie basis functions in Equat ion (:3..5) t o b e alrnost t h e same. The choice of which of
t h e two opposirig blocks shoulcl be part i t ioned horizontailj- or vertically is arbitrarj-.
In this work. as shown in Figure 13.7. t h e t o p and b o t t o m bIocks are parti t ioned in
t h e horizontal direction. and the left and right blocks a r e parti t ioned in t h e vertical
direction.
Figure :3.7: The partitioning of t h e blocks adjacent t o a missing block.
T h e weight matr ices in Equat ion (:3.4) a r e diagonal matr ices and t h e diago-
6 3
na1 e lements of each ma t r ix can b e considered as a n S-point sequence. For t h e case
discussed above. each of t h e 8-point sequences is a piece-wise constant function with
two values ( e . g . . a n d w L 2 ) -4s will b e seen in C'hapter 5. using piece-wise con-
s t an t secpiences improves t h e reconst ructiori of diagonal edges bu t m a y genera te some
blockiness in t he reconstructed images. If t h e 8-point seqiiences m e n t ioned above a r e
chosen a s linear ramps. t h e es t imat ion equa t ion can be wri t ten as
w-licre rnr. m ~ . n z ~ ancl r n ~ a r e t h e increments assigneci t o t h e weights of t h e cor-
rcsponding blocks. Iralues of t h e unknown parameters (zctls a n d r n , s ) a r e obtained
b>- minimizing t h e total border e r ro r described in Equation (3.2). C-sing t h e above
est imat ion. t he reconstructed images will not contain much blockiness but t h e diag-
onal edges may b c snioothed. O t her types of S-point sequences c a n also be assigned
t o eacli block. This. howevcr. would increase t h e computat ional complesi t j - of t h e
nlethod. Indeccl. ou r results show t h a t t h e improvement in t h e visiial cliialitj- of t h e re-
const riictcd images cloes not just if!- t h e addi t ional complexi t? of iising more corn ples
weighting schemes.
C-sing the diagonally ad jacent blocks can help improve t h e reconstriiction of
diagonal edges. This. Iiowever- would increase t h e computat ional coinplexity of t h e
c r ros concealnient me t hod bj. a fac tor of S. l loreover. t h e diagonally adjacent blocks
a r e fart lier from t h e missing block t h a n t h e horizontal a n d vertical acljacent blocks.
Thiis. t heir correlation wit h t h e missing block is usually less t han t h a t of t h e horizontal
or \-ertical adjacent blocks. Therefore. including diagonally adjacent blocks in the
error coriceatment would likely not prot-ide a signi ficant irnprovement in reconstruction
qriality. In fact. oim simuIations show that. in many cases. the increase in quality of
the reconsttucted image as a resiilt of using diagonaliy adjacent blocks does not justify
the increase in computational complesity.
The \veiglits w r ~ . t c ~ l . w ~ l . WB-. L C L ~ . U L ~ . W R ~ ancl C L ' R ~ . which are required for
cstimating the rnissing block iisiiig Equation ( 3 . 4 ) . are determinecl by minimizing the
total border error ,ai\-en by Equation ( 3 . 2 ) . Since the total border error is a quaciratic
fiinction of the ahove mentioned weights. the weights are obtained hy solving a set of
linear ecl~iat ions. Likewise. if Eqiiation (3 .6) is iised as t h e estimator. the total borcler
error is a quaciratic function of the iinknown parameters q - 1 . m ~ l . tcgl. n - 1 ~ 1 . W L I .
m L I . t rRl ancl m R I . Therefore. the values of the latter paranieters can also be obtained
hy solving a set of linear ecluations. Thiis. the associated computational complesity
of oiir niethod is eqiial to that required for solving a set of S linear eqiiations with 8
~inkriowns. besicles the miiltiplications and acldi t ions requiretl to cornpute the equation
coeffcients from t h e available da ta and the ones necessary in Ecpations (i3.4) or (3.6).
Table 3.1: Computatiorial cornplesitj. of various reconstruction rnethods.
1 Met Iiod I Xiirnber of additions I Sumber of multirîlications I
Method of [6'2] ' 1
4069 7-10 1920
Slet hod of [65] , Pronosed method
82-36 108'2 1 720
h-e nest compare the computational cornplexit- i.e. t he reqiiired niiniber of
actdit ions and multiplications. of our met hod wit h o t h e r methods reviewed in C h a p t e r
2. Generally. ecige-based met tiods are computat ional ly intensive since t tiey reqriire
estimation of edge directions. Some other methods like [66] are compi i t a t iona l l~ very
simple but their performance levels are worse than t h a t of otir methoci. Therefore.
we nes t compare oiir method wit h only t hose proposed in [ 6 ï ] . [65] and (621. T h e
nrrnlber of additions and multiplications requireci for restoring 64 pixel values a r e $60
and 960 respcctively using the niethod proposecl in [67] and 17'20 and 1920 respectively
i~s ing oiir met hod. The method proposed in [62] requires S2-56 additions a n d -1069
miiltipIications t o restore the 6-1 missing pixel values in the spatial domain. T h e
method proposecl in [65] is unable t o restore al1 of t h e 64 coefficient in t h e DCT
clornain. I t restores u p to '28 DCT coefficients. The missing pisel values a re ohtained
iising thc inverse DCT- The reqiiirecl number of addit ions ancl miiltiplications for tliis
mettiocl assuming a fast inverse DCT a l g x i t h m a re 1082 and 7-10. respectively. Table
:3.1 siimmarizes t tie cornparison results. T h e cornputat ional demand of oiir met hod is
larger than t hat of [67] and (6.51 and less t h a n tha t of [62] and edge-based methods.
However. orir methocl can reconstruct image details 1%-hich the methocl presented in
[65] is unable t o recover. and it can reconstriict diagonal edges whicli t h e nietliod
presented in [67] carinot restore. This. in most applications. justifies the additional
conipri ta t ional complesity.
3.4 A Statistical Method for Reconstruction of Miss-
ing Texture Information
The s ta t istical m e t h o d t h a t ive propose for t h e reconstruction of missing t ex tu re infor-
mation ernploys a h l a x i m u m A Posteriori (.\lt\P) est imator . \Ve consider a Slarkov
Random Field ( M R F ) ~ v i t h a n eight-pixel clique a round each pisel as t h e image a
priori niodel (see t h e riiarkeed pixels in Figure 3.8) [10 11. The reason for seiecting a n
eiglit-pixel clique in t h e manne r shown in Figure 3.8 will becorne clear later. Tlie
potential fiinction of Eqiiation (2.6) can be wri t ten as
Figure 3.S: .A pixel. i t s clique c a n d t h e eight directions. Tlic complcment of t h e cliqiic. cf is t h e d a r k arca.
irherc c is t h e cliciiie a n d D(siSj. xbSl) is t h e difference betiveen the i-aliies of t h e pisel
in position (i.j) a n d t h e pisel in i t s clique nt position (k.1). and ir;.,-k,l is t h e weight
assignecl t a t his difference.
Combining the SIAP estimator of Equation (2.4) with Equations (2.5) and
( S . 7 ) . t h e restoration of missing data results in the following minimization problem
ivliere .M is the set of a11 missing pixels in the frame. For the case p ( r ) = x2 (Gauss-
lIarko\- Random Field (GMR F ) moclel). the above rninimizat ion probiem xields a
tiniquc global solution. since p ( s ) is a conves function. In fact. the estimatcd value
of a pisel is gii-en by
~vherc c is t h e clique and c' is its complement shown in Figure 3.S. If al1 the weights
are selected equal to each ottier. the above ecpation is simplified to
where r t , ancl r1,l are the numbers of pixels in the clique and its complement. respec-
tii-el'- (each S for the shon-n cliclue). In other words. the estimated value of a pixel is
t h e average of the pisels in its clique and the clique's complement (sec Figure 3.8).
Tlie i-alties of missing piscls in a block can be estimated iising Equation (3.9) itera-
t ively. This is in fact the application of t lie iterative conditiorial mode (tC.\i) mcthod
proposecl in (1021. In the ICSI met hod. the .\L.-\P problem is not solvecl but only ap-
prosirnateci. Depending on t lie sclected neighborhood systern pixels are divided into
groupssçtich that those in one groiip are mutually independent ancl the a posteriori
of eacli group is masimized. This perniits fast convergence and parallel synchronoiis
implementat ion. Since the fiinction p ( . ) is conves. the iterative met hod will converge
to the glo1)al minimum.
The cost function. p ( x ) . may also be selected as t h e Huber function given hy
wlierc is a threshold. T h e image model using th i s cost function is called a H u b e r
Jlarkov Random Field ( HMRF) model - For th i s case it is difficult to ob t a in a closed
forni espression for the es t imated value of a missing pixel [10:3]. T h e solution t o Ecliia-
t ion 3 3 . however. can be obtained by n ieans of t h e ICll methocl. T h e performance
of t lie JI.AP e s t i ~ n a t o r hased on H l i R F mode l can be described as follo\vs. \Vhen t Iie
valtics of t h e pisels i n t he cliclue a r e close to each other. t h e missing pixel is se t to
t h e average of t hose pixels. When t h e pixel vaIues a r e not similar. a voting proceclure
is performed ancl t he est iniated value is se lected such t h it is close t o t h e \-aliie of
tlic major i ty of the neighboring pixels (a m e d i a n Iike performance). T h i s behavior
prcvcnts t lie appearance of pixel values t h a t a r e different fronl t heir neighbors. which
in t iirri linlits t h e performance of t h e e s t i m a t o r in t h e reconstruction of edges. Be-
sides. HSI RF hased reconst riiction algorit hms a r e computat ionalIy corn ples anci t heir
suhoptirna1 \-ersion. median filtering. does no t yield unique soltitions for t h e missing
\ali ies [!O-11. -4 very simple c s a m p l e dep i c t ed in Figure 3.9 reveals t h e perfornianre
of a li:\P cçt imator using t h e GMRF a n d HNRF models. The value of the cente r
pixel . r , , is missing. :Issume tha t t h e pisel values a r e p and q a s shown in t h e figure.
p << q . a n d there is a \-ertical line in t h e image . Using t h e G M R F model. t h e value
of t h c missing pixel is
Figure 3.9: A missing pixel in
L-sing t hc H.\IRF moclel for t his specific e sample .
Ob\-iotisly. both of t h e es t imated values a r c
a vertical line.
we u-il1 have
(3.12)
close t o the rnajority of t h e neigh-
horing pisels. Thiis. none of the above mentionecl models (i.c.. CiAlRF a n d H M R F )
is ab le t o de tec t t h e presence of t h e vertical edge a n d reconstruct t h e missing pixel
\-altic based o n t ha t edge. L-siial1~-. rel-ing only o n t h e local image characteristics in
t h e reconstruction procediire causes s o m e of t h e image at t r ibutes t o be ignored o r
niisintcrpreted [lO5]. Here. we aclopt a GMRF wi th an eight pisel clique (shown in
Figure i3.8) as t h e image n-priori model . Howex-er. t h e weight corresponding t o t h e
cliffcrence between a pixel ancl each of t h e pixels in i ts clique (u . , , - k .~ in Equat ion
(3.9)) is selected aclaptivell-. hased o n t h e likelihood of a n edge in t h e direction of t h e
siiI>jcct pair of pisels. T h e rationale behind th i s selection is t o give more weight t o
t h e ciifference between t h e pixels in t h a t direction ivhich will cause t h e 1-dues of t h e
pixels in t h a t direction t o get closer to eacli o ther . T h e likelihoocl of eclges in each of
t h e ciglit clirections is cornputed using blocks a round t h e missing block. In th i s way.
t h e a\-aiiable information in a larger a r e a is esp lo i ted ivithout increasing t h e o rde r of
t lie G.\I R F model (which increases dramaticaI1y t h e computational complesi t y ) . To
determine the likelihood of edges in each of the eight directions. edges in the blocks
siirroiinding the missing block whose directions impIy t hat t hey pass t hrough the
rnissing block are determineci using
for CL-ery pisel in the blocks to the left. right. top and bottom of the rnissing block.
The niagnit ude and angiilar direction of t lie edge at pixel (i-j) are
Y, O = arctan(-). gr
tvherc 0 cletcrmines if the edge nt pixel (i.j) passes through the missing biock- Since
t here are eight pisels in the clique. the value of 19 is roiincled to one of the eight direc-
t ions eclually spaced in the range from O" to lSQO. There is a counter cm, ( m = l .?.....S)
for cach of t hc eight directions. If the cstcnsion of an eclge at pixel (i.j ) belonging to
one of the neighboring blocks passes through the missing area. the counter for that
particular direction is incrernentecl by the arnount of G . This procediire is repeated
for al1 the pixels in the blocks to the top. bottom. left and right of the rnissing block
( i f applicable). To remove the spurious edges resulting from the type of the edge
cletcct or eniployed. the values of the counters are t lires holded.
Thrre are eiglit pixels in the clique of each pixel and eight clircctions for the
cletccted edges. Each pisel in the clique of a pixel corresponds to a direction. In our
proposecl method. the value for ZL';,,~;,J is selected baçed on the eclgc coiinter of the
direction corresponding to (i-j) and (k.1). i-e..
\\-here 3 is a constant and cm is the corrntcr corresponding to direct ion m. and direct ion
rn corresponds to the direction formed b ~ - (i.j) and (k.1). Tlius. oiir statistical error
conceal ment. met hod can be summarized as follows:
1. Determine the edges in the neighboring blocks and assign eacli to one of eight
eclually spaced directions. C'ompiite the countcr for each direction,
2. L-se Equation (3.1:3) to find a set of weights for each missing block.
3. Use Equation (13.9) to obtain an estiniate of each missing piscl emplo~-ing the
weights obtained in the previous step. and
4. 1 teratively re-estimate the missing pixels using Equation (3.9) until convergence.
In t lie case where adjacent blocks are lost . the reconstruction algorithni is appliecl re-
cursi velj.. Blocks wi t h t lie masimum nurnber of correctly decodecl neighboring blocks
are reconst riicteci first. and the rest of blocks are reconst ructed recilrsivcly. This giiar-
ant ces t hat the best possible estimation accuracy. when missing blocks are rest ored
one by one. is achieved. Sote that. since the cost function is conves. convergence is
g~iaranteed.
For intra coded blocks, our >I.AP estimation method is ernployed d o n g with
the da ta of neiglihoring blocks. For inter coded blocks. prediction error signais of
the neighhoring blocks are used together with the JlXP estimator (or the method
proposecl in Section 4-13). to find an estimate of the prediction error of t h e niissing
block. The estirnatecl value is then added to the initial estimate (as shown in Figure
:3. '3) . Since the predict ion error signal consists most ly of high frequency components.
the second stage wil l improve the video reconstruction quality. especiallv around
ed ges .
Tlic niimbers provided beiow for t h e computational cornplexity correspond to blocks
and sitb-blocks of sizes 16 x 16 and 8 x S. respectively. The cornpiitational load of
t l i ~ proposed M'IP estimator consists of t hose compiitat ions recliiired in aclapt ing
the weights of the C i l l R F niodel. which are approsimately lS-100 operations for a
rnissing block. assuming four available neighboring biocks. The estimation of missing
pisels using Equation (3.9) is an iterative procediire which. for each iteration. requircs
approsimateIy 130 operations. O n average. 80 itcrations are recpired for the algorithm
to con\.erge for a bIock.
3.5 Initial Texture Estimate of a Missing Inter Coded
Block
The purpose of this stage of oiir mcthod is to obtain an initial estimate of the
missing texture of an inter cocled block iising information [rom the prcvioiis frame. If
t lie motion vector of the rnissing block is receivcd correct ly (e-g.. because theéncoded
texture data has been separated from the motion da ta via partitioning). then the
estiniate is set to the motion compensation predicted block. N'hen the motion vector
Figure 13-10: (a) The four siib-blocks. ( h ) their corresponding sub-blocks in the pre- vious franie ancl the blocks connected to them.
is not available. tve do not enlploy ttic motion vectors of adjacerit blocks to estimate
the missing motion vector due to the reasons mentioned in Chapter 2. instead. we
find an estimate of the missing block such that image continuit'. inside the block
and across its boundaries is preservcd [106]. To do this. sub-blocks adjacent to the
niissing block. i-e.. ul . LI?. 11 and 12 in Figure XlOa. are consiciered. First. for each
of t hese siib-blocks. a corresponding sub- block in the previous frame is deterinineci.
T h corresponding sub-Mock is found by searctiing a srnail area arountl the point
corresponding to the center of each of the sub-blocks u l . u2. I L . I l in the previous
franie, The siim of absolutc differences is iised as the measure of similaritj-. The four
siib-blocks ui. u 2 . Il. 12 ancl their corresponding sub-blocks u l f . trl ' . I l f . 12' in the
prek-ious frarne are shown in Figures :3.10a and 3.10b. For example. t l 1' is t Iie sub-
l~lock corresponding to ul. Then. four blocks. namelj- XI. X2. X3. and X4. which
are connected to ul' . n2'. Il'. and I2' (respectively). are cietermined. To obtain an
initial estirnate of the missing block that smoothly connects to the rest of t h e image.
a block from the above four blocks that minimizes the squared sums of border errors.
between the estimatecl block and its adjacent above and left blocks. is selected. Thus
2 x = nrg min c . x 1 .X2 .X3 .X4
where
Each of the border errors c~ and EL is clefined in terms of pixels bl-
where the vector p ~ b consists of the bottom line pixels of the block to the top of
the missing block and p ~ , consists of the right column pixels of the block to the left
of niissing block. The vectors Xt and xl are those elements of the estimated block
X that correspond to the pixels in its top row and left column. respectively. If the
missing block belongs to the upper-most row of the frame. only the left border error
is considered in Eqtiation (3.15). Sirnilarly. for missiiig blocks lying in the Ieft-most
colunin of t hc frame. only the tipper border error is considered in Equat ion (3.15). If
the first block of a frame is missing. the initial estimate is set t o the corresponding
block in the previous franie.
The estimation of motion of the sub-blocks has a high computational overhead
whicli can possi bly int roduce unacceptable rcyuirements on decoder. The corn puta-
tional 01-ertiead can be reduced if the search for displacement of each of the sub-blocks
is restricted to a set of candidate motion vectors. This is a decoder option and can be
used to t rade performance against corn put at ional complexity [ l o i ] . The set consists
of the motion vector of the block corresponding to the missing block in the previoiis
franie. the motion vectors of a\-ailable neighboring biocks. the median of the motion
vectors of available neighboring blocks. the average of the motion vectors of a\-ailable
ncighboring hlocks. and the zero motion vector [LOS].
3.5.1 Computationd Complexity
Tlie cornput at ional load of t his stage of the proposed met hoc1 consists of t hose com-
piitatiotis reqiiired for L ) estimating tlie displacement of each of the sub-hlocks. and
2 ) computing the error of Ecluation (3.14). For motion estimation. we tised a spirai
search method using an area of 16 x 16. This requires approsimatcly 197000 opera-
tions. If t he searcii for the displacement of each of t h e stib-blocks is restricted to the
set of candidate motion vectors as i ras esplained. the required niimber of operation
will reducc to 6100. The number of operations required t o finci t h e total error of
Eqtration (3.1-L) for four b1ocks is approsinlately :380.
C'learly. t he computational load of oirr error concealment methotl is quite rea-
sonable. Our simulation esperiments confirm that the run time of otir method is
indeed. acceptable. In fact . real-t inie decoding (e-g.. 10 frames/sec for QCIF vicleo
seqiienccs) is still possible or1 a Pentium 300 l I H z PC.
3.6 Summary
In t his chapter. reconstruction of missing texture i r i transmission ~f coded visual data
~i-as investigated. A two stage method waç introduced for concealment of errors in
76
int ra codecl tes t lire. In the first stage. if applicable. effects of loss of coded data of a
block on the rest of the blocks of the frame is removed. This is a key contribution of
the proposed reconstruction argori t hm since. unlike previously publishcd algorit hms.
it considers the depenclencj- between coded bIocks. In the second stage. the missing
textures corresponding to t h e missing blocks are reconstructed. W e proposed a deter-
niinistic and a statistical method for the restoration of missing testure information.
The cleterministic methocl achieves a good perfor~nance level in the reconstruction of
ectges. The stat istical met hoc1 uses an aclaptive IIRF as the image a-priori model. The
adaptation enables the estimation procedure to incorporate more information without
increasing the order of A I RF. The met hod proposed for concealment of errors in intcr
coded testure information also consists of two stages. First the information in the
acljaccnt frame is used to find an initial estimate of the missing testure information.
Sest . the spatial error concealrnent method is applied to the preclict ion error of the
biocks adjacent to the missing blocks in order to find an estiniate of the prediction
crror of the niissing block, The result is then combined with the initial estimate of
the rnissing block to find its best approsimation.
Chapter 4
Error Concealment: Shape
Informat ion
To r a i s t n e w q u t s t i o n s . ne w poss ib i l i t i r s . lo regard o ld p r o b l ~ m s fronz rr
n e t r r r n g l ~ r e q u i r e . ~ c r e a t i m imcrginatzon and m a r k s rval rrdccrnces i n x i -
EIICE.
-4 lbe rt E i n s t e i n
4.1 Introduction
A s was nientioned in Chapter 2. the new generation of conipression standards (e-g..
11PEC;-4) support an object-based representatioii of vicleo bj- allowing t lie coding
of t h e shape information of arbitrarily shaped video objects along with the objects'
testiire and motion information. The concealment of errors in t h e texture inforrna-
tion was disciissecl in the previous chapter. In this chapter. ive present a n efficient
concealment method for errors in the shape information. This suhject has. so far.
not been addressed in t h e Iiterature. .As in the previous chapter. ive discuss error
concealment methods for intra coded and inter coded shape data. For intra coded
s h a p e . the spatially adjacent information is iised to estimate the missing sliape in-
formation. =ln adaptive llarkov Ranciom Field ( J IRF) . which is designed for binarj-
shape information. is proposcd as the image a-prion' modcl. The proposecl image
mode1 is used along with a 1l.AP estimator t o recover the missing sliape informa-
tion. For inter coded s h a p e information a reconstruction methoci simiIar to t he one
that \vas proposed in the previous chapter is devised. It should be noted that con-
ceaIment of errors in inter coded shape data is more critical tlian that of the inter
coded one. simpIx because the intra coded shapes are used for prediction and thiis
t h e errors in them propagate to the follou-ing inter coded shapes. Sections 4.2 and
4.:3 present the proposed error concealment niet hods for intra coded anci inter coded
s h a p t information. respectivel~.
4.2 Concealment of Errors in Intra Coded Shape
Informat ion
For reconstruction of missing intra coded sliape information. ive e m p l o ~ a statistical
mct hocl. As esplainecl in Chapter 2. statistical met hods for error concealment assume
t hat the pixel values in an image o r video signal are realizations of an underlying
randoni process. The I IXP estimation can t hen be employed to yield the most likely
image given the obserx-ed image data and the image mode1 [Z. 104]. Here. ive use a
Il-AP estiniator for restoring missing shape information. Since the shape information
is biriary. ive propose an appropriate form of hlarkov Random Field (SIRF) as the
image niodel.
As shown in C'hapters 2 and 3. the N=\P estimation of missing data in an
image assiirning AIRF as the image a-priori mociel and t h e conditiond probabilit-
,ai yen h ~ - Eqiiation 2.3 can be espressed bu the following rninimization problem
x = min C \,,.-(z,, j. 1: .J i . j € M
wlicre .M is the set of al1 missing pisels in the image and V is the potential t'unction.
Tlic potential fiinction characterizes the relationship between a group of pixels bl-
assigning larger costs to configurations of pisels which are less likeiy to occiir. The
choice of the potential function is crucial to the performance of the image modeI.
('oriinionly. the potential functions are selected to be of the form
ivhere c is a clique. p is a fiinction called the cost function. and rc;,j,k.l is the weight
assignecl to the difference between the pixel ~ a l u e s ri, and [W]. Tlie cost fiinction
p ( . ) . i n fact. encourages the pixels that a re spatially close to each other to have the
sarnc ïaliin '. The shape information is binary and r i , can only assume one of two
valucs. Therefore. WC select the cost function with the following form
! Refer to Cliapters 2 and 3 for a complete discussion of MRF
80
wherc 3 is a positive constant .
Figure -1.1: =\ pixel. i ts cliclue c and t h e eight directions. T h e complement of t h e clique. cf is t he d a r k area..
For t h e clique. ive aclopt an eight pixel neighborhood shown in Figure -1. 1. The
iveight corresponding t o t h e difference between a pixel a n d o n e of t h e pisels in i ts
clique IL-,.,-^., in Equat ion (4.1 ) ) is selected adaptively. based on t h e likelihood of a n
edge in t he direction of t h e subject pair of pixels, The rationale behinci this selection
is to dive more weight t o t h e difference betwecn t h e piseIs in a direction which n-il1
cause t h e r a l i i n of t h e pixels in tha t direction t o b e t h e same. Assuniing the values
rc;,,k.l t o hc integers. t h e estimatecl value of a pixel given t h e values of t he pixels
aroiintl it ( in t h e clique a n d i t s complement) , will be
wliere cf is t h e cornpleinent of t h e clique shown in Figure 4.1. Each of the t e rms
ri.^ - Ji+ i . j+m ) in t h e right hand s ide of Equatiori (-1.2) will contr ibute L? to t h e
whole sumrnation i f t h e e s t ima ted value of t h e missing pisel ( I i , ) i s clifferent from
t h e corresponding neighboring pixel a n d will cont r ibu te zero t o t h e whole
sumrnat ion if t h e estimateci valiie of t h e missing pisei is equal t o the value of t h e
corresponding neighboring pisel. Therefore. t o minimizc t h e \-alcie of the right hand
s ide of Equation (4.2). t h e number of terrns with a n e s t ima ted \ d u e that is different
from tha t of its neighboring pisel. should b e minimizeci. Thus. t h e estimatecl value
shoitld be equal t o t h e median of t h e following vector
T h e likelihood of edges in each of t h e eight directions (shown in Figure 4.1)
is cornputeci using blocks a round t h e missing block. In tliis way. t h e available s h a p e
information in a larger a r ea is exploited in t h e concealment proccss. To assess t h e
likelihood of edges in each of t h e eight directions. edges in t h e blocks siirrounding t h e
missing block. whose direct ions imply t h a t they pass th rough t h e rnissing block. a r e
clet erniined-
Since t h e edge information of t h e s h a p e d a t a is embeddcd in its borders. ive
first scpara te t he borders of t h e shape d a t a in t h e adjacent blocks. To d o this. ive
use a niorphological t ransforni caIled t h e boundary t ransform [16]. If a11 the four
neighhoring pixels (above. bellow. left and right ) of a pisel a r e insiclc the shape. tlien
t h e pixel is declared inside t h e shape. O t her~vise. t h e pisel belongs to the border of t h e
shape. After finding t h c boundary of t h e shape using t h e above describecl transform.
a 3 x 3 window is cen te red at each border pisel and t h e angle of the best line-fit t o
t h e border pixels in t h e window is computed. T h i s in fact gives t h e direction of t h e
cdgc a t t h e pixel in t h e cen t e r of the window. Figure 4.2 shows a typical window a n d
Figure 4.2: X :3 x 3 pisel window. the border pixels in it (shaded) and the best Iine-fit.
the best line-fit and the angle of tlie line. There are eight counters corresponding
to eight directions shown in Figure 4.1. The counter corresponding to tlie direction
of the detected edge (best line-fit ) is incremented if the extension of the best line fit
passes through the missing block. The procedure is repeated for al1 the border pisels
in the blocks to t h e right. left. below and above the missing block ( i f applicable). The
weights required in Equation (4.3) are then obtained by
where cm is the counter corresponding to direction m. and the direction m corre-
sponds t o the direction formed bj- (i: j ) and (k: 1). Finally: the proposed shapc error
concea lme~t met hod can be sumrnarized as follows:
1. Determine the edges in the neighboring blocks using the border transforrn and
the best-Iine-fit and assign t hem to eight equally spaced direct ions. Cornpute
the corresponding counter for each direction.
2. Use Equation (4.4) to find a set of weights for each missing block,
3. Ese Equation (4.:3) to obtain an estimate of each missing pixel employing the
weights obtained in the previous step. and
4. Iteratil-el- re-estimate the missing pixels using Equation (-1.3) until convergence.
In the case where adjacent blocks are lost. the reconstruction algorithm is applied re-
cursii-ely. Blocks with the rnaxim~rn number of correctly decoded neighboring blocks
are reconst.ructed first. and the rest of the blocks are reconstructed recurciveIy. This
ga ran t ee s that the best possible estimation accurac- when missing blocks are re-
stored one by one. is achieved.
The computational load of the proposed method consists of those computations
required in finding the boundary pixeIs and the slope of the best line-fit for each of
t h e boundary pixels and estimation of missing pixels using Equation (4.3). The first
two i-alues depend on the complexity of the shape in the blocks adjacent t o a missing
block. The estimation of missing pixels using Equation (4.3) is an iterative procedure
rvhich. for each iteration. requires counting the number of 1's and 0's which can
he performed very fast. On average. 6 iterat,ions are required for the algorithm to
converge for a bIock. Clearly, the computational load of our error concealment method
is quite reasonable. Our simulation experiments confirm this fact.
4.3 Concealment of Errors in Inter Coded Shape
Informat ion
.A simple concealment method for inter coded shape da ta is to replace the shape
of the missing blocks with those of blocks corresponding to the same location in the
84
previous frame. .An alternative method is to est imate the missing motion vectors from
the motion vectors of surrounding macroblocks a t the decoder and use the est imated
\-ectors to replace the rnissing shape from the previous frame. ket anot her method
is to perform a motion estimation a t the decoder similar to the method that \vas
proposed in Chapter 3 for texture information. Hoivever. our esperiments showed
t hat for shape information a motion estimation a t the decoder. similar to what is
dorie for texture. is unlikely to improve the qualit. of reconstruction. In fact. the
motion vector of the macroblock above a lost macroblock provides a good estimate of
the missing motion vector and can be employed for motion compensated concealment
of missing shape information. This is because motion ivithin an object is regular
and therefore the motion vectors of available blocks b e h g i n g to an object are good
estimates of the motion of a missing block. This is in agreement with what has been
observed in [log].
4.4 Summary
In this chapter. methods for concealing errors in intra and inter coded shape infor-
mation were iiitroduced. The proposed method for intra coded shape da ta employs
a SIAP estimator along ivith an MRF as the image a priori model. The 41RF is de-
signed for the binary shape information, and its parameters are adapted based on the
information embedded in the neighboring blocks. Motion compensated concealment
is used for inter coded shape data.
Chapter 5
Spatial and Temporal Error
Concealment: Performance
Creat ir i ty incolces breaking out of ~stabl ished patterns i n order lo look al
things in a different c a y .
Edward De Bono
In this chapter. we examine the performance of the error concealment methods de-
veloped in Chapters 3 and 4. In Section 5.1. we evaluate the performance of the
proposed method for reconstruction of intra coded texture in baseline JPEG coded
images. Since there is dependency between coded blocks in baseline JPEG coded
images, both stages shown in Figure :3.1 should be employed to reconstruct the miss-
ing parts of the image. The stages are 1) removing the effects of loss of a block on
t h e rest of the image and 2) the reconstruction of the missing block. For the second
stage. Ive ernploy both the deterministic and the statistical methods and compare
t heir performances. In Section 5.2, we study the performance of the proposed error
concealment method on H.263 coded video sequences. For intra coded blocks. ive use
the statistical method presented in Section 3.4. Concealment of inter coded blocks
consists of two stages. In the first stage. we use t h e information in the previous
frame to obtain initial estimates of the missing blocks using the method discussed
in Section :3..'j. In the second stage. a 1 lAP estimator. which ernploys an adaptive
AIRF as the image a-priori model. is used for refinernent of the initial estimates. In
Section 5.13, we first revieic various scenarios that can occur when decoding an error-
corrupted SIPEG-4 video bit stream. Obviously the methods discussed in Chapter 3
and studied in Sections 5.1 and 3.2 can be employed to reconstruct the missing tes-
turc information in 31PEG-4 coded data. Therefore. we focus on the missing shape
data and show the results of our method for reconstruction of sliape information on
1IPEG-4 \.ide0 coded data. In each of the cases (JPEG. H.26:3 and IIPEG-4). ive
compare the performance levels of our methods with those of methods proposed in
the literature. It sliould be added that the mettiods developed in Chapters :3 and
4 are applicable to other compression standards (e.g.. 5fPEG-2) as well. The main
reason for considering H.263 and MPEG-4 is the application of these standards in
communication over low bit rate unreliable channels.
5.1 Error concealment of JPEG coded images
N-e here evaluate the reconstruction method presented in Sections 3.2, 3.:3 and 3.4.
Such a method consists of estimation of DC values of the missing blocks, removal of
the stripes and reconstruction of the missing pixel values (employing the deterministic
and the statistical methods). in terms of reproduction quality. It should be added
t Iiat in the case where adjacent blocks are lost. Our proposed reconstruction method
is appiied recursively. start ing wi t h blocks wi t h the maximum number of correctly
decoded neighbors.
Figure 5.1 shows the 51" 512. 8 bits/pixel images LESA and PEPPER coded
and decoded using baseline .JPEG. Figure 5.2 shows the image LESA with 10% rniss-
i r i g blocks and the image PEPPER with 3% missing blocks. The missing blocks are
colored in white so that their positions are clearly shown. To decode the image
data with missing blocks, the DC value of each missing block is estimated using the
method developed in Section :3.2 and the missing blocks are primarily reconstructed
using tlieir estimated DC values. The results are shown in Figure 5-53. The stripes
appearing in the figure are due t o errors in DC value estimation. Figure -5.4 shows the
images of Figure 5.3 after appl-ing the stripe removal algorithm and before the final
reconstruction stage. Cleariy. most of the error effects in estimating the DC values
are rernoved.
('4 'igure 5.1: Original images: (a) Lena; (b) Peppe
i r e 5.2: Images: (a) Lena with 10% missing data; (b) Pepper with 3% missing data. The missing blocks are shown in white to show their positions clearly.
Figure 5 . 3 : Decoded images: (a) Lena with 10% missing data; (b ) Pepper with 3% missing data. The value of each pixel of the missing blocks is replaced with the estimated DC value of that block.
Figure 5.4: 1
(b)
?ictures after removing the stripes: (a) Lena; er.
After removing the stripes, the texture of missing blocks should be recon-
structed. Figure 5.5 shows the images LENA and PEPPER after reconstructing the
data of missing blocks using piecewise constant weights (Equation (~3.4)) . Figure 5.6
shows the same images after reconstructing the data of missing blocks usirig linear
ramp weights (Equation (3.6)). We also applied our statistical method for recon-
struction of rnissing texture data to the image shown in Figure 5-13 (a) . Figure 5.7
sliows the reconstruction result. Sforeover. we compare the results of our method
(baseci o n Equation (3.4)) for reconstructing the data of the missing blocks wit h the
ones proposed in [67] and [62] in terms of visual quality. Figure 5.8 shows the image
LESA after reconstructing the data of missing blocks using the rnethod proposed in
[6ï]. Figure 5.9 shows the reconstruction result using the met hod proposed in [62] for
the image LESA. Comparing Figures 5.5 and 5.9. it is clear that the visual quality
of the results of the method proposed in [62] is very close to Our method. However.
our met hod perforrns better in the restoration of diagonal edges. and this can be
seen in the restored blocks around the edge of the hat. .As was discussed in Section
3.13.1. t h e cornputational complexity of our method is also less than that of [62]. Fi-
nally. Figure 5.10 shows the reconstruction result obtained using diagonally adjacent
I3locks (besides the horizontal and vertical adjacent blocks). The improvement in the
reconstruction of diagonal edges is small, and it comes a t a cost in computational
com plexi ty.
Figure 5-11 shows a magnified picture of an area of Figure 5.5a. Figure 5.12
shows a magnified picture of the same area of Figure 5.1 1 for Figure 5.6. Figures
5-14. 5.1.5 and 5.16 show magnified pictures of the same area of Figures 5.1 1 and 5.12
for Figures 5.S. .5.9 and .i.lO. respectivel. By comparing Figures 5-14 and 3-15 witli
Figures -5.11 and 5-12. it is clear that our met hod (wit h both piece-wise constant and
linear ramp weights) improves the quality of the reconstruction of edges. lloreover.
comparing Figures 5.11 and 5-12. it becomes clear that using weights that var? as
liriear ramps reduces t h e blockiness of the output image but smoot hes the edges.
Figure 5.5: Reconst ructed images after removing the stripes and restoration of missing hlocks using the method based on Equation (3.4) (piecewise constant rveights).
Figure 5.6: Reconstructed images after removing the stripes and restoration of missing blocks using the method based on Equation (3.6) (linear ramp weights).
Figure 5.7: Reconstructed image using Our proposed statist ical met hod.
Figure 5.8: Reconstructed images using the method proposed in 1671.
Figure 5.9: Reconstructed image using the method proposed in [62].
Figure 5.10: Reconstructed image when diagonally adjacent blocks are used in addi- tion to horizontally and verticallÿ adjacent blocks.
Figure 5.1 1: Slagnified picture of part of the image LENA shown in Figure 5.5.
Figure 5-12 Magnified picture of the same area shown in Figure 5.1 l for the image LESA of Figure 5.6.
Figure 5.13: Magnified picture of the same area shown in Figure 5.1 1 for the image LESA of Figure 5.7.
Figure 5.14: Magnified picture of the same area shown in Figure 5.11 for the image LESA of Figiire 5.8.
Figure 5.1.5: Magnified picture of the same area shown in Figure 5.1 1 for the image LESX of Figure 5.9.
Figiire .5.16: Magnified picture of the same area shown in Figure 5.1 1 for the image LEKA of Figure 5.10.
Table -3.1: PSXR results (in dB) for the 512 x 512 LENA and Pepper images. The nurnbers are for the images after removing the stripes and reconstructing the missing data.
Table 5.1 prorides the PSKRs for the images LEM and PEPPER obtained using
our mcthod (based on Equation ( 3 . 4 ) ) . as well as the PSNRs of the correctly JPEG
decoded images. for several different block loss rates. Figures 5.5 and 5.6 and Table
5.1 demonstrate tliat our method performs quite ~atisfactorily~ bot h subjectil-ely and
objectively.
Moreover. w e compare the results of our deterministic method (based on Equa-
tions (13.4) and (3.6)) and Our statistical rnethod for reconstructing the data of the
missing blocks with the ones proposed in [67]. [ T T ] and [6'2] in terms of PSSR. Table
5.2 compares the PSSR values for t h e 512 x 512 LEKA image with about 10% rnissing
blocks for various restoration methods. The numbers are for the image after remov-
irig t h e stripes and reconstructing the missing data. As can be seen from the Table
the mettiod proposed in [77] slightly outperforms our deterministic rnethod in terms
of PSSR. Hoivever. our deterministic method compares favorably to that of [ii] in
terms of computational compiexity.
Loss Percentage LES'.-\
PEPPERS
0% 35.7 :34,7
1% 29.3 '25.9
3% 28.6 '23.4
10% 37.3 '22.0
Table 5.2: PSXR results (in dB) for t h e 512 x 512 LEXA image for various recon- struction methods. The numbers are for the image after removing the stripes and reconstruct ing the missing data.
1 Met hod 1 PSNR (dB) 1 Proposed in [62] Proposed in (671 Proposed in [77j
Using diagonally adjacent blocks
. ,
26.50 26.64 27.63 26. 13
Our proposed based on Equation 3.4 Our proposed based on Equation 3.6
Our proposed statistical method
'27.46 1 '26.10 27-85
5.2 Error concealment of H.263 coded video se-
quences
To study the performance of the proposed methods in H.263 coded video sequences.
QCIF (176 x 144 pixels/frame) vide0 sequences a t a temporal resolution of 10 frames
per second are coded at 64 kbps and then decoded in the presence of slice/GOB errors.
The size of the blocks is 16 x 16. that of the sub-block is 8 x S.
AS explained in Chapter 2. t o make the compresseci data more error resilient.
most of the standard-compliant video compression systems partit ion a video frame
into Groups of Blocks (GOBs) or **slices-. which are coded independentl) Tlierefore.
the output bit stream usually consists of segments separated with markers. where
each segment corresponds to the coded data of the blocks in a GOB or a slice. When
channel errors occur. the decoder usually discards the erroneous data between two
markers surrounding the erroneous data. effectively discarding the GOB or the slice.
Then. loss of data of a slice does not affect the rest of the compressed video sequence.
Here. it is assumed t hat the frames of the coded video sequence are partitioned into
GOBs or slices. and thus. the missing da ta belongs to blocks of a GOB or a slice.
lloreover. it is assumed t hat the decoder knows the locations of the missing blocks.
This information (e.g.. the checksurn information) can be obtained from the network
or it can be inferred. for esample. by detecting the semantic or syntactic violations
as a result of errors [59 1 101.
To simulate the channel errors. the following tasks are performed. Coded
video information is first grouped into packets, where each video packet consists of
the coded data of a GOB or a slice. The video packets are then multiplexed with
audio information according to the H.223 standard. We used a n H.223 multiplexing
simulator which receives video packets. simulates audio t raffic. applies errors to the
multiplexed bit s tream accordinp to an error pattern stored in a file. and outputs the
packets [ I l l ] . The error pattern that ive ernployed corresponds t o a wireless channel
[1 1 y. Burst errors rc-il1 most likely not corrupt two consecutive video packets. since
audio packets are inserted between thern. The erroneous bit stream is decoded such
tliat the effect of errors will appear as missing slices/COBs.
For inter coded blocks. our error concealment consists of 1) obtaining an ini-
tial estimate of the missing block using information from the previous frarne and 2 )
estimate the prediction error using the adaptive MRF model. We compare the per-
formance of the above method with two other methods: 1) a replacement method.
ivhere a missing block is replaced wit h the same block in the previous frarne. and 2)
a median method. where the motion vector of a missing block is set to the median of
motion vectors of blocks to the left. above and above-right of it. and the estimated
motion vector iç used t o obtain a motion compensation block which serves as the
concealment of the missing block. Here. we consider only the case of missing GOBs
(i.e, it is assumed tha t each packet contains a GOB).
Shown in Figure 5-17 are the concealment PSNR values for the three methods
rnentioned above for the video sequence FOREMAK wit h a 10% packet (GOB) loss rate.
-4s can be seen, for video sequences with a large amount of motion like FORESIAS.
the performance of the proposed method and the median method are close to each
other and both are better than the replacement method. Although the performance
Figure 5-17: PSNR values for image sequence FOREMA'; with 10% GOB missing for different concealment methods.
of the above methods is comparable in terms of PSXR. the video sequences obtained
using the proposed met hod are more visually appealing t han the ones obtained using
t h e other two methods. Figures .j.lS(a). J.lS(b). and J.lS(c) show the reconstructed
video franle (inter coded) of the video sequence FOREMAN. As can be seen in the
images. the replacement method does not generate good results. especially around
the nose area. Moreover. the concealment result of the proposed method is clearly
Letter than that of the other two methods.
For intra codcd blocks, ive compare the performance of our method (detailed
in Section 3.4) with that of four other efficient methods 1) a MAP estimator using
C;MRF as the a-priori mode1 where each missing pisel is basically set to the average
of the pixels around it [El. 2) a suboptimal version of the M A P estimator using the
Figure 5-18: An inter coded frame of the sequence FOREMAN (frame 22) concealed by the (a) replacement, (b) median, and (c ) proposed methods.
HJIRF model where a missing pixel is set to the median of pixels around it [104].
3 ) a deterministic method proposed in [6i] where each pixel in a daniaged block is
interpolated from the corresponding pixels in its four neighboring blocks such that
the total squared border error is minimized. and 1) a deterministic met hod proposed
in [62] where the missing data is restored such that the variation between adjacent
pixels wi t hin the damaged block and t heir spat ially neighboring pixels in adjacent
blocks is minimized. Since for H.263 coded video. the size of the missing blocks are
comparable to the size of the image. the deterrninistic method proposed in Chapter
13 [vil1 not perform acceptabIe. This is mainly because the pixels in the area around
the missing block will be weakly correlated to the pixels in the damaged region.
In this experiment. we employ different video sequences and different packe-
tization schemes. In the first set of simulation experiments. we assign a slice which
consists of one block. to a packet. Figure 5.19(a) shows a frame of the image se-
quence FOREhl,\Y encoded and decoded iising an H.26:3 compliant coder. Figures
.S. 19(b). 5.19(c). .5.19(d). 5.19(e). 5.19(f). and 5.19(g) show (respectively) the same
frame 1) missing approximately '20% of the packets (blocks), 2) reconstructed us-
ing the non-adaptive GXIRF model. 3) reconstructed using the suboptimal H&LRF
model. 1) reconstructed using the met hod proposed in [67]. 5) reconstructed using
the method proposed in [62]. and 6) reconstructed using our proposed adaptiw hlRF
error concealment met hod. By comparing t h e figures. the superior performance of
our method becomes obvious. This performance advantage is demonstrated in the
areas of the frarne that correspond to adjacent missing blocks. Moreover, blocks that
contain edges are reconstructed very well by our method.
Figure 5.19: -4 frame from the video sequence FOREMAN, (a) original, (b) missing blocks. reconstructed iising (c) a GMRF model, (d) a suboptimal HMRF model. (e) t h e rnethod proposed in [67], ( f ) the method proposed in [62]? and (g) our adaptive JIRF model.
Table 5-13: PSXR comparison of different met hods for the video sequence FORELIAS.
GOB missing 25.3
Met hod G 391 RF H 34 RF 3lethod proposed in [67] Method proposed in [62] Adaptive MRF
In the next set of simulation experiments. we assign the coded da ta of a GOB.
which consists of 11 blocks. to each of the video packets. Figure a.ZO(a) shows a frame
of t h e \-ide0 sequence F O R E M A N wit h two missing packets ( GOBs). approsimatel-
ISR loss rate. Figures 5.2O(b) and .j.2O(c) show the error concealment result obtained
iising the GhlRF model and suboptimal HMRF models. respectively. Figures 5.20(d)
and j.'lO(e) show the result obtained using the rnethods proposed in [65] and [62].
respectively. Figure .52O(f) shows the result obtained using our adaptive SIRF model.
Clearl~.. our proposed met hod periorms best in reconstruction quality. part icularly in
retrieving the edges.
For a quantitative evaluation, TabIe 5.3 provides the PSKR values of the above
concealment met hods (for bot h packetization cases) for the video sequence FOREXI..\N.
The table demonstrates that our method outperforrns the other methods by at Ieast
2 dB. The concealment results for another video sequence (AKIYO) with different loss
pattern. are also provided in Figures 5-21, and 5.22. The PSNR values are quoted
in Table 5.4. It is clear from the shown reconstructed video frames and the PSXR
values t hat the proposed met hod has a clear performance advantage in concealing the
effects of errors.
No loss 32.1 32.1 32.1 32.1 3'2.1
20% Block loss 26.3 25.4 23.2 2.5-2 28.6
(4 ( f )
Figure 5.20: -A frame from the video sequence FOREMAN (a) with missing GO&. reconstructed using (b) a GMRF model. (c) a suboptimal HMRF model. (d) t h e method proposed in [67], ( e ) the method proposed in [62]! and ( f ) our adaptive XIRF rnodel.
Figure 0.21: A frarne from the video sequence AKIYO (a) original (b) with missing blocks. reconstructed using ( c ) a GMRF model, (d) a suboptimal HMRF model. (e) the method proposed in [67], ( f ) the method proposed in [62] , and (g) our adaptive AlRF model.
Figure 5.22: A frame from the video sequence AKIYO (a) with missing GOBs, recon- structed using (b) a GWRF model, ( c ) a suboptimal HMRF model: (d) the method proposed in (671, ( e ) the method proposed in [62], and ( f ) our adaptive MRF model.
iMet hod GMRF HMRF Method proposed in [67] Method pruposed in [62] Adaptive MRF
Table .5.4: PSNR cornparison of different methods for the video sequence AKIYo.
1 No loss 33.52 33.5'2 33-52 33.52 33.52
20% Block loss 27-03 27.07 24.4 16.3 28.53
GOB rnissing 26.54 25.79 22.9 24.3 26.95
5.3 Error concealment of MPEG-4 coded video se-
quences
Here. ive discuss the various scenarios tha t can occur when decoding a n erroneous
XIPEG-4 coded video bit s t ream. I t is assumed t h a t t h e error resilience tools of
LI P EG-4 (ment ioneci in Chapter 2): more specifically using video packets and data
parti tioning. a re applied during t h e encoding. W e consider the intra-video object
planes (1-L'OPs) and inter-video object planes (P-VOPs) separately.
Quantimion Sbpc & Motion Tufurc RQI)IIPE. I=INUmbaI Vduc IMoti~t~DataIMkI 1-1
Figure 5.23: Video packet s t ruc ture of MPEG-4 in (a) 1-VOPs and (b ) P-VOPs.
In the P-VOPs. t h e shape a n d motion da ta in a packet a re separated from t h e
texture d a t a with a motion marker as seen in Figure 5.23. If a n error occurs in t h e
t e s t ure part of a video packet. t he decoder can use t h e motion information t o replace
the missing texture with t h e tex ture in t h e previous VOP. If t h e error occurs in t h e
motion/shape part, then t h e whole video packet is discarded. A simple concealment
can be done by replacing t h e shape a n d t e s tu re of t h e missing macroblocks wit h t hose
of nlacroblocks corresponding t o t h e same location in t h e previous VOP. An alter-
native method is t o e s t ima te t h e missing motion vectors from t h e motion vectors of
surrounding macroblocks a t t h e decoder and use the es t imated vectors t o replace t h e
missing texture and shape from the previous VOP [log] or use the method proposed
in Section 3 . 5 [113: 1141.
\Ve tested Our proposed shape reconstruction met,hod on various corrupted
AIPEG-4 bit streams ivith various packet sizes which contain vide0 ohjects coded at
-5000 bits/VOP. Error simulation is performed at the decoder: While decoding the
bit stream. the decoder ignores the video packets randomly with a given packet loss
percent age. Then the proposed concealment algorit hm is applied to the erroneous
1-VOP binary shape data to recover the missing shape blocks-
The similarity of the erroneous and error concealed shape data to the originaI
shape data is measured using the following value
where nd is t h e number of pisels that are different betiveen the restored shape and
t h e original one: and nt is the total number of in the bounding box. nd is in
fact t h e Hamming distance between the restored shape and the original one.
Figure 5.24: The shape of the first 1-VOP of the AKIYO video object.
Figure 5.24 shows the binary shape da ta of the first 1-VOP of the AKIYO video
11s
Figure 5-25: The shape of the .AKrYo video object missing 30% of the shape blocks.
Figure 5.26: T h e reconstructed shape of the first 1-VOP of the A K I Y O video object.
ohject. T h e bounding box of the VOP is of size 272 x 208. The length of each
\-ide0 packet is selected to be 500 bits. The number of blocks in each video packet
is different and depends on the size o l coded data of each biock. For example. in
this case. the first ten packets have 9: 14. 3, 14: 2, 15. 2 . 2 . 13. and :3 macroblocks.
respecti\rely. Figure 5.25 shows the shape of the 1-VOP with 90% of t h e shape blocks
rnissing. corresponding to 6 missing video packets. The result of the proposed error
concealment method is presented in Figure 5.26. The shape similarity measure JI is
SS% for the erroneous shape data shown in Figure 5.25. and 99% alter using the
proposed shape concealment method.
Figure 5-27: The shape of the first 1-VOP of the BREAXf video object.
Figure -5.28: The shape of the BREAM video object missing 25% of the shape blocks.
Figure 5.27 show the binary shape data of the first 1-VOP of the video object
BRE.A.\~. The size of the bounding rectangle is 272 x 192. The size of' the video packets
is selected to be 700 bits. The coded 1-VOP contains 44 video packets. Figure 5.28
shows the binary shape information of the 1-VOP with 2.5% of the blocks missing.
The shape data after concealment is given in Figure 5.29. T h e similarity measure p
is SG% before? and 99% after? concealment.
Lasto w e apply our concealment technique to the shape of the first 1-VOP of
the \VE.aTHER video object. as presented in Figure 5.30. The size of the VOP is 160
x 22-4. The size of the video packets is set to 1000 bits. Figure 5.31 shows the shape
Figure 5.29: The reconstructed shape of the first 1-VOP of the BREAM video object.
Figure .5.30: The shape of the first 1-VOP of the WEATHER video object.
of the 1-VOP when 3.5% of the macroblocks are missing. Figure 5.32 shows t h e result
of the proposed error concealment method. The value of 7 is S i % and 99'7~ before
and after the coricealment of shape information, respectively.
5.4 Summary
In this chapter, we studied the performance of the error concealment methods pro-
posed in Chapters 3 and 4 on baseline JPEG, H.263 and klPEG coded visual data.
Figure 5.3 1 : The shape of the \VE..\THER video object missing 35% of the shape blocks.
Figure 5-32: The reconstructed shape of the first 1-VOP of the it'E.4THER video object.
In each case. we also compared the performance levels of our proposed methods rvit h
t hose of similar met hods proposed in the literature. The proposed concealment met h-
ods achieve very good computation-performance tradeoffs. as demonstrated via our
esperiment al results.
Chapter 6
Conclusions and Future Work
6.1 Summary of Thesis Contributions
The problem of error control and concealment in i-isual communication is becoming
i ncreasingly important because of the growi ng interest in image/video delivery over
iirireliable channels sucli as wireless networks and the Internet- This thesis has ad-
clressed the problern of post processing for concealment of errors resulting frorn t h e
transmission of coded visual information in error prone environrnents. r\lthoiigh most
of the proposed methods a re applicable to more popular visual coding systems. we
have focused on the 11PEG-4 framework where the bit Stream consists of a collec-
tion of coded testure (DCT coefficients). shape and motion information '. W e have
devcloped error concealment methods for each of these components. For texture com-
ponent of coded visual data. we have considered intra and inter coded testure. The
niet liod int roducecl for concealment of errors in int ra coded t e s t ure. unlike previously
published algorithms. considers the dependency between coded blocks in a frame. It
uses a deterministic or a statistical technique for the restoration of missing testiire
information. The deterministic technique achiews a good performance level in the
reconstruction of edges. The statistical technique uses an adapt ive MRF as the iniage
cl-priori rnoclel in a SI-AP estimator. 'The adaptation cnablcs the estimatiori procc-
dure to incorporate more information wi t hoiit a dramatic increase in computat ional
complesi ty. The met hod proposed for concealment of errors in inter-coded testure
Lises t he information in the adjacent frame to find an initial estirnate of the missing
test tire. If the mot ion vectors associated wit h the missing blocks are availahle. motion
compensation is uscd to provide good estimatcs. Otlierwise. a novel algorithm which
preserves image continuity is used t o find the initial estimate of the missing texture.
The initial estimate is then combined with the estimate of the precliction error of the
missing block to fincl the best approximation of the missing texture information.
Error concealment for shape information has. so far. not been addressed in the
l i tcrat ure. \k proposecl efficient concealnient nlethods for shape information in t his
thcsis, .A t\-\I'\P estiniator. which uses an adaptive >[RF as the image n-priori model.
is used t o estimate the missing shape information.
\\'e esamirieci the performance of our error concealment met hods on the bit-
'The hybrid motion conipensated DCT-based coder (e-g.. NPEG-2) is a special case of the hIPECL.1 one. where the video objects are rectangular in shape
streams of baseline JPEC; coded images. H.263 coded vide0 sequences. and 3IPECL-I
cocfed ride0 data. R-e evaluated the performance of our proposed methocls bj+ using
objective criteria such as PSXR. and by subjective assessment of reconst riicted im-
ages. .\Ioreo\-er. i v e compared t h e performance of our proposed met hod tri t h that of
other methods available in literature. The methods presented in this thesis achieve
consistent ly good perforniance-computation t radeoffs which makes t hem siiitable for
real t ime communication over error prone nctworks. Employing the proposed error
concealment methods can lead to acceptable visual quality at loss rates hç high as
20%.
The main contributions of this thesis can bc summarized as follows:
An error concealment methoci for texture that consiclers the dependenc~- be-
tween coded blocks.
A determinist ic technique for reconstruction of missing texture informat ion t hat
achie\.es a goocl performance Ievel in the reconstruction of edges.
O .A statistical technicl~ie for reconstruction of missing texture that uses an adap-
tif-e AIRF as the image a-priori mode1 in a LIAP cstimator. The adaptation
enables t Iie estimation procecliire to incorporate more information without a
dramatic increase in computational coniplesity.
An efficient concealment niethods for shape information wtiich uses an adaptive
AlRF. designed for binarj- shape information. as the image a-priori modcl.
6.2 Future Research Directions
The problern of error control and coiicealment in visual communication will continue
to be of high importance because the ciemand for visual communication will continue
to grow. Thus. research in error coiitrol and concealment algorithms and tooIs d l
continue to be an active area. Among the possible research topics t o extend and
improve the ~ o r k of this thesis are:
Derivation of bounds for the estimation error in reconstruction of missing d a t a
of image and video signal.
lIore advanced interactive error concealnient methods ut ilizing t h e cooperation
bctween the encoder and the decoder.
Fast implementation of error concealment met hods for by using task-split tech-
niques (e.g.. pipelines).
0 -+II t hough more e fec t i ve error concealment approaches are st il1 neetleci. more
crnphasis should be placeci a t joint design of the encoding algorithm. transport
protocol. and post-processing met hod to minimize the combinecl distort ion due
to hoth compression and transmission.
Bibliography
[ I I K. R. Rao a n d .J. .J. Mirang. Techniques and Standards /or Irnnge. I'idco and
.4udio C'oding. Engiewood Cliffs. SJ: Prentice-Hall . 1996.
[2] ISO/IEC. Digital compression and coding of contin uous ton t still images: I-e-
quir~rncnis and guid l inr .~ International S t a n d a r d Organization. 1994.
[3] \$:. Pennebaker a n d .J. .\litchell. dPEG Still Image Cornpr~s.sion Sfnndnrd. Seir
York: Van S o s t r a n d Reinholcl. 19913.
[-l] O. Egger. P. Fleury. T. Ebrahinii. a n d 11. Kunt. a-High-performance compression
of visual information. a tutor ia l rcview. par t 1: St i l l pictiires." Proc. of th€ IEEE.
rot. $7. pp. 976-101 1. .lune 1099.
[5] B. Ci. Haskell. P. Ci. Ho~rard. Y. A. Lecun. A. Puri . J . Ostermann. 51. R.
Civanlar. L. Rabiner. L. Bottoii. a n d P. Haffner. -*Image and video coding-
cmcrging s t anda rds a n d heyond." IEEE Trans. on Clrcirit.r; and Sysferns /or
\Idco Technology. vol. 8 . pp. C-14-8:37. Yov. 1998-
[fi] T. El>raliimi and )1. l iunt . -Visual data compression for miiltirnedia applica-
tions." Proc. o/ the IEEE. vol. 86. pp. 1109-1 12.5. J u n e 1998.
[7] G . K. U;allace. -The jpeg st il1 pictiire compression standard.- Conimunications
of the .4C-11. vol. 134. pp- 30-44. Apr. 1991.
[XI .J. In. S. Shirani. and F. Iiossentini. .*On R D optimized progressive image coding
using JPEC; .- IEEE Transactions on fninge Proc~ssing. vol. S . pp. 1630- 1698.
Sov. 1999.
[9] .i. In. S. Shirani. and F. Kossentini. -JPEC cornpliant efficient progressive image
- coding. in Proccedings of the International Conference on ,-lcoustics, Speech.
nrd Signal Processing. (Seat t k ) . pp. 2633-26136. 199s.
[ I O ] D. L e Ciall. -.\IPEC;: a video compression s tandard for multimedia applica-
tions-*' C'ornmunications of the -4C'M. vol. :34. pp. 46-58. Apr. 1991.
[ I 11 I i . Sayood. introduction lo data cornpr~ssion. San Francisco. C'alifornia: Morgan
Iiaufnlann Publiçliers. 1996.
[ l?] \:. Bliaskaran and K. Iïonstantinides. lrnagc and Ir'ideo Compression Standards:
.-llgorifhms and Archif ecf u r ~ . Boston: Iiluwer Xcademic Piiblisliers. 1995.
[1:3j B. C;irocl. -Rate-const rained motion est irnat ion." in SPIE Proc. I wual Corn-
nl u n icat ions and Inlnge Proce.i;sing. vol. 2308. pp. 10%- 103-1, 1994-
i1.11 \\.'. C'hung. F. Kossentini. and A I . Smith. -.An efficient motion estimation tech-
.. . n ique based on a rate-distortion criterion. in Proceedings o j the lnt~rnationnl
Conference on . - I C O ~ L S ~ ics. Speech. and Signal Processing. vol. - 1 . (.Atlanta. C; .A.
[--S.=\). pp. 1926-1929. 1Iay 1996.
[16] F. Iiossentini. Y. Lee. 51. Smith. and R. \\%rd. -Predictive RD-constrained
motion estimation for very low bit ra te video coding.- IEEE Transactions on
S ~ f e c f cd . d r ~ n s in ~ommunicat ions . vol. 15. pp. 1 ï32-li63. December 1997.
[16] A. II. Jain. Fundamcntnb of Digital Image Processing. Englewood Cliffs. Xew
.Jersej-: Prentice-Hall. 1989.
- [l ï] .J. .J. S. Ja-nt. -Signal compression based o n rnotlels of hiiman perception.
Proc. of the IEEE. pp. 1S85-1-13?. Oct 1993.
[IS] .-1. Segall. -Bit allocation and encoding for vertor sources.- IEEE Transnctiona
on Infornzation Theory. vol. IT-22. pp. 162-169. I\Iar. 1976.
[19] 1-. Shoham and A. Gersho. -Efficient bit allocation for an arbi trary set of
q u a n t i ~ e r s . ~ IEEE Tram. o n Acovst ics. Speech. and Signal Proc~s.cirtg. vol. 136.
pp. 144.3-14.73. Sept. 19SS.
[-O] T. \\'iegand. 11. Lightstone. D. Slukherjee. T. Campbell. and S. .\[itra. -Rate-
Distort ion Opt imized 5Iocle Selectiori for Ver? Low Bit Rate Vidco C'ocling and
t h e Emerging I-1.263 Standard.- IEEE 7'rczn.s. on C'irruits and SpIe ms for I l d ~ o
Tcchnologg. pp. 1S2-190. Apr. 1996.
[ Z l ] S. Farvardin ancl d. \Y. .\lodestino. -Adaptive buffer-instrumented entrop?--
codecl quant izer performance for menioryless s o i ~ r c e s . ~ IEEE Trcr nsact ion.+ on
In for~rnation Thcory. vol. IT-32. pp. 9-22. Jan . 1986.
['>" 1. Shoham and A. Gersho. -Efficient bit allocation for an arbitrary set of yuan-
tizers." IEEE T r w n s . .Jcoirst. Spicech Signal Proc&.si.. vol. ASSP-36. pp. 1445-
14.53. Septenlber 1988.
[23] H. Sun. W. Kwok. 11. Chien. and J . Ju. -Mpeg coding performance improve-
ment b ~ - jointlj- optirnizing coding mode decisions and rate control." IEEE - I'ran.~. on Circuits and S g d ~ r n s for I'i'deo Technology. vol. L . pp. 449-A%. .June
19%'.
[24] A. Schuster and --1. liatsaggelos. -Fast and efficient mode and qiiantizer select ion
in the rate-distortion sense for H.263." in SPIE Proc. I'i.sual C'omrnunications
clnd Image Processing. vol. 27-27. pp. CS-1-79.5, 1996.
[25] le. Lee. F. Iiossentini. and R. \\'ad. -Efficient 51PECL'L encoding of inter-
laced video..' Cunndian .Journa l of Elect rical and C'omput c r Engineering. vol. Z3.
pp. 61-61. . lune 1998.
['>(il 1.. Lee. F. Iiossentini. R. liard. and 11. Smith. -To~vards SIPEG4: An improved
II .%Il- basecl vidco codcr." Signal Proc~ssing: lm ngc C'ont 111 urr ica t ion. Sptcial
I.wué on .\IPEG-4. vol. 10. .July 1997.
7 ISO/IEC. -Information te ch no log^. - coding of moving pictures and associated
audio for digital storagc niedia at up to about 1.5 mbits/s: Video.- 11 172-2.
1993.
2 ISO/IEC. -Information technology - generic coding of rnoving pictures and
associated atidio information: Video." 1:3SlS-2. 1995.
[29] IT Li-T. -Vide0 coding for low bit rate communication.- Recommendat ion
t1.2653, 1996.
[:IO] [TL--Tm -Vide0 coding for Iow bit rate communication." Recomniendation H.2613
\'ersion 2. 1998.
[3 11 T. Sikora. -The .\IPEG-4 video standard \-erification mode1 .*. in IEEE Trnns.
o n Circuits and S y s t ~ m s for Iiidco Tech~ofogy- vol. 1. pp. 19-Rl. Febriraru 1997.
[:El . J . Osterman and -4. Puri. -Xatiiral and synthetic video in 1IPEC;-4." in Pr-oc.
/EEE In!. C'on f. .-lcoust .. Spc~ch . anci Signal Processing. vol. 5 . pp. 3805-3808.
May 199s.
[:H] 1. JTC' 1 /SC'2S/\VG 1 1. -1IPEG-4 s p t e m s FAQ: version 7-0a.- X2521. October
1998.
[Xi] 1. JTCI /SC'29/\jiG 1 1 . -1fPEC-4 requirements. version 9.'- S'L4.56. Octoher
109S.
[37] 11 P EG--1 Systerns Croup. T o d i n g of audio-visual ohjects: Systems.'* ISO/ IEC
.JTCl/SC29/\.\:G 1 1 K2201. 11ay 199s.
[dd] 1. JTC 1 /SC'29/\4;C; 1 1. "Description of IIPEG-4:- S 1-1 10. Octobcr 1996.
[99] R. Haralick and L. Shapiro. "Image segmentation techniques.'* in Cornputer
Il-isiorz. Graphics and image Processing. pp. 100- l Z . 198.5.
1 :E
[-101 A. Iiatsaggelos. L. P. Iiondi. F. Meier. -1. Ostermann. and Ci. Schuster. -.\lPEG-
4 and rate-distort ion-based shape-coding technicl~ies.- in Proc. of th E IEEE.
vol. 86. pp. 1029-1051. .June 199s.
[-!LI AI.-. V. Group. -Coding of audio-visual objects: Vicieo.^ ISO/IEC
.JTCl/SC'2S/\\:G I 1 S2202. hfarch 1998.
[41] If.-. V. Group. *hIPEG-4 video verification mode1 version 8.0.- ISO/IEC
.JTC I/SCB/\\:C; 11 51796. July 1997.
[-131 1'. Wang ancl Q. F. Zliii. 'Error control and concealnwnt for video comrnunica-
tion: .A revicw." Proceedings of thc IEEE. pp. 97'4-997. -\la'- 1998.
[4-l] S. Farber. B. Girard. and J . Lyillasenor. '-Estension of ITC--T reconimmnedation
H.263 for error-reiIierit video transmission.- IEEE corn rn unication nzagnzine.
L-01. 36. pp. 120-128. Jurie 1998.
[-I.31 L. El. Kieu and K. S. Sgan. Tell-loss concealment techniques for layercd video
cociecs in an XTlI netivork.- IEEE Trnns. on Image PI-ocf.z'sing. vol. :3. pp. 666-
677. Sept. 1994.
[AG] R. Aravincl. R. C'ivanlar. and A. R. Reibman. --Packet loss resilience of 1lPEG-
'1 scalal,le 1-ideo cocling alg~rithrns.~. IEEE Trans. on Circuits and Syderns f i r
Iridco T~chnofogy. vol. 6 . pp. 570-5d0. Oct. 1996.
[-171 5 1. T. Orcliard. Y. \\:ang. V. Vaishampayan. and A. R. Reibnian. -Reclunclancy
raie-distort ion analysis of mui tiple description cotling using pair wise correlat ing
-. - transforms. in Proceedinp of International C'onference on Image Procéssing.
(Santa Barbara). pp. 608-61 1. Oct. 1997.
[48] Y. \Vang. 41. T. Orchard. and .A. R. Reibnian. -SIuitiple description image
coding for noisy channels by pairing transform coefficients." f roc . IEEE 1.d
IP-orkshoy .\lutirnedia Signai Proccssing. pp. -119-424. .lune 1997.
[49] )'. Wang. 11. T. Orcharcl. and A. R. Reibman. -Optimal pair \vise correlat-
i ng t ransfornis for muitipk description coding." in Proceedings of Intcrnntionnl
C'or~ f c r ~ n c e on Imngc Processing. vol. 1 . (Chicago). pp. 679-683. Oct. 1998.
[30] 1.. Cioyal. J . liovacevic. R. Arean. and AI. Vet terli. -Slult iple description trans-
.. . form coding of images. in Proceedings of International Con ferencc on Inzagr
Proc~.i.iing. (Chicago). pp. 674-67s. Oct . 1998.
[ . i l ] Ci. Rej-es. -4. R. Reibman. J . Chuang. and S. Chang. -Video transcoding for
rcsilierice in wireless channels." in P r o c ~ ~ d i n g s of Int~rnationcri Cenfcrc n c ~ on
/rrtagc Processing. vol. 1 . (Chicago). pp. 3138-342. Oct 199s.
[ X ] Ci. C'ote. S. Shirani. and F. Kossentini. -Robiist h.263 video communication over
packet lossy networks." in Procecdings of International Conjcrencc on I lnage
Pt.occssing. vol. 2. ( Iiobe. Japan). pp. .5:33-539. Oct 1999.
[53] Ci. Cote. S. Silirani. and F. Iiossentini. -Optimal mode selection ancl syncliro-
nization for rohust vide0 communications over error prone net\vorks.-- ncc~p ted
for publication in the IEEE Journal on Sdcctcd Amas in C'orn nz unicntions.
[.i-I! \\-. Lam and A. R. Reibman. -Self-s-nchronizing variable length codes for image
transmission.- in Proc. IEEE Int. C'onf. =Icoi~at.. Speech, and Signal Processing.
i.01. 6. pp. 477-480. 190'2.
[55] D. Rcdmill and S. G - Iiingsbury. -The EREC: a n error-resilient technique for
coding variable-lengt h blocks of data." IEEE Trana. on Image Proc~ssing. vol. 5 .
pp. .56.5-574, april 1996-
[.XI A. li. Katsagelos. F. Ishtiaq. L. P. liondi. SI. C. Hong. 11. Banham. and
.. . .J. Brailean. -Errer resilience and concealment in \-ide0 cocling. in Proc.
EC-SIPC'O--98. ( Rhodes. Cireece). pp. 211-228. Sept. S- 11 1998.
1571 C'. Parthasarath-. J . W. Ilodestino. and K. S. Vastola. -Design of a transport
coding scheme for higii-qiiality video over ATSI networks.'- IEEE Trcrns. on
C'ircirit.~ and Sysfcrna /or \-ide0 Twhnology. vol. 7. pp. 3-58-376. April 1997.
(5S] 11. . iiliansari. A. .Jalali. E. Dubois. and P. Ilermclstein. -Law bit-rate video
transmission over fading channels for wireless microcellular systerns.- IEEE
TI.QI~.s. or? ClrcuiC.< crrzd Syslema for I,ldeo Tcchnology. voi. 6 . pp- 1-1 1. Feb.
9 R. Talluri. -Error resilient video coding in
Cornm. .\/nyazinc. 1-01. 26. pp. 112-119. June
t tie 1.1 P EG-4 standard.'- IEEE
1998.
[GO] J . Liang and R. Talluri. -Tools for robust image and \-ide0 coding in .JPEC;2000
ancl 1I PEG-4 st aiiclards..- i.01. 36-53. pp. 40-5 1. January 1990.
[Gl] S. Shirani. F. liossentini. and R. Ward. -Errer concealment methods: A corn-
parati\-e study." in iEEE Canadian Conference o n efectrical and cornputer en-
gincering. (Edmonton. Canada). pp. S3.5-840. .lune 1999.
[62] Y. \i:a:ang. Q. F. Zhu. and L. Shaw. -11 asimally smoot h image recorery in trans-
form cocling." IEEE Trans. on Communications. vol. 41. pp. l.j-L-I-l-E 1. Oct.
1 99:3.
[&3] \\-. Zhu and )'. \4:ang. -The use of second order deri\-atives based smoot hness
measiire for error concealment in t ransfer based codecs.- in SPIE Proc. ITi.sutd
C'omn~iinicalions a n d Image Proces.iing. vol. '2501. pp. 120.7-1214. Feh. 199.5.
[(i-l] Q. F- Zhu. \-. Wang. and L. Shaw. "Image reconstruction for Iiyhrid video
cociing systems.-' in Pr-oc. IEEE Data C'ornpr-ession Co nJ . pp. 229-238. l\[arch
1992.
[G5] .J. \\:. Park. J . \Y. Kim. ancl S. Ir. Lee. -DC'T coefficients recovery-basecl error
concealment technique and its appIication to the 1IPEC;-2 bit streani error."
IEEE Tran.5. on Clrcuits and Systems for lri&o Tech nology. pp. 8-1.5-8.51. De-
cernher 1997.
[G6] \\'. 11. Lani and A. Reibman. -.An error concealment algorithm for images
subject to chan~iel errors.- IEEE Tr.ans. on Imcrgc Procés.sing. pp. 5:3:3-.54'L.
hlay 199.5.
[ f i i l S. S. Heniami ancl T. H. Y. Meng. "Transiorm coded image reconstruction
espIoi t ing interblock correlat ions.? IEEE Trnnsnct ions on Image procéssinq,
vol. -1. pp. 1023- 1027. 3uly. 199.5.
[GJ] P. Salama. S. B. Shroff. E. .J. Coyle. and E. J . Delp. "Error concealment tech-
niques for encodecl video st reams.- in Proceedings of International Conjerence
on Image Proce.r;.sing. pp. 9-12. 1995.
[69] V. \\éerackocody. C. Podilchuk. Y. Zhou. and A. Est rella. Transmission of J PEC; - coded images over wireless channels. in SPIE Proc. I-isual C'ornrnunicatiotzs
and Image Procesaing. pp. 157- 168. 1995.
[TOI G. J u . 11. \V. 1Iarcellin. and .CI. M. K. Liu. -Recovery of video in the presence
.* . of packet loss iising intericaving and spatial redundancy. in Procccdinp of the
Intct-national C o n f ~ r c n c c on Acoustics. Speech. and Signal Pr-oce.wing. pp. 105-
108. 1906.
[ i l ] 51. C;lianhari and V. Seferidis. T e l l loss concealmerit in ATSI video coclecs."
IEEE Tr-ansacfions on Circuit and sgstem i n i~chnology. vol. 3 . pp. 238-
2-17. .June 19913.
[Z] P. Salama. S. B. Shroff. and E. J . Delp. -t\ baysian approach to error conceal-
ment in encoclecl vidco strcams.- in Procccdings o f In tcrnnt ionni Cor2 f~rcncé o n
l m n g e Proc~ssinq, pp. 49-52, 1996.
[73] R. Veldliiiis. Restorntion O/ Lost Sarnp1c.s in Digital Signais. Engleu-ood Cliffs.
Sew Jerse!.: Prent ice- HaIl. 1990.
[T4] .A. C'. liokaram and et al. -Interpolation of niissing data in image secltiences.-
IEEE Transactions imnye processing. vol. -1. pp. 1509-1519. Kov. 199.7.
FI. R. Rabiee. H. Radha. and R. L. Iiashyap. -Errer concealment of still image
ancl video st ream $vit h mrrlti-direct ional recursive non-linear fi1ters.- in Proceed-
ings oJ the international Conference on ;Icoustic.s. Speech. and Signal Process-
ing. pp. 37-40. 1996.
H. Sun and W. Kwok. "Concealment of damaged block t ransform coded images
using projection onto conves setsl" IEEE Tran.sncti0n.s imagc proce.iising. vol. 4.
p p . 4704'7'7. April 199.5.
\4:. 1I;wok and H. Sun. -1Tulti-directional interpolation for spatial error con-
cealment ." /EEE Tran.snctions on consumer- c lecf ronics. vol. 39. pp. -15.5-460.
Airgiist 199:3.
L. Capodiferro. S. Puledda. and G. .Jacovitti. -1lissing block recover? by linear
pattern proPa@ ion." in Proce~ding of Picl ure Coding Sympo.iiu m. pp. 3-53-359.
Sept. 1997-
IV. Zcng and B. Liu. -Ceometrical-st ructirre- based directional filtering for error
concealment in iniage/video transmission." in SPIE Proc. i I.sirnl COmm u~licn-
tions nrid fmnge Proc~ssing. pp. 14.3-1.56. 1995.
\\-. Zeng and B. Liu. "Geornet ric-struct ure-based error concealment !vit h novel
applications in block-based low-bit-rate coding." iEEE Trnns. o n CFi-cuits and
Sgsfcrns for l l d r o Technology. vol. 9. pp. 6-48-665. June 1999.
11. Chien. W. Sun. and W. Iiwok. -.A temporal ancl spatial POC5 based error
concealment algori t hm for the SIPEG encoded video sey uence.- in SPIE Proc.
I*isunl Communications and I m a g ~ Processing. vol, 2501. pp. 16s-174. Feb.
1 09.5.
[S-] \\'. S. Lee. 11. R. Frater. ancl d. F. Arnoldet-al. --Spatial temporal concealment
of lost blocks in coded \-ideo." in Proceedings of International COnferenc~ on
Irnngic Proce.wing. vol. 13. (Chicago). pp. 477-481. Oct 1998.
[S3j 2. \\;ang. Y. Yu. and D. Zhang. -Best neighhorhood mat ching: An information
loss restorat ion technique for block- based image codi ng sys tems." IEEE T I - m .
on Image Pr-oc~ssing. pp. 10.56- 106 1. .lui'- 19%.
- 1 S. Ikng a n d )-.-Hm Hu. T o d i n g artifacts rernoval iising biasecl anisotropic diffii-
sion." in Proccc~dings of Int érnational Con fer~nce on [mage Prmxssing. pp. :346-
349. 1997-
[S5] Q. F. Zhu. Y. Wang. and L. Shaw. -Image reconstriictior, for hybrid vicleo
cotling systems." in Proc. IEEE Data Comprt.ision Con/. . (Snowbircl. UT).
p p . 219-2138. Mar. 1992.
[SG] Ci. D. Sampson. D. V. Papadimitriori. and C'. Cliamazas. --Postprocessing of
block-cociecl images at Iow bit rates.'* in Procecdings of the Inter-nnlional Con-
f c rc-nce o n .-tcoilst ics. Spcech. and Signal Processing, pp. 1-4. 1906.
[ST ] J . Ho. S. Sinaceur. F. Li. ri. Tarn. and 2. Fan. ~~Reiiioval of hlocking and
ringing artifacts in transforrn coded images.- in Proceedings of thc International
CWon/'êr-encc on Acoust ics. Speech. and Signal Processing. pp. '356.5-2.56S. 1997.
[Sa] H. C . Iiim and H. W. Park. -Signal adaptive postprocessing for blocking effects
reduction in JPEG image." in Procecdings o j the Internntionnl Conjkrrncr on
.-Lcoustics. Speech. and Signal Processing. pp. 4 1-44. 1996.
[89] Z. Fan and F. Li. -Reducing artifact in JPEC. decompression segmentation
" - ancl smoothing. in Procecdings of thc Int~rnational Conference on ..tcou..;tics.
Speçch. and Signa€ Processing. pp. 17-20. 1996.
.. [!)O] C'. J . liiio and R. J. Hsieii. -Adaptive postprocessor for block encoded images.
IEEE 'Tr-an.5. on Circuits n nd Systems for I'ideo Technofogy. pp. 2%-30-1. Aug
199.5.
[9 11 R. C1<astagno and J . A. Villaroel. spline-based adaptive filter for the r e m o d
of the blocking artifacts in image seciilence coded at ver>- ION- bit rate." in
Procc~dings of the [nternationd Conferrr~ce on ~+~cou.sCics. Speech. and Signal
Processing. p p . 45-48. 1996.
[92] H. Paek and S. [ y . Lee. -A projection bwed post processing technique to reduce
blocking artifact iising a priori information on DCT coefficients of adjacent
hlocks." in Pmcedin ip of th c International Con fercnce on .-lcousf ics. Speech.
and Signal Procrssing. pp. 53-56. 1996.
[93] D. C'. Yotila and H. \\:ebb. -Image restoration by t h e nietbocl of conves projec-
.. tions: Part 1-theory. IEEE Trnnsactions on medical Imageing. vol. 1. pp. 81-
9-1. Oct. 198'2.
[94] C'. Derviaus. F. X. Couctous. SI. G. Gazalet. P. Corlay. and '1. Ciharbi.
postprocessing technique for block effects elirnination using a perceptual dis-
- tor t ion meastire. in Proc~edings 01 the Inlrrnationaf confer-encc o n .-koustics.
S p ~ c c h . and Signal Processing. p p . 300 1 -ROO4. 1997.
[95] Z. Xiong. 31. T. Orchard. and Y. Q. Zhang. -.A deblocking algori thm for JPEC;
compressed images iising overcomplete wavelet representat ion." IEEE Trans.
on Clrcuits and Sya/ems fcr Video Technology. pp. 433--146. Apr. 1997.
[W] T. C'. Hsiing. D. P. K. Lrin. a n d \V. C . Siti. -.A cleblocking technique for JPEC;
decoded image using wavelet t ransorm modulus maxima representaion." in Pro-
cecdings of t h t I n t ~ ~ n a t i o n a l C o r z f e r ~ n c ~ on rlcou.otic.s. Speech. und Signal Pro-
crssing. p p . .% 1 -.564. 1996.
1971 T. C'lien. --The p s t . present. a n d fu tu r e of image and miiltitlimentional signal
processing." I E E E Signal Proc~s s ing .\.fagnzin. vol. 15. pp. 21-58. JIarch 1998.
[9S] -4. Rosenfelti. Irrznge .\lodels. Seu- Y+ork. 31.: .-\cademic Press. 1 %O.
[99] S. C e m a n a n d D. Geman. --Stochast ic rclasation. Gi bbs clistri but ion. ancl the
Bayesian restoration of images.- /EEE Trnr~i;. Pattern -4 nnl. and JIacfl. [nt..
\ v o l . 6. p p . 721-741. Sovernber 1984.
[100] S. Çhirani. F. Iiossentini. and R. \Varcl. -Reconstriictiori of baseline JPEG
codcd images in error prone envi ronmerits." acctpf ed /or publication i n f ll e IEEE
Trnnsnct io 1 2 s o n Irnage Proce.wing.
[IOl] S. Sliirani. F. Iiossentini. and R. Uard. -.A concealrnent niethod for video com-
niunicat ions in a n error prone environment ." acceytcd /or publication in the
IEEE Jour-na1 o n Sflected .Pt /-cas in C'ornrn unictrtions.
[LO?] J . Besag. -On the statistical analysis of dirty pictures.- .Journal Royal Stntist.
Soc. B. vol. 48. pp. '2.59-302. May 1986.
[103] P. Salama. S. Shroff. E. .J. Coyle. and E. .J. Delp. Error conccalrnent in encoded
cidco streams. in Signal recovery techniques for image and vicleo compression
ancl transmission. Ed. S. P. Galatsanos and A. K. Katsaggelos. Boston: Kluwer
Acaciemic Piiblishers. 1998.
[104] P. Salama. 3. B. Shroff. and E. J . Delp. -A fast suboptimal approach to error
conceaiment in encoded video st rcams." in Proc~edings of Inter-national Con-
j ~ i v n c c on Image Proc~ssing. pp. irol. I I . 101-104. 1997.
[105] S. Shirani. F. Kossentini. and R. \Vard. --An adaptive rnarkov randorn field
hased error concealment method for video communication in an error prone
.. - erivironment, in Proceedings of the Jnternational Con f e r t n c e on . - I C O U S ~ ~ C S .
Spctch. and Siynal Proc~ssing. vol. VI. pp. 31 17-3120. lfarch 1999.
[IO61 S. Siiirani. F. Iiossentini. and R. \\:ard. -Reconstriiction of motion vector miss-
.. - ing n-racroblocks in H.26:3 encocIec1 video transmission over lossy networks. rn
Procc~dings o f fnternationnl C'on ference on Image Proce.r;sing. pp. -LSï--l9 1. vol.
III . October 199s.
[lOT] J. Zhang. .J. F. Arnold. M. R. Frater. and JI . R. Pickering. --C-ideo error conceal-
nient using decoder motion vector estimation ." in IEEE TE.\'C'O.\-- Speech and
Imngc Technology for Cornputer and Telécornnzt~nications. pp. 777-ïdO. 1997.
[ los] \4:. A I . Lam. A. R. Reibman. and B. Lin. -Recot-ery of lost or erroneously
- receivecl niotion vectors. in Procerdings of t h e International C'onjerence on
-4coil.i-tics. Speech. and Signal Processing. vol. V . p p . 4 17-120. 1993.
jlol)] 1 R. Frater. IV. S. Lee. and d. F. Arnold. -Error concealment for arbitrary
stiaped video ohjects." in P r o c d i n g s of Interna!ionaI C'on ference on Image
Proc~ssing. vol- 13, (Chicago). pp- 507-.7I 1. Oct. 199s.
[ l I O ] Ci. Côté and F. Kossentini. -Optimal intra coding of blocks for rohost video
cornn~~inication over t h e internet.- EL-ROSIP Journal for- I-kunl Cenzmunica-
lion. Sp~cial Issue on Réal-time I'idco orer- thc Intcrrr~t. vol. 15. pp. 25-34.
Scpt. 1990.
(1 111 Ci. Siillivan. -.A simple ï ideo packet rnux sirnolator program for video streams
in H.26:3/1\[ using XLB mux of H.233 annes B." [TC-T Sfudy Group 2.5. l'ideo
C'oding Erpc ;.t C;r.oup. Docum ER! Ql.5-F- 16. Yovember 1996.
[ I l - ] Ericsson. -WCDMr\ error patterns at 64 kh/s.- [ T ( - - T Study Group 16. Miil-
f i n ~ d i n Er-rrt irtcrls and Sy.slenw Ezpc ri Gr-O u p . J une 19%-
[1 l:3] S. Shirani. B. Erol. and F. Iiossentini. "Error concealnient for mpeg-4 video
corrimunication in an error prone environment.- in Proc. IEEE In!. Conf.
.-\CO 11.5t.. Sptxch. and Signal Processing. ( to appear). 2000.
[11-11 S. Shirani. B. EroI. and F. Iiossentini. ".-1 concealment rnethod for sliape in-
formation in MPEG-4 coded video sec~uences." acccptcd 101. publication itz the
IEEE Transaction on Multimedia.
[ l i 51 .I. .A. \'-ebb. -Steps toward archi tectwe-independent image pr0cessing.l IEEE
C'ompcitcr dlagnrir~. pp. 2 1-31. February 1997.
[ l lF] R. Chellappa. J l a r k o r Randorn Fields. T'hheory a n d .4pp l ica t ions . California:
Academic Press. Inc., 1993.
[ I l i ] L. O. S. Herodotoii and A. Venetsanopoulos. -Image interpolation using a
simple gi hbs ranclom fielcl model.- in Procced ings of l n t ~ r n n t i o n n l Con ference
on i m a g e Process ing . vol. 1 . (Washington D.C.). pp. -194-49s. Oct 1995.
[IlSI H. .J. Siegel. J . B. Armstrong. and D. \V. Watson. -llapping cornputer-vision-
relatcd t asks ont0 reconfigiirable paraHel processing systenis." lEEE C'omp ute r
,\f(~gnzin. pp. 54-6R. February 1992.
Appendix A
Parallel Implement at ion of
MRF-Based Error Concealment
Method
A.1 Introduction
I t was sliown in C'Iiapters 2 and 3 that reconstruction of niissing tex ture inforniation
lising 1lRF-based l f AP estimation resillts into a minimizat ion problern which is usii-
a l I ~ - solved using i terat ive met hocls. Alt hough under specific condit ions t hey con\-erge
to t lie global mininuim. iterative methods a r e t ime consuming and prohibitil-e in ap-
plications where real t ime implenlentation is required ( a s in most video processing
applications). T h e high speecl recluired in many of the image processing applications
cari he achievecl 1- employing parallel processing techniques [lL5].
A paralle1 implemeritation for the simulated anneal ing algori t hm. which has
Figure A.1: A pisel. i t s clique ( d a r k ) a n d t h e cornplement of t h e clique ( in shatle).
heen used t o obtain t h e 41.-\P e s t i m a t e of a noisy image. has been proposed in [116].
In [ I l i ] . image interpolation is performed using a s imple .\[RF a s t h e image model.
Tlic i terat ive rnethocl required for 'c1XP est imat ion is app ros ima ted b ~ * non-iterative
noiiliriear filtering operations. thereby reducing t h e computa t iona l cornplesity of t lie
interpolation process.
In t his appendis . we present fast implementations for t h e lIRF-basecl error
concealmerit met hods proposed in C h a p t e r :3 using parallel processing techniques.
Since the pisel values in a n image a r e restricted t o a finite se t (e.g.. O t o '2.55). t h e
proposecl implementations replace mos t of the reqtiirecl a lgebraic operat ions wit h
lookiip tables. reducing t lie exccution t ime.
For t h e sake of simplicit?. ive assume t h a t in
x = min C C p ( ~ i . ~ - k . / D ( . r ; , . . r i , [ ) ) . r''' i . jE,u k.(Ec
rr;.,-k.l = 1. V ( i . j ) . (k. 1 ) . l loreover . lets consider a two ( ins tead of eight ) pixel
ncighborhood for tlie cliqiie. T h e clique is illustrated in F igure -4.1. T h e estension
t o a n eight-point clique is straightforu-ard. Finally. it is sufficient to assume tha t 71
in Equat ion (2.7) is a positive integer value since t he pixel values a r e positive integer
.A parallel machine consists of a nurnber of processing elements ( PEs). T h e
PEs are usually arranged in a niatris configuration with each PE connected to its
ncighboring PEs through a bus (see Figure A.2). lloreover. each PE usually has its
own riiemorj.. Parallel machines can operate in t wo modes: single instruction. niultiple
data (SI I ID) or multiple instruction. multiple data (1II l ID) . In SIIID. al1 active PEs
esecute the same instruction sj-nchronous1~- on their own data. Onlj- a single copy
of the program is stored in this niode and -sencl- and -receiveW operations betiveen
PEs are automatically synchronized. In the 1 I I I ID mode. on the other hand. t here
is no constraint on opcrations t hat can be performed concurrently and PEs operate
as'-nchronously. The SIMD mode is easier to program and requires lcss memory but
is not as flexible as the I I I I ID niode. Choosing the proper parallelism mode and
in~~le rnen ta t ion tins a significant impact on the performance of the application [118].
.ksiirning each missing block consists of -Y x -Y pixels. in our proposed imple-
mentation. :Y2 PEs are enip10'-ed for estimating tlie missing pisel values. Eacli PE is
rcsponsihle for reconstructing the value of a pisel in tlie missing block. If the number
of adjacent missing pisels esceeds the number of PEs. a groiip of pisels is assigned
to cach PE. In that situation. each PE is responsible for restoring the values of t h e
pixels in the group. The operations ~erforrned b-. each of the PEs and the way it
commiinicates with the PEs around it. is detailed below for the G l I R F and HMRF
nioclcls.
A.2 GMRF Mode1
.As was cliscussed in Chapter 3. for the C.\IRF moclel the estimated value of a pixel
is given bj- Ecpation (3.9). In other words. the estimated value of a pixel is the
(weighted) average of the pisels in its clique and tlie clique's complement. For this
model. the estimated value of a pixel has a simple closed form expression in terms of
acljaccnt pixels. Therefore. the estimation result c m be compiited efficientlu and there
is t hereiore no need for a parallel implementation. Hoivever. ive will discus the parallel
iniplemcnt at ion of the Cil1 RF based error conceainient since i t will help developing
tlie parallel iniplementation of error concealment methocls iising other forms of SIRF
models.
Figure A.?: Sim~lified diagram of a parallel processing machine.
For paraIlel implementat.ion of error concealment usiiig the G3.1 RF motlel. each
PE commiiriicates witli the PEs responsible for restoration of pixels i r i the clicliie
ancl its complement of the pixel that the PE is restoring (see Figure ."\:3). Having
the values of t lie pisels in the clique and its complement. the surnmation of them is
Figure A.3: The connection of adjacent PEs for e r ror concealment based o n t h e LIRF niodels.
calciilated bj- t h e PE. Since each pixel value is restricted t o a n integer niirn'ber between
O and 25.5. t h e surnniat ion can onlj- have 2.56 x 4 different values. For eacli srimmation
\-allie. t h e est imat ion result which is t h e surnmation divided by four. is s tored in a
lookup table. T h i s lookiip table. schernatically shown in Figiire A . - I . requires LI<
Bytes of mernory a n d can be shared a m o n g al1 t h e PEs. T h e opera t ions perforrned
by each of t h e P E s a r e identical. Hence. a ver)- efficient SIMD implementat ion is
possible.
if t h e restoration of each pixel could be clone independent ly of t h e value of
o thc r pixels. t h e proposed parallel implementat ion could lead t o a speecliip factor
equal t o t h e to ta l nurnber of P E s (.V2). However. t h e estirnated value of a pixel
depends o n t h e \*alue of pixels aroirncI i t and therefore sorne in te r -PE d a t a t ransfer is
rcquirecl. Thiis. t h e speedup factor will b e less t h a n .Y2. If t h e t i m e t o perform an
in te r -PE t ransfer is denoted by t t h e speedup factor of t h e parallel irnplc-mentat ion
over t h a t of t h e uni-processor will be equa l t o
SUM ESTIMATION
Figure A.4: T h e lookup tab le iised in t h e e r ror concealment based o n t h e C;lIRF model.
where f,,, is t h e t ime recluired t o perform t h e procedure detailcd above (calculating
t h e siim ancl using t h e lookup table) . T h e speedup calculation is based on t h e as-
sumpt ion t h a t t he uni-processor a n d each PE a r e of equa l comput ing potver. If t h e
t i m e t o perform inter-processor t ransier is ignored. t h e speedup factor of t h e proposed
parailel implementat ion compared to a serial implementat ion of t h e s a m e algorithm
will be equal t o t h e n u m b e r of PEs which is .Y2.
A.3 HMRF Mode1
Tliere a r e t ~ v o sources for cornputat ional c o m p l e s i t ~ in t h e lI.L\P es t imat ion empIoving
t h e H l I R F model. First . the es t imated value of a pisel depends o n t h e values of t h e
pixels in i ts clique which a r e iisually unknown. Second. t h e function p ( . ) has diffcrent
definitions for 1.7-1 > 7 a n d 1x1 < -1. L7sualb. i terat ive methods a r e iised t o ohtain t h e
es t imated value of pisels iinder these conditions. hence a reaI t i m e processing rnay
not be fcasible. Ive address t h e first problem hy proposing a parallel implementation
1.50
for II1,lRF-based MAP estimation. To address the second problem. ive use bit-wise
operations and lookup tables.
Figure .-1..5: The connection of adjacent PEs in the error concealment hased on HAIRF
For paraIlel implementation of error concealment methods using the HMRF
model. each PE communicates with the PEs responsible for restoration of pixels in
the clique ancl its complenient of the pixel that the PE is restoring (see Figiire -4.5).
To determine which part of the functiori p ( . ) should be used in computing the cost, we
dc\doped t h e following met hod. Corresponding to the value of each of the pisels in
the clique ancl its complement. there is an intcrval called an associate interval defined
as the follou-ing set
~vtiere a; is the valiie of ith pisel. If the intersection of the associatc intervals of
the four pixels arountl a specific pixel is not ernpty. the estiniated valiie of t h e pisel
is equal to one of the valiles in t h e intersection. If intersection of t h e associate
intervals of the four pixels is ail empty set. cfifferent combinations of three associate
intervals are testecf for an intersection. If the intersections of al1 conibi~iations of
t hree di fferent associate intervals are empty. combinat ions of two associate intervals
Figure A.6: T h e bit pattern for a pisel value of 98 and 5 = 2.
arc being checked- If intersections of al1 two associate inter\-als are al1 ernpty. t h e
cstiniatecl value of the pixel is one of the values in t h e associate intervals of the pixels
in t h e clique. However. this case is very iinlikely a n d corresponds to scatterecl d a t a
which obviously carinot prof-ide a reliablc estimate. I n n-tiat folloivs. fvc cliscuss a
method for fincling the intersection of the associate inter\-& efficiently.
T h e associate interval of each pixel value consists of a t most 27 + 1 integer
nirrnhers with values hetiveen O and 2.55. The associate interval of each pixel value
can be stored in a 2.56 bit-wide binary number as follou-. The least significant bit
ancl t he most significant bit in the binary nurnber represent the pixel value of O ancl
2.5.5. respectively. The bits corresponding to the values in the associate interval of
a specific pixel value are set to 1 a n d al1 other bits are set to zero. Thereiore. a
spccific hinary ni~rnber is generated for every pisel value and a particular 7. Dire to
t h e fact tliat the pisel values can assume only 2.56 different integer niimbers. t here are
o n l ~ - 2.56 clifferent binary niirnbers. These numbcrs can be stored as a look-iip table
consisting of 236 elements each of t hem 2.56 bit \vide. Figure A.6 sho\vs an elenlent
of tliis table for a pixel value of 9s and *, = 2. This look-iip table. calletl associate
intervals look-up table. recluires G4Ii of meniory and c a n be shared bet,ween al1 the
P Es.
Figure A.7: The lookup table for t h e Hube r function with 7 = 2.
Ttic intersection bctween t h e associate i n t e r \ a l s of pixels around a specific
piscl can be de te rmined hy a series of bit-wise ;\>:D operations. T h e set of possible
candidates for t h e es t imat ion resiilt consists of t h e \-alties corresponding t o ones in
t h e result of X S D operations. =\mong t h e values in t h e candida te set . t lie value t h a t
niinimizes t h e potent ia l functioii (Ecluation (3.7)) is t h e est imat ion result. To find
the values in t h e canclidate set t hat niinimizes t h e potent ia l fiinction efficient15 t h e
value of t h e H u b e r function (Eqiiation 2.7) is also c o m p u t e d for clifferent pixel values
( O to 25.3) a n d s tored in a look-up table called cost fi inction look-up table (see Figure
.-1.7)- Sincc p ( . ) is a n cvcn furiction. only t h e function \ -dues for s betweeii O a n d 2.55
shoiild be storecl in t h e cost function look-up table. This tab le can be shared between
al1 t he PEs.
A n e s a m p l e can clarifj- t h e abovr proceclure. Let's assiinie the valucs of t h e
pixels a round a specific pixel t o be 98. 99. 100 a n d 37 a n d 7 = 2. These values a r e
available to the PE t hrougti conimunication wi t h t h e neighboring PEs. The biiiary
representation of associate i n t e n d of t h e pisel value of 98 will have L in positions
corresponding t o 96. S i . 9s. 99 ancl 100. For t h e ~ i s e l values of 99. 100 ancl -57 these
positions will h e 91. 9s. 99. 100. 101 a n d 98. 99. 100. 101. 102 and .55. 56. -57. .?S.
.59. rcspectively. These bit patterns a r e obtained using t h e associate intervals look-up
table. .An A S D operation performcd on al1 four bit pat terns results in zero value.
Among different combinations of three different bit patterns. only the resiilt of the
A S D operat ion on t h e first three bit pa t te rns will have a non-zero value. The 1s
in the result of t h e A S D operation correspond t o t h e pixel valiles of 9s. 99 a n d 100
whicli const i tute t h e candidate set. For each of t h e values in the canclidate se t t h e
t-alue of t h e potential function (3.7) is cornputeci using t h e cost function look-iip as
fol lo\vs:
the est imation residt is the value t ha t minimized t h e potential function which for t his
esample will bc 98.
Finallj-. t h e proposecl met hod c a n be siimmarized as follon-s ( t h e ope ra t ions
are pcrformeci in parallel in each of t h e PEs):
1. Obta in t h e associat,e interval of t h e pixel values in t h e clique ancl its complemcnt
enlptoying t h e associate intervals look-up table.
2. Find t h e candidate set by performing a series of :\XI) operations on the associate
i nt ervals.
13. Cornpute t h e valiics of the potential h n c t ion ( Ecluat ion S.7) using cost functioti
look-up table.
4. Select the value in the candidate set corresponcling to the minimum value of
the potent ial funct ion compiited in the previoiis step.
For HMRF model. the number of A?;D operations and hence the esecution tirne
of est iniation of each pixel is data dependent. S I N D and JIILID implerrientat ions are
bot h concei\-able in t his case with different trade offs. In SIlID. since al1 the PEs are
synchronized. the process shouid wait for t h e PE with t.he highest number of AND
operations to finish its operations. In hII31D implementation. on the other hand.
each processor has its own program and can perform the operatioris without waiting
for other PEs. C'learl~.. the speecl of 1 I N D implementation is greater tlian or ecltiaI
to t hc SI3ID iniplenientation. but i t requires more mernory space.
The speedixp factor of the parallel inipiementation of H l 1 RF mode1 wsiimi ng
SIJID niode over that of a uni-processor will be eclual to
Lvliere t,,, is the time required t o perform the procediire listed above. Again. it is
assumecl t hat the uni-processor and each PE are of eclrial compiiting power. Ignoring
t , , , comparecl to t , , . the speedup factor of the proposecl parallel implementation
cornparcci to a serial implementation of the samc algorithm will be eqiial to the
niiniher of PEs whicli is ,Y2,