Better restore the recto side of a document with an ... fileBetter restore the recto side of a document with an estimation of the verso side: Markov model and inference with graph

Better restore the recto side of a document withan estimation of the verso side: Markov model

and inference with graph cuts

Christian Wolf

Laboratoire d’InfoRmatique en Image et Systèmes d’informationLIRIS UMR 5205 CNRS / Université de Lyon

INSA de Lyon, bâtiment J. Verne20, Avenue Albert Einstein - 69622 Villeurbanne cedex

http://liris.cnrs.fr

June 23rd 2008

http://liris.cnrs.fr

Introduction Prior model Observation model Iterative graph cuts Pre- and postprocessing Results

Plan

1 Introduction

2 The prior model

3 The observation model

4 Iterative graph cuts - the posterior probability and itsmaximization

5 Pre- and postprocessing

6 Results

Christian Wolf - http://liris.cnrs.fr/christian.wolf Restore the recto by estimating the verso 2 / 41


Plan

1 Introduction

2 The prior model




6 Results



The problem

Blind ink bleed-through removal (scans of the verso side are notavailable).

The verso contents must be replaced by background.



Ink bleed-through removal

Problem : image segmentation. The classification decision foreach pixel is based on

the color/gray value of each pixelthe local structure of the image.

input recto labels verso labels

The framework : Bayesian estimation with two hidden Markovrandom fields (MRF)



Bayesian estimation

ESTIMATION

PRIOR KNOWLEDGE LIKELIHOOD OF THE DATAGIVEN THE HIDDEN VARIABLES



Plan

1 Introduction

2 The prior model




6 Results



A classical hidden Markov random field

F

D

Observed

variable

Hidden

variable

Clique of the

Prior model

Clique of the

observ. model

P(f ) =1Z

exp {−U(f )/T}



Why two hidden label fields ?

input segmentation

The prior should regularize fields which directly correspondto the natural process “creating” the contents.A correct estimation of the covered verso pixels, through thespatial interactions encoded in the MRF, helps to correctlyestimate verso pixels which are not covered by a recto pixel.



The double hidden Markov random field

F1

F2

D

Observed

variable

Hidden

variable

Clique of the

Prior model

Clique of the

observ. model



The joint probability : dependence of recto and verso

w/o obser-vations, f 1

and f 2 areinde-pendent

with ober-vations,they aredependent

P(f 1, f 2, d)

=1Z

exp{−(U(f 1) + U(f 2) + U(f 1, f 2, d)

)/T}

=1Z1

exp{−U(f 1, f 2)/T

}·

1Z2

exp{−U(f 1, f 2, d)/T

}= P(f 1, f 2)P(d|f 1, f 2)

= P(f 1)P(f 2)P(d|f 1, f 2)



The prior of a single field : the Potts model

U(f ) =∑{s}∈C1

αfs +∑

{s,s′}∈C2

βs,s′δfs,fs′

δi,j ={

1 if i = j0 else

Discontinuity preserving



Plan

1 Introduction

2 The prior model




6 Results



The observation model (Likelihood)

The observation model characterizes the degradation processwhich is applied to the theoretical “perfect” image resulting in theobserved image.We suppose that there are two independent degradationprocesses for each side, followed by a combination of the two“virtual” observations :

D = φc(φ1(F1), φ2(F2))

The combination process φc assumes 100% opaque ink : a rectopixel f 1

s = 1 completely covers the verso pixel f 2s , whatever its

label.



A Gaussian observation model

P(d|f 1, f 2) =∏

s

N (ds; µs,Σs)

µs =

µr if f 1

s = textµv if f 1

s = background and f 2s = text

µbg else

Σs =

Σr if f 1

s = textΣv if f 1

s = background and f 2s = text

Σbg else



Plan

1 Introduction

2 The prior model




6 Results



The posterior probability and its maximization

P(d) does notdepend on f 1, f 2

=⇒We can maxi-mize the jointprobabilityinstead ofthe posteriorprobability

P(f 1, f 2|d) = 1P(d) P(f 1, f 2)P(d|f 1, f 2)

∝ P(f 1, f 2)P(d|f 1, f 2)

U(f 1, f 2, d)

=∑{s}∈C1

α1f 1s +

∑{s,s′}∈C2

β1s,s′δf 1

s ,f 1s′

+∑{s}∈C1

α2f 2s +

∑{s,s′}∈C2

β2s,s′δf 2

s , f2s′

+∑{s}∈C1

12(ds − µs)

TΣ−1s (ds − µs)



The posterior probability and its maximization

Rewrite of the energy in terms of unary func-tions U1 and two types of binary functions U2and U′2

U(f 1, f 2, d)

=∑{s}∈C1

[α1U1(f 1

s ) + α2U1(f 2s )]

+∑

{s,s′}∈C2

[β1

s,s′U2(f 1s , f

1s′) + β2

s,s′U2(f 2s , f

2s′)]

+∑{s}∈C1

U′2(f1s , f

2s ; ds)



Minimization with minimum cut/maximum flow

[Kolmogorov et al., PAMI 2004] : a function ofbinary variables composed of unary terms andbinary terms is graph-representable, i.e. it canbe minimized with algorithms based on the cal-culation of the maximum flow in a graph, if andonly if each binary term E(x, y) is regular, i.e. itsatisfies the following equation :

E(0, 0) + E(1, 1) ≤ E(0, 1) + E(1, 0)

Depending on the observation ds of site s thismay or may not be the case for the observationmodel.




Iterative optimization algorithm :

Fix/estimateInitialize lables, e.g. with k-meansrepeat• Fix f 1, estimate optimal f 2

• Fix f 2, estimate optimal f 1

until happy/dead/running out of patience

Sub problem can be solved exactly withminimum cut/maximum flowEquivalent to the α-expansion movealgorithm for a single MRF with adapatedinteraction potentials.




Joint estimation of a subset of the graph GDetermine regular sites :Hs = 1⇐⇒ U′2(f

1s , f

2s , ds) is regular

Initialize lables, e.g. with k-meansrepeat• Fix f 1

s for all s and f 2s for Hs = 0, estimate

optimal f 2 for all s and f 2s for Hs = 1

• Fix f 2s for all s and f 1

s for Hs = 0, estimateoptimal f 1 for all s and f 1

s for Hs = 1

until happy/dead/running out of patience

=⇒ larger standard move !




Iterativealgorithm : onefield is fixed (e.g.f 1, the other fieldis estimated (e.g.f 2).

Regular sites(Hs ←− 1) arejointly estimated.

=∑{s}∈C1:Hs=0 α1U1(f 1

s )+

∑{s}∈C1:Hs=1 α1U1(f 1

s )+

∑{s}∈C1

α2U1(f 2s )

+∑{s,s′}∈C2:Hs=0∧Hs′=0 β1

s,s′U2(f 1s , f

1s′)

+∑{s,s′}∈C2:Hs=1∧Hs′=1 β1

s,s′U2(f 1s , f

1s′)

+∑{s,s′}∈C2:Hs 6=Hs′

β1s,s′U2(f 1

s , f1s′)

+∑{s,s′}∈C2

β2s,s′U2(f 2

s , f2s′)

+∑{s}∈C1:Hs=0 U′2(f

1s , f

2s ; ds)

+∑{s}∈C1:Hs=1 U′2(f

1s , f

2s ; ds)



The cut graph

D

F2

F1

Dependency graph

F2

source(0/non− text)

sink(1/text)

F2

F1

source(0/non− text)

sink(1/text)

Cut graph α-exp move Cut graph proposed algorithm



Plan

1 Introduction

2 The prior model




6 Results



Estimation of the model parameters

Estimation of all parameters from the median filtered initialsegmentation (k-means).

Estimation of the likelihood parameters : empirical mean andvariance (covariance matrix).Estimation of the MRF parameters : least squares (Derin etal.)

Alternatives : Iterated Conditional Estimation,Expectation-Maximization, the meanfield theory etc.



Estimation of the MRF parameters

Least squares method [Derin et al., 1987]. Input data : pairs ofdifferent labels fs and f ′s with the same neighborhood Ns

θTp [N(f ′s , fNs)− N(fs, fNs)] = ln

(P(fs, fNs)P(f ′s , fNs)

)N(fs, fNs) = [ δfs,0,

δfs,1,γ(fs, fwe) + γ(fs, fea)γ(fs, fno) + γ(fs, fso)γ(fs, fne) + γ(fs, fsw)γ(fs, fnw) + γ(fs, fse) ]T

Least squares solution :

Nθp = p θp = N+p N+ = (NTN)−1NT























Label identification

AssumptionMost space on the document page is occupied bybackgroundThe ink is 100% opaque and therefore a recto text pixelcompletely covers a verso pixel.

A B

(a) (b)



Replacement of the verso pixels

��

�� !!!"""###$$�%%

&&&&''''(((())))**�++,,,---...///00�11

222233334444555566�77888999:::;;;<<�==

>>>>>>??????

@@@@@@AAAAAA

BBBBBBBBBBBBBBBBBBBBB

CCCCCCCCCCCCCCCCCCCCC

D D D DD D D DD D D DD D D DD D D DD D D DD D D DD D D DD D D DD D D DD D D DD D D DD D D DD D D DD D D DD D D DD D D D

E E E EE E E EE E E EE E E EE E E EE E E EE E E EE E E EE E E EE E E EE E E EE E E EE E E EE E E EE E E EE E E EE E E E

F F F F F F FF F F F F F FF F F F F F FF F F F F F FF F F F F F FF F F F F F FF F F F F F FF F F F F F FF F F F F F FF F F F F F FF F F F F F FF F F F F F F

G G G G G G GG G G G G G GG G G G G G GG G G G G G GG G G G G G GG G G G G G GG G G G G G GG G G G G G GG G G G G G GG G G G G G GG G G G G G GG G G G G G G

H H HH H HH H HH H HH H HH H HH H HH H HH H HH H HH H HH H HH H HH H HH H HH H HH H HH H H

I I II I II I II I II I II I II I II I II I II I II I II I II I II I II I II I II I II I I

J J JJ J JJ J JJ J JJ J JJ J JJ J JJ J JJ J J

K K KK K KK K KK K KK K KK K KK K KK K KK K K

L L LL L LL L LL L LL L LL L LL L LL L LL L LL L LL L LL L LL L LL L LL L LL L LL L LL L L

M M MM M MM M MM M MM M MM M MM M MM M MM M MM M MM M MM M MM M MM M MM M MM M MM M MM M M

N N N N N N NN N N N N N NN N N N N N NN N N N N N NN N N N N N NN N N N N N NN N N N N N NN N N N N N NN N N N N N NN N N N N N NN N N N N N NN N N N N N NN N N N N N NN N N N N N NN N N N N N N

O O O O O O OO O O O O O OO O O O O O OO O O O O O OO O O O O O OO O O O O O OO O O O O O OO O O O O O OO O O O O O OO O O O O O OO O O O O O OO O O O O O OO O O O O O OO O O O O O OO O O O O O O

P P P PP P P PP P P PP P P PP P P PP P P PP P P PP P P PP P P PP P P P

Q Q Q QQ Q Q QQ Q Q QQ Q Q QQ Q Q QQ Q Q QQ Q Q QQ Q Q QQ Q Q QQ Q Q QR R R R R R RR R R R R R RR R R R R R RR R R R R R RR R R R R R RR R R R R R RR R R R R R RR R R R R R RR R R R R R RR R R R R R RR R R R R R RR R R R R R R

S S S S S S SS S S S S S SS S S S S S SS S S S S S SS S S S S S SS S S S S S SS S S S S S SS S S S S S SS S S S S S SS S S S S S SS S S S S S SS S S S S S ST T T T T T TT T T T T T TT T T T T T TT T T T T T TT T T T T T TT T T T T T TT T T T T T TT T T T T T TT T T T T T TT T T T T T TT T T T T T TT T T T T T T

U U U U U U UU U U U U U UU U U U U U UU U U U U U UU U U U U U UU U U U U U UU U U U U U UU U U U U U UU U U U U U UU U U U U U UU U U U U U UU U U U U U U

V V VV V VV V VV V VV V VV V VV V VV V VV V V

W W WW W WW W WW W WW W WW W WW W WW W WW W W

X X XX X XX X XX X XX X XX X XX X XX X XX X XX X XX X XX X XX X XX X XX X XX X XX X XX X X

Y Y YY Y YY Y YY Y YY Y YY Y YY Y YY Y YY Y YY Y YY Y YY Y YY Y YY Y YY Y YY Y YY Y YY Y Y

Z Z Z ZZ Z Z ZZ Z Z ZZ Z Z ZZ Z Z ZZ Z Z ZZ Z Z ZZ Z Z ZZ Z Z ZZ Z Z Z

[ [ [ [[ [ [ [[ [ [ [[ [ [ [[ [ [ [[ [ [ [[ [ [ [[ [ [ [[ [ [ [[ [ [ [

\ \ \ \ \ \ \\ \ \ \ \ \ \\ \ \ \ \ \ \\ \ \ \ \ \ \\ \ \ \ \ \ \\ \ \ \ \ \ \\ \ \ \ \ \ \\ \ \ \ \ \ \\ \ \ \ \ \ \\ \ \ \ \ \ \\ \ \ \ \ \ \\ \ \ \ \ \ \\ \ \ \ \ \ \\ \ \ \ \ \ \\ \ \ \ \ \ \

] ] ] ] ] ] ]] ] ] ] ] ] ]] ] ] ] ] ] ]] ] ] ] ] ] ]] ] ] ] ] ] ]] ] ] ] ] ] ]] ] ] ] ] ] ]] ] ] ] ] ] ]] ] ] ] ] ] ]] ] ] ] ] ] ]] ] ] ] ] ] ]] ] ] ] ] ] ]] ] ] ] ] ] ]] ] ] ] ] ] ]] ] ] ] ] ] ]

^ ^ ^^ ^ ^^ ^ ^^ ^ ^^ ^ ^^ ^ ^^ ^ ^^ ^ ^^ ^ ^

_ _ __ _ __ _ __ _ __ _ __ _ __ _ __ _ __ _ _

` ` `` ` `` ` `` ` `` ` `` ` `` ` `` ` `` ` `` ` `` ` `` ` `` ` `` ` `` ` `` ` `` ` `` ` `

a a aa a aa a aa a aa a aa a aa a aa a aa a aa a aa a aa a aa a aa a aa a aa a aa a aa a a

b b b b b b bb b b b b b bb b b b b b bb b b b b b bb b b b b b bb b b b b b bb b b b b b bb b b b b b bb b b b b b bb b b b b b bb b b b b b bb b b b b b bb b b b b b bb b b b b b bb b b b b b b

c c c c c c cc c c c c c cc c c c c c cc c c c c c cc c c c c c cc c c c c c cc c c c c c cc c c c c c cc c c c c c cc c c c c c cc c c c c c cc c c c c c cc c c c c c cc c c c c c cc c c c c c c

d d d dd d d dd d d dd d d dd d d dd d d dd d d dd d d dd d d dd d d d

e e e ee e e ee e e ee e e ee e e ee e e ee e e ee e e ee e e ee e e e

fffffffffffffffffffff

ggggggggggggggggggggg

h h h hh h h hh h h hh h h hh h h hh h h hh h h hh h h hh h h hh h h hh h h hh h h hh h h hh h h hh h h hh h h hh h h h

i i i ii i i ii i i ii i i ii i i ii i i ii i i ii i i ii i i ii i i ii i i ii i i ii i i ii i i ii i i ii i i ii i i i

j j j j j j jj j j j j j jj j j j j j jj j j j j j jj j j j j j jj j j j j j jj j j j j j jj j j j j j jj j j j j j jj j j j j j jj j j j j j jj j j j j j j

k k k k k k kk k k k k k kk k k k k k kk k k k k k kk k k k k k kk k k k k k kk k k k k k kk k k k k k kk k k k k k kk k k k k k kk k k k k k kk k k k k k k

l l ll l ll l ll l ll l ll l ll l ll l ll l l

m m mm m mm m mm m mm m mm m mm m mm m mm m m

n n n n n n nn n n n n n nn n n n n n nn n n n n n nn n n n n n nn n n n n n nn n n n n n nn n n n n n nn n n n n n nn n n n n n nn n n n n n nn n n n n n nn n n n n n nn n n n n n nn n n n n n n

o o o o o o oo o o o o o oo o o o o o oo o o o o o oo o o o o o oo o o o o o oo o o o o o oo o o o o o oo o o o o o oo o o o o o oo o o o o o oo o o o o o oo o o o o o oo o o o o o oo o o o o o o

p p p pp p p pp p p pp p p pp p p pp p p pp p p pp p p pp p p pp p p p

q q q qq q q qq q q qq q q qq q q qq q q qq q q qq q q qq q q qq q q qr rr r�ss

t tt t�uuv vv vww

xxyyzz{{

||}}

~ ~~ ~��

� ��

� � � � � � � � � � � � � ��

� � � � � � � � � � � � � ��

� � � � � ��

� � � � � ��

� � � � � � � ��

� � � � � � � ��

� ��

� ��

� � � � � ��

� � � � � ��

� � � � � � � ��

� � � � � � � ��

� � � � � � � � � � � � � ��

� � � � � � � � � � � � � ��

� ��

� �� LEVEL 2

B

C

LEVEL 0

A

LEVEL 1

Leaves :

Ms =

1 f 1s = backgr. ∧

f 2s = backgr.

0 else

Os = ds

Parent nodes :

Ms =∑

s′∈s− Ms′

Os =

∑s′∈s− Ms′Os′

(∑

s′∈s− Ms′)

s− : children of site s



Plan

1 Introduction

2 The prior model




6 Results



Experiments

3 types of experiments :Synthetic images : recall/precision on pixel levelReal images, evaluation by humans : ranking of methodsReal images, evaluation by OCR : Levenstein cost andrecall/precison on character level

2 types of optimization :Simulated annealingBoykov and Kolmogorov’s implementation of minimumcut/maximum flow



Experiments on synthetic images

Noise level Nr. of classes Errorσ = 10 K-Means (k=3) 0.25

Single MRF 0.03Double MRF 0.01

σ = 15 K-Means (k=3) 1.40Single MRF 0.23Double MRF 0.08

σ = 15 K-Means (k=3) 3.56Single MRF 0.73Double MRF 0.31

Inference with simulated annealing



Experiments on real images

Input K-means

Single MRF Double MRF




Input K-means





Input K-means





Input K-means





Method Ranked 1 Ranked 2 Ranked 3K-Means 18 10 36Single-MRF 13 39 12Double-MRF 33 15 16Total 64 64 64

Binomial-test : P-value of 0.00197. Significance level ofα = 0.025 =⇒ reject of H0.



Real images : input and k-means (graph cut)



Real images : single and double MRF (graph cut)



Real images : Improvement of OCR results (graph cut)

103 pages, inference with graph cuts. OCR : Google Tesseract.Recall Precision Cost

No restoration 84.25 60.71 80,512K-Means (k=3) 90.19 73.26 47,611Double MRF 90.45 75.57 42,314

1 page, inference with graph cuts. OCR : Google Tesseract.Recall Precision Cost

No restoration 91.65 70.72 502Niblack 91.03 70.64 492Sauvola 91.75 76.66 349K-Means (k=3) 92.68 77.10 326Source separation RGB 80.72 54.53 911Single MRF 94.22 81.68 275Double MRF 94.43 83.27 262



Computational complexity

1026×1557 2436×3320K-Means (k=3) 3.2 18.9Single MRF 9.1 40.6Double MRF 12.6 67.5

Inference with graph cutsExecution time in seconds (Pentium-M 1.86Mhz 1Gb RAM).



Conclusion and Outlook

The double MRF model is more natural and more powerfulthan the single MRF model.The double MRF model using binary labels naturally leads toan efficient graph cut implementation.The MRF methods based on classification beat the sourceseparation based methods in terms of OCR improvement.Perspectives :• Creation of a in-homogeneous “adaptive” observation model,

which increases the performance on larger images. Thismodel needs to take into account several text colors, as wellas the page bending process and other different kinds ofdegradation.• A hierarchical Markov model.• A discriminative model taking into account texture.


Documents

Better restore the recto side of a document with an ... fileBetter restore the recto side of a document with an estimation of the verso side: Markov model and inference with graph