Upload
arnab-sinha
View
656
Download
0
Tags:
Embed Size (px)
DESCRIPTION
"Texture" provides the perceptual information about the surface, nature etc. about the visual objects. Study in texture learning and synthesis with a mathematical model will hopefully provide us the mathematical nature of visual perceptiveness. On the other hand, Markov Random Field, nonparametric density estimation and their applications in the real world problems, are becoming popular in both research and industrial fields. The reason for this popularity is because of the mathematical models have more robustness, flexibility and simplicity. The research problems (given this background) are order estimation and large computational complexity. In my PhD thesis I have tried to solve these issues for the application in homogeneous texture synthesis.
Citation preview
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Fast NMRF based texture synthesis algorithms
Arnab [email protected]
April 16, 2009
Thesis Supervisor: Dr. Sumana GuptaACES-205, Dept. of EE, Indian Institute of Technology Kanpur, India
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Earlier MethodsResearch problems in NMRF-tex-syn algorithms
1 OutlineEarlier MethodsResearch problems in NMRF-tex-syn algorithms
2 Ph.D. Research WorkOrder Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
3 Conclusion and Possible Future Directions
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Earlier MethodsResearch problems in NMRF-tex-syn algorithms
The significance of texture synthesis
• What defines texture ?• Locally varying intensities and/or color values• The local variations can be found perceptually similar within the total region
• Texture Synthesis:
Given a small texture exempler, synthesizean arbitrary sample of texture, so that thesynthesized texture is visually similar to theoriginal sample.
Original D104 Texture
Synthesized texture should look alike the original texture
• Application of texture synthesis in -• Image segmentation, classification, synthesis, etc.• Content-based image retrieval
• Development of high-level computer vision algorithms• Animation of real scenes• Perceptual analysis• Computationally fast and efficient handling of objects
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Earlier MethodsResearch problems in NMRF-tex-syn algorithms
Texture synthesis: Difficulty
Figure: Spectrum of Natural Textures
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Earlier MethodsResearch problems in NMRF-tex-syn algorithms
Brief History of Models
Linear
Image Domain Model Mixed Domain Model Transformed Domain Model
Texture Synthesis Algorithm
Non−Linear Models
Hard−limitedProcessJacovitti et al. (1998)
Non−Linear
Circular Harmonic Func+ Hard−limited GaussianCampasi and Scarano (2002)
Heeger and Bergen (1995)
Portilla and Simoncelli (2000)3.
2. Zhu et al. (2000)1. Zhu et al. (1997)
Chellappa and Kashyap (1985)2D−NCAR,
2D−Wold Francos et al. (1993)
2D− MAEom (1998)
Efros and Leung (1999)
Ashikhmin (2001)Wei and Levoy (2000)
Tonietto et al. (2005)
We are workingwithin thisFramework
Sampling Process
Hidden Markov TreeFan and Xia (2003)
Zhang et al. (1998)Wavelet + AR
Paget and LongstaffNNMRF
(1998) Charalampidis (2006)
Gaussian
Mathematical Models
Intuitive Models
Non−Linearity Introducedby Histogram Equalization
Pixel−based
Patch−based
Popular Methods
Kwatra et al. (2003)
Wu et al. (2004)Patch−based sampling with wavelet transformation as a feature set for graph−cut algorithm
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Earlier MethodsResearch problems in NMRF-tex-syn algorithms
Description of N-MRF model
• S is the lattice
• Ys is the random variable at site s ∈ S
• Concept of Neighborhood system:
• s < ℵs
• r ∈ ℵs ⇔ s ∈ ℵr
• Circular neighborhood: ℵs = {r ; s.t., |r − s|2 ≤ o2}• say, Xs = {Yr ; r ∈ ℵs }
s = (i,j), site
1st order neighbors
2nd order neighbors{ }
• Say, Y(s) = {Yr ; r , s}, r , s ∈ S
• Definition of MRF: P(Ys |Y(s)) = P(Ys |{Yr ; r ∈ ℵs })• parameteric model for P(Ys |Xs)• semi-parameteric model for P(Ys |Xs)• non-parameteric model for P(Ys |Xs)
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Earlier MethodsResearch problems in NMRF-tex-syn algorithms
Description of N-MRF model: Kernel Density Estimation
Definition of KDE, [Scott(1992)]
• single dimensional: P(x) = 1N∑N
i=1 Kh(x − xi)
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Earlier MethodsResearch problems in NMRF-tex-syn algorithms
Description of N-MRF model: Kernel Density Estimation
Definition of KDE, [Scott(1992)]
• single dimensional: P(x) = 1N∑N
i=1 Kh(x − xi)
• multi-dimensional: P(X) = 1N∑N
i=1∏d
j=1 Khj (X(j) − Xi(j))
• where, in case of Gaussian kernel, Khj (X(j) − Xi(j)) = 1N√
2πhjexp{− (X(j)−Xi (j))
2
2h2j
}
• and, hj = σjN−1/(d+4)
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Earlier MethodsResearch problems in NMRF-tex-syn algorithms
Texture Synthesis Algorithm
Some definitions• Input texture field: {Ys }, where, s ∈ Sin
• Output texture field: {Yq}, where, q ∈ Sout
• Definition of LCPDF: P(Yq |Xq) =
∑s∈Sin
KhY(Ys−Yq)KhX
(Xs−Xq)∑s∈Sin
KhX(Xs−Xq)
Iterative Conditional Mode (ICM) algorithm
• Evaluate P(Yq = y |Xq), for y = 0,1, . . . ,255 gray values.
• Assign Yq = y, for which the above conditional probability is maximum
Local Simulated Annealing
• Define a Confidence field, Cq; q ∈ Sout , and a matrix Φq = DIAG{Cr ; r ∈ ℵq}• KhX (Xs − Xq; Φq) = exp{−(Xs − Xq)T Φh,q(Xs − Xq)}, and Φh,q = ΦqHX ≈ hΦq
• Updation rule for the confidence field• Cq = min{1, 1
|r∈ℵq |∑
r∈ℵq Cr + u × e}• where, u is a random number and e is a constant scale factor
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Earlier MethodsResearch problems in NMRF-tex-syn algorithms
Texture Synthesis Algorithm
Approximate Independent Conditional Mode (ICM) algorithm
• Ds,q = (Xs − Xq)T Φq(Xs − Xq)
• Define Sq = {r ∈ Sin} ⊂ Sin, s.t., ∀r ∈ Sq, Dr ,q = constant.
• Assign Yq = yr , where r is sampled from the set Sq randomly.
WqX q
{X }
X q X s( − ) Xq X s( − )t
X q X s( − ) Xq X s( − )t
Input texture
Output Texture
Input Neighborhood Vectors
Output Neighborhood
Confidence Field
Output Confidence Vector Vector
s
Matrix
Similarity Measure
N−MRF :
SinSout
q q
C
WL alg :
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Earlier MethodsResearch problems in NMRF-tex-syn algorithms
Research Problems
• Order estimation• Large Computational Complexity
• Computational complexity ∝ d, the dimension of the neighborhood vector.• Computational complexity ∝ (M × N), the input image size• Computational complexity ∝ I, the number of iterations required to attain global
convergence
Order 4 Order 8 Order 14
Original Texture
computational complexity of texture synthesis algorithmis proportional to ’d’
Nei
ghbo
rhoo
d ve
ctor
dim
ensi
on ’d
’
Model order ’o’
0
1000
2000
3000
4000
5000
6000
7000
8000
0 10 20 30 40 50
Figure: Effect of order on the synthesis results and computational complexity
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Order estimation from two fundamental frequencies
Xi
Yi
Yj
Xj
X
Y
o
c
b
a
d
Figure: Points a,b , c, d are the four corners of texton defined by the fundamental spatial periodvectors [Xi Yi ] and [Xj Yj ]. The major diagonal o gives the order of causal circular neighborhoodand o/2 gives the order of non-causal circular neighborhood.
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Extraction of the parameters
• Dimitri’s algorithm• estimate the two fundamental frequecies from the two-dimensional DFT of the texture
sample.• computational complexity is of the order of image size.
• Hays’s algorithm• it estimates the two fundamental spatial vectors from the correlation function• the algorithm is iterative• computationally expensive with respect to Dimitri’s algorithm
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
D52D35
D20 D21
order = 4 order = 9order = 18 order = 23
order = 48 order = 21 order = 8 order = 22
Figure: Comparison of estimated order through Dimitrios and Hays’s methods with (NR) texturesynthesis results
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
A new neighborhood system
Xi
Yi
Yj
Xj
X
Y
neighborhood
neighborhoodproposed Non−causal
circular Non−causal
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Results: Proposed neighborhood system
Circular Neighborhood Proposed Neighborhood
D65
Circular Neighborhood Proposed Neighborhood
D104
D64 D95
D3 D67
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Two approaches
Computational complexity affected by
1 the neighbourhood dimension, d, and
2 the number of input pixels, N
Reduction methodologies
1 Dimensionality reduction methodologies, e.g., Principal Component Analysis(PCA) – to reduce the effect of d
2 A data structure for fast search
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
With Dimensionality reduction methods
How the distance metric Ds,q looks after projection
(Xq − Xs)T Φq(Xq − Xs) ≈ (X̂q − X̂s)
T Φq(X̂q − X̂s)
= [PTr Pr (Xq − Xs)]
T Φq[PTr Pr (Xq − Xs)]
= [Pr (Xq − Xs)]T PrΦqPT
r [Pr (Xq − Xs)]
= (Zq − Zs)T Ψq(Zq − Zs)
• What is Ψq = PrΦqPTr ?
• Is it reducing the computational complexity ?
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Simulated Annealing for Principal components
Original Ds,q
Ds,q = (Xq − Xs)T Φq(Xq − Xs)
where, Φq = DIAG{W1,W2, . . . ,Wd }
Proposed D̂s,q
D̂s,q = (Zq − Zs)T Φ̂q(Zq − Zs)
where, Φ̂q = DIAG{W1,W2, . . . ,Wk }
WHY ? Because we need only
• a steady increase in the value of confidence, and
• the starting value has to be ”0” and ending value has to be ”1”
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Results
Table: Comparison of dimensionality; Original dimension, |ℵs | = d; Reduced dimension, k(<< d), ηis the ratio of computational complexities between earlier and proposed one
Texture Type Texture order d k η
NR D20 20 1516 60 21.9405NR D3 30 2820 580 3.7926NR D21 25 1960 56 29.2642NR D22 20 1256 177 6.3042NR D35 28 2452 287 6.6612NR D36 22 1516 258 5.1024
ST D7 27 2288 511 3.6438ST D13 24 1792 131 11.6006
NR+ST D18 32 3208 95 25.5666NR+ST D4 29 2628 465 4.4755NR+ST D5 29 2628 179 11.6262
IN/STR D15 23 1652 167 8.4897IN/STR D42 26 2120 293 5.9699
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Results continued ...
D20
d = 1516 k=60
D7
d = 2288 k = 511
D21
d=1960 k=56
D22
d=1256 k = 177
D13
d=1792 k=131
D42
d=2120 k = 293
NNMRF Proposed Algorithm NNMRF Proposed Algorithm
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
With Fast Kernel Density Estimation
Assumption: The h parameters along all the directions are equal.
Source data vector
Target data vector
X
Y
R=100
To calculate KDEat this target pointwe only needthese two points
• Let Rn = {ts : ||ts − tn || ≤ R}• P(tn) = 1
N∑N
s=1 KH(ts − tn)
• P(tn) = 1N∑
s∈Rn KH(ts − tn)
• Let Nn = |{s < Rn}|
Err(tn ,R) =1N
∑
s<Rn
KH(ts − tn)
≤ Nn
NKH(R)
≤ KH(R)
• Rel err(R) =max(Err)
max(Probability)
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Earlier FKDE algorithms
• Improved Fast Gaussian Transform (IFGT)[Yang et al.(2003)Yang, Duraiswami, Gumerov, and Davis]
• kd-tree based FKDE [Gray and Moore(2003)]
• Reconstructionhistogram [Zhang et al.(2005)Zhang, Tang, and Kwok]
Reconstructionhistogram
• Clustering: {Clusti ; i = 1 . . .M}• Let ni as the number of source data vectors within ith cluster
• P(tn) = 1N∑M
i=1 KH(tn − Clusti)ni
• KDE of tn given the source data points at cluster centroids with a weight factorni/N
• flexibility ?
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Improved Fast Gaussian Transform
• P(tn) =∑||tn−Clusti ||≤RIFGT
KH(tn − Clusti)f(tn ,Clusti)
• P(tn) = 1N∑||tn−Clusti ||≤RIFGT
∑s∈Clusti KH(tn − ts)
RIFGT
this overlapDue to
we need to consider thissource cluster
RIFGTthe radius threshold has to be To consider the source cluster
RIFGTTheand cluster shape
can vary with the overlap size
R
Source data clusters
Target data vectors
In effect it can include some source clusterwhich was not needed at all
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
kd-tree-based FKDE
Build up the kd-tree and Search according to the radius R.
e
e
e
e
e
e
ee e
e
e
ee
e e ee
ee
e
ee
e e e
e
e
e
e e
e
e
� -
6
?
-
6
6
?
--
--
X
Y
Span
inY
dir
ecti
on
Span in X direction
σX > σY
Partition
Sub-spaces
tn
Rmax_rec_err
Error Rrec_err,nR> R+ max_rec_err
ReconstructionR
Centroid
Hyper−Sphere
Hyper−Rectangle
Eigen vectors
Y
X
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Why do we need another algorithm for FKDE
Table: Why do we need another algorithm for FKDE ?
C-FKDE KD-FKDEAdvantage Clustering algorithm provides more Due to the hyper-plane boundary,
compact representation of the data one can use original radius Rspace for strict error bound
Disadvantage optimal RIFGT has to estimated kd-tree is not a good clusteringfor every tn, maximum RIFGT algorithm, therefore it does notcan increase computational provide compact representation
complexity of the data space
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Principal Directive Divisive Partitioning (PDDP)[Boley(1998)]
• project each source data point within the present space onto the first principaldirection (eigen vector corresponding to the largest eigen value).
• partition the present space into two sub-spaces with respect to the mean (ormedian) of the projected values.
IV V VI VII
II III
I
2nd
1st
3rd
4th 5th
6th
7th
8th
Hieararchical BoundariesSource data
III III
VI
VII
1st 2nd 3rd 4th 5th 6th 7th 8thV
IV
Tree structure of the nodes
Leaf nodes
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
FKDE based on PDDP
1 If the present node is a leaf then evaluate KDE.
If D > R => return 0lbElse if D > R => return 0Else process
rec_err Else process both childrenif D > R => process left child
vnmr
vnmr
Target data vector
D
D
2nd Direction
1st Direction
Projection of terget point
Projection of source data
Source data vectors
rec_err
Dlb
(Process child) (Which child to process)
n(t )
Projected target data vectorProjected mean vector
(a) Target point is outside
Drec_errT is within left childDrec_err
Return 0Else
Process its children
If R <
If D > RReturn 0
For the right child
lb
If R < Drec_errElse Return 0
Else Process its children
D
T
1st Direction
2nd Direction(Boundary for Partition)
Dlb
If D < R process both children
(b) Target point is inside
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
FKDE based on PDDP
1 If the present node is a leaf then evaluate KDE.
2 Is there any need to go further for the children of the present node ?
If D > R => return 0lbElse if D > R => return 0Else process
rec_err Else process both childrenif D > R => process left child
vnmr
vnmr
Target data vector
D
D
2nd Direction
1st Direction
Projection of terget point
Projection of source data
Source data vectors
rec_err
Dlb
(Process child) (Which child to process)
n(t )
Projected target data vectorProjected mean vector
(c) Target point is outside
Drec_errT is within left childDrec_err
Return 0Else
Process its children
If R <
If D > RReturn 0
For the right child
lb
If R < Drec_errElse Return 0
Else Process its children
D
T
1st Direction
2nd Direction(Boundary for Partition)
Dlb
If D < R process both children
(d) Target point is inside
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
FKDE based on PDDP
1 If the present node is a leaf then evaluate KDE.
2 Is there any need to go further for the children of the present node ?
3 Which child node (left or right or both) of the present node to process further ?
If D > R => return 0lbElse if D > R => return 0Else process
rec_err Else process both childrenif D > R => process left child
vnmr
vnmr
Target data vector
D
D
2nd Direction
1st Direction
Projection of terget point
Projection of source data
Source data vectors
rec_err
Dlb
(Process child) (Which child to process)
n(t )
Projected target data vectorProjected mean vector
(e) Target point is outside
Drec_errT is within left childDrec_err
Return 0Else
Process its children
If R <
If D > RReturn 0
For the right child
lb
If R < Drec_errElse Return 0
Else Process its children
D
T
1st Direction
2nd Direction(Boundary for Partition)
Dlb
If D < R process both children
(f) Target point is inside
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Comparison between the FKDE algorithms
(g) Image consideredfor creating the data set
1
2
3
4
(i,j)
(i−1,j)
(i,j+1)
(i,j+1)
{1,2,3} X RGB ==> 9 dimensions{1,2,3,4} X RGB ==> 12 dimensions
{1,2} X RGB ==> 6 dimensions{1} X RGB ==> 3 dimensions
(h) Creation of the dataspace
Figure: Data set creation for FKDE analysis
Table: Time comparison
Dimension 3 6 9 12KDE: Time (sec) 981.78 1204.77 2529.22 2668.58
PDDP-FKDE: Time (sec) 11.3 23.39 52.55 65.79KD-FKDE: Time (sec) 22.55 33.88 110.62 228.44
IFGT-FKDE: Time (sec) 399.79 425.45 209.33 384.08
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Comparison between the FKDE algorithms
Table: Comparative analysis of FKDE algorithms
FKDE Dimension Maximum Maximum Relative Radiusalgorithms probability Error Error threshold
PDDP-FKDE 3 4.33531e − 05 0.0003e − 04 0.0007 14.54456 1.32977e − 10 0.0000e − 10 0.0000 28.16229 9.64769e − 18 0.0014e − 18 0.0001 47.569312 1.71367e − 23 0.0000e − 25 0.0000 59.9263
KD-FKDE 3 4.33531e − 05 0.0005e − 04 0.0011 14.54456 1.32977e − 10 0.0001e − 10 0.0000 28.16229 9.64769e − 18 0.0014e − 18 0.0001 47.569312 1.71367e − 23 0.0000e − 25 0.0000 59.9263
IFGT-FKDE 3 4.33531e − 05 0.2618e − 04 0.6040 18.18076 1.32977e − 10 0.5389e − 10 0.4053 35.20289 9.64769e − 18 0.4183e − 18 0.0434 59.461612 1.71367e − 23 0.132e − 25 0.0007703 74.9079
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Computationally efficient Texture synthesis algorithm with FKDEalgorithms
Problems
1 how to include the effect of Wq (the temperature field) within the PDDP-based treestructure for the implementation of FKDE, and
2 there are two joint densities corresponding to {Yq ,Xq} and Xq; therefore, itrequires two FKDE structure, which is not computationally efficient.
Inclusion of Wq
• Starting State:{Wq,i = 0} ⇒ P(Xq; Wq) = constant⇒ P(Xq) is uniform⇒ Each Xs has equal effect upon Xq⇒ Every Xs should be considered in the KDE⇒ Rnew is very large
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Computationally efficient Texture synthesis algorithm with FKDEalgorithms
Problems
1 how to include the effect of Wq (the temperature field) within the PDDP-based treestructure for the implementation of FKDE, and
2 there are two joint densities corresponding to {Yq ,Xq} and Xq; therefore, itrequires two FKDE structure, which is not computationally efficient.
Inclusion of Wq
• Starting State:{Wq,i = 0} ⇒ P(Xq; Wq) = constant⇒ P(Xq) is uniform⇒ Each Xs has equal effect upon Xq⇒ Every Xs should be considered in the KDE⇒ Rnew is very large
• Ending State:{Wq,i = 1} ⇒ P(Xq; Wq) = P(Xq)⇒ Rnew = R
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Computationally efficient Texture synthesis algorithm with FKDEalgorithms
Problems
1 how to include the effect of Wq (the temperature field) within the PDDP-based treestructure for the implementation of FKDE, and
2 there are two joint densities corresponding to {Yq ,Xq} and Xq; therefore, itrequires two FKDE structure, which is not computationally efficient.
Inclusion of Wq
• Starting State:{Wq,i = 0} ⇒ P(Xq; Wq) = constant⇒ P(Xq) is uniform⇒ Each Xs has equal effect upon Xq⇒ Every Xs should be considered in the KDE⇒ Rnew is very large
• Ending State:{Wq,i = 1} ⇒ P(Xq; Wq) = P(Xq)⇒ Rnew = R
Rnew =Rcq
abs(vn −mr ) ≤ Rnew
⇒ abs(vn −mr ) ≤ Rcq
⇒ abs(vn −mr )cq ≤ R
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Results with comparisons
Original NNMRF IFGT kd−tree Proposed
D102
D49
D20
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Results with comparisons
Original NNMRF IFGT Proposedkd−tree
D53
D104
D4
D82
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Results with comparisons
Original NNMRF IFGT kd−tree Proposed
D110
D60
D93
D97
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Results with comparisons
Table: Time taken in texture synthesis: input texture size 128 × 128 and output texture size256 × 256
NNMRF C-FKDE KD-FKDE PDDP-FKDEhours 8 5 8 6
minutes 7 55 34 0seconds 39 12 56 41
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Maximum Log-Pseudo-likelihood
LPL =∑
s∈Sin
log[P(Ys |Xs)]
For 1st order neighborhood system For 2nd order neighborhood system
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
How to estimate MLPL ?
• parametric MRF model
• non-parametric MRF model: what should be the kernel ?• Gaussian kernel: as used in [Paget and Longstaff(1998)]• Dirac-delta kernel• Some other solution
Effect of kernel upon the MLPL estimate
LPL is not saturating rather it is increasing
LPL
Order
−80000
−70000
−60000
−50000
−40000
−30000
−20000
−10000
0 5 10 15 20 25 30 35 40
D102: Near regularD104: Near regular
D110: StochasticD60: StochasticD93: Stochastic
LPL is getting saturated before 2 ordernd
Order
LPL
−450
−400
−350
−300
−250
−200
−150
−100
−50
0
0 1 2 3 4
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Why does the original LPL measure, not saturate ?
Why ?
• original LCPDF: p(Yq |Xq) =∑
s∈S Kh (Ys−Yq)Kh (Xs−Xq)∑q∈S Kh (Xs−Xq)
• Changing terms with order:• hy = σy N−1/(d+4): changes due to change in d and N, with order• In case of LCPDF the normalizing term becomes:
√2πhy ;
• Moreover, hy also affect the argument within the exponential term.
• One can not neglect this term.
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
A new definition for LCPDF
p(Ys |Xs) =
∑q∈S δ(Ys − Yq)Kh(Xs − Xq)∑
q∈S Kh(Xs − Xq)
Two reasons in the support for this new definition
• From the texture synthesis algorithm point of view
• From a numerical point of view
0 50 100 150 200 250 3000
0.002
0.004
0.006
0.008
0.01
0.012
0.014
0.016
0.018
Gray Levels
Pro
babi
lity
calc
ulat
ed w
ith G
auss
ian
kern
el
D104 Near Regular Texture
D110 Stochastic Texture
Probability = 0.0002963
probaility = 0.003544
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Results: D104
D104
2 4 6 8
10 14 16
18 20 22 24
12
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Results: D9
D9
31 5 7
11 13 15
17 19 21 23
9
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Results for Near-regular textures
Original NNMRF Our Synthesis Algorithm
D104 D20
D22 D34
o = 12 o = 18
0 = 13 o = 17
Original NNMRF Our Synthesis Algorithm
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Results for Stochastic textures
Original NNMRF Our Synthesis Algorithm
D4 D9
D93 D97
O = 10 O = 9
O = 16O = 9
Original NNMRF Our Synthesis Algorithm
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Results for Some other textures
Original NNMRF Our Synthesis Algorithm
D53 D55
D80 D82
O = 17 O = 14
O = 12
Original NNMRF Our Synthesis Algorithm
O = 14
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Problem Definition
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Problem Definition
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Problem Definition
Texture synthesis
HOW ?
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Motivation
Applications of Inverse Texture Synthesis
• Understanding of Textures
• Content-based image/video retrieval
• Perceptual Image/Video compression• Computer Vision Tasks
• Perceptual understanding of textures within the image• Creation of animation – Collecting information from natural images/sequences• Perceptual Understanding of temporal texture – such as, dance sequence, walk
sequence, music sequence etc.
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Definition of Objective Functions
According to N-MRF model
• Distance between two LCPDF’s evaluated w.r.t. both input and output texturepatches.
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Definition of Objective Functions
According to N-MRF model
• Distance between two LCPDF’s evaluated w.r.t. both input and output texturepatches.
• What distance function ?
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Definition of Objective Functions
According to N-MRF model
• Distance between two LCPDF’s evaluated w.r.t. both input and output texturepatches.
• What distance function ?
• Computationally expensive
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Definition of Objective Functions
According to N-MRF model
• Distance between two LCPDF’s evaluated w.r.t. both input and output texturepatches.
• What distance function ?
• Computationally expensive
According to N-MRF model: Intuitively
• Size of the output patch
• Do the input neighborhood vectorsexist within output patch ?
• M × N
• 1|Sin |∑
s∈Sinmin{||Xs − Xq ||2, where
q ∈ Sout }
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Problem with these two objective functions
Neighborhood
A B
Scaled up versions of solutions
approximatelysame
difficult to find within "B"Deformation/variation within "A"
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Three objectives
• Say Sout = {s ∈ Sin, s.t., (si − i)2 ≤ M2 and (sj − j)2 ≤ N2}• Define S {i,j,M,N}in = S − Sout
• First objective finds neighborhood from input texture within the output texture
• Second objective finds neighborhood from output texture within the input texture,excluding the part of Sout
F1 =1|Sin |
∑
s∈Sin
min{||Xs − Xq ||2; q ∈ Sout }
F2 =1|Sout |
∑
q∈Sout
min{||Xq − Xs ||2; s ∈ S {i,j,M,N}in }
F3 = M × N
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Multi-objective Framework
minx
f1(x)f2(x)...
fm(x)
such thatinequality constraints: gj(x) ≥ 0, j = 1, 2, ..., J
equality constraints: hk (x) = 0, k = 1,2, ...,K
solution space: xLi ≤xi ≤ xU
i , i = 1, 2, ...,N
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Multi-objective Framework
minx
f1(x)f2(x)...
fm(x)
such thatinequality constraints: gj(x) ≥ 0, j = 1, 2, ..., J
equality constraints: hk (x) = 0, k = 1,2, ...,K
solution space: xLi ≤xi ≤ xU
i , i = 1, 2, ...,N
DominationA vector x ∈ RN is said to dominate y ∈ RN if both the conditions stated below holdtrue:
fi(x) ≤ fi(y), ∀i ∈ [1 . . .m]
∃ j ∈ [1 . . .m], such that, fj(x) < fj(y)
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Pareto-optimal Front
F1
2F
2FF1
F1
2F
2nd
obje
ctiv
e fu
nctio
n
1st objective function
Worst in Best in
Best in Worst in
All are optimalsolutions
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Genetic Algorithm
Why not classical optimization algorithms
F1 =1|Sin |
∑
s∈Sin
min{||Xs − Zt ||2; t ∈ Sout }
F2 =1|Sout |
∑
t∈Sout
min{||Zt − Xs ||2; s ∈ S {i,j,M,N}in }
F3 = M × N
Genetic Algorithm
• It is an intuitive algorithm, biologically inspired,
• Based upon the phylosophy of survival of the fittest
• Crossover, mutation operators
• There are many algorithms, we choose NSGA [Srinivas and Deb(1994)]
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Redundancy of objectives
F1 F2 F3
F1 + − −F2 − + +F3 − + +
Table: Conflict matrix: In each case of Fabric.0014, D22, Fabric.0009, and Grass textures the sametrait has been observed.
600 700 800 9000
50
100
150
200
250
300
F1
F2
(a) F1 conflicts F2
0 2000 4000 6000 80000
50
100
150
200
250
300
F3
F2
(b) non-conflicting F2 and F3
0 2000 4000 6000 8000600
650
700
750
800
850
900
F3
F1
(c) F1 conflicts F3
Figure: Two-dimensional views of Pareto-optimal front for the Fabric.0014 texture: the positivity ornegativity of the cross correlation between the objective functions can be understood
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Results: D22
2500 3000 3500 40000
100
200
300
400
1
2
3 4 5
F1
F2
ExtractedExemplerOriginal Textured
Region
ExtractedExempler
SynthesisResult
SynthesisResult
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Results: Fabric.0009
F1
1st solution
2nd
3rd
Original Textured Region
F2
500
1000
1500
2000
2500
3000
1900 1950 2000 2050 2100 2150 2200 2250
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Order Estimation from Fourier DomainReduction of Computational complexityOrder Estimation : RevisitedInverse Texture Synthesis
Results: Grass
F1
Synthesized textures fromthe extracted solutions
1st solution
2nd solution
3rd solution
Original Textured Region
F2
0
20
40
60
80
100
120
140
160
180
200
260 280 300 320 340 360 380 400 420
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Conclusion• The problem of Order estimation• The problem of computational complexity reduction
• With the incorporation of Dimensionality Reduction methodology, e.g., PCA• With fast estimation of Kernel Density Estimation with an improvised data structure
• An inverse application of texture synthesis with NMRF model• Objective functions• Analysis of objective functions• Multi-objective framework
Possible Future Directions• Order estimation for in-homogeneous textures or globally varying textures
• three-dimansional variation of surface• structural variation• time specific variation
• How to incorporate a control field within the texture analysis
• How to choose a particular solution from the multi-objective framework, dependingupon the application in hand
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
D L Boley.Principal direction divisive partitioning.Data Mining and Knowledge Discovery, 2(4):325–344, 1998.
Alexander G. Gray and Andrew W. Moore.Nonparametric density estimation: toward computational tractability.IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(8):1344–1348, January 2003.
R. Paget and I. D. Longstaff.Texture synthesis via a noncausal nonparametric multiscale markov random field.IEEE Transactions on Image Processing, 7(6):925–931, June 1998.
David W Scott.Multivariate density estimation - theory, practice and visualization.Wiley interscience, 1992.
N. Srinivas and Kalyanmoy Deb.Multiobjective optimization using nondominated sorting in genetic algorithms.Evolutionary Computation, 2:221–248, 1994.
Changjiang Yang, Ramani Duraiswami, Nail A Gumerov, and Larry Davis.Improved fast gauss transform and efficient kernel density estimation.In Proceedings. Ninth IEEE International Conference on Computer Vision, 2003.
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms
OutlinePh.D. Research Work
Conclusion and Possible Future Directions
Kai Zhang, Ming Tang, and James T Kwok.Applying neighborhood consistency for fast clustering and kernel densityestimation.In Proceedings of the 2005 Computer Society Conference on Computer Visionand Pattern Recognition (CVPR’05), volume 2, 2005.
Arnab Sinha [email protected] Fast NMRF based texture synthesis algorithms