Fast BTF Texture ModellingM. Haindl, J. Filip

Institute of Information Theory and AutomationAcademy of Sciences of the Czech Republic

Prague, Czech Republic, CZ182 08

Abstract— This paper presents a novel fast model-based algo-rithm for realistic multispectral BTF texture modelling poten-tially capable of direct implementation inside the graphical cardprocessing unit. The algorithm starts with range map estimationof the BTF texture followed by the spectral and spatial factori-sation of an input multispectral texture image. Single orthogonalmonospectral band-limited factors are independently modelled bytheir dedicated Gaussian Markov random field models (GMRF).We estimate an optimal contextual neighbourhood and param-eters for each GMRF. Finally single synthesised band-limitedfactors are collapsed into the fine resolution monospectral imagesand using the inverse Karhunen-Loeve transformation we obtainthe smooth multispectral texture. Both multispectral and rangeinformation is combined in a bump mapping or alternativelya displacement mapping filter of the rendering hardware. Thepresented model offers huge BTF texture compression rationwhich cannot be achieved by any other sampling-based BTFtexture synthesis method.

I. I NTRODUCTION

Photo realism in virtual or mixed reality scenes cannotbe accomplished without nature-like colour textures coveringvisualised scene objects. These textures can be either smoothor rough (also referred as the bidirectional texture function -BTF). The rough textures which have rugged surfaces do notobey Lambertian law and their reflectance is illumination andview angle dependent as it is illustrated in Fig.1. Textures canbe either digitised natural textures or textures synthesised froman appropriate mathematical model. The former simplistic op-tion suffers among others with extreme memory requirementsfor storage of a large number of digitised cross-sectioned slicesthrough different material samples. Sampling solution becomeunmanageable for rough textures which require to store thou-sands of different illumination and view angle samples forevery texture.

Several intelligent sampling methods [1], [2], [3], [4], [5],[6], were proposed with the aim to diminish these huge mem-ory requirements. The method [1] constructs texture in coarse-to-fine fashion, preserving conditional distribution of filteroutputs over multiple scales, while another multiscale method[4] uses histograms of filter responses. The quilting method [3]is based on the overlapping tiling and subsequent minimumerror boundary cut. Similarly the algorithm [5] uses regulartiling combined with a deterministic chaos transformation. Allthese methods are based on some sort of original small texturesampling and the best of them produce very realistic synthetictextures. However these methods require to store thousandsimages for every combination of viewing and illuminationangle of the original target texture sample, they often produce

visible seams, some of them are computationally demandingand they cannot generate textures unseen (unstored) by thealgorithm.

Fig. 1. Relationship between illumination and viewing angles withintexture coordinate system.

Synthetic textures are far more flexible, extremely com-pressed (few parameters have to be stored only), they may beevaluated directly in procedural form and can be designed tomeet certain constraints or properties, so that they can be usedto fill an infinite texture space without visible discontinuities.

Several monospectral smooth texture modelling approacheswere published, e.g., [7], [8],[9], [10], [11], among them alsofew colour models, e.g., [12], [13], [14], [15], [16] and somesurvey articles are available [17], [18] as well. Previous colourtexture modelling methods either mapped colour to gray tones[19] sacrificing considerable amount of image informationor used 3D models [13],[14] with corresponding problemsof robust nonlinear estimation of large amount of modelparameters.

Modelling multispectral images requires three dimensionalmodels however if we are willing to sacrifice some informationa 3D model can be approximated with a set of much simpler2D models without compromising its visual realism. Amongsuch possible models the Gaussian Markov random fields areappropriate for texture modelling not only because they donot suffer with some problems of alternative options (see[20], [17], [18], [15] for details) but they are also easy tosynthesise and still flexible enough to imitate a large set ofnatural and artificial textures. While the random field basedmodels quite successfully represent high frequencies presentin natural textures low frequencies are much more difficult forthem. One possibility how to overcome this drawback is to usea multiscale random field model.

Multiple resolution decomposition (MRD) such as Gaus-sian/Laplacian pyramids, wavelet pyramids or subband pyra-

mids [21] present efficient method for the spatial informationcompressing. The hierarchy of different resolutions of animage provides a transition between pixel-level features andregion or global features and hence such a representationsimplify modelling a large variety of possible textures. Un-fortunately Markov random fields in general, and GaussianMarkov random fields in particular are not invariant to multipleresolution decomposition (MRD) even for simple MRD likesubsampling and the lower-resolution images generally losetheir Markovianity. Fortunately we can avoid computationallydemanding approximations [22] of a non-Markov multigridrandom field by Markov random fields because there is noneed to transfer information between single spatial factorshence it is sufficient to analyse and synthesise each resolutioncomponent independently.

We propose a novel algorithm for efficient rough texturemodelling which combines an estimated range map with syn-thetic multiscale Markov random field based generated smoothtexture. The texture visual appearance during changes ofviewing and illumination conditions simulated using the bump/ displacement mapping technique. The obvious advantage ofthis solution is the possibility to use hardware support of bump/ displacement map techniques in contemporary visualisationhardware.

The rest of the paper is organised as follows. After adescription of range map estimation technique, we discussthe underlaying multiscale Markovian smooth texture modeltogether with its parameter estimation and synthesis solutions.Results are reported in Section 3, followed by a conclusionand discussion about future work in Section 4.

II. RANGE MAP ESTIMATION

There are several methods for determining a surface rangemap. The most accurate range map can be estimated by directmeasurement of the observed surface using correspondingrange cameras (e.g. laser or structured light based). How-ever this method requires special hardware and measurementmethodology. Hence several alternative approaches for rangemap estimation from surface images were developed. One ofthem is aPhotometric Stereowhich estimates surface rangemap from at least three images obtained for different positionof illumination source while the camera position is fixed. Itsobvious disadvantage is the requirement to obtain at least threemutually registered images. An alternative option is to use amethod from theShape from Shadinggroup.

The range map estimation of texture images within this pa-per was performed by shape from shading algorithm publishedby Frankot & Chellappa [23]. This method exploits the factthat image intensityYr of each pixel observed in texture imageis given according to Lambertian (in our implementation)reflectance mapR :

Yr = R(zr1 , zr2 ,L,C, ρ) (1)

= ρL1zr1 + L2zr2 + L3√

L12 + L2

2 + L32√

1 + z2r1

+ z2r2

where zr1 and zr2 are surface slopes in directionsr1 andr2, L is vector from surface to illumination source,C isvector from surface to the camera andρ is albedo of observedmaterial. The algorithm ignores multiple reflections, assumesknown vectors C,L and spatially invariant reflectance map.The advantage of this method is fast range map estimationfrom single surface image illuminated by single light sourcefrom known position. In comparison with other methods, as forexample Horn & Brooks [24], this algorithm includes constrainwhich enforces integrability of surface slopes (2). These slopesare then updated in each iteration step.

∂2Zr

∂r1∂r2=

∂2Zr

∂r2∂r1(2)

where Zr is the unknown surface height in the locationr = (r1, r2). This integrability condition is performed in theFourier coefficient representation as follows:

C(u, v) =12π

∫ +∞

−∞

∫ +∞

−∞Zr · ej(ur1+vr2)dr1dr2

Zr =12π

∫ +∞

−∞

∫ +∞

−∞C(u, v) · e−j(ur1+vr2)dudv

Based on these assumptions the reconstruction of surfaceheight is performed in the Fourier space with coefficientsobtained by the above equation. Frankot & Chellappa proved[23] that the algorithm maps closed convex sets into closedconvex sets in each iteration step and thus the estimated sur-face slopes are integrable. Finally the inverse transformationleads to a surface range map. The Lambertian reflectanceassumption may deteriorate the range map estimation forsome non-Lambertian texture surfaces, although results in [23]demonstrate still acceptable estimates even in these cases.

III. SMOOTH TEXTURE MODEL

Modelling general multispectral (e.g., colour) texture im-ages requires three dimensional models. If a 3D data spacecan be factorised then these data can be modelled using aset of less-dimensional 2D random field models, otherwise itis necessary to use some 3D random field model. Althoughfull 3D models allows unrestricted spatial-spectral correlationmodelling its main drawback is large amount of parameters tobe estimated and in the case of Markov models (MRF) alsothe necessity to estimate all these parameters simultaneously[13]. The factorisation alternative is attractive because it allowsusing simpler 2D data models with less parameters (one thirdin the three-spectral case of colour textures).

Spectral factorisation using the Karhunen-Loeve expansiontransforms the original centred data spaceY defined on therectangularM × N finite toroidal latticeI into a new dataspace with K-L coordinate axesY . These new basis vectorsare the eigenvectors of thed × d second-order statisticalmoments matrix Φ = E{Yr,•Y

Tr,•} where the multiindex

r has two componentsr = [r1, r2], the first component isrow and and the second one column index, respectively. Theprojection of random vector Yr,• (the notation• has the

meaning of all possible values of the corresponding index)onto the K-L coordinate system uses the transformation matrixT = [uT

1 , . . . , uTd ]T which has single rowsuj that are

eigenvectors of the matrixΦ.

Yr,• = T Yr,• (3)

Components of the transformed vectorYr,• (3) are mutuallyuncorrelated and if Yr,• are Gaussian they are also inde-pendent hence each transformed monospectral factor can bemodelled independently of remaining spectral factors. Texturemodelling does not require computationally demanding MRDapproximations (e.g. [22]) because it does not need to propa-gate information between different data resolution levels. It issufficient to analyse and subsequently generate single spatialfrequency bands without assuming a knowledge of some MRF-type global multi-grid model. Input multispectral image isfactorised using (3) intod monospectral imagesY•,i fori = 1, . . . , d. These components are further decomposed intoa multi-resolution grid and each resolution data are indepen-dently modelled by their dedicated GMRF. Each one generatesa single spatial frequency band of the texture. An analysedtexture is decomposed into multiple resolutions factors usingLaplacian pyramid and the intermediary Gaussian pyramidY

(k)•,i which is a sequence of images in which each one

is a low-pass down-sampled version of its predecessor. TheGaussian pyramid for a reduction factorn is

Y(k)r,i =↓nr (Y (k−1)

•,i ⊗ w) k = 1, 2, . . . , (4)

whereY

(0)•,i = Y•,i ,

↓n denotes down-sampling with reduction factorn and⊗ is theconvolution operation. The Laplacian pyramidY (k)

r,i containsband-pass components and provides a good approximation tothe Laplacian of the Gaussian kernel. It can be constructed bydifferencing single Gaussian pyramid layers:

Y(k)r,i = Y

(k)r,i − ↑

nr (Y (k+1)

•,i ) k = 0, 1, . . . , (5)

where↑n is the up-sampling with an expanding factorn.Single orthogonal monospectral components are thus de-

composed into a multi-resolution grid and each resolutiondata are independently modelled by their dedicated GaussianMarkov random field model (GMRF) as follows. The Markovrandom field (MRF) is a family of random variables with ajoint probability density on the set of all possible realisationsY of the latticeI, subject to following conditions:

p(Y•,i) > 0, ∀Y , (6)

and

p(Yr,i |Ys,i∀s ∈ I \ {r}) = p(Yr,i |Ys,i∀s ∈ Ir,i) , (7)

and Ir,i is a contextual support set of thei-th monospectralfield.

If the local conditional density of the MRF model (8) isGaussian, we obtain the continuous Gaussian Markov randomfield model (GMRF):

p(Yr, |Ys,i∀s ∈ Ir,i) =

(2πσ2i )−

12 exp{−1

2σ−2

i (Yr,i − µr,i)2} , (8)

where the mean value is

E{Yr,i |Ys,i∀s ∈ I \ {r}} = µr,i =

µr,i +∑

s∈Ir,i

as,i(Yr−s,i − µr−s,i) (9)

andσi, as,i ∀s ∈ Ir,i are unknown parameters. The 2D MRFmodel can be expressed as a stationary non-causal correlatednoise driven 2D autoregressive process:

Yr,i =∑

s∈Ir,i

as,iYr−s,i + eri,i (10)

where the noiseer,i is random variable with zero mean. Theer,i noise variables are mutually correlated

Rei= E{er,ier−s,i}

=

{σ2

i if s = (0, 0),−σ2

i as,i if s ∈ Ir,i,0 otherwise.

(11)

Correlation functions have the symmetry propertyE{er,ier+s,i} = E{er,i er−s,i} hence the neighbourhoodsupport set and their associated coefficients have to besymmetric, i.e. s ∈ Ir,i ⇒ −s ∈ Ir,i andas,i = a−s,i .

A. Parameter Estimation

The selection of an appropriate GMRF model support isimportant to obtain good results in modelling of a givenrandom field. If the contextual neighbourhood is too small itcan not capture all details of the random field. Inclusion of theunnecessary neighbours on the other hand add to the computa-tional burden and can potentially degrade the performance ofthe model as an additional source of noise. We use hierarchicalneighbourhood systemIr,i, e.g., the first-order neighbourhoodis Ir,i = {r − (0, 1), r + (0, 1), r − (1, 0), r + (1, 0)}, etc.An optimal neighbourhood is detected using the correlationmethod [25] favouring neighbours locations corresponding tolarge correlations over those with small correlations.

Parameter estimation of a MRF model is complicated bythe difficulty associated with computing the normalisationconstant. Fortunately the GMRF model is an exception wherethe normalisation constant is easy obtainable however eitherBayesian or ML estimate requires iterative minimisation ofa nonlinear function. Therefore we use the pseudo-likelihoodestimator which is computationally simple although not effi-cient. The pseudo-likelihood estimate foras parameters hasthe form

γa,i = [as,i ∀s ∈ Ir,i] =

[∑∀r∈I

XTr,iXr,i]−1

∑∀r∈I

XTr,iYr,i , (12)

where Xr,i = [Yr−s,i ∀s ∈ Ir,i] and

σ2i =

1MN

MN∑r=1

(Yr,i − γa,iXTr,i)

2 . (13)

B. Model Synthesis

Several possibilities exist [17] for a finite lattice GMRF syn-thesis. The most effective synthesis method uses the discretefast Fourier transformation. GMRF can be generated [9] fromY•,i = F−1{Y•,i} + Ui, where Ui the mean vector of thewhole filed and Y•,i is generated from the Gaussian gen-eratorN (0, NMSY (r, i)) . SY (r, i) is the associated powerspectrum [20] andN ×M is the underlying generated latticesize. Single GMRF models synthesise spatial frequency bandsof the texture. Each monospectral fine-resolution component isobtained from the pyramid collapse procedure (inversion pro-cess to (4),(5)). Finally the resulting synthesised multispectraltexture is obtained from the set of synthesised monospectralimages using the inverse K-L transformation:

Yr,• = T−1Yr,• (14)

IV. RESULTS

We have tested the algorithm on BTF colour textures fromthe CUReT [26] database, several our and the University ofBonn (Fig.2) BTF measurements. Figs.2,4 show corduroy, up-holstery and knitwear textures synthesised using the presentedmethod for three illumination angles. For comparison, theincluded corresponding smooth texture synthesis results usingthe same sets of models demonstrate clear improvement ofthe presented approach. All these figures demonstrate alsothe colour quality of the model. The Fig.3 demonstratesthe improvement achieved using the displacement mappingtechnique (the leftmost three examples) and the comparisonbetween measured and estimated range map results. Our syn-thesised images manifest comparable colour quality with themuch more complex 3D models [14]. The multi-scale modelsdemonstrate [12] their clear superiority over their single-scale counterparts while the colour quality is comparablebetween single-scale and multi-scale alternative models. Thesynthesised smooth intensity image is identical for all viewingangles but differs for illumination angle changes.

V. SUMMARY AND CONCLUSIONS

Our testing results of the algorithm on all available BTFdata are encouraging. Some synthetic textures reproduce givenmeasured texture images so that both natural and synthetictexture are almost visually indiscernible. Even the not sosuccessful results can be used for the preattentive BTF texturesapplications. The overwhelming amount of original colourtones were reproduced realistically in spite of restricted spec-tral modelling power of the model. The multi-scale approach

is more robust and allows far better results than the single-scale one if the synthesis model is inadequate (lower ordermodel, non stationary texture, etc.). The proposed methodallows huge compression ratio (unattainable by alternativeintelligent sampling approaches) for transmission or storingtexture information while it has still moderate computationcomplexity. The method does not need any time-consumingnumerical optimisation like for example the usually employedMarkov chain Monte Carlo methods. The replacement of thebump filter technique with the novel graphics cards feature -the displacement mapping is expected to further significantlyimprove the visual quality of our algorithm results.

The presented method is based on the estimated model incontrast to prevailing intelligent sampling type of methods,and as such it can only approximate realism of the originalmeasurement. However it offers unbeatable data compressionratio (tens of parameters per texture only), easy simulation ofeven non existing (previously not measured) BTF textures andfast seamless synthesis of any texture size.

ACKNOWLEDGEMENTS

This research was supported by the EC project no. IST-2001-34744 RealReflect, grant No. A2075302 of the GrantAgency of the Academy of Sciences CR and partially by thegrant MSMT No. ME567 MIXMODE. The authors wish tothank R. Klein of the University of Bonn for providing uswith the corduroy BTF measurements.

REFERENCES

[1] J. De Bonet, “Multiresolution sampling procedure for analysis andsynthesis of textured images,” inProc. SIGGRAPH 97. ACM, 1997,pp. 361–368.

[2] A. A. Efros and T. K. Leung, “Texture synthesis by non-parametricsampling,” in Proc. Int. Conf. on Computer Vision (2), 1999, pp.1033–1038. [Online]. Available: citeseer.nj.nec.com/efros99texture.html

[3] W. Efros, A.A. Freeman, “Image quilting for texture synthesisand transfer,” inSIGGRAPH 2001, Computer Graphics Proceedings,E. Fiume, Ed. ACM Press / ACM SIGGRAPH, 2001, pp. 341–346.[Online]. Available: citeseer.nj.nec.com/efros01image.html

[4] J. Heeger, D.J. Bergen, “Pyramid based texture analysis/synthesis,” inProc. SIGGRAPH 95. ACM, 1995, pp. 229–238.

[5] Y. X. B. G. H. Shum, “Chaos mosaic: Fast and memory efficient texturesynthesis,” Redmont, Tech. Rep. MSR-TR-2000-32, 2000.

[6] J. Dong and M. Chantler, “Capture and synthesis of 3d surface texture,”Texture 2002, vol. 1, pp. 41–45, 2002.

[7] J. Besag, “Spatial interaction and the statistical analysis of latticesystems,”Journal of the Royal Statistical Society, Series B, vol. B-36,no. 2, pp. 192–236, February 1974.

[8] R. Chellappa, “Two-dimensional discrete gaussian markov random fieldmodels for image processing,” inProgress in Pattern Recognition 2,L. Kanal and A. Rosenfeld, Eds. North-Holland: Elsevier, 1985, pp.79–112.

[9] R. Kashyap, “Analysis and synthesis of image patterns by spatialinteraction models,” inProgress in Pattern Recognition 1, L. Kanal andA.Rosenfeld, Eds. North-Holland: Elsevier, 1981.

[10] J. Portilla and E. Simoncelli, “A parametric texture model based onjoint statistics of complex wavelet coefficients,”International Journalof Computer Vision, vol. 40, no. 1, pp. 49–71, 2000.

[11] J. Grim and M. Haindl, “Texture modelling by discrete distributionmixtures,” Computational Statistics & Data Analysis, vol. 41, no. 3-4,pp. 603–615, 2003.

[12] M. Haindl and V. Havlıcek, “Multiresolution colour texture synthesis,”in Proceedings of the 7th International Workshop on Robotics in Alpe-Adria-Danube Region, K. Dobrovodsky, Ed. Bratislava: ASCO Art, June1998, pp. 297–302.

light → φi = 0o light ↓ φi = 100o light ← φi = 180o range map (measured)

range map (estimated)

Fig. 2. A synthetic rough corduroy texture example using the bump map technique. The upper row contains original texture measurement, the second rowcontains corresponding smooth synthetic textures and the bottom row demonstrates the resulting rough textures, respectively.

φi = 0o φi = 90o φi = 180o φi = 0o φi = 90o φi = 180o

Fig. 3. Synthetic corduroy textures using the displacement mapping technique. The leftmost three textures exploit the measured range map while the remainingones use the estimated range map.

[13] J. Bennett and A. Khotanzad, “Multispectral random field models forsynthesis and analysis of color images,”IEEE Trans. on Pattern Analysisand Machine Intelligence, vol. 20, no. 3, pp. 327–332, March 1998.

[14] ——, “Maximum likelihood estimation methods for multispectral ran-dom field image models,”IEEE Trans. on Pattern Analysis and MachineIntelligence, vol. 21, no. 3, pp. 537–543, 1999.

[15] M. Haindl and V. Havlıcek, “A multiresolution causal colour texturemodel,” in Advances in Pattern Recognition, Lecture Notes in ComputerScience 1876, F. J. Ferri, J. M. I. nesta, A. Amin, and P. Pudil, Eds.Berlin: Springer-Verlag, August 2000, ch. 1, pp. 114 –122.

[16] M. Haindl and V. Havlıcek, “A multiscale colour texture model,” inPro-ceedings of the 16th International Conference on Pattern Recognition,R. Kasturi, D. Laurendeau, and C. Suen, Eds. Los Alamitos: IEEEComputer Society, August 2002, pp. 255–258.

[17] M. Haindl, “Texture synthesis,”CWI Quarterly, vol. 4, no. 4, pp. 305–331, December 1991.

[18] ——, “Texture modelling,” inProceedings of the World Multiconferenceon Systemics, Cybernetics and Informatics, B. Sanchez, J. M. Pineda,

J. Wolfmann, Z. Bellahse, and F. Ferri, Eds., vol. VII. Orlando, USA:International Institute of Informatics and Systemics, July 2000, pp. 634–639.

[19] A. Gagalowicz, S. Ma, and C. Tournier-Laserve, “Efficient models forcolor textures,” inProceedings of Int. Conf. Pattern Recognition, Paris,1986, pp. 412–414.

[20] M. Haindl, “Texture synthesis,” Centrum voor Wiskunde en Informatica,Amsterdam, The Netherlands, Tech. Rep. CS-R9139, 1991.

[21] in Multiresolution Image Processing and Analysis, A. Rosenfeld, Ed.Berlin: Springer-Verlag, 1984.

[22] B. Gidas, “A renormalization group approach to image processing prob-lems,” IEEE Transactions on Pattern Analysis and Machine Intelligence,vol. PAMI-11, no. 2, pp. 164–180, 1989.

[23] R. T. F. R. Chellappa, “A method for enforcing integrability in shapefrom shading algorithms,”IEEE Trans. on Pattern Analysis and MachineIntelligence, vol. 10, no. 7, pp. 439–451, 1988.

[24] B. K. P. H. M. J. Brooks, “The variational approach to shape fromshading,”Computer Vision Graphics Image Processing, vol. 33, pp. 174–

light → φi = 0o light ↓ φi = 90o light ← φi = 180o range map

light → φi = 0o light ↓ φi = 90o light ← φi = 180o range map

Fig. 4. Two examples of synthetic rough (BTF) textures (textile, knitwear) using the presented method. The upper rows in each example contain originaltextures (θi = 60o) illuminated with tilt angles 0, 90 (100) and 180 degrees, respectively and the range map obtained from the original. The bottom rowscontain the corresponding synthesised rough textures (the rightmost image is one of smooth synthesised results using the same model).

208, 1986.[25] M. Haindl and V. Havlıcek, “Prototype implementation of the texture

analysis objects,”UTIA AV CR, Praha, Czech Republic, Tech. Rep.1939, 1997.

[26] S. K. J. Dana, K.J. Van Ginneken B. Nayar, “Reflectance and textureof real-world surfaces,”ACM Transactions on Graphics, vol. 18, no. 1,pp. 1–34, January 1999.

[27] L. Liang, C. Liu, Y.-Q. Xu, B. Guo, and H.-Y. Shum, “Real-time texturesynthesis by patch-based sampling,”ACM Transactions on Graphics(TOG), vol. 20, no. 3, pp. 127–150, 2001.

[28] X. Liu, Y. Yu, and H. Shum, “Synthesizing bidirectional texture func-tions,” ACM SIGGRAPH, 2001.

[29] X. Tong, J. Zhang, L. Liu, X. Wang, B. Guo, and H.-Y. Shum,“Synthesis of bidirectional texture functions on arbitrary surfaces,”ACMTransactions on Graphics (TOG), vol. 21, no. 3, pp. 665–672, 2002.