NONPARAMETRIC TEXTURE ANALYSIS WITH SIMPLE SPATIAL OPERATORS

M. PIETIKÄINEN AND T. OJALA

Machine Vision and Media Processing Group, Infotech Oulu,University of Oulu,

P.O. Box 4500, FIN-90401 OULU, FINLAND

email: mkp@ee.oulu.fi skidi@ee.oulu.fi

Abstract - Recently, we have developed a nonparamet-ric approach to texture analysis based on simple spatialoperators like local binary patterns and signed gray lev-el differences. Very good performance has been ob-tained in various texture classification andsegmentation problems. This paper overviews our ap-proach and presents examples to demonstrate its effi-ciency. Our results suggest that complementaryfeatures based on distributions of local spatial patternsand contrast play very important roles in texture dis-crimintation.

1. INTRODUCTION

Texture analysis is important in many applications of com-puter image analysis for classification, detection, or seg-mentation of images based on local spatial variations ofintensity or color. Many different approaches to textureanalysis have been proposed. Among the most widely usedtexture measures are those derived from gray level cooc-currence matrices or difference histograms, “ texture ener-gy” measures obtained by local li near transforms, andfeatures based on multi-channel Gabor filtering or Markovrandom field models [1-3].

Most of the approaches to texture classification quantifytexture measures by single values (means, variances etc.),which are then concatenated into a feature vector. The fea-ture vector is fed to an ordinary statistical pattern recogni-tion procedure or neural network to perform classification.In this way, much of the important information containedin the whole distributions of feature values might be lost.

Recently, we have developed a nonparametric approach totexture analysis based on simple spatial operators like localbinary patterns and signed gray level differences. Verygood performance has been obtained in various textureclassification and segmentation problems, see e.g. [4-13].The specific problems of texture analysis in surface in-spection are discussed in [10].

This paper overviews our approach and presents some ex-amples to demonstrate its eff iciency. The results suggestthat the distributions of local spatial patterns and contrastplay very important complementary roles in texture dis-crimintation.

2. TEXTURE DESCRIPTION WITH SIMPLE SPATIAL OPERATORS

2.1. Local Binary Patterns

Ojala et al. [5] introduced the Local Binary Pattern (LBP)texture operator shown in Fig. 1. The original 3x3 neigh-borhood is thresholded by the value of the center pixel.The values of the pixels in the thresholded neighborhoodare multiplied by the weights given to the correspondingpixels. Finally, the values of the eight pixels are summed toobtain a number for this neighborhood. The LBP histo-gram computed over a region used for texture description.LBP provides us with knowledge about the spatial struc-ture of the local image texture.

LBP is invariant against any monotonic gray scale trans-formation. The method is rotation variant like most exist-ing texture measures. A rotation invariant version of LBPis proposed in [8,11]. LBP does not address the contrast oftexture which is important in the discrimination of sometextures. For this purpose, we can combine LBP with a

1 0 0

1 0

1 0 1

1 0 0

8 0

32 0 128

6 5 2

7 6 1

9 3 7

P1 P2 P3

P4 P0 P5

P6 P7 P8

example thresholded weightspixels

LBP = 1+8+32+128 = 169C = (6+7+9+7)/4 - (5+2+1+3)/4 = 4.5

Fig. 1. Computation of Local Binary Pattern (LBP) andcontrast measure C.

simple contrast measure C also shown in Fig. 1 and con-sider joint occurrences of LBP and C.

LBP and LBP/C perform well also for small image regions(e.g., 16 x 16 pixels), which is very important e.g. in seg-mentation applications.

He and Wang introduced the Texture Spectrum operator[14], which is similar to LBP, but it uses three levels (i.e.two thresholds) instead of two levels used in LBP. Thisleads to a more ineff icient representation and implementa-tion than with LBP (6561 bin values instead of 256), andaccording to our tests the three-level operator does not per-form any better than LBP. The Texture Spectrum operatorusually also needs a user defined delta value for setting thethreshold values, which makes it dependent on the grayscale variance.

2.2. Signed gray-level differences

Recently, Ojala et al. [12] showed that an approach basedon multidimensional distributions of signed gray level dif-ferences of neighboring pixel values is very powerful fortexture classification. The advantages of gray-level differ-ences over the traditional cooccurrence method are: (1) thedifferences fall mainly within a narrower range than thegray levels, due to the high correlation between gray levelsof adjacent pixels, consequently providing a more compactdescription of texture; (2) the signed differences are not af-fected by changes in mean luminance. In comparison tothe commonly used absolute differences, the signed differ-ences contain more information about image texture andconsequently are more powerful.

In our experiments, the classification performance of two-,four-, and eight-dimensional difference distributions hasbeen evaluated. For computing cooccurring differenceswithin 3x3-pixel subimages,

we estimate following distributions,

The LBP operator is similar to p8, when differences are ex-pressed with two levels. If three levels are used, p8 resem-bles the Texture Spectrum operator [14].

The volume of the difference space for an image with Ggray levels equals (2G-1)k, where k=2,4,8, corresponding

to the distribution we are estimating. If we would straight-forwardly describe the difference space with a k-dimen-sional histogram, we would obtain, even with modestvalues of G, very large histograms that are computationallyexpensive and suspect to statistical unreliabilit y.

Instead of reducing G, for example, with simple requanti-zation of each coordinate, we partition the k-dimensionaldifference space using vector quantization [13]. For thispurpose we employ a codebook of N k-dimensional code-words, which have indices n = 0,1,...,N-1. The codebook istrained with the optimized LVQ1 training algorithm [15],by selecting random vectors from each of the samples inthe training set. The small black and white rectangles inFig. 2a correspond to the locations of the codewords, whenthe difference space of p2 is quantized with a codebook of384 codewords. The indices of the codewords correspondto the 384 bins in the histogram (Fig. 2b). Our method ofvector quantization is computationally much simpler thanthat used by Valkealahti and Oja [16].

g4 g2 g3

g5 g0 g1

g6 g7 g8

p2 g1 g0– g2 g0–,( ) (1)

p4 g1 g0– g2 g0– g3 g0– g4 g0–, , ,( ) (2)

p8 g1 g0– g2 g0– … g8 g0–, , ,( ) (3) 0 383bin index

(a)

(b)

Fig. 2. The difference space of p2 and its quantization witha codebook of 384 codewords (a) and the difference histo-gram of a 64x64 texture sample (b). The indices of the 384codewords correspond to the 384 bins in the histogram.

We describe the difference information of a texture samplewith a difference histogram. The mapping from the differ-ence space to a difference histogram is straightforward.Given a particular k-dimensional difference vector, the in-dex of the nearest codeword corresponds to the bin indexin the difference histogram. In other words, a codebook ofN codewords produces a histogram of N bins. The differ-ence histogram of a texture sample is obtained by search-ing the nearest codeword to each vector present in thesample, and incrementing the bin denoted by the index ofthis nearest codeword.

3. EXPERIMENTS WITH TEXTURE CLASSIFICA-TION

The 32 Brodatz textures used in the experiments are shownin Fig. 3 [16,17]. The images are 256x256 pixels in sizeand they have 256 gray levels. Each image was dividedinto 16 disjoint 64x64 samples, which were independentlyhistogram-equalized to remove luminance differences be-tween textures. To make the classification problem morechallenging and generic, three additional samples weregenerated from each sample: a sample rotated by 90 de-grees, a 64x64 scaled sample obtained from the 45x45 pix-els in the middle of the ‘original’ sample, and a sample thatwas both rotated and scaled. Consequently, the classifica-tion problem involved a total of 2048 samples, 64 samplesin each of the 32 texture categories [16].

The texture classifier was trained by randomly choosing,in each texture class, eight ‘original’ samples, togetherwith the corresponding 24 transformed samples, as mod-els. The other half of the data, eight ‘original’ samples andthe corresponding 24 transformed samples in each texture

class, was used for testing the classifier. In the classifica-tion phase a test sample S was assigned to the class of themodel M that maximized the log-likelihood measure

where Sn and Mn correspond to the sample and modelprobabili ties of bin n, respectively.

We estimated distributions p2, p4 and p8, by partitioningthe difference space with a codebook of 384 codewords.The codebook was trained with the standard optimizedLVQ1 training algorithm, by selecting 100 random vectorsfrom each of the 1024 samples in the training set. As a ruleof thumb, statistics literature often suggests 10 entries perbin for a histogram to be statistically reliable. Thereforewe chose to use 384 codewords, for it produces differencehistograms of 384 bins, which corresponds nicely toroughly 10 entries per bin, given the effective sample sizeof 622 (the one pixel border is excluded in the computationof differences).

In experiments we obtained classification accuracies of93.3% for p2, 95.7% for p4 and 96.8% for p8, respectively.These results are excellent considering the difficulty of theproblem. The simple LBP operator achieved an accuracyof 94.4%, i.e. only 2.4% less than the much more complexp8 operator.

For comparison, experiments with features based on stand-ard GMRF models and Gabor filtering using differentmask sizes, implemented as in [18], and two different clas-

L S M,( ) Sn Mnln

n 1=

N

∑= (4)

Fig 3. Brodatz textures used in classification experiments.

sifiers were carried out. The best result for the GRMF ap-proach was 68.2%, obtained with a standard 6th ordersymmetric mask and a multivariate Gaussian discriminantclassifier. The best combination of GMRF features ob-tained from features computed using all models from the1st order to 6th order achieved 87.7%, providing an esti-mation for the best accuracy a method based multi resolu-tion GMRF might achieve. The best standard Gaborfeatures, extracted with a filter bank of three differentwavelengths and four different orientations in a 11x11neighborhood, using a 3-NN classifier achieved an accura-cy of 88.0%. The best combination of Gabor features ob-tained from features computed using all mask sizesbetween 7x7 and 17x17 pixels achieved an accuracy of90.2%, indicating the best accuracy a multiscale Gabor ap-proach could achieve.

These results are significantly poorer than those obtainedwith our methods, even though a larger neighborhood wasused to compute these features instead of the 3x3 neigh-borhood used in our approach. The rather poor results ob-tained especially with the GMRF method are partly causedby the histogram equalization that was used to remove ef-fects of unequal brightness and contrast. Many textureanalysis papers have neglected this kind of gray scale nor-malization needed for a fair comparison of different meth-ods.

4. EXPERIMENTS WITH UNSUPERVISED TEXTURE SEGMENTATION

Recently, an unsupervised texture segmentation algorithmusing the LBP/C texture measure and nonparametric statis-tical test was developed by Ojala and Pietikäinen [6]. Themethod has performed very well i n experiments. It is notsensitive to the selection of parameter values, does not re-quire any prior knowledge about the number of textures orregions in the image, and seems to provide significantlybetter results than existing unsupervised texture segmenta-tion approaches. The method can be easily generalized,e.g., to util ize other texture features, multiscale informa-tion, color features, and combinations of multiple features.

The segmentation method consists of three phases: hierar-chical splitting, agglomerative merging and pixelwise clas-sification. First, hierarchical splitt ing is used to divide theimage into regions of roughly uniform texture. Then, anagglomerative merging procedure merges similar adjacentregions until a stopping criterion is met. At this point, wehave obtained rough estimates of the different textured re-gions present in the image, and we complete the analysisby a pixelwise classification to improve the localization.The method does not require any prior knowledge aboutthe number of textures or regions in the image, as most ex-isting approaches do. Fig. 4 ill ustrates the steps of the seg-mentation algorithm on a 512 x 512 mosaic containing fivedifferent Brodatz textures.

The mosaic in Fig. 5, which is 384 x 384 pixels in size, iscomposed of textures taken from outdoor scenes [19]. Forthis image, our method gives a very good segmentation re-sult with an error of 2.1%.

Fig 5. Segmentation of a mosaic of textures.

Fig 4. Main sequence of the segmentation algorithm.

hierarchical agglomerative pixelwiseclassificationsplitting merging

(a) (b) (c) (d)

We also applied the texture segmentation method to natu-ral scenes [20], an example is shown in Fig. 6. The texturesof natural scenes are generally more nonuniform than thehomogeneous textures of the test mosaics. Also, in naturalscenes adjacent textured regions are not necessarily sepa-rated by well-defined boundaries, but the spatial patternsmoothly changes from one texture to another. Further, wehave to observe the infinite scale of texture differencespresent in natural scenes; choosing the right scale is a verysubjective matter. For these reasons there is often no ‘cor-rect’ segmentation for a natural scene, as is the case withtexture mosaics.

The invariance of the LBP/C transform to average graylevel shows in the bottom part of the image, where the seais interpreted as a single region despite the shadows. Theresult obtained is very satisfactory, considering that impor-tant color or gray scale information is not utilized in thesegmentation.

5. DISCUSSION AND CONCLUSIONS

The statistics based on gray level differences between pairsof pixels provide a simple and powerful tool for textureanalysis. The histograms of signed gray level differencescontain a more compact texture description than the ordi-nary approach based on coocurrences of pixel values, andare not affected by changes in mean luminance. In compar-ison to the commonly used absolute differences, the signeddifferences contain more information about image textureand consequently are more powerful.

Cooccurring differences provide useful additional infor-mation about local interpixel dependencies. The computa-

tional problems caused by multidimensional histogramscan be efficiently reduced by transforming multidimen-sional histograms into one-dimensional histograms vialearning vector quantization.

The 8-dimensional signed gray level difference operatorcan be regarded as a generalization of the LBP operator.LBP contains information about the spatial structure of lo-cal image texture. It encodes various simple feature detec-tors (edge, curve, corner, curve end, spot) at differentorientations in a single operator. LBP is invariant againstany monotonic gray scale transformation. The basic LBPmethod is sensitive to texture orientation, but this propertyappears to be very useful in texture segmentation for find-ing accurate boundaries between neighboring regions.

LBP does not address the contrast of texture which is alsoimportant for texture discrimination. For this purpose, wecan combine LBP with a simple contrast measure C. LBP/C has provided excellent results in our texture segmenta-tion experiments. The contrast operator is invariant withrespect to average gray level, which is very important con-sidering e.g. the needs of texture segmentation. The C op-erator could be replaced with other contrast measures,including (signed) gray level differences or local variance.

Our results suggest that complementary features based ondistributions of local spatial patterns and contrast play veryimportant roles in texture discrimination. There are studieson human perception suggesting that textural segmentationmight occur as a result of differences in the first-order sta-tistics of textural elements and their parts, see e.g. [21].Our approach resembles this model: simple feature detec-tors are used to measure significant textural properties andthe segmentation is performed by comparing textural prop-erties of neighboring regions.

The division of a texture analysis process into independent“ texture channels” measuring, for example, spatial patternand contrast information, leads to computationally verysimple implementations. For example, the sum of the log-likelihood measures computed for a LBP histogram andfor a 1-dimensional signed difference histogram (p1) pro-vided an accuracy of 95.9% in our classification problem,which is only 0.9% less than obtained with the 8-dimen-sional distribution p8.

The importance of the spatial pattern and contrast informa-tion in texture discrimination is also supported by our ear-lier findings. The joint pair of a rotation-invariant versionof LBP and a local variance measure provided the best per-formance in our experiments with rotation-invariant tex-ture classification [11]. The LBP measure and a measurebased on absolute gray level differences were best featuresfor two different test image sets in our experiments withmultichannel texture description, providing together nearlyas good performance as an “optimal” combination of all 12measures used in the study [9].

Fig 6. Segmentation of a natural scene.

The presented scheme can be generalized by adding otherperceptually significant feature channels, like directionali -ty and color. Further, we could consider a particular featureat multiple scales. Information obtained from differentchannels can then be combined in various simple ways, de-pending on the needs of the given application.

Our approach is annoyingly simple compared to the recentmainstream research in texture analysis. In the light of theresults of our work we cannot help wondering if the re-search has focused too heavily on a few theoretically im-pressive (and consequently computationally complex)paradigms?

6. ACKNOWLEDGEMENTS

The financial support provided by the Academy of Finlandand the Technology Delelopment Center is gratefully ac-knowledged.

7. REFERENCES

[1] Van Gool, L., Dewaele, P. and Oosterlinck, A. “Textureanalysis anno 1983,” Computer Vision, Graphics, andImage Processing, Vol. 29, 1985, pp. 336-357.

[2] Haralick, R.M. and Shapiro, L.G. “Computer andRobot Vision,” Vol. 1, Addison-Wesley, 1992.

[3] Tuceryan, M. and Jain, A. K., “Texture analysis,” inHandbook of Pattern Recognition and Computer Vision,Second Edition, eds. C.H. Chen, L.F. Pau, P.S.P. Wang,World Scientific Publishing Co., Singapore, 1993.

[4] Harwood, D., Ojala, T., Pietikäinen, M., Kelman, S.and Davis, L. “Texture classification by center-symmet-ric autocorrelation, using Kullback discrimination of dis-tributions,” Pattern Recognition Letters, Vol. 16, No. 1,pp. 1-10.

[5] Ojala, T., Pietikäinen, M. and Harwood, D. “A compar-ative study of texture measures with classification basedon feature distributions,” Pattern Recognition, Vol. 29,No. 1, 1996, pp. 51-59.

[6] Ojala, T. and Pietikäinen, M., “Unsupervised texturesegmentation using feature distributions,” Pattern Recog-nition, Vol. 32, No.3, 1999, pp. 477-486.

[7] Ojala, T., Pietikäinen, M. and Nisula, J., “Determiningcomposition of grain mixtures by texture classificationbased on feature distributions,” Int. J. Pattern Recogn.Artif. Intell., Vol. 10, 1996, pp. 73-82.

[8] Ojala, T. “Nonparametric texture analysis using simplespatial operators, with applications in visual inspection,”Acta Universitatis Ouluensis, C 105, 1997.

[9] Ojala, T. and Pietikäinen, M. “Nonparametric mul-tichannel texture description with simple spatial opera-tors,“ Proc. 14th Int. Conf. on Pattern Recognition,Brisbane, Australia, 1998, pp. 1952-1056.

[10] Pietikäinen, M., Ojala, T. and Silven, O.“Approachesto texture-based classification, segmentation and surfaceinspection,“ in Handbook of Pattern Recognition andComputer Vision, Second Edition, eds. C.H. Chen, L.F.

Pau, P.S.P. Wang (World Scientific, Singapore, 1998/99).[11] Pietikäinen, M., Ojala, T. and Xu, Z., “Rotation-invar-

iant texture classification using feature distributions,”Pattern Recognition, 1999, in press.

[12] Ojala, T., Valkealahti, K., Oja, E. and Pietikäinen, M.”Texture discrimination with multidimensional distribu-tions of signed gray level differences,” Submitted forreview, 1999.

[13] Ojala, T., Pietikäinen, M. and Kyllönen, J. “Gray levelcooccurrence histograms via learning vector quantiza-tion,” The 11th Scandinavian Conference on Image Anal-ysis, Kangerlussuaq, Greenland, 1999, in press.

[14] He, D.C. and Wang L. “Texture unit, texture spectrumand texture analysis, ” IEEE Trans. Geosci. Remote Sens-ing, Vol. 28, 1990, pp. 509-512.

[15] Kohonen, T., Kangas, J., Laaksonen, J. and Torkkola,K. “LVQ_PAK: A program package for the correct appli-cation of learning vector quantization algorithms,” Proc.International Joint Conference on Neural Networks, Bal-timore, 1992, pp. 1725-1730.

[16] Valkealahti, K. and Oja, E. “Reduced multidimen-sional cooccurrence histograms in texture classifica-tion,“ IEEE Transactions on Pattern Analysis andMachine Intelli gence, Vol. 20, 1998, pp. 90-94.

[17] Brodatz, P. “Textures: A Photographic Album for Art-ists and Designers,” Dover Publications, New York,1966.

[18] Smith, G. and Burns, I. “Measuring texture classifica-tion algorithms,” Pattern Recognition Letters, Vol. 18,1997, pp. 1495-1501. http://www.cssip.elec.uq.edu.au/~guy/meastex/meastex.html.

[19] Jain, A. and Karu, K. “Learning texture discrimina-tion masks,” IEEE Transactions on Pattern Analysis andMachine Intelli gence, Vol. 18, 1996, pp. 195-205.

[20] Panjwani, D.K. and Healey, G. “Markov random fieldmodels for unsupervised segmentation of textured colorimages,” IEEE Transactions on Pattern Analysis andMachine Intelli gence, Vol. 17, 1995, pp. 939-954.

[21] Beck, J., Prazdny, S. and Rosenfeld, A. “A theory oftextural segmentation,” in Human and Machine Vision,eds. J. Beck, B. Hope and A. Rosenfeld, Academic, NewYork, 1983.