Real-time in-plane rotation estimation of pharmaceutical tablets:a feature based approach

Blaz Bratanic1, Bostjan Likar1,2, Franjo Pernus1,2, Dejan Tomazevic1,21Sensum, Computer Vision Systems, Tehnoloski park 18, 1000 Ljubljana, Slovenia

2University of Ljubljana, Faculty of Electrical Engineering, Trzaska 25, 1000 Ljubljana Sloveniablaz.bratanic@sensum.eu

Abstract

Visual quality inspection plays an important rule inquality control. With visual inspection it is ensured thatthe inspected surface meets the requirements; that thesurface is free of irregularities, such as cracks, dents,and scratches. Detection of irregularities can, however,be a challenging task, especially on complex surfaces,where it is necessary to differentiate between surface ir-regularities and expected surface variations. Differen-tiation is done by comparing a sample with a referencemodel, which requires an accurate spatial alignment be-tween the two. In this paper we focus on visual qualityinspection of pharmaceutical tablets. While inspected,tablets are mechanically constrained, thus alignmentsimplifies to in-plane rotation estimation. For in-planerotation estimation we propose a method using his-tograms of oriented gradients (HOG) and a nearest-neighbor regression in HOG feature space. Method wasevaluated on four datasets of pharmaceutical tablets,varying in size, shape and color. The results showthat the proposed method is superior in robustness, withcomparable accuracy to the methods previously used forrotation estimation of pharmaceutical tablets.

1 Introduction

All pharmaceutical tablets must be uniquelymarked, to enable unique identification and to avoidhazardous mix-ups. Due to imperfect production pro-cess a compliant visual appearance is ensured by visualinspection [1], [2]. Visual inspection of pharmaceuti-cal tablets is a difficult task, especially of tablets withcomplex geometries, because it is necessary to differen-tiate between defects and expected surface variations.Differentiation is done by comparing a sample with anideal reference model which, however, requires an ac-curate spatial alignment between the two.In this paper, we address the spatial alignment of

pharmaceutical tablets for visual quality inspection ofpharmaceutical tablets [1]. While inspected, tabletsare mechanically constrained. This simplifies the align-ment to in-plane rotation estimation (Fig. 2). Never-theless, rotation estimation of pharmaceutical tabletsis a challenging task, because tablets vary in shape,color, texture, and because of high speed requirements.The task is additionally challenging because of normalintra-tablet variability which is a result of imperfectproduction process.Spiclin et. al [3] proposed the following three

registration methods for in-plane tablet rotation es-timation: Direct Pixel Matching (DPM), PrincipalAxes Matching (PAM), and Circular Profile Matching(CPM). The first method (DPM) evaluates the nor-malized cross-correlation of a sample image and therotating reference image. A rotation angle is deter-mined by the maximum value of the normalized cross-correlation. By contrast, circular profile matching isbased on extraction and alignment of circular profiles.Circular profiles are obtained by the radial integration(I(φ) =

∫ r2r1

I(−r sinφ, r cosφ) dr) within a ring cen-tered at the tablet center. By integration the 2D imageI(x, y) is reduced to 1D profile I(φ) – the angle is esti-mated by 1-D cross correlation of reference and samplecircular profile: Ir(φ)∗Is(φ). PAM as the third methodestimates the angle by matching principal axes of thereference and a sample object [4]; method is suitableonly for objects with distinctive assimetric structures.Morever, this approach requires additional verificationstep, due to residual ±π sign ambiguity.

Described methods used for rotation estimation arerather basic, because of speed requirements (< 5 msper estimation). Methods work sufficiently well onlyif center estimations are accurate (DPM, CPM) or forobjects with distinctive shape asymmetry (PAM). Fur-thermore, due to global similarity measure (i.e. cross-correlation), methods are affected by large irregular-ities that distort the tablet appearance – leading toinaccurate spatial alignment. An inaccurate spatialalignement further leads to erroneous inspection re-sults, and finally to lost profits.

In order to increase position invariance and align-ment robustness, we propose a feature based approachto in-plane tablet alignment.

Figure 1. Mechanically constrained inspectedproducts on a rotating drum.

MVA2013 IAPR International Conference on Machine Vision Applications, May 20-23, 2013, Kyoto, JAPAN14-16

423

Illumination Illumination

Camera

φ

Figure 2. Illumination and camera setup. Cameraview is parallel to the tablet rotation axis

2 Rotation estimation

Prior to alignment, all the tablets are segmented us-ing method proposed by Mozina et. al [5], by which anobject center and an object’s bounding window are ob-tained. Rotation is estimated with a nearest-neighboursearch in HOG feature space. A rotation angle φ (Fig.2) for each sample image Is is estimated by mappingthe sample image to HOG feature space then searchingfor a nearest neighbour in the reference set T (φ). Thereference set contains HOG feature representations forreference object at different angles, and is constructedby rotating the reference image Ir by φ; we assumea similar object appearance, regardless of the rotationangle.

The method is described in three steps. First, webriefly describe Histograms of Oriented Gradients, sec-ond we describe a procedure for reference set construc-tion, and finally we describe the rotation estimation.

2.1 Histograms of Oriented Gradients

Dalal and Triggs [6] proposed Histograms of Ori-ented Gradients, to answer the question of feature setsfor robust visual object recognition. Histograms of Ori-ented Gradients feature set is obtained from a grid ofspatial cells; each spatial cell contains a k-bin weighted(with absolute gradient) histogram of vector orienta-tions. Spatial cells are concatenated into larger blocksof N×M cells and normalized to a L1 (

∑i |xi| = 1) or

a L2 (√∑

i x2i = 1) unit length. Finally, vectors over

all blocks are merged to a HOG feature vector (Fig. 3).Subsequently, Zhu et. al [7] proposed an integral image(Fig. 4) approach to HOG calculation. This approachenables a constant time calculation of histograms oforiented gradients over arbitrary rectangular regions.For integral image based HOG calculation, a separateintegral image is stored for each histogram bin. First,an image Ib is initialized for each histogram’s bin, thena gradient orientation at each image’s pixel is bilinearlyinterpolated; results are stored in the correspondingimage. Lastly, an integral image for each histogrambin is calculated from each image Ib. Value for each el-ement of HOG vector is independently calculated fromthe corresponding integral image – calculation of HOGvector thus requires 4× k access operations.

DC

BA

Figure 4. The value I(x, y) at any point (x, y) isthe sum of all the pixels above and to the leftof (x, y) inclusive, I(x, y) =

∑x0

∑y0 i(x

′, y′) - in-tegral of pixel values within arbitrary rectangleABCD is obtained by I(D)+I(A)−I(B)−I(C).

2.2 Reference set construction

A reference set is built from a single reference imageIφ rotated in r degree steps around the rotation axis.Step size r is user determined and determines the the-oretical upper bound of rotation estimation accuracy.Next, each rotated image Iφ is transformed to a HOGfeature space Iφ → τφ. As a result of mapping imagesIφ, φ = [0, . . . , 2π] to HOG feature space τφ we obtaina reference set T(φ).

HOG features are calculated over a deterministicallyconstructed multi-scale set of cells W ; each scale ina set determines the size of cells that uniformly tilethe window. By iteratively proceeding from largest tothe smallest scale, cell size is reduced by 2× betweentwo neighbouring scales, - forming a multi-scale set ofcandidate cells.

Reference set T(φ)Samplefeaturevector τ s

Sampleimage

d(T(φ), τ s)

Figure 5. Rotation angle is estimated by a nearestneighbour search in feature space.

2.3 Rotation estimation

With a reference set T(φ), a rotation angle for eachsample image Is is estimated as follows: first, a sampleimage is mapped to a HOG feature space Is → τ s,with features calculated over all the cells w ∈ W . Arotation angle is then estimated (Fig. 5) by finding thenearest neighbour in reference set T(φ):

φestimate = argminφ

d(T(φ), τ s) (1)

where d(T(φ), τ s) is an Euclidean distance betweenfeature vectors τ s and T(φ).

424

Cell

Block

Blocks overlaping

Feature vectorτ = [. . . , . . . , . . . ]

Input imageCalculate imagegradients

Accumulate weighted votes overspatial cells and normalize withinsliding block

Merge vectorsfor all blocks

Figure 3. Histograms of Oriented gradients.

2.3.1 Implementation details

Method was implemented in C++ and compiledwith Visual Studio. The same parameters were usedfor all the datasets. Three scales were used to constructa multi-scale cell set; half object size at start then dec-imating cell size by 2× with each increasing scale. A9-bin histogram was calculated for each separate cell,which was merged to a 2×2 block of cells and normal-ized to L1-norm. The reference set was constructedwith a 1 degree step. For nearest-neighbour search, weused an exhaustive search. Use of a space-partitioningdata structure (e.g. kd-tree [8]) was not feasible, dueto high dimensionality of a HOG feature vector.

3 Experiments and results

The proposed method, denoted as HBM, was evalu-ated on four datasets provided by Spiclin [3]. Eval-uation datasets were captured with a trilinear line-scan camera and a fixed illumination on a SensumSPINE inspection system (Sensum). An illuminationand camera setup is shown in Fig. 2. Datasets con-tain 514, 219, 212, and 183 images of tablets RTD73,ELP20, 500, and LEK respectively, differing in size,color, and shape (Fig. 6).To allow objective comparison of the proposed

method, we evaluated registration robustness and ac-curacy by evaluation criteria proposed by Spiclin et.al [3]. The robustness of the method was estimated by

(a) (b) (c) (d)

Figure 6. Evaluation datasets. The 500 datasetcontains 212 images of oblong tables (c).Datasets RTD75 (b), ELP20 (c), and LEK (d)contain 514, 219, and 183 images of round tabletsrespectively.

the percentage of successful registrations (Hits); suc-cessful is a registration with absolute angular error lessthan 5◦. Registration accuracy was measured by themean absolute angular error (Eq. 3).

An absolute angular error ε is the difference betweenreference angle φref and an estimated angle φestimate:

ε = |φref − φestimate|. (2)

A reference angle φref is obtained with 5 manuallypositioned corresponding points {xr

i , yri } on reference

and points {xsi , y

si } on sample image. Angle is obtained

by minimizing the mean square distance between cor-responding points as a function of angle φref :

[xri

yri

]=

[cosφref sinφref

− sinφref cosφref

] [xsi

ysi

]

ε =5∑

i=1

(xri − xs

i cosφref − yis sinφref )2+

5∑i=1

(yri − xsi sinφref − ysi cosφref )

2

∂ε

∂φref=cosφref

5∑i=1

(xsiy

ri − xr

i ysi )+

sinφref

5∑i=1

(xrix

si + yri y

si ) = 0

φref =arctan

∑5i=1(x

siy

ri − xr

i ysi )∑5

i=1(xrix

si + yri y

si )

(3)

Results for all the methods are presented in Table1. In addition, Fig. 7 shows a cumulative sum of suc-cessful registrations with respect to maximum absoluteangular error.

4 Discusion

Method was evaluated on four different datasets ofpharmaceutical tablets. Registration of pharmaceu-tical tablets is a challenging task, because tablets areproduced in various shapes, sizes, colors, and with var-ious imprints. Further registration complexity arises

425

Table 1. Registration results for HOG based ro-tation estimation and the three reference regis-tration methods

RTD75 ELP20 500 LEK

Error (◦)DPM 1.17 1.17 1.58 0.86

PAM 2.44 2.71 0.80 2.31

CPM 0.72 0.54 0.83 0.75

HBM 0.60 0.74 0.57 1.27Hits (%)

DPM 100 100 94.3 99.5

PAM 56.0 21.5 86.8 44.3

CPM 100 100 100 99.5

HBM 100 100 100 100Time (ms)

DPM 109 184 227 191

PAM 1.01 1.42 1.66 1.46

CPM 3.18 3.49 3.01 3.15

HBM 1.52 1.56 1.53 1.57

0 1 2 3 4 50

20

40

60

80

100

Maximum absolute error ε (◦)

Successfulregistrations(%

)

HBMCPMDPMPAM

Figure 7. Cumulative percentage of successfulregistrations as a function of maximum absoluteregistration error.

from imperfect illumination and mechanical manipu-lation, and various surface irregularities. Evaluatedmethod was compared with three methods describedby Spiclin et. al [3]. Registration speed was recorded,however, the running times cannot be directly com-pared, as the methods were executed on different com-puters. Nevertheless, we can still infer that the pro-posed method is a feasible option for real-time rota-tion estimation. The percentage of successful regis-trations for the proposed method is higher than forcompared methods. The DPM method is directly af-fected by the outliers caused by product’s defects andby irregular illumination. By contrast CPM integratesvalues over the tablet radius, thus increasing robust-ness, and performance, but decreasing discriminativepower. This is particulary pronounced with near sym-metric imprints. The PAM method simplifies rotationestimaton by matching the principal axes, but is useful

only for objects with distinctive asymmetric structures,and has a residual ±π sign ambiguity. Accuracy, eval-uated only for successful registrations, had an upperbound at 5◦. The accuracy of the proposed methodis comparable with the accuracy of CPM and DPMwhile accuracy of PAM is considerably worse. All themethods, other than PAM, are theoretically boundedby the sampling resolution.

5 Conclusion

In this paper we propose a registration method forin-plane rotation estimation of pharmaceutical tablets.A feature based registration is performed, by con-structing a reference feature set of rotated referenceimages in mapped feature space. A rotation angleof a sample is then estimated by finding its nearest-neighbour in that reference set.

Method was evaluated and compared on fourdatasets of pharmaceutical tablets. The results showthat the proposed method is a feasible option for rota-tion estimation in industrial environments. Althoughthe evaluation was made on datasets of pharmaceuti-cal tablets, we consider the method useful for rotationestimation of arbitrary objects.

References

[1] M. Bukovec, Z. Spiclin, F. Pernus, and B. Likar, “Au-tomated visual inspection of imprinted pharmaceuticaltablets,” Measurement Science and Technology, vol. 18,pp. 2921–2930, Sept. 2007.

[2] M. Mozina, D. Tomazevic, F. Pernus, and B. Likar,“Automated visual inspection of imprint quality of phar-maceutical tablets,” Machine Vision and Applications,vol. 24, pp. 63–73, 2013.

[3] Z. Spiclin, M. Bukovec, F. Pernus, and B. Likar, “Im-age registration for visual inspection of imprinted phar-maceutical tablets,” Machine Vision and Applications,vol. 22, pp. 197–206, Jan. 2011.

[4] N. M. Alpert, J. F. Bradshaw, D. Kennedy, and J. A.Correia, “The principal axes transformation–a methodfor image registration,” J Nucl Med, vol. 31, pp. 1717–1722, Oct. 1990.

[5] M. Mozina, D. Tomazevic, F. Pernus, and B. Likar,“Real-time image segmentation for visual inspection ofpharmaceutical tablets,” Machine Vision and Applica-tions, vol. 22, pp. 145–156, 2011.

[6] N. Dalal and B. Triggs, “Histograms of oriented gra-dients for human detection,” in Computer Vision andPattern Recognition, 2005. CVPR 2005. IEEE Com-puter Society Conference on, vol. 1, pp. 886 –893 vol.1, june 2005.

[7] Q. Zhu, M.-C. Yeh, K.-T. Cheng, and S. Avidan, “Fasthuman detection using a cascade of histograms of ori-ented gradients,” in Proceedings of the 2006 IEEE Com-puter Society Conference on Computer Vision and Pat-tern Recognition - Volume 2, CVPR ’06, (Washington,DC, USA), pp. 1491–1498, IEEE Computer Society,2006.

[8] J. L. Bentley, “Multidimensional binary search treesused for associative searching,” Commun. ACM, vol. 18,pp. 509–517, Sept. 1975.

426