Embed Size (px)
Citation preview
H Dayal Sujeevan Ranadewa
159.793
Project for BSc (HONS)
01257358
Application of Machine Vision in Fruit Grading
Table of Contents
Page Number
1. Introduction
2. Literature Review
3. Methods
3.1 Image Acquisition and Database
3.2 Proposed System Architecture
3.3 Threshold Based Segmentation
3.4 Edge Detection
4. Colour Segmentation
4.1 Colours
4.2 Algorithm
4.3 Differentiating between defects and stems/calyxes
5. Results and Discussion
6. Conclusion
7. References
8. Bibliography
9. Appendices
2
1 Introduction
The use of Machine Vision for the inspection and grading of fruits and vegetables has
increased during the recent past. Recent research in this area have been pioneered by
Unay and Gosselin (2004) who presented a quality sorting method for ‘Jonagold’ bi
colour apples and also presented several other important works such as stem and
calyx recognition to differentiate these from true defects. Fresh market fruits like
apples and oranges are graded into categories based on several factors such as size,
colour, shape and presence of defects/bruises/blemishes. The growing need to supply
graded quality food products within a short period of time has given high priority to
Automated Grading of Agricultural Products. There have been many attempts by
manufactures around the world to produce sorting machines capable of grading fruits
not just by size, colour but also capable of recognising additional features such as
bruises, blemishes and other defects. However, automated sorting of fresh fruits based
on defects is not yet efficient or forthcoming. Inspection of fresh fruit quality presents
numerous challenges as aspects such as appearance cannot be defined clearly and the
degree of irregularity that is acceptable is not uniform.
Computer Vision based sorting of fresh market fruits is a difficult task but more
emphasis is being put in this area due to factors such as increasing the speed of
grading and reducing the human error. Some of the defects present in fresh market
fruits are frost damage, flesh damage, hail damage, scald, russet, rot and bruise. Much
of the study in finding a machine vision automated grading system involves
segmenting these defects from healthy tissue. ThresholdBased segmentation and
Region Growing Algorithm to separate defected area from healthy tissue are ways to
tackle this. Any Machine Vision based system developed to detect defects in fruits
would have the following steps. They are Image Acquisition, PreProcessing, Defect
3
Segmentation, Feature Extraction and Fruit Classification. Some of these can be
combined and they can also overlap (Unay & Gosselin, 2004).
Although ThresholdBased segmentation of defects would be able to easily detect the
defected area of a fruit, there are numerous other challenges. Stem ends of fruits and
Calyxes are natural parts of fruits and these appear as dark blobs on images like true
defects. Any segmentation applied to detect the defects should be refined such that
stem ends/calyxes are not confused with actual defects and recognised appropriately.
Other minor difficulties would be bicoloured skin in some forms of fruit (apples)
causing more difficulties in segmentation due to colour transition areas.
Research in this area involves training a statistical classifier such as a Neural
Network, Nearest Neighbour Classifier or Adaptive Boosting (AdaBoost) such that
given a number of characteristics (presence of defects, shape including area,
perimeter, circularity) of a fruit, an appropriate overall classification can be given as
Healthy or Defected. Unay and Gosslin (2005) used Linear Discriminant Classifier
(LDC), kNearest Neighbour (kNN), AdaBoost and Support Vector Machines (SVM)
in the classification step of Stem and Calyx Recognition on ‘Jonagold’ apples by
Pattern Recognition. This would be ideal as an accurate classifier of this nature can
lead to excluding the human participation for the whole process and thus being
completely automatic.
4
2 Literature Review
Commercial grading systems of fruits acquire images of fruits while they go through
the grading machine. In order to grade fruits accurately, number of requirements have
to be met. Namely, the images should cover the entire surface of the fruit, high
contrast has to be created between the defected area and the healthy tissue and must
also maintain low variability of the healthy tissue (Destain & Leemans, 2003). In
commercial systems fruits placed on conveyor belts or rollers rotate while moving
and are observed from above using a camera (Destain & Leemans, 2003). It has also
been found that 80% of the fruit surface can be observed with only 4 images (Molto,
Blasco & Benlloch, 1998). Lighting system plays a vital role in these applications.
The purpose of the lighting system is to provide radiant light with tailored spectral
characteristics and uniform spatial repartition (Destain & Leemans, 2003). This is
especially necessary in identifying defects and stems/calyxes. Yang (1993) used
structured lighting combined with a diffuse lighting system especially to make the
calyxes and stem ends of fruits clearly visible and increase their contrast.
As stated earlier, the contrast between fruit and the background should be high
enough to simplify the localisation of the fruit which is usually carried out by
thresholding (Destain & Leemans, 2003). In monochrome images (greyscale) the
apples appear to be light grey and mean luminance decreased from the centre of the
fruit to the boundaries (Yang, 1994). Although the defects are darker than the healthy
tissue their contrast, shape and size vary strongly and as such simple techniques such
as thresholding or background subtraction would give poor results (Yang, 1994).
Yang & Marchant (1996) used a flooding technique to detect patch like defects after
applying median filter followed by Gaussian filter. These patch like defects were
considered as catchment basins in this flooding technique. Active contours algorithm
was used further enhance the detection of defects and minor over segmentation was
5
removed by applying threshold over depth or volume of the basins (Yang &
Marchant, 1996). However these studies did not concern bicolour fruits (bicolour
apple varieties).
Leemans et al. (1999) applied the Bayes’ theorem on bicolour apples and the
probability distributions of the healthy tissue and the defects were considered as non
gaussian and modelled numerically using a kernel method. Different statistical
techniques have also been proposed to segment colour images. Supervised techniques
tend to reproduce the classification given by the human while unsupervised
techniques produce a classification of their own (Destain & Leemans, 2003). Yang
(1993) used neural networks to classify segmented areas as patch like defects,
elongated defects or nondefective areas although the question of proper grading of
fruit was not addressed. Most research that has been done in fruit grading did not
consider the management of several images representing the entire surface of the fruit
and some researchers did not consider bicolour apples characterised by green
(‘ground colour’) and red (‘blush’ colour).
Unay and Gosselin (2004) proposed a quality grading system that segments the
defected regions on fruit by pixelwise processing using artificial neural network.
Statistical features such as average, median, standards deviation, textural features
such as 1st invariant moment of Hu and shape features such as area, perimeter and
circularity were calculated, so each fruit is represented by 13 features. Supevised
classifiers such as linear discriminant, nearest neighbour, fuzzy nearest neighbour,
adaboost and support vector machines were tested where the last two were found to
perform the best with 90% accurate recognition.
Unconventional methods have also been introduced by many researchers. Cao, Wang
and Li (2005) presented a method to segment fruit images using the Fuzzy Art Model.
The segmentation process of this model used fuzzy clustering which was based on
6
neural network combining adaptive resonance theory and colour texture
characterization. The cluster system was made up of a modified fuzzy art model and a
pattern group model. To further improve the discrimination of classification features,
colour texture descriptors were chosen as main classification features. Experimental
results showed that this method achieved satisfactory results.
It is also interesting to note the research that has gone into detecting defects in
different types of fruit varieties such as citrus fruits (oranges and mandarins) and
freshcut star fruits. Blasco, Aleixos and Molto (2006) proposed a regionoriented
segmentation algorithm to classify sound peel, the stem and defects in oranges and
mandarins. The algorithm starts with detection of regions in the image with a
homogeneous colour. A seed is created in the middle of the region which grows by
iteratively by adding neighbourhood pixels that accomplish the criterion of colour
homogeneity. Largest region is assumed to be sound peel and the rest of the regions
are blemishes. This algorithm was able to detect 95% of the defects present.
Other related areas such as detecting defects in vegetables and agricultural robotics
have also been researched and experimented recently. Milella, Reina, Foglia and
Gentile (2004) introduced an agricultural automation system for radicchio (vegetable
grown in Italy) harvesting through visual localization of the plants in the field. This
included image acquisition, conversion from RGB to HSL, thresholding in the Hue
and Luminance plane, OR operation between binary images, morphological
operations, particle filtering and radicchio localization in the image reference
frame(x,y). The study also covered postharvesting automation of fennel (vegetable
grown in Italy) through visual identification. This involved image acquisition, RGB to
HSL conversion, extraction of the plant from the image, processing the hue plane,
individuation of the orientation of the plant on the conveyor belt and calculation of
lines for the cutting operation.
7
3 Methods
3.1 Image Acquisition and Database
Images were acquired using a 3.1 Mega Pixel Digital Camera and transferred to the
machine host albcache.massey.ac.nz (IP address 130.123.225.69) linux server where
Open CV Library is readily available. The programs were tested on an Intel® Core™
2 Duo machine with 1.8 GHz CPU and L2 cache. The Open CV library was
developed by researchers working for Intel. This is freely available on the web and it
can work with different platforms and operating systems (Barczak, 2006). The
database that was used for the project consists of approximately 200 images. The
resolution of most of the images is 640 * 480 pixels meaning that each image has
307200 pixels. Some of the images were taken at a higher resolution of 2048 * 1536.
The database consists of images of apples taken on a dark uniform coloured low
intensity background. The images of apples were taken in a diffusely illuminated
background. As can be seen on Image 1, the slight blurring of the image is due to the
camera. The blurring did not prove to be a significant problem at latter stages and
therefore did not need functionality to address deblurring.
8
Fig. 1. A screen shot of some of the collected apple images
Image 1
3.2 Proposed System Architecture
The grading of apples using machine vision is arbitrarily divided in to several steps.
These are normally image acquisition, segmentation of regions of interest,
interpretation of the regions of interest and fruit classification. The proposed system
architecture consists of the following main steps. It is expected that some steps may
overlap with others and some may be combined to form one whole step.
9
Fig. 2. Proposed system architecture
Thresholding is proposed to be used to force the pixels of the background to be zero
and thus to be completely dark in colour. The defects, stem ends and calyxes are then
segmented in the object segmentation phase using colour segmentation. Then methods
will be developed to extract information to differentiate between the segmented areas
for true defects and stem ends/calyxes in the feature extraction stage which will be
further refined in the differentiation stage of stem ends/calyxes to decide whether the
segmented areas found through colour segmentation are true defects or stem ends or
calyxes. Classification phase will make use of the entire previous steps to ascertain
whether the fruit contains any skin defects or not. Algorithms to conduct the object
10
Background Removal
Colour Segmentation of Objects using Algorithm 1
Feature Extraction
Classification using Algorithm 1 & 2 and Quality Decision
Image Acquisition
Differentiating stems/calyxes and defects using Algorithm 2
segmentation using colour segmentation and also to differentiate between
stems/calyxes and true defects would be proposed and discussed in detail.
3.3 Threshold Based Segmentation
A digital image is defined as a two dimensional matrix. Each element has co
ordinates (x, y) and a value f(x, y). The coordinates correspond to the position of a
certain element in the image. In a grey scale image the coordinates denote a
brightness value between 0 and 255 where as in a colour image each value may
actually be translated in a set of values that are somehow combined such as in the
RED, GREEN, BLUE colour space. Each element of the matrix is often called a pixel.
Unay and Gosselin (2004) states that thresholding can be applied within a
neighbourhood if each pixel or globally for defect segmentation. The intensity of a
pixel is set at 0 if this is under a set value (e.g. 128) at set at 255 if the intensity of the
pixel is above the level of the threshold. The thresholding technique that would be
used for this purpose can be written as follows.
g (x, y) = 0 if i(x, y) <= 128
g (x,y) = 255 if i(x, y) > 128
11
Image 2 Image 3
Fig. 3
As seen in image 3, which is the resultant thresholdbased segmentation image of
image 2, this method is able to accurately segment the pixels in the apple which tend
to be darker. These areas are usually blemishes or defects which tend to be darker in
colour than the healthy tissue. The method of thresholding forces pixels whose values
are lower than a set value to be zero for all 3 channels Red, Green and Blue. As per
image 3, this method has clearly segmented the defected tissue and the stem area from
the rest of the healthy skin. However, its inability to distinguish between the defected
areas and stem area is a major disadvantage. On monochrome images the apples
appear in light grey and the average luminance varies with its colour and luminance
decreases from the centre of the fruit to its boundaries. The defects are generally
12
darker than the healthy tissue but the contrast, size and shape of the defects vary
strongly. Yang (1994), therefore suggests that simple techniques such as thresholding
or background subtraction would give poor results while.
3.4 Edge Detection
Special features such as lines, corners and circles have important applications in
machine vision (Gonzales & Woods, 2002). Before these features can be found, edges
need to be detected. The goal of doing this would be to mark the points in a digital
image where luminous intensity changes sharply. There are many methods used in
edge detection and most can be grouped in to two. They are searchbased and zero
crossing based. Searchbased methods detect edges by looking for maxima and
minima in the first derivative of the image whilst zerocrossing based methods search
for zero crossings in the second derivative of the image in order to find edges usually
the zerocrossings of the Laplacian.
Edges in an image correspond to the zero crossings of the Laplacian. Laplacian of a
Gaussian blurs the image and make some less important edges disappear. This method
is especially useful when there are too many edges that are detected, hence subdues
the important edges that are of interest (Gonzales & Woods, 2002). A Sobel mask
would be capable of finding the gradients of an image. The Canny edge detection uses
a Gaussian followed by Sobel edge detection. The edges are further improved by
finding their direction. Once the lines/edges are found a hysteresis process is used to
improve the quality of the edges such as replacing the broken edges.
These edge detection methods may be used to find edges of fruit defects, stem ends or
calyxes. During the initial stages of the project various ways of detecting and/or
13
segmenting the objects of interest within a fruit, were carried out. As seen in image 5,
edge detection has been able to segment and detect the boundaries of the apple as well
as the defected areas and the calyx. This is a reasonable and easy way of detecting
these areas but due to the reasons discussed in 2.3 this may not be entirely sufficient
and lacks clarity.
Image 4 Image 5
14
4 Colour Segmentation
4.1 Colours
Grey Scale images contain information only about the brightness of the pixels and the
shape of the objects in it (Gonzales & Woods, 2002). As colours hold additional cues
about the objects, image processing in colour properties can be very useful in some
applications (Gonzales & Woods, 2002). In image acquisition cameras can be
notoriously selective and can make colours disappear or change under the prevalent
lighting conditions (Barczak, 2006). The human eye is capable of seeing light
frequencies between Infra Red and Ultra Violet and as such this range is narrow.
However, cameras are sensitive to different ranges of light frequencies and may have
capabilities that the human eye has not (Gonzales & Woods, 2002).
Mixture of different light frequencies can yield similar colours/ resultant colours to
our eyes in perception or to the cameras in image acquisition (Gonzales & Woods,
2002). Therefore a mixture of Red and Green may result in yellow although this may
be different to pure yellow obtained from a single light frequency. The human eye has
different sensors called ‘cones’ which specialise in certain frequency ranges (Barczak,
2006). Hence by using RED, GREEN and BLUE any other colour in the visible
spectrum can be made.
The standard specifications of representing colours are called colour spaces. They
are RGB or GBR (RED, GREEN & BLUE colour space), HSI or HSV (this is
composed of HUE, SATURATION and INTENSITY or VALUE) and YUV and
YUQ (Y representing LUMINANCE or SHAPE and UV representing
CHROMINANCE or COLOUR).
15
The project was conducted entirely on images specified in the RGB colour space. In
the RGB space usually one byte per colour is used and therefore each colour has 256
states or levels. As this amounts to 24 bits, a maximum of 224 colours can be
represented in the RGB space. In cases where colours are an important property of an
object in the image, the colours can be used to separate or segment regions of interest
in the image. Practical applications where colour segmentation is used include skin
colour segmentation, fruit sorting to ascertain the quality of the fruit, quality
assurance etc. It has also been found by using different colour spaces certain groups
of colour can be more easily segmented (Barczak, 2006).
4.2 Algorithm
The algorithm that was developed to segment the fruit’s surface into healthy and
defected areas can be summarised as follows. The threshold levels for the RED,
BLUE and GREEN channels for segmentation were chosen after painstaking
experimenting with the collected database of images so that the chosen levels would
enable accurate uniform segmentation across the board in most images. GIMP (GNU
Image Manipulation Program) which is a freely distributed piece of software was
used during this stage and aided in determining the chosen threshold levels. GIMP
colour picker provides the individual threshold levels of RGB of a chosen pixel as per
below images.
16
17
Fig. 4 Apple 19.jpg with the individual levels of RED, GREEN and BLUE extracted
by GIMP’s colour picker
Algorithm 1
If there are 3 arguments {
Load the Image as per Argument 2
Create a clone of this image for pixel manipulation
Start from x, y at position (0, 0) and run through to entire width and height of
the image and at each pixel check the following {
If the value of the RED channel is < 10, this pixel is likely to be part of
the background, make all 3 channels values 0 to exclude the
background. Make this pixel BLACK
Else If the value of the RED channel is > 10 and GREEN channel is >
70 and GREEN > RED and (RED – BLUE) is > 40 then this pixel is
part of the apple’s healthy skin. Make this pixel completely GREEN
Else If the value of the RED channel is > 55 and RED > GREEN and
(RED – BLUE) is > 40 then this pixel is part of the apple’s healthy
skin. Make this pixel completely RED
Else If RED is > 10 and BLUE >15 and (RED – BLUE) and (GREEN
– BLUE) is < 50 and (RED + GREEN) < 400, this pixel does not
18
represent RED or GREEN and is likely to be from a defect or stem end
or calyx. Make this pixel completely BLUE
Else If the value of RED + GREEN + BLUE > 255, then this pixel is
still likely to be part of the healthy skin. Make this pixel completely
RED
Else If RED + GREEN + BLUE > 30, this is likely be part of a Defect
or Stem or Calyx. Make this pixel completely BLUE or BLACK
Else if the pixel does not belong to any of the above it is likely to be
part of a defect or background. Make this pixel BLACK
Save the cloned image in the file denoted by the 3rd argument
}
Image 6 Image 7
19
As seen on Image 6, the apple image contains the stem area as well as a slightly dark
defected path from skin defect. Image 7 shows colour segmentation has been able to
successfully segment the mostly healthy skin area into RED and the stem area into
BLACK and the defected patch into BLUE blob.
Image 8 Image 9
20
21
Fig. 5 Screen shot of some of the colour segmented images produced by Algorithm 1
As seen on Image 8, the image contains the stem area as well as a slightly dark
defected path from a skin defect. Image 9 shows the colour segmentation has
successfully segmented the mostly healthy skin area into RED and GREEN and the
stem area into BLACK and the defected patch into BLUE blob.
The colour segmentation is geared such that the chosen threshold levels for the 3
channels and the conditions would enable to accurately segment the background in to
DARK BLACK, healthy fruit skin to RED and GREEN, defected skin into BLUE and
stem ends and calyxes in to DARK BLACK.
However, it is entirely possible that stem ends and calyxes are also segmented in
BLUE rather than BLACK and defects segmented in BLACK rather than BLUE. This
is possible as the intensity of the darkness of defects, stem ends and calyxes can vary
from image to image. Therefore, further refinement is needed to ascertain whether
BLUE and BLACK blobs are representative of defects or stem ends & calyxes, as this
would determine whether the apple is ok or not and free of skin defects.
4.3 Differentiating between defects and stems/calyxes
A segmented image with no difference between stems/calyxes and defects presents a
practical problem. The commercial systems use a video sequence where the apple
rolls over a dark area and a simple optical flow approach is used to link the various
frames. These systems use two or more cameras to get simultaneous images as
otherwise areas on the sides of the apple might be missed. Although a similar
approach is desirable, this was deemed too complex. Instead of a sophisticated
approach to rebuild the image of the apple a simple comparison of areas could be
done as follows.
22
Sometimes the calyx and the stem might just be visible on opposite sides if the apple
is in a certain position in relation to the camera. In this case there may be two dark
black/blue regions and the distance between these two blobs may be close to the
diameter of the apple. However with one camera the calyx and stem would not be
visible simultaneously, unless the apple is distorted.
In other cases where a defect appears, there are always two dark black/blue regions
which are much closer than the diameter of the apple, one region representing a calyx
or stem and the other is a defect. However for this method to succeed one must have
4 – 5 images of the every apple as there is a good chance by rolling the apple at
chance, the defects will appear together with either the calyx or stem end. Therefore,
if two dark black/blue areas that are closer than the diameter of the apple are found,
the apple certainly contains a defect. On the other hand if there is only a single dark
black/blue area in all images of the apple, these are representative of the stem
ends/calyxes and there is a good chance that there are no defects and the apple is ok.
The question remains, how can the distances between segmented dark blobs and
diameter of the apples be calculated to ascertain the presence or absence of
defects? There are many ways to do this. Two different ways are proposed. Method 1
is a generic method and Method 2 is specifically designed for the purpose of the
project and also provides alternatives.
Method 1
The centre of the mass of each segmented dark blob can be calculated. Suppose there
are 2 dark blobs then their centres may be (x1, y1) and (x2, y2). Once the centres are
found the distance between the two centres is,
sqrt ( (x2 – x1) ^2 + (y2y1) ^2 )
23
Likewise the distances between all the blobs can be calculated. To calculate the
diameter of the apple, a simpler thresholding function for the whole apple could be
done in greyscale. If the centre of the mass is found for the whole apple, the distance
between the centre of the mass to the pixels on the edge (the thresholded image’s edge
can be found using Canny Edge Detection) can be computed. Now, by computing all
the distances between the centre and the edge, the radius could be the average of the
maximum and minimum distances. Alternatively the distance could be the average of
all the distances measured between the centre of mass and each pixel at the edge. If
the apple is distorted, the radius could be a bit further from what would be measured
by hand, but would be accurate enough for most cases. Finally compare the distances
between the blobs and diameter of the apple to determine the presence or absence of
defects.
Method 2
Fit circles/ellipses on the colour segmented image produced by Algorithm 1 and
calculate distances based on the centres of the fitted circles. (This can be done
using Hough Transforms)
Hough Transform
Hough Transform is used to confirm whether a subset of pixels that belong to edges
form a certain geometry to be classified as straight lines. Hough transform uses the
steps mentioned below in deriving straight lines .
Obtain a binary image, the edges marked with 1
Create a , pl ane to represent the straight lines passing by point (x,y)
24
Extract the points of the , pl ane that are above a certain threshold
Examine the continuity of the chosen lines
(Gonzales & Woods, 2002).
Hence the straight line equation can be written as
x cos ( ) + y sin ( ) =
In OpenCV the function cvHoughLines2 ( ) can be used to implement the Hough
transform. Hough transforms can also be used for circles, ellipses etc. However it
must be borne in mind that parameters have to be represented with accuracy
otherwise points that do not fall within the curve might be considered to be so. Also,
transforms of points that belong to the same curve might never intersect.
Alternatively, can use the OpenCV function cvFindContours ( ) and then fit a circle
on the contours found. This method was used in the project.
The colour segmented image produced by Algorithm 1 would contain the dark blobs
as well as stem or calyx also segmented in dark. Mark the centres of all the circles and
ellipses fitted. Code for Algorithm 2 would find the centres of all the segmented
blobs plus the whole apple through the fitted circles. Now calculate the distance
between all the centres fitted on the circles using the following equation.
sqrt ( (x2 – x1) ^2 + (y2y1) ^2 )
For each fitted circle must also calculate the radius. The ellipse with the largest radius
would be the whole apple’s radius and can now calculate the diameter of the apple.
Distances between all the centres of the fitted circles can be stored in an array. And
now can search through the array and find the largest distance. If the largest distance
25
between the blobs is less than the diameter of the apple and there are more than 1
fitted circle, there is a skin defect/blemish on the apple’s surface.
Method 2 was used for this part of the project and is by far the most interesting part of
the process as it involves several calculations and is logically thought. The output
image given by the 1st algorithm with the colour segmentation is fed as the input to
the 2nd algorithm which will ascertain whether the colour segmented image contains
any defects/stem ends or calyxes and will produce an output image with centres of the
fitted circles marked and the circles fitted in greyscale. The 2nd algorithm, which uses
Method 2 can be summarised as follows.
Algorithm 2
for (i=0;i< 4 or 5; i++) {// Number of images per apple must be at least 4 if not 5
At image i
Fit a circle for all contours found in colour segmented image[i]
Calculate all the centres (x, y) of the fitted circles and mark in the image
Store the radius of the fitted circles
Find the diameter of the Apple
Find the distances between all the centres of the fitted circles
IF (number of fitted circles is more than 1) AND (maximum distance
between centres of the fitted circles is < Diameter of the Apple)
Then Apple is likely to have defects
IF (number of fitted circles is 1 (other than for the whole apple)
26
Then the segmented area is likely to be from a stem end or calyx, Apple is
likely to be OK. No defect found.
If (maximum distance between centres of the fitted circles is = = Diameter of
the Apple) Then stem and calyx both found on the same image! Apple
might be defected as this cannot happen!
}
After going through all 4/5 images of the same apple as above
If only one or no dark area found in all images NO DEFECT PRESENT
If there are more than 1 dark area in at least 1 image out of the 4/5 colour segmented
images DEFECT PRESENT
The results of the 2 algorithms run on the Apple11.jpg are described as follows.
Image 10 Image 11
27
After going through all 4/5 images of the same apple as above
If only one/no dark area found in all 4/5 images NO DEFECT PRESENT
If there is more than 1 dark area in at least 1 image out of the 4/5 colour segmented
images & distance between the two dark areas in that image is less than the diameter
of the apple DEFECT PRESENT
Image 10 presented the original image of the apple which is Apple11.jpg in the
collected database. Algorithm 1 produces the colour segmented image output.jpg as
shown on image 11. The output image clearly separates the healthy apple skin in RED
and GREEN whilst the defected patch is separated in BLUE and the stem end is
separated in BLACK.
The output.jpg is fed as the input image to Algorithm 2 which produces the
output19.jpg and ascertains whether segmented blobs represent defects or stem ends
or calyxes. Image 12 represents the resultant output of Algorithm 2 and the terminal
represented in figure 3 details the findings from the algorithm on this image.
Image 12
28
Figure 6
Image 12 and Figure 3 portrays the strength of the 2nd Algorithm written clearly to
distinguish whether there is sufficient proof to deem whether defects are present in
the apple based on image examined. Image 12 has accurately fitted 5 ellipses with the
centre of the ellipses marked accurately with a cross. However, 2 circles fitted on the
lower left areas on Image 12 are not desired. These are from the two green coloured
segmentations from Image 11 which have been deemed as ellipses in Algorithm 2.
The bi – colouring of healthy skin in RED and GREEN in Algorithm 1 was done so
that the natural colours of bicoloured apples would be preserved in the colour
segmentation.
This is purely complementary and at step 2 of the process where segmentation has to
be further refined, it has lead to unnecessary marking of ellipses. If the colour
segmentation at Algorithm 1 is confined to healthy skin, segmented in just one colour
(RED or GREEN not BOTH) the problem is likely to be alleviated. Therefore,
29
algorithm 1 was refined to segment the healthy skin in one colour, RED. Image
Apple11.jpg (Image 10) was subjected to the refined Algorithm 1 and Image 13 was
obtained which has healthy skin segmented in RED only.
Image 10 Image 13
Now Image 13 can be subjected to Algorithm 2 for Defect or Stem Calyx refinement.
Image 14 which is the resultant out image from Algorithm 2 now only has 3 fitted
circles which are for the defected area, stem end and whole apple. The two undesired
fitted circles as seen on Image 12 have been successfully eliminated through
refinement of Algorithm 1.
30
Image 14
Figure 7
Logic of Algorithm 2 applied on results (Image 14 and Figure 4)
31
Figure 4 represents the terminal which specifies the results of Algorithm 2. It has
correctly fitted 3 ellipses as seen on Image 14 and initially states the possibility of 2
skin defects, where as there is only 1 defected area in the apple. However, upon
calculating the distances between segmented blobs/centres of circles it has found that
all the distances are less than the diameter of the apple and as there are at least 2 fitted
ellipses representing at least two dark black/blue regions which are much closer than
the diameter of the apple, as per the logic there is a skin defect present. This is correct
in comparison with the original Image 10. The algorithm and logic is therefore correct
and can now be practically applied.
32
5 Results and Discussion
Background removal and object segmentation processes were applied on each of the
images in the collected database. During the initial stages of the experiments the
background not being captured in complete dark resulted in parts of the background
segmented in RED, GREEN or BLUE. This was overcome by further enhancement to
the images before subjecting to the colour segmentation. The latter images were
obtained bearing this in mind, such that Algorithm 1 would be able to compel the
background to be completely dark and thus not interfere with the segmentation.
The results from Algorithm 1 and Algorithm 2 were convincing. Segmentation in
Algorithm 1 was obtained by thresholding the pixels to obtain the dominant colour
RED/GREEN in healthy skin and BLUE/BLACK in background or blemishes. The
colour segmentation produced by Algorithm 1 was able to accurately segment the
healthy skin of the bi colour apples in to RED & GREEN (this was subsequently
changed to RED for pixels both showing high RED/GREEN levels as segmenting in
both RED and GREEN gave rise to problems at Algorithm 2). The Algorithm 1 also
successfully segmented skin blemishes, patches, stems and calyxes into either BLUE
or BLACK. This formed the basis of the identification and segmentation of defects on
bi colour apples. The results were satisfactory as Algorithm 1 segmented the areas in
the apple images as desired in 95% of the cases.
33
Fig. 7. Examples of object segmentation using Algorithm 1 (Thresholding). The
colour segmented images are displayed in resolution 640 * 480.
However at this stage of the experiments no differentiation had been made between
defects/blemishes and stem ends/calyxes which presented a practical problem.
Although attention was drawn to methods that ‘rebuild’ the image, a simple
comparison of areas was done.
By finding two dark areas side by side at a distance less than the diameter of the apple
would indicate a defect while the presence of one dark area could be the stem end or
calyx and therefore indicates a healthy surface. With one camera the calyx and the
stem would not be visible simultaneously, unless the apple is distorted. This logic
provided the backbone for Algorithm 2 which uses the colour segmented image
produced by Algorithm 1 as the input and provides a grey scale image with circles
fitted on the apple, defected area, stem and calyx. The kernel states the final
determination and classification of the apple based on Algorithm 2. However this
decision cannot be made by examining just one image and hence needs up to 4/5
images of the same apple. It is known that there is a good chance by rolling the apple
at chance the defect/blemish appears together with either the stem or calyx. This
meant that if two dark areas are found that are closer than the diameter of the apple
certainly there is a defect/blemish. On the other hand if there is only a single dark area
34
in all the images of the same apple (4/5 images per apple at least), it is concluded that
this represents either a stem or calyx and thus the apple is good and free of surface
defects.
Once a defect is recognised in one image, there is no need to examine further in other
images of the same apple, however if no defect is recognised must go through all 4/5
images of the same apple to classify with credibility.
The progression of the whole process is summarised as below and must be followed
to obtain a credible classification.
35
Step 1: Apply the Algorithm 1 on the RGB
image of apple
&
36
Step 2: Examine the resulting image. If there are dark blue/black segmented areas > possibility of stem/calyx or defects/blemishes in image. Needs further
Step 3: Apply the Algorithm 2 on the colour segmented image obtained at step 2. Examine the greyscale image and also the
Fig. 8. The flow chart of the defect detection and classification
Results from Algorithm 2 were satisfactory in 85% of the cases. Even through for
some stand alone single images of apples the classification Algorithm 2 gave false
results, when the results from 4 5 images of the same apple were combined and
overall result analysed in relation to the logic presented for “Differentiating between
defects and stems/calyxes” the result was accurate. Therefore instead of a
sophisticated method, a simple comparison of areas could be done to accurately
differentiate and classify.
37
Step 4: If the terminal states that a defect has been found > Stop at Step 4 as evidence is sufficient! Apple has a Defect
If the terminal states that no defect has been found and the Apple is OK AND if go back to Step 1 and redo the process with another image of the same apple.
If, after having subjected 4 or 5 images of the same apple, at every time terminal output is No Defect/ Apple is OK> stop as evidence is sufficient
The 15% of cases that are misclassified or unrecognised were examined thoroughly to
find the similarities of these images (defects, SCs) and the following were observed.
(1) If a stem end or calyx is located far from the centre of the fruit especially close to
the edges of the apple then it has a high probability to be unrecognised.
(2) Sometimes healthy tissue closer to the edge of the fruits were segmented in
blue/dark indicating a defect/stem/calyx due to illumination artifacts related to the
varying slope of fruit surface to the camera.
(3) If two or more objects are touching (stem/calyx and defect) then they are likely to
be recognised as one object and this leads to inaccurate determination at Algorithm 2
(4) If an object is partially segmented due to erosion, it will most likely be incorrectly
recognised or unrecognised.
(5) If the defect/blemish is of a certain nature such as russet or mild bruises, they are
unlikely to be recognised and segmented as these present a very similar colour to the
healthy tissue of bi colour apples.
Although this method recognises defects, the presence of a defect does not mean that
it cannot be marketed. For example the European Commission’s marketing standard
states taking into account only the fruit size and skin quality skin defects must not
extend over 2.5cm2 in area and the minimum diameter of the fruit should be 65mm
for an apple to be regarded as healthy (Unay and Gosselin, 2004). This can lead to the
calculation of a defect ratio (defected area / fruit area) which can be used as a
classification tool to ascertain whether the apple is rejected or accepted for marketing
reasons. This can be further extension to the project.
Decision Ratio
Area Defected / Area fruit = A defected / ( . (d/2) π 2 = 2.5cm 2/ ( . (65mm/2) π 2)
Decision Ratio (r) 0.07534≈
38
6 Conclusion
External quality grading of apple fruits by computer vision is still a largely open,
complex and extremely challenging problem. Accuracy of this task largely depend on
several key features such as quality of images, precision of objects in images,
recognition of stem ends/calyxes and distinguishing between stem/calyxes and
defects. In this project a clear recognition of defects was introduced through threshold
– based colour segmentation and compelling the resultant image to segment healthy
bicolour skin with the dominant colour GREEN/RED and thereby segmenting the
defects and SCs in either BLUE or BLACK.
The fruit area is extracted from the background through thresholding. Two algorithms
were developed and the 1st Algorithm performed the colour segmentation. Defect
detection in fruit grading also involves recognition of stem ends and calyxes as
defects must not be confused with these. Therefore 2nd Algorithm was developed to
cater for these and includes a logical step with comparison of areas and distances
between segmented objects. The results from both algorithms were encouraging and it
is recognised that the progression from Algorithm 1 to Algorithm 2 and Algorithm 2
overall after comparison of 4 – 5 images may result in the overall classification of the
apple as either free of defects or defects present. The results showed 85% of the
cases were can be classified correctly.
The database in composed of images of stem ends/calyxes, defects and apple on the
whole with various orientations relative to the camera. As the logic behind Algorithm
2 requires rolling the apple at chance to obtain satisfactory results, images were
obtained in such a manner rather than at planned orientations relative to the camera.
39
7. References
Barczak, A. (2006). Machine Vision Study Guide. Institute of Information and
Mathematical Sciences Massey University. 1 – 70.
Blasco, J., Aleixos, N., & Molto, E. (2006). Computer vision detection of Peel defects
in citrus by means of region segmentation algorithm. Journal of Food Engineering,
81, 535543.
Retrieved July 07, 2007, from Science Direct Global database.
Cao, Y.K., Wang, C.L., & Li, Y.F. (2005). Fruit Images Segmentation Based on
Fuzzy Art Model. Retrieved
March 12, 2007 from
http://www.ieeexplore.ieee.org/iel5/10752/33894/01614742.pdf
Destain, F., & Leemans, V. (2003). A realtime grading method of apples based on
features extracted from defects. Journal of Food Engineering , 61, 8389.
Retrieved July 02, 2007, from Science Direct Global database.
Gonzalez, R. C., & Woods, R. E. (2002). Digital Image Processing. New Jersey:
Prentice Hall.
Milella, A., Reina, G., Foglia., & Gentile, A. (2004). Computer vision applications in
agricultural robotics. Retrieved
June 15, 2007, from
http://www.giulio.reina.unile.it/GR/M2VIP_1.pdf
40
Moltó, E., Blasco, J., & Benlloch, J. V. (1998). Computer vision for automatic
inspection of agricultural produces. In SPIE Symposium on Precision Agriculture and
Biological Quality. 1–6 November 1998, Boston, MA, USA
Q. Yang, Finding stalk and calyx of apples using structured lighting. Computers and
Electronics in Agriculture 8 (1993), pp. 31–42
Q. Yang, An approach to apple surface feature detection by machine vision.
Computers and Electronics in Agriculture 11 (1994), pp. 249–264
Q. Yang and J.A. Marchant, Accurate blemish detection with active contour models.
Computers and Electronics in Agriculture 14 (1996), pp. 77–89
Unay, D., & Gosselin, B. (2004). A Quality Sorting Method for ‘Jonagold’ Apples.
Retrieved
April 16, 2007, from
http://www.tcts.fpms.ac.be/publications/papers/2004/ageng04_dubg.pdf
Unay, D., & Gosselin, B. (2004). Artificial Neural NetworkBased Segmentation And
Apple Grading By Machine Vision. Retrieved
May 12, 2007, from
http://www.ieeexplore.ieee.org/iel5/10242/32661/01530134.pdf?arnumber=1530134
Unay, D., & Gosselin, B. (2005). Stem and Calyx Recognition on ‘Jonagold’ Apples
by Pattern Recognition. Retrieved
March 12, 2007, from
http://www.tcts.fpms.ac.be/publications/regpapers/2006/unay_jfoodeng_dubg.pdf
41
V. Leemans, H. Magein and M.F. Destain, Defect segmentation on ‘Jonagold’ apples
using colour vision and Bayesian method. Computers and Electronics in Agriculture
23 (1999), pp. 43–53
8. Bibliography
Barczak, A. (2006). Machine Vision Study Guide. Institute of Information and
Mathematical Sciences Massey University. 1 – 70.
Gonzalez, R. C., & Woods, R. E. (2002). Digital Image Processing. New Jersey:
Prentice Hall.
42
9. Appendices
Code for Algorithm 1
#include "cv.h"
#include "highgui.h"
#include <math.h>
#include <stdio.h>
#define TRESHOLD 6
char* filename;
IplImage *image = 0, *image2 = 0,* auximage=0;
uchar* pixel;
int sc=0, apple=0, defect=0;
int main( int argc, char** argv ){
if (argc == 3) { filename=argv[1];} //Remember might need to change the # of
arguments
else exit(0);
if( (image = cvLoadImage( filename, 1)) == 0 )
return 1;
image2=cvCloneImage(image);
//Access pixels of the colour image and then segment the background by darkening
for (int pos_y=0;pos_y<image2>height;pos_y++){
for (int pos_x=0;pos_x<image2>width;pos_x++){
pixel=&((uchar*)(image2>imageData+image2
>widthStep*pos_y))[pos_x*3];
43
if(pixel[2]<10) {// The value of the RED Channel tend to be
rather small in the BLACK Background. So if a pixel's Red Colour Value is under
pixel[0]=0;// 10 this pixel is likely to be part of the
background
pixel[1]=0;
pixel[2]=0;
}else if( ((pixel[2] >10) && (pixel[1] > 70 )) && (pixel[1]
pixel[0] >40) &&(pixel[1]>pixel[2]) ) {
pixel[0]=0; //BLUE
pixel[1]=255;//GREEN
pixel[2]=0;//RED
apple ++;
}else if( ((pixel[2]>10) && (pixel[2] > 55 )) && (pixel[2]
pixel[0] > 40) && (pixel[2] >pixel[1]) ) {
pixel[0]=0; //BLUE
pixel[1]=0;//GREEN
pixel[2]=255;//RED
apple++;
}else if( ( (pixel[2]>10) && (pixel[0] > 15) ) && ( (pixel[2]
pixel[0] < 50) && (pixel[1] pixel[0] <50) ) && (pixel[1] + pixel[2] < 400) ) {
pixel[0]=255;//BLUE
pixel[1]=0;//GREEN
pixel[2]=0;//RED
defect++;
}else if ( pixel[0] + pixel[1] + pixel [2] > 255 ) {
pixel[0]=0;//BLUE
pixel[1]=0;//GREEN
pixel[2]=255;//RED
44
apple++;
}else if (pixel[0] + pixel[1] + pixel [2] > 30){
pixel[0]=0;//Blue This may have to be changed to either
BLUE or BLACK depending on picture quality
pixel[1]=0;//Green
pixel[2]=0;//Red
}else{
pixel[0]=0;//Blue
pixel[1]=0;//Green
pixel[2]=0;//Red
}
}
}
cvSaveImage (argv[2] , image2);
cvReleaseImage(&image);
cvReleaseImage(&image2);
return 0;
}
Code for Algorithm 2
#include <cv.h>
#include <highgui.h>
#include <math.h>
#include <stdio.h>
#define TRESHOLD 30
#define RADIUSCOUNTER 20
int xcounter[RADIUSCOUNTER];
int ycounter[RADIUSCOUNTER];
45
double radiuscounter[RADIUSCOUNTER];
double distancebetweenblobs[100];
int FindCircles(IplImage* input_img, int x,char * resultimage);
double blobdistance(int x1,int x2, int y1,int y2);
double findmaximum(double array[ ], int a);
int main(int argc, char** argv)
{
IplImage* src;
int contourtotal=0;int counter=0;
double diameter=0;double distance=0;
char * resultimage;
if( argc == 3 && (src=cvLoadImage(argv[1], 0))!= 0){
resultimage=argv[2];
contourtotal=FindCircles(src,0,resultimage);
printf("\nThe total number of elipses found is %d\nNumber of Defects found
is likely to be %d\n",contourtotal,contourtotal1);
diameter =findmaximum(radiuscounter, contourtotal) * 2;
printf("The Diameter of Apple is %f\n",diameter);
if (contourtotal <= 1) {
printf("Only one blob found, this is LIKELY to be the stem end or
calyx,\nno defects found, APPLE is OK\n");
exit(0);
}
for(int i=0;i<contourtotal;i++){ // Put in an array the distances between all the
blobs
46
for(int a=0;a<contourtotal;a++){
if(i != a) {
distance=blobdistance(xcounter[i],ycounter[i],xcounter[a],ycounter[a]);
distancebetweenblobs[counter]=distance;
//printf("Distance is %f\n",distance);
//if (distance >= diameter) printf("Stem End and Calyx
both found in the same image\n");
//else printf("Skin Defect Found\n");
counter++; //maximum #' of distances between
blobs...contourtotal * (contourtotal 1)
}
}
}
printf("Distances between blobs are\n");
for(int a=0; a<counter;a++){
if(a%5==0) printf("\n");
printf(" %f",distancebetweenblobs[a]);
if(distancebetweenblobs[a] >=diameter) printf("Stem End and Calyx
both found in the same image\n");
}
if( (findmaximum(distancebetweenblobs, counter) < diameter) &&
(contourtotal >1) ) {
printf("\n\nThe distances between blobs are all less than the Diameter
of the Apple. Can conclude, Skin Defects are found on the Apple surface!\n");
}
}else{
47
printf(" Missing Arguments\n");
}
return 0;
}
int FindCircles(IplImage* input_img,int x ,char *resultimage){
CvMemStorage *storage;
int header_size, i, count,length, width;
int circlecounter=x;
CvSeq *contour;
CvBox2D * myBox;
CvPoint *PointArray;
CvPoint2D32f *PointArray32f;
CvPoint myCenter;
myBox= (CvBox2D *) malloc(sizeof(CvBox2D));
header_size = sizeof(CvContour);
storage = cvCreateMemStorage (1000); // For FindContours.
IplImage* tempimage=cvCreateImage(cvSize(input_img>width ,input_img
>height ),IPL_DEPTH_8U,1);
if(input_img>nChannels==3) cvCvtColor(input_img, tempimage,
CV_RGB2GRAY);
else cvCopyImage(input_img,tempimage);
cvCanny(tempimage, tempimage, 60, 70,3);
cvFindContours (tempimage , storage,
&contour,header_size,CV_RETR_EXTERNAL,CV_CHAIN_APPROX_SIMPLE);
48
while(contour!=NULL){
if(CV_IS_SEQ_CURVE(contour)){
count = contour >total;
PointArray = (CvPoint *)malloc(count *
sizeof(CvPoint));
cvCvtSeqToArray(contour, PointArray,
CV_WHOLE_SEQ);
PointArray32f = (CvPoint2D32f *) malloc((count + 1)
* sizeof(CvPoint2D32f));
for (i=0; i<count1; i++){
PointArray32f[i].x = (float)(PointArray[i].x);
PointArray32f[i].y = (float)(PointArray[i].y);
}
PointArray32f[i].x = (float)(PointArray[0].x);
PointArray32f[i].y = (float)(PointArray[0].y);
if (count>=TRESHOLD){
cvFitEllipse(PointArray32f, count,myBox);
myCenter.x = (int) myBox>center.x;
myCenter.y = (int) myBox>center.y;
length =(int)myBox>size.height;
width = (int)myBox>size.width;
double radius=(length + width) /4; //Radius of
the elipse/circle is length+width / 4 of myBox
float myAngle= myBox>angle;
if ((myCenter.x > 0) && (myCenter.y >0))
{
CvPoint pt1, pt2, pt3, pt4;
pt1.x= myCenter.x+2;
49
pt1.y=myCenter.y2;
pt2.x= myCenter.x2;
pt2.y=myCenter.y+2;
pt3.x=myCenter.x2;
pt3.y=myCenter.y2;
pt4.x=myCenter.x+2;
pt4.y=myCenter.y+2;
xcounter[circlecounter]=myCenter.x; //Add the
x,y and radius of each of the elipses found to the respective arrays of X,Y
ycounter[circlecounter]=myCenter.y;//Radius
radiuscounter[circlecounter]=radius;
cvCircle(input_img,myCenter,(int)length/2
,CV_RGB(0,255,255));
printf("The length is %d and the width is %d
and the radius is %f\n", length, width,radius);
cvLine(input_img,pt1,pt2,CV_RGB(0,255,255),2,8);
cvLine(input_img,pt3,pt4,CV_RGB(0,255,255),2,8);
cvCircle(input_img,myCenter,(int)length/2,CV_RGB(0,255,255));
circlecounter=circlecounter +1 ;
}
}
free(PointArray32f );
50
free(PointArray );
}
contour = contour>h_next;
}
free (contour);
free (myBox);
cvSaveImage(resultimage,input_img);
cvReleaseImage(&tempimage);
cvReleaseMemStorage(&storage);
return circlecounter;
}
double blobdistance(int x1,int y1, int x2,int y2){
double squareroot;
int xdifference=x2 x1;
int ydifference=y2 y1;
//printf(" The x1,y1 and x2, y2 are %d %d %d %d\n", x1,y1,x2,y2);
squareroot = sqrt((xdifference * xdifference) + (ydifference * ydifference));
//printf("This is the squareroot %f\n",squareroot);
return squareroot;
}
double findmaximum(double array[ ], int a){
double maximum=array[0];
for(int x=0;x<a;x++){
if (array[x] > maximum) maximum=array[x];
51
}
return maximum;
}
Acknowledgements
The project was completed as partial fulfilment of course requirements for BSc
(HONS) in Computer Science at Institute of Information and Mathematical Sciences
at Albany Campus of Massey University. I would like to thank my supervisor and
dedicated mentor Mr. Andre Barczak of Department of Computer Science at IIMS for
the support, guidance afforded to me during the tenure of the project and for all the
invaluable reviews and feedback. I would also like to thank all my lecturers for their
invaluable teaching, guidance and support provided to me during the course of my
studies at Massey University.
52
