Automatic fruit recognition: a survey and new results ...ben-shahar/ftp/papers/Agrobotics/Fruit... · The sensor used is a laser range-"nder giving range/attenuation data of the sensed

*Corresponding author. Tel.: #34-91-8711900; fax: #34-91-8717050.

E-mail address: [email protected] (A.R. JimeH nez)

Pattern Recognition 32 (1999) 1719}1736

Automatic fruit recognition: a survey and new results usingRange/Attenuation images

A.R. JimeH nez!,*, A.K. Jain", R. Ceres!, J.L. Pons!

!Instituto AutomaH tica Industrial (CSIC), Ctra. N-III Km. 22.8 LaPoveda, 28500 Arganda del Rey, Madrid, Spain"Department of Computer Science, Michigan State University, East Lansing (MI), USA

Received 10 April 1998; received in revised form 29 September 1998

Abstract

An automatic fruit recognition system and a review of previous fruit detection work are reported. The methodologypresented is able to recognize spherical fruits in natural conditions facing di$cult situations: shadows, bright areas,occlusions and overlapping fruits. The sensor used is a laser range-"nder giving range/attenuation data of the sensedsurface. The recognition system uses a laser range-"nder model and a dual color/shape analysis algorithm to locate thefruit. The three-dimensional position of the fruit, radius and the re#ectance are obtained after the recognition stages.Results for a set of arti"cial orange tree images and real-time considerations are presented. ( 1999 Published by ElsevierScience Ltd on behalf of the Pattern Recognition Society. All rights reserved.

Keywords: Range images; Shape recognition; Contour extraction; Circular Hough transform; Agriculture

1. Introduction

1.1. Automatic vision systems in agriculture

The use of computers to analyze images [1] has manypotential applications for automated agricultural tasks.But, the variability of the agricultural objects makes itvery di$cult to adapt the existing industrial algorithmsto the agricultural domain. The agricultural systemsmust support this #exibility, and methods for includingdomain knowledge in algorithms should be studied asa rational way to cope with this variability.

There are many processes in agriculture where deci-sions are made based on the appearance of the product.Applications for grading the fruit by its quality, size orripeness are based on its appearance, as well as a decisionon whether it is healthy or diseased. Humans are easilyable to perform intensive tasks like harvesting and prun-ing using basically the visual sensory mechanism. Thissuggests that a system based on a visual sensor should beable to emulate the interpretation process of the humanvisual recognition system.

The current areas of image analysis research in agricul-ture can be classi"ed into two main groups: Researchtools and Decision-making (Fig. 1) [2]. The "rst group ofimage analysis systems includes applications like plantgrowth monitoring, orphometry of new cultivars or bio-logical cell counts. This type of tool allows a researcherto e$ciently gather the data automatically. The user

0031-3203/99/$20.00 ( 1999 Published by Elsevier Science Ltd on behalf of the Pattern Recognition Society. All rights reserved.PII: S 0 0 3 1 - 3 2 0 3 ( 9 8 ) 0 0 1 7 0 - 8

Fig. 1. Image analysis applications in agriculture.

monitors the performance of the system and can inter-vene when the system misinterprets an image. Theseimage processing tools also allow features to be mea-sured automatically which would be too time-consumingto do manually. The second group of image analysissystems must provide information to guide the mechan-ical equipment. Such systems support two di!erentgroups of applications, Grading and Guidance. The use ofimage processing for grading is being applied to manyproducts, including oranges, potatoes, apples, carrots,green peppers, tomatoes and peaches. The grading maybe for size and shape, color, or the presence of defects.Current guidance research includes harvesting oranges,tomatoes, mushrooms, apples, melons and cucumbers.The guidance research also focuses its attention on navi-gating robot vehicles using machine vision strategies orother simple sensors in order to obtain autonomousmobile capabilities.

The techniques used in the above applications aresuccessful under the constrained conditions for whichthey were designed, but the algorithms are not directlyusable in other applications. In principle, computers are#exible because they can be re-programmed, but in prac-tice it is di$cult to modify the machine vision algorithmsto run for a slightly di!erent application because of theassumptions made to achieve robustness and speed fora speci"c application [3].

1.2. Robotic harvesting

The automatic harvesting of citrus has been doneentirely by hand and the cost of this labor #uctuatesaround 25% [4], 30% [5] and 33% [6] of the totalproduction costs. So, an e$cient robotic system couldreduce the production costs signi"cantly and this is oneof the reasons why the use of an automated roboticsystem for harvesting is so attractive. The other reason isto improve the quality of the fruit that would make theproduct more competitive.

The con"guration of the trees signi"cantly alters thepercentage of visible fruits in the tree. For tree rowcon"gurations, with a hedge appearance, the visibility ofthe fruit can reach 75}80% of the actual number of fruits[4], which is much better than the 40}50% of visibility

for conventional plantings. So, a recon"guration of thecrops should be considered in order to reach the degreeof pro"tability expected when automating a harvestingtask.

There are several techniques used for the harvesting offruits which are not appropriate for the fresh fruit marketdue to the damage caused to the fruit during its collec-tion. These techniques include the shaking of tree limbsor tree trunks, oscillating forced-air removers and thecomplementary chemical treatment. Fruits are usuallybruised when striking limbs during the landing. So, thereis a need for a non-aggressive method to perform theharvesting of fruits as delicately as possible. The manualpicking is the most delicate way to perform the harvest-ing, but it is expensive and time-consuming.

The use of robots to pick tree fruits was "rst proposedby Schertz and Brown [7] in a review of mechanicalcitrus harvesting systems. The basic concepts of roboticharvesting were established in this paper. One of theseconcepts was the line-of-sight approach to fruit picking.This consists of the following three steps: (1) to visuallylocate the fruit with an optical sensor, (2) to guide thefruit detachment device along the line of sight to the fruit,and (3) to actuate the device when the fruit is contacted.A robotic system based on the Schertz approach consist-ing of a simple robotic arm, a B/W TV camera and acontrol computer was built for the harvesting of apples[8]. The TV camera was used to locate the fruit attachedto an arti"cial canopy. The control computer directed therobot arm along the line-of-sight to the targeted fruituntil a contact was made by a mechanical whisker. Nodetachment device was implemented.

D'Esnon and Rabatel [9] presented the "rst version ofthe apple picking robot, known as MAGALI. The robotconsisted of a hollow tube mounted in a vertical supportframe. Attached to the end of the tube was a rotatingcup-e!ector used to detach a fruit from a simulated appletree canopy. The hollow tube could slide in and out,rotate left and right, and move up and down the supportframe. A B/W camera was attached to the support frameto detect the fruit. When the fruit was detected, the tubewas aligned with the fruit. The tube would extend outuntil a contact with the fruit was detected by a re#ectancesensor in the end-e!ector. The cup would rotate behind,cutting the stem and allowing the detached fruit to rolldown to hollow tube into a collection bin.

Other extensive research has been directed at usingrobots for a variety of agricultural harvesting tasks:grapes [10,11], asparagus [12], cucumbers [13], mush-roms [14] and apples [15]. Kawamura investigated theharvesting of tomatoes and used a stereoscopic visionsystem to obtain the three-dimensional location [16].

A second version of the MAGALI robot was construc-ted in 1986 [17]. The new design included a sphericalmanipulator, a camera at the center of the rotation axesand a vacuum grasper. MAGALI is a hydraulically

1720 A.R. Jime&nez et al. / Pattern Recognition 32 (1999) 1719}1736

actuated vehicle, self-propelled and totally self-guided inthe pathways by four ultrasonic telemeters.

An Italian company, AID Catania, designed and builta prototype of a citrus harvesting autonomous robotwith a single arm, driven by a vision system which wasoperated both in the laboratory and in the orange grove[5,18]. This robot has a cylindrical coordinate electricaldriven arm which supports a goal-oriented smart end-e!ector. The end-e!ector is made of a mobile helix anda "xed cylinder. An infrared proximity sensor indicatesthat the orange is at the right distance. Approximately65% of the located fruits were successfully detached. Theauthors also presented their plans to develop a multi-armrobot for automatic picking of oranges, but no morereferences were found about this research.

Harrell presents the design of a citrus picking robotCPR [19]. The robot consists of a single arm with aspherical coordinate system whose joints are actuated byservo hydraulic drives. The rotating-lip picking mecha-nism (PM) includes, in a small cavity at the end of thearm, a CCD video camera, an ultrasonic ranging trans-ducer to provide distance information to objects in frontof the PM, light sources and the rotating lip to cut thestem of the fruit.

The Japanese company, Kubota [20] developed afruit-picking robot which uses a mobile platform to ap-proximate a small four-degrees-of-freedom manipulatorto the detachment area. The gripper had a mobile vac-uum pad to capture the fruit and to direct it towardsa cutting device, an optical proximity sensor, a strobo-scope light and a color camera, with everything protectedby a fork-shaped cover.

The Spanish}French CITRUS project to harvest or-anges, includes an agronomical study, the developmentof a visual system to locate the fruit, the design andcontrol of a harvesting arm, the integration of the grasp-ing and cutting device and the "eld test [3]. There aretwo versions of the robot: one with cylindrical coordinatesystem and a more sophisticated version with sphericalcoordinates. This second version is the same robot usedin the second design of the MAGALI fruit harvester. Thegrasping method used is based on a vacuum sucker andto detach the fruit, a spinning movement is used.

For the harvesting of apples, the AUFO robot wasdeveloped as the Central Enterprise for the Organizationof Agriculture and Food Industry [15]. This robot wasdesigned to use six arms with a movement in a verticalplane due to the use of only two horizontal axes per arm.To sweep the whole volume of the tree, the robot plat-form is moved around the tree by small angular shifts.The position of the apples is computed by a triangulationtechnique using two color cameras.

The harvesting of melons was studied and a prototypeharvester was constructed to selectively harvest thesefruits [21}23]. The system consists of a robot with a Car-tesian manipulator mounted on a frame moved by a trac-

tor. The robot vision system is used to locate the melonsand to guide the attaching device towards the fruit.

Stepanov presents a review of di!erent robotic systemsdeveloped in Russia under di!erent projects [24]. TheMAVR-1 is an autonomous grape robot, the MOP-1 isa vegetable harvesting robot to harvest melons, pump-kins and cabbage and the MIIP-1 is a fruit picking robotto collect oranges and apples.

The AGRIBOT is a Spanish project [25] to harvestfruits with the help of a human operator who has themain responsibility of the fruit detection task. The oper-ator using a joystick moves a laser pointer until the laserspot is in the middle of the fruit. The three-dimensionalcoordinates are recorded and the parallelogram manipu-lator is controlled towards the fruit. A gripper systembased on a pneumatic attaching device and an opticalproximity sensor is used to detach the fruit.

Nowadays, the harvesting of agricultural products islimited to crops which ripen at the same time and whichdo not need individual or delicate treatment [26]. Selec-tive harvesting could increase the e$ciency of produc-tion, and improve the fruit quality.

1.3. Fruit detection review

One major di$culty in developing machinery to selec-tively harvest fruits is to determine the location, size andripeness of individual fruits. These speci"cations areneeded to guide a mechanical arm towards the object.The computer vision strategies used to recognize a fruitrely on four basic features which characterize the object:intensity, color, shape and texture. In the following para-graphs, a review of di!erent approaches is presented.This review is sorted chronologically in order to under-stand the evolution of research in this area.

Schertz and Brown suggested that the location of fruitsmight be accomplished by photometric information, spe-ci"cally by using the light re#ectance di!erences betweenleaves and fruits in the visible or infrared portion ofthe electromagnetic spectrum [7]. Ga!ney determinedthat &&Valencia'' oranges could be sorted by color usinga single wavelength band of re#ected light at 660 nm[27]. This technique was capable of distinguishing be-tween normal orange, light orange and green fruits.

The "rst computer vision system for detecting applesconsisted of a B/W camera and an optical red "lter, andused the intensity data to perform the analysis [8]. In the"rst step, a thresholding is done to obtain a binary image.This binary image is smoothed to eliminate noise andirrelevant details in the image. Finally, for each of thesegments, the di!erence between the lengths of the hori-zontal and vertical extrema are computed. So, a round-ness measure is obtained as well as the centroid andradius values. Then, the density of the region is computedby placing a window, whose size is determined by themean value of the extrema, on the centroid. If the density

A.R. Jime&nez et al. / Pattern Recognition 32 (1999) 1719}1736 1721

of the region is found to be greater than a present thre-shold, the region is accepted as an apple.

Grand D'Esnon developed a vision system, for theMAGALI robot, to detect apples using a color camera[17]. An analog signal processing system was able toselect points of a given color within the image. But, thisvision system required a protective covering to get a darkbackground. In the second version of this system, threecolor cameras were used with di!erent optical "lters.A more detailed description of this new version is givenby Rabatel [28]. The vision system is based on theanalysis of three spectrum bands chosen after a spectro-photometric study in the visible and close infra-redbands. The three color CCD cameras and the threedi!erent "lters (950, 650 and 550 nm) are used to obtainthree intensity images. Some ratio features (with refer-ence to the image "ltered at 950 nm) are used to decidewhich pixels belong to a fruit or to a leaf. After a prelimi-nary study based on the spectral properties of the appletree's leaves and the apples (Golden Delicious (yel-low}green), Red Delicious and Granny Smith (green)), itwas possible to recognize even the green mature apples.The extension of this work to other varieties of apples orfruit trees involves individual spectral studies for eachrecognition problem. No quantitative data is presentedbut the authors declare that not all the fruits are recog-nized and there are failures in the detection. Using a sim-ilar technique, the harvesting of tomatoes with machinevision was investigated by Kawamura [29].

Whitaker presents a system to recognize and locategreen tomatoes in a natural setting [6]. An intensityimage with 256 gray levels is used. The analysis is notbased on the intensity level, but uses shape information.The circular Hough transform (CHT) is applied to a bi-nary edge and direction images. The results obtained arevery sensitive to the user-speci"ed threshold value, andthe best results for a 99% threshold value are 68%correct detection and 42% false detection. The contour ofthe leaves is one of the major problems, since the analysisalgorithm interprets them as possible fruits. The authorsrecognized that, at that time, the algorithm was computa-tionally intensive on a serial processor and can not beperformed in real time.

The AID robot vision system was implemented torecognize oranges by preprocessing the color image withan electronic "lter and locating the fruits by recognizingdistributions of the orientation of maximum gradients[5]. A color camera pixel with an arti"cial lighting isused. An analog electronic "lter enhances the image andduring digitization, 6 bits are used to codify the pixelvalue which is proportional to the closeness of the actualpixel hue to a present reference hue. With this pseudo-gray image, a gradient image and a direction image arecomputed using the Sobel operator. Finally, the sceneinterpretation is done through searching for a match withan object model previously stored. This gradient direc-

tion template is moved step by step throughout thedirection image. Approximately 70% of the visually rec-ognizable fruits were detected. This was one of the "rststudies that attempted to recognize spherical forms in theimage, in this case through the orientation of gradients.This technique was also used, together with a method ofsegmentation by region growing and a search for spheri-cal patterns [30].

Slaughter and Harrel [31] introduced a method tolocate mature oranges based on color images. This sys-tem uses the Hue and Saturation components of eachpixel obtained using a color camera and arti"cial light-ing. So, there is a two-dimensional feature space and twothresholds are employed based on the maximum andminimum values for the saturation and the hue compo-nents. This leads to a linear classi"er that can be dis-played as a square region in the feature plane. Approxim-ately 75% of the pixels were correctly classi"ed. Thisalgorithm (in software) took 2.5 s/image and the authorssuggested a hardware implementation to increase theperformance.

Sites [32] presents a system to recognize ripe applesand peaches. This intensity-based method uses a B/Wcamera and color "lters (630}670 nm) to increase thecontrast between the fruits and the background. Arti"ciallight is used and most of the images are recorded undernight operation. The whole method can be divided into"ve steps: (1) thresholding based on a constant 37%value, (2) smoothing by a binary "lter, (3) segmentationby an eight-neighbor connected component labeling,(4) feature extraction (area, perimeter, compactness,elongation), and "nally (5) classi"cation by a linear deci-sion function or a nearest-neighbor method. Classi"ca-tion results around 89}90% are obtained working atnight and for mature fruits. During the day, an 84%classi"cation accuracy is declared and at least 20% offalse detections. Analysis of the preliminary tests resultedin a selection of a 4.5 mm2/pixel "eld of view resolution,which was able to provide the necessary geometricdetails.

Slaughter and Harrel [33] extended their earlier studyby using the RGB components recorded by a colorcamera as features and a traditional Bayesian classi"ermethod to segment the fruit pixels from the backgroundpixels. So, each pixel has three components (R, G, B) andeach of them is classi"ed as belonging to a fruit or to thebackground. No arti"cial lighting or optical "lters areused. The tests show that 75% of the pixels are correctlyclassi"ed. Harrel et al. [34] present a method to estimatethe size and position of the fruit region which containedan initial valid pixel.

Texture can also be used to segment objects of interestfrom the background. Some fruits have textures di!erentfrom their leaves, some are smooth while others arerough. Texture analysis has been used and might be away to locate some speci"c fruits [35].


The European Eureka Project CITRUS ROBOT, in-volving both &&Instituto Valenciano de InvestigacionesAgrarias'' (Valencia, Spain) and CEMAGREF (Montpel-lier, France), investigated the robotic harvesting of or-anges [4]. Several techniques for the vision system weredeveloped, but none of them was able to recognize non-mature fruits. Three methodologies were used. In the "rstone, a B/W camera in conjunction with a red "lter(630 nm) and two synchronized #ashlights were em-ployed to obtain a uniformly illuminated scene which isas much independent as possible of the environmentalconditions. With the use of a fast thresholding algorithm,80% of the visible fruits were detected but a high rate offailures was found. In the second approach, two B/Wcameras instead of one, and two red and green "lters (630and 560 nm) for each camera were utilized. Computingthe ratio between the gray levels of both the images, thethreshold method works and is independent of the lumin-osity level (the two #ashlights are also used here). Ap-proximately 80% of the fruits were successfully detectedand approximately 10% were false detections. Finally, inthe third experiment, they used a color camera withoutarti"cial illumination. Each pixel with its three RGBcomponents is considered a pattern and a Bayesian clas-si"er is used, similar to the method presented by Slaugh-ter and Harrel [33,34]. Success and failure rates ofapproximately 90 and 5%, respectively, for the visiblefruits were reported. These results were not completelysatisfactory since these performance indices are onlyvalid for mature fruits and the three vision systems pre-sented do not cope with green oranges.

A vision system for the harvesting of melons has beeninvestigated under a close collaborative research betweenthe Purdue University (USA) and The Volcani Center(Israel). In the "rst attempt [36], a B/W camera is used toobtain intensity images of the melon crop. The visiontechnique is divided into two steps. First, there is ananalysis step to identify the melon and its position andsize; this "rst stage performs an image enhancement,a thresholding, a parameter extraction and hypothesisgeneration. Shape and texture parameters in the neigh-borhood of the hypothesized position are computed toobtain the "nal candidates. The second stage performsa knowledge-directed evaluation using rules whichallows to avoid noisy detections and to eliminate mul-tiple occurrences. If the second step is not employed,approximately 89% of success and relatively high rates offalse detections are found, but when using the know-ledge-based rules, 84 and 10% rates are obtained, respec-tively.

The AUFO project, for the harvesting of apples, in-cludes a stereo vision system that uses two color camerasseparated by a certain distance and having a convergingposition [15]. Firstly, there is a segmentation of bothimages based on a threshold value. The regions obtainedare grouped and the mean position per region obtained.

For all the possible pairs of segments between bothimages, the three-dimensional position is computed. Thetechnique used to compute the position is a simple tri-angulation algorithm divided in two steps. The "rst stepgives the X}> position using the projection on the X}>horizontal plane and the second step computes theheights or Z coordinates from each camera viewpoint. Ifthe di!erence between this heights is lower than 40 mm,then an object is considered to be present. Only 41% ofthe visual fruits are detected correctly and some falsedetections appear.

A general vision system for the above melon harvestingproblem is presented by Dobrousin [37]. The visionsystem is divided into two subsystems, a far-vision anda near vision. The far-vision subsystem uses a B/W cam-era to locate the X}> coordinates of the melon. Thenear-vision subsystem uses a B/W camera and a linearlaser source to extract the distance or Z coordinate, sothat a picking arm can be guided. In this work, only themethodology used for the far-vision subsystem is shown.Several images are captured in di!erent blowing condi-tions to avoid occlusion of the melons from the leaves.These images are "ltered, segmented by a histogram-based thresholding, cleaned by a morphological erosionparameter and "nally all the images are integrated byperforming a logical OR operation. The resulting imageis analyzed and some features (shape, area, size) areextracted from each segment. Finally, a rule-based classi-"cation is applied to obtain the valid fruits. Approxim-ately, 80% of the melons are detected and thesegray-level routines have been integrated in a real-timepipelined system. The authors also propose the use ofinfrared images to detect the di!erences of temperaturesthat should exist between the leaves, the soil and themelons.

Benady and Miles present a description of the near-vision subsystem for the melon harvestor robot [26].This system, as explained above, uses a laser line projec-tor to illuminate the scene. This line of light when con-tacting the surface of a melon is recorded as a curved line;the deformation of the initial straight line indicates thedistance to the object by a triangulation analysis. Thistriangulation system is used to get one pro"le at everypreviously present distance gap. These pro"les (not con-tours) are analyzed using the CHT to obtain a matrix ofvotes indicating the candidates for being the center ofa melon. To get the most probable candidates the distri-bution of votes around a pixel is used instead of theabsolute value of votes. For increasing the e$ciency ofthe algorithm, some domain speci"c rules are used. Theserules rely on the following parameters: the expected size,the shape, the position of the ground, and the heightvalue of the presumed fruit pixels that must belong eitherto the surface of the melon or to leaves covering the fruit.All the fruits that were visually discernible were detectedby the system, and no false detection occurred.


Table 1Summary of the most important vision systems for agricultural harvesting. No food inspection systems are included

Research reference Type of fruit Sensor and Detection method applied" Detects Success/falseaccessories! green fruit detection rates#

[8] Apples B/W#F Intensity (Thr#FE#RC) No NR[17] Apples Color Color (Thr) No NR[17,28] Apples 3 Color#3 F Color (Ratio#Thr) Yes 50%/O0%[6] Tomatoes B/W Shape (Edge#CHT) Yes 68%/42%[5] Oranges Color#F#L Shape (Gradient#Matching) No 70%/NR[31] Oranges Color#L Color (Hue&Sat#LC) No 75%/NR[32] Apples and

PeachesB/W#F#L Intensity (Thr#FE#LC) No 84%/20%

[33,34] Oranges Color Color (RGB#BC) No 75%/NR[4] Oranges B/W#F#2L Intensity (Thr) No 80%/High%[4] Oranges 2 B/W#2F#2L Intensity (Ratio#Thr) No 80%/10%[4] Oranges Color Color (RGB#BC) No 90%/3}5%[36] Melons B/W Intensity (Thr#FE#RC) No 84%/10%[15] Apples 2Color Color (Thr#Stereo) No 41%/NR[37] Melons B/W#Blower Intensity (Thr#FE#RC) No 80%/NR%[21] Melons Laser&B/W#

BlowerShape (Pro"le#CHT#RC) Yes 100%/0%

[38] Oranges B/W#L Shape (Concv#Thr&Fitting) Yes 75%/8%[39] Tomatoes Color Color (Hue&Sat#Thr) No 90%/NR%

!(B/W"Black/White camera, Color"Color camera, F"Filter, L"Arti"cial lighting)."Thr"Thresholding, FE"Feature extraction, LC"Linear classi"er, BC"Bayesian classi"er, RC"Rule-based classi"er,

RGB"Red-Green-Blue feature space, Hue&Sat"Hue-Saturation feature space, CHT"Circular Hough Transform, Gradi-ent"Gradient image, Concav"Concavity image, Pro"le"Pro"le image).

#(NR"Not Reported).

For the purpose of detecting oranges during the initialstages of maturity, a system reported by the Spanish-French CITRUS ROBOT project [38] uses #ashlampsand a B/W camera to obtain an intensity image of thescene that must have concave surface where a fruit ispresent. This approach uses the shape information andnot only the intensity levels, like previous work, to detectspherical objects. The algorithm can be divided into twosteps. The "rst stage computes another image indicatingthe degree of concavity. The raw image is thresholded toconsider only those pixels which have certain curvatureand thereby reducing the computing time required forthe next step. The second stage consists of "tting anellipse to the initial image for all the points that passedthe threshold. This "tting gives an error index indicatingthe goodness of the "t in two directions, and "nally thisinformation is weighted and used in conjunction with thethresholded image to obtain the "nal segmented image.This system recognizes oranges in the "rst stages ofmaturity and results of 75 and 8% of success and falsedetection rates, respectively, are reported. The false de-tections are mainly due to the presence of sky or patchesof sky. The processing time per fruit is about 20 s andaround 3 min for each scene.

A robotic system for greenhouse operation, AG-ROBOT, was developed at CIRAA in Italy [39]. The

vision system used for this project is based on a colorcamera that supplies the HSI color components. Hue andSaturation histograms are employed to perform a thre-sholding to segment the image. The three-dimensionalinformation is obtained by a stereomatching of two dif-ferent images of the same scene. About 90% of the ripetomatoes are detected and the most frequent errors aredue to occlusions.

There is a study for the recognition of partial circularshapes which was tested for the detection of brokenbiscuits in sorting applications [40]. In this work also,the technique is applied to the recognition of oranges ina tree using a color camera. Since the oranges are matureand the leaves are green, the image has enough contrastto apply an edge detection procedure and a contourimage is obtained. The technique presented can bedivided in two steps: an initial segmentation of contoursobtaining groups of pixels with constant curvature, anda second step of contour segment grouping to obtaincircle candidates and their parameters (radius, center andratio of visible contour). The method works very wellwhen a good contour image is obtained, like in the biscuitapplication, but there are serious problems for the detec-tion of fruits since the contour due to the occlusion of anorange by another orange or by a leaf generates falsecandidates.


A major problem in segmenting intensity or gray-levelimages lies in the selection of the threshold value thatdistinguishes an object from the background [41]. Thisvalue depends on the illumination of the scene and thereis no a priori knowledge about it because the illumina-tion conditions can vary randomly. For instance, a fruitin the sun appears ten times brighter than a leaf in thesun, a fruit in the shade appears four times dimmer thanthe leaf in the sun.

Most of the above vision systems (Table 1) give thetwo-dimensional position of the fruits. The thirddimension about fruit location is usually obtained bymoving the gripping device throughout the line-of-sightuntil the presence of the fruit is detected. This detection isperformed using di!erent sensors like touch sensors[8,17] or ultrasonic sensors [31,33,34]. Some approachesuse stereoscopic vision to indirectly compute the positionof the fruit [16,15,39]. The use of a sensor whichdirectly gives the three-dimensional informationreduces the computing time required to perform a stereo-scopic matching or simplify the task of directing therobot arm towards the fruit. In this sense, the onlyapproach using a 3-D measurement system was present-ed for the harvesting of melons [26], but it was necessaryto use a traditional camera-based stage to obtain theX}> coordinates due to the small "eld of view of the3-D sensor.

2. Objectives

The main objective of this research is to develop animage analysis system capable of locating near-sphericalfruits (oranges, apples, peaches) in natural tree sceneswhile meeting the following requirements:

(1) The system should be able to recognize and locateboth ripe and close-to-leaf-color fruits (green fruits).

(2) The method should be applicable in situations wherecertain areas of the fruit are not visible due to partialocclusion by leaves or by overlapping fruits.

(3) The system should be robust enough for operating inthe presence of di$cult conditions like bright sunre#ections, shadows, variable lighting conditions,night operation and small noisy patches of sky in thebackground.

(4) The system output must supply the three-dimen-sional position, the approximate size of the fruit andan index indicating the degree of ripeness of the fruit.This information allows a robot harvester to performselective harvesting.

(5) The algorithm must operate in real time on a general-purpose sequential processor with the support ofspecial image processing boards. A processing time of1 s per fruit is considered to be acceptable.

3. Methodology

A general data #ow of the fruit recognition system isgiven in Fig. 2. The natural scene is sensed and digitizedby a three-dimensional scanner. This sensor which will bedescribed in the next section, gives the spherical coordi-nates of each scene point as well as a value indicating theattenuation of the laser energy due mainly to the dis-tance, the surface type and orientation of the sensedsurface. So, for each full scan, four digital images areobtained. Two images represent the azimuth and elev-ation angles (AZ(x, y) and E¸ (x, y)), the distance orrange is included in RANG(x, y) and the attenuation is inA¹¹E(x, y). As can be seen, no natural lighting shadowsappear since an active sensor is used and the laser beamis, in this case, the light source.

After the above image extraction, an image processingand generation process is carried out. An image enhance-ment technique is applied to the RANG(x, y) andA¹¹E(x, y) to increase the quality of these images. But themost interesting aspect is based on the sensor model pre-viously computed using a set of di!erent kinds of surfaces atdi!erent distances and orientations. This model allows usto know the re#ectance of the surface, which onlydepends on the type of surface, but not on the distanceand orientation of the sensor with respect to the sensedobjects. So, the re#ectance image REF¸(x, y) theoret-ically give us an image whose pixel values depend only onthe energy absorbing ability of the object surface. Thisimage could be used as an ideal one, but the need forcomputing the surface normal with high precision at eachpixel, leads to a noisy image when non-soft surfaces arepresent.

The same model permits to obtain another imageAREF(x, y) (apparent re#ectance), which does not re-quire the estimation of the surface normal. This image issimilar to an intensity image obtained with a TV-camerausing a red "lter and also utilizing a high-power lightingsystem placed along the axis of the camera. But, ourAREF(x, y) image has an advantage over the formerimage; based on the scene knowledge, distances to validsensed points are known, and so high value pixels can berejected if they are outside this range. With this know-ledge-based image transformation, AREF(x, y) is notperturbed by patches of sky, objects far away or pointsbelonging to the soil. So, the AREF(x, y) image can onlybe compared with images obtained after a classi"cationhas been done to distinguish between the objects and thebackground, using color TV-cameras and arti"cial illu-mination. Finally, we can conclude that the AREF(x, y)image has a quality at least as good as the best TV-images which we have been able to obtain.

The image analysis process uses three input imagesRANG(x, y), A¹¹E(x, y) and REF¸(x, y) to detect theposition of the fruit (Pos(x, y)), its approximate radius(Rad), the distance from the origin of the 3-D scanner to


Fig. 2. General data #ow diagram of the fruit detection process.

the center of the fruit (Dist), and the mean re#ectance (Re-)of that fruit can be used to determine its degree of ripeness.

This information allows us to perform a selective har-vesting based on the size and the ripeness of the fruits. So,only the desired type of fruit is selected to be detached.The "nal information supplied to the AGRIBOT robotcontrol system is the (X, >, Z) Cartesian coordinatesof the center of the fruit and the localization accuracyexpected.

3.1. The 3-D sensor

The 3-D sensor consists of a point laser range-"nderand a tilt/pan mechanism to direct the laser for scanningthe desired area of the scene. The range sensor is com-

mercially available from a German company, SICKOptic Electronic (DME 2000). The sensor is based on theprinciple of phase shift between emitted and returnedamplitude modulated laser signal. Following are some ofthe main technical features of this sensor:

f Resolution: 1 mm.f Consistency: 1}25 mm (depends on the target re#ec-

tance).f Accuracy: $5 to $65 mm (depends on the target

re#ectance).f Max. measuring range: 2047 mm (Con"gured to

measure from 600 to 2647 mm).f Wavelength: 670 nm (red color).f Laser class: 2.


Fig. 3. Some examples of range and attenuation images for two di!erent arti"cial orange tree scenes. At the top, from left to right: scenewith four mature oranges; the range image and the attenuation image. At the bottom, another sequence for a scene with four greenoranges.

f Measuring rate: 29 ms (100 ms when also measuringthe attenuation).

f Light spot size: 3 mm (measuring distance 2 m).

The scanner mechanism is programmable to allow toselect the desired area of the scene to be scanned and thespatial resolution needed. The spatial resolution variesfrom the center of the image to the boundary, since theangular resolution is constant. This fact does not deformthe shape of the fruits due to the symmetric shape of thesespherical objects. So, there is no need for any type ofcorrection before processing the captured image. Thespatial resolution used in the set of images recorded forthis study ranges between 1.5 to 3 mm/pixel. A spatialresolution of 3 mm/pixel is appropriate to have a detailedinformation about the objects shape.

The sensor supplies several digital and analog signals,but two of them are the most useful: the range to thesensed surface and the signal attenuation. Some rangeand attenuation images are shown in Fig. 3, and theintensity image of the same scene obtained with a photo-graphic camera is also displayed for comparison. The sizeof these images are 100 by 100, and the time required tocapture them is around 1000 s since the measurementtime is 100 ms. The slow scanning speed is not admissiblein a practical application and a faster sensor must beused for a practical recognition system.

The range and attenuation signals can be used toderive additional information about the scene based on

the model of the laser range-"nder. This model allowsus to obtain the re#ectance, the appearance re#ectance,the precision and the standard deviation of the digi-tized pixel. A more detailed description of the sensormodel is given in Appendix A. Using a visible red laserwavelength, like in the present work where we use a670 nm/laser, there is a contrast between green/bluesurfaces and red/yellow/orange/white objects. Thisfact is interesting when a color analysis algorithm isemployed, since the objects belonging to the secondgroup of colors are easily separated from a green/bluebackground. Also, the sensitivity of the sensor when thecolor surface changes from green to red, gives a good clueto deduce the degree of ripeness. These reasons suggestthe use of a red laser source instead of an infrared orgreen laser. But, if only the shape of the scene is going tobe analysed to recognize the objects, the infrared telem-eter versions are preferred since the attenuation is lowerand it is independent of the surface color and thereforethe accuracy of the range data is good throughout thewhole image.

3.2. Image processing and image generation

This stage of processing has two basic goals: the gen-eration of new images for an easier analysis and therestoration of these images. Fig. 4 shows a detaileddiagram indicating the #ow of information and thetransformation process. Most of the algorithms are based


Fig. 4. Data #ow diagram of the image processing and image generation process. At the top are actual input images (RANG(x, y),A¹¹E(x, y)) and at the bottom, the output images (PREC(x, y), RANG(x, y), AREF(x, y), REF¸(x, y)).

on the model of the laser range-"nder illustrated inAppendix A.

The range image is almost noise-free when the attenu-ation of the image is low, or in other words, if the scenesurfaces have a good re#ectance. Since this property ismodeled by a function relating the attenuation of thesignal with the standard deviation of the range measure-ment, an adaptive approach can be implemented tochange the restoration coe$cients as the attenuation foreach pixel of the image varies. This technique is able toremove outliers, smooth surfaces and preserve jumpboundaries. The knowledge of the standard deviationexpected for the neighborhood of each pixel andthe di!erence with the actual value give us the informa-tion required to take the most appropriate restorationdecision.

The model of the precision of the range measurementand the re#ectance as a function of the attenuation allowus to generate two new images which will be used infuture processing steps to obtain the position precision of

the fruit and to determine the re#ectance of the fruitwhich indicates the ripeness of the fruit.

The apparent re#ectance image is computed based onthe apparent re#ectance model and some domain speci"cknowledge which give us the necessary support to elimin-ate bright areas that are not created by a fruit or that areoutside the working volume of the robot manipulator.This image is "nally smoothed by a low-pass "lter accom-plishing a good-quality image. The "nal apparent re#ec-tance image is much better than the intensity images ob-tained using a red optical "lter and a B/W camera, since nostrange bright areas appear and there is no need of arti"cialillumination due to the active property of the laser sensor.

3.3. Image analysis approach

This image analysis approach is characterized by theuse of two di!erent images of the same scene: AREF(x, y)and RANG(x, y). These pictures were obtained with thesame sensor, so a direct pixel-to-pixel correspondence


Fig. 5. Data #ow diagram of the dual image analysis approach.The left side represents the color analysis and the right siderepresents the shape analysis.

exists between both images allowing an easier integrationof the individual analysis results. For each of these im-ages, a di!erent analysis algorithm (color and shape) isapplied and "nally a high-level integration is performedin order to take into account both results (Fig. 5).A scene-based knowledge is incorporated to reduce thecomputing time required by the algorithms and to makethe analysis task more robust and immune to noisydisturbances. This information includes the expectedfruit radius interval (30}50 mm), the expected distance tothe fruits (1}2.5 m), the maximum predicted re#ectancevalue of the tree leaves (0.3 for perpendicular incidence)and the angular resolution of the processed image.

The apparent re#ectance image, AREF(x, y), is seg-mented by thresholding at a preset value based on thescene knowledge, so the background pixels are set tozero. The remaining non-zero values are clustered bya labeling procedure based on the Euclidean distancebetween pairs of pixels. During this stage, the maximumapparent re#ectance (minimum distance error) of eachcluster is used to compute the distance to the closestpoint of the fruit. The re#ectance image is employed toaverage the clustered pixels, obtaining an approximateestimate of the re#ectance of the object surface, which canbe used to know the ripeness of the fruit. The positionand radius estimation is based on the extrema positionvalues in the vertical and horizontal directions inside thecluster. The detected clusters without a minimum num-ber of pixels belonging to it are rejected as valid fruit inorder to eliminate the possibility of random small areasof a highly re#ective non-fruit object. Since the size sup-plied by the former method tends to be smaller than thereal size, a range image-based exploration is done start-ing from the previously computed radius value. Oncethe "nal radius is calculated, this value is added to theprevious distance to the fruit's surface to obtain thedistance to the fruit center. Finally, a rule-based rejectionalgorithm is applied to reduce the chance of false detec-tion. This rule is based on the range image and states thatno pixels can be found inside the area of the candidatefruit, with range values greater than the estimated dis-tance to the fruit center. If some candidate violates thisrule then it is not considered a fruit candidate anymore.

The previous algorithm based on the apparent re#ec-tance image and also on the range image which re"nes theresults, basically only detects mature fruits. Such fruits arecharacterized by an apparent re#ectance of 0.3 or higher.This method is not time consuming and allows a quickdetection of three-dimensional fruit position, its size andthe ripeness of the fruit based on the re#ectance value.

A more time-consuming method is based on the shapeof the fruits detected in the range image (RANG(x, y)).A special pseudo-edge detection algorithm is applied todetect steep slopes corresponding to proximity regions tothe boundaries of the fruit, but rejecting the step bound-aries which mainly belong to leaf-to-leaf transitions. Ba-

sically, a gradient map and a direction map are com-puted, but instead of thresholding the image lookingfor the highest values of the gradient, a sandwichthresholding is used based on the values whose selectionis explained in Appendix B. This edge extraction methodgives a set of pixels which is employed to performa specially designed CHT.

The Hough transform is a well-known method forextracting shape information from edge images [42}46].The circular version identi"es the center and radius ofprobable arcs or circular edges. The use of edge image aswell as the direction of the gradient allows us to performthis transform more e$ciently in time and more robustlyagainst false detections. One of the major problems ofthis method is the selection of the threshold value todistinguish between a good candidate to be a circle centerand an insu$ciently voted candidate. We select the high-est voted pixels until a 1% percentage of the total pixels isreached, and a later clustering technique groups the votesto highlight the stronger candidates whose votes couldhave been spread over a certain area due to the non-perfect spherical shape of the fruits. This spreading of thevotes due to the imperfect shape of the fruits could causea high density of pixels with medium votes but none ofthem with enough value to be considered a valid candidate


after the thresholding. If a low enough threshold value ischosen, the medium voted pixels are considered, and ifa posterior clustering technique is applied summing allthe votes for each pixel inside a cluster, the imperfectspherical shape of the fruits is overcome and a robustsystem is obtained.

This CHT not only manages a matrix of accumulatorsor matrix of votes, it also accumulates, for each pixel inthe image, the average distance and re#ectance of thesurrounding pixels which vote for each pixel. This in-formation allows a quick computation of the distance tothe center of the fruit and an estimation of the re#ectanceof the surface of the object which will be used to calculatethe ripeness. Appendix C shows some corrections to thedistance to the center of the fruit, which are needed, due tothe special features of the edge extraction stage, in orderto obtain more accuracy in the distance measurements.

The clustering algorithm is similar to the one used inthe processing of the apparent re#ectance image, but isadapted to manage several images of votes for eachradius tested and the distance and re#ectance votematrix. In this clustering process, the "nal radius, dis-tance and re#ectance are estimated taking into accountthe pixels belonging to each cluster. Clusters withouta su$cient number of votes are rejected to eliminate theappearance of random clusters because of the low initialthreshold values. Finally, as in the color analysis process,the same rule-based rejection algorithm is applied toreduce the chance of the false detections.

The results obtained by the color and shape analysismethods are integrated in order to obtain a "nal resultwith the contributions of both methods (see Fig. 5),resulting in a higher amount of correct detections, butwithout spurious detections produced when the samefruit is recognized by both the methodologies. In thiscase, the position, radius and distance information pro-vided by shape analysis is considered more precise andthe re#ectance is supplied by the result obtained from thecolor analysis method.

Fig. 6 shows some intermediate images, for two treescenes, obtained using color (left side) and shape (rightside) analysis. The four images displayed for the coloranalysis are from top to bottom: the AREF(x, y) image,the thresholded image, the result of clustering and thedetected fruits overlaid on the AREF(x, y) image. The"ve images displayed for the shape analysis are from topto bottom: RANG(x, y) image, a binary version of thegradient image after the two-limit thresholding, thematrix of votes for one of the radius tested, the matrix ofvotes after the clustering and the detected fruits overlaidon the RANG(x, y) image. The objects detected are integ-rated and superimposed over the photographic version ofthe tree scene. Some position shifts occur due to the non-perfect pixel-to-pixel correspondence between these im-ages that were recorded with di!erent sensors and fromslightly distinct observation angles.

Both the images in Fig. 6 include four fruits. In the leftimage the citrus are mature and in the right image thecitrus are green. Since the color of the images in the rightscene is green, the color analysis did not detect any fruit,but we can notice the existence of some specular re#ec-tion in the middle of the fruit that is "nally rejectedbecause of the small size of the clusters. Three fruits arefound in the right scene with the shape analysis, so onefruit is not detected. Looking at the vote image after theclustering, four candidates are present but one of them isa false detection, but fortunately the rule-based rejectionstep eliminates the false detection.

4. Results and discussion

4.1. Experimental results

A set of 15 images were captured by scanning anarti"cial orange tree, containing a total of 38 oranges.This test set of images is not exhaustive considering thenumber of fruits, but contains the most typical con"gura-tions of occlusion and overlapping that are frequentlyfound in a real fruit scene. The test set includes about58% of mature fruits and about 42% of green oranges.

The color analysis method is able to recognize every-one of the mature fruits but obviously none of the greenfruits are detected due to their similarity with the color ofthe tree leaves. False detections, possibly appearingbecause of the presence of bright objects, branches orbackground, are not found showing the robustness sup-plied by the AREF(x, y) image and the rejection stages(size-clustering-based and rule-based rejections).

The shape analysis method recognizes mature fruits aswell as green fruits, but presents di$culties for detectingthe fruit when less than 30% of its contour is not visible.This fact leads to some error in detection of the fruits, butlike in the color method, no false detections are founddue to the robustness supplied by the rejection stages.Table 2 shows the detection results. The overall classi-"cation results show that approximately 87% of thevisible fruits (to a human) are detected and no falsedetections were found. These results do not mean that thesystem is free of false detections; some false detectionscould occur under certain circumstances but its probabil-ity is very low. The recognition system performance (inour case, 87%) varies with the percentage of green fruits;the overall correct detection results ranges from 74% fora set of only green fruits to 100% for orange, red oryellow color fruits.

A specular re#ection is detected in the centerof the fruit even for the green variety. This informationcould be used to increase the performance of the system(some tests indicate 90% for only green fruits) but itimposes some constraints about the fruit surface or skinto be detected and reduces the general spherical object


Fig. 6. Some intermediate images during the image analysis process. At the top-left and top-right, two photographs are shown. Each ofthese images is processed by the color and shape analysis and the "nal results are displayed by overlapping circumferences with thecomputed radius.

Table 2Recognition results for the test set of mature and green orangeswith di!erent degrees of occlusion

Analysis method Partial success/ Final success/failure rate failure rate

Color 58%/0%87%/0%

Shape 74%/0%

applicability to only shiny spherical objects. For thisreason, the specular information was not taken into ac-count in the recognition stages.

4.2. Real-time considerations

The algorithms have been executed on a Pentium-90 MHz processor without special image processinghardware. The software was written in Matlab code


and the Matlab interpreter was used to execute the algo-rithms spending an average of 10 s to recognize eachfruit. The software was not compiled to generate a fastercode, so the timings reported can be improved to copewith the requirements of a real-time application. Anaverage processing time of 1 s/fruit is expected usingcompiled programs and an image processing board.

The use of the color and shape analysis proceeds astwo sequential stages instead of two parallel stages. Anadditional step to remove the objects detected by thecolor stage, can improve the speed of detection since theinput image to the shape analysis becomes simpler. Theshape analysis is approximately 10 times slower than thecolor analysis, and its processing time depends on thenumber of edge pixels in the input image forwarded tothe CHT process. The complexity of the CHT is propor-tional to the number of edge pixels. This sequentialcon"guration can lead to a system with reduced process-ing times for images with mature fruits.

4.3. Future work

Future work should be focused on the improvement ofthe shape recognition stage so that it is able to detectmore number of spherical objects. This way, the overalllocation performance would not depend on the maturitystage of the fruit. To ful"l this requirement, the rangeimage should not only be analyzed by its contour shape,but by pro"le shape or by the curvature of the surfaces.This additional analysis could improve the correct detec-tion rates to a hypothetical maximum limit of 95% of thevisible fruits, but its real-time achievement should bestudied.

The compiled version of the recognition and locationsystem will have to be integrated in the AGRIBOT robotto allow the system to locate fruits in an automatic mode.As it was explained, this system was originally designedto locate the fruits manually. Now, the system couldwork automatically and only the non-detected fruitscould be pointed manually if the additional labor costs,due to the manual operation, are considered advisable.

There is a need for performing a ripeness study tocorrelate the re#ectance information obtained for eachfruit with its ripeness. This study should supply a set oftables or functions, one for each type of fruit or varietyconsidered, relating the re#ectance value with the ripeclassi"cation.

5. Summary

A review of di!erent vision systems to recognize fruitsfor automated harvesting is presented. This survey ofrecent works in this "eld should be useful to researchersin this interesting area. Current research proves the feasi-bility of practical implementations of these computer

vision systems for the analysis of agricultural scenes tolocate natural objects under di$cult conditions. Somebasic considerations about the distributions and charac-teristics of the fruits in natural orange crops are dis-cussed.

The research reported here explores the practical ad-vantages of using a laser-range "nder sensor as the maincomponent of a three-dimensional scanner. This sensorsupplies two sources of information, the range to thesensed surface and the attenuation occurred in theround-trip travel. A model of the attenuation process ispresented and used to restore images and to derive addi-tional information: re#ectance, apparent re#ectance,range precision and the range standard deviation. Theapparent re#ectance image and the range image are usedto recognize the fruit by color and shape analysis algo-rithms. The information obtained with both the methodsis merged to "nd the "nal fruit position. The three-dimensional information with its precision, the size andthe average re#ectance of the image is the "nal informa-tion obtained for every fruit. This information allowsa selective harvesting to improve the quality of the "nalproduct for the fresh fruit market.

Some experimental results are presented showing thatapproximately 74% of the green fruits are detected andthis correct location rate is improved as the amount ofmature fruits in the scene increases, reaching a 100% ofcorrect detection over the visible fruits. No false detec-tions were found in the test images used. Future workcould be directed to extract more shape information fromthe range image to improve the detection results.The integration of the recognition methods with theAGRIBOT harvesting system will be reported in futurepublications.

Acknowledgements

This research was done at the PRIP laboratory of theComputer Science Department at Michigan State Uni-versity and was sponsored by the Spanish National Pro-gramme PN93 (CICYT-TAP93-0583).

Appendix A. The laser range-5nder model

The main goal of this section is to derive a mathemat-ical expression which is able to model the behavior of thelaser range-"nder when the operational conditionschange. The attenuation signal supplied by the sensormust depend on the distance r to the object, the re#ec-tance properties of the target surface and the angle h be-tween the laser optical axis and the normal to the targetsurface.

Let oddenote the di!use re#ectance coe$cient which is

the ratio between the re#ected di!use radiant #ux and the


Fig. 7. The angle h between the surface normal and the laseraxis is necessary to obtain the re#ectance of the surface.

incident radiant #ux Fi(3.4]10~3 w). The values of this

coe$cient ranges from 0 for a black surface to 1 for anideal white surface. o

ddepends on the wavelength of the

light, but in our case a "xed wavelength, 670 nm, will beutilized.

The di!use re#ected radiant intensityId

(w/s rad)depends on the incident radiant #ux F

i, the di!use

re#ectance coe$cient od, and the incident angle h.

Using the cosine Lambert law, the following expression isfound:

Id"

Fi

podcos h. (A.1)

The fraction of the received laser signal which passesthroughout the optical system of the sensor is denoted bya. The a value range from 0 to 1 for ideal optics. Theoret-ically, this value must be a constant, but for our sensorthe transmission rate changes when the distance to thesensor varies:

a(r)"a1[a tan(a

2r)]2. (A.2)

The area of the optical surface for the signal receptionis represented by A

r(908 mm2). The solid angle ) cap-

tured by the sensor is equal to AR/r2. The radiant #ux

captured by the laser range-"nder is a function of Id,

r and ). The following equation expresses this relation-ship:

Fc"aI

d)"A

aARFi

p Bodcos hr2

. (A.3)

Finally, Atte, the signal supplied by the sensor ona decibel unit scale, can be modeled in the following way:

Atte"20 log10A

Fi

FcB

"20 log10A

Fi

(a (r)ARFi/p) (o

dcos h/r2)B , (A.4)

Atte"20 log10A

pr2

a1[a tan(a

2r)]2A

Rodcos hB . (A.5)

This model is directly employed to obtain the follow-ing equation to compute the di!use re#ectance coe$c-ient:

od"

pr2

a1[a tan(a

2r)]2A

Rcos(h) 10Atte@20

. (A.6)

To compute the re#ectance coe$cient, it is necessaryto know the distance r, the signal attenuation Atte, andthe angle h (Fig. 7). The "rst two parameters are obtaineddirectly by the sensor, but for computing h there is a need

for analyzing the range image and produce a surfacenormal image. Due to error in computing surface nor-mals, we obtain noise re#ectance images.

If the term related to h is placed on the left-hand side ofEq. (A.6), the apparent re#ectance is obtained, which ismuch easier to calculate:

odcos(h)"

pr2

a1[a tan(a

2r)]2 A

R10Atte@20

(A.7)

In a previous work [47], the following dependenciesbetween the signal to noise ratio, SNR, and the capturedradiant #ux, F

c, are exhibited:

SNR"Sgj¹hc

Fc"A

a(r)gjARFi¹

phc

odcos hr2 B

1@2, (A.8)

where h is the Planck constant, c is the speed of light("3]108 m/seg), j the laser beam wavelength"0.67 lmand g the photocathode quantic e$ciency.

Taking into account that the standard deviation andthe precision of the range measurements are inverselyproportional to the SNR, the following two expressionsallow us to estimate these parameters:

pr"(1.45]10~5)10(0.054)Aten#0.5 , (A.9)

Precision"(13.8]10~6)10(0.062)Aten#8 . (A.10)

Appendix B: The maximum and minimum gradients foredge extraction

Two values are calculated to perform the thresholdingof the gradient image obtained by applying a Sobeloperator. The goal is to obtain a set of pixels belonging tothe boundaries of the spherical object. This set of pixelswill be used to perform the CHT, but to reduce thepossibility of error, only the pixels within a certain


Fig. 9. Drawing showing the parameters related in the correc-tion of the distance to the center of the spherical fruit.

Fig. 8. The two slope limits depicted, produce a two-bit-wideedge ring when a sphere is present.

surface slope interval are considered. The pixels withslopes higher than a maximum value are not consideredsince these abrupt transitions could be due to leave-to-leave jumps. Pixels with slopes below the minimum valueare also not taken into account. The two threshold valuesare computed so that a two-pixel-wide ring is alwaysobtained when spherical objects exist. The outermostpixel of the contour of the sphere is not considered forgetting rid of the step edge which could be a source ofundesired edges.

If N is the number of pixels existing in the radius ofa sphere (Fig. 8), then we obtain the following relation-

ship: y"JN2!x2 for a spherical object. To obtain theslope function, y is derived with respect to x obtaining

dy

dx"

!x

JN2!x2, (B.1)

and the gradients for the pixels N!1 and N!3 are

Gradient}max"K

dy

dx Kx/N~1

"

N!1

JN2!(N!1)2, (B.2)

Gradient}min"K

dy

dx Kx/N~3

"

N!3

JN2!(N!3)2. (B.3)

Appendix C: Distance to the object correction when usingthe CHT

Appendix B describes the selection of the two thre-shold values to obtain the edge image. Since we are notusing the real boundary of the spherical object, the com-putation of the distance to the center of the object d isequal to d ' (distance computed using the CHT) plus anerror e. If da is the angular resolution of the image andN is the number of pixels in the radius of the sphere

which it is being searched by the CHT, then we candeduce the following relations (see Fig. 9):

d@"r cos(da (N!2))

d@ is the distance without correction, (C.1)

e"JN2!(N!2)2 (da r) e is the error produced.

(C.2)

Finally, the corrected distance is computed by the follow-ing expression:

d"d@#e"r cos(da (N!2))#JN2%(N!2)2 (da r).

(C.3)

References

[1] A.K. Jain, P. Flynn (Eds.), Three-Dimensional Object Rec-ognition Systems, Elsevier, Amsterdam, 1993.

[2] R. Tillet, Image analysis for agricultural proceses: a reviewof potential opportunities, J. Agric. Engng Res. 50 (1991)247}258.


[3] A.K. Jain, C. Dorai, Practicing vision: integration,evaluation and applications, Pattern Recognition J.,to appear.

[4] F. Juste, F. Sevilla, Citrus: A European project to study therobotic harvesting of oranges, in: Proc. 3rd Int. Symp.Fruit, Nut and Vegetable Harvesting Mechanization, De-nmark, Sweden, Norway, 1991, pp. 331}338.

[5] P. Levi, R. Falla, R. Pappalardo, Image controlledrobotics applied to citrus fruit harvesting. Procedures,ROVISEC-VII, Zurich, 1988.

[6] Whitaker, Miles, Mitchell and Gaultney, Fruit location ina partially occluded image, Trans. ASAE 30 (3) (1987)591}597.

[7] C.E. Shert, G.K. Brown, Basic considerations in mechaniz-ing citrus harvest, Trans. ASAE (1968) 343}346.

[8] E. Parrish, A.K. Goksel, Pictorial pattern recognition ap-plied to fruit harvesting, Trans. ASAE 20 (1977) 822}827.

[9] A. Grand D'Esnon, Robot harvesting of apples, in: Proc.AgrMation I, ASAE and SME, St. Joseph, MI, 1984,pp. 210}214.

[10] F. Sevilla, F. Sittichareonchai, J.M. Fatou, A. Constans,A. Brons, A. Davenel, A robot to harvest grape: a feasibil-ity study, ASAE Paper No. 89-7084, St. Joseph, MI 49085,1989.

[11] F. Sevilla, P. Baylou, The principles of robotics in agricul-ture and horticulture, in: Progress in Agricultural Physicsand Engineering, C.A.B. International, Bedford, UK, 1991,pp. 119}147.

[12] D.S. Humburg, J.F. Reid, Field performance for machinevision for selective harvesting of asparagus, Appl. EngngAgric. 2 (1) (1986) 2}5.

[13] K. Amaha, H. Shono, T. Takakura, A harvesting robot ofcucumber fruits, ASAE Paper No. 89-7053, St. Joseph, MI49085, 1989.

[14] R. Tillet, Initial development of a mechatronic mushroomharvester, in: Proc. Int. Conf. &&Mechatronics: DesigningIntelligent Machines'', 12}13 September, Institution ofMechanical Engineers, Cambridge, 1990, pp. 109}114.

[15] L. Kassay, Hungarian robotic apple harvester, ASAE Pa-per No. 92-7042, Charlotte, NC 21}24 June, 1992.

[16] N. Kawamura, N. Kiyoshi, F. Tasteshi, U. Motonobu,Study of fruit harvesting robot and its application onothers works, in: Proc. Int. Symp. on Agricultural Mechand Int. Coop. in High Tech. Era, Japanese Society ofAgricultural Machinery, 1987.

[17] A. Grand D'Esnon, G. Rabatel, R. Pellenc, Magali: a self-propelled robot to pick apples, ASAE paper No. 87-1037,St. Joseph, MI 49085-9659, 1987.

[18] E. Blandini, First approaches to robot utilisation for auto-matic citrus harvesting, in: Dodd, Grace (Eds.), Land andWater Use, Balkema, Rotterdam, 1989.

[19] R.C. Harrel, P.D. Adsit, T.A. Pool, R. Ho!man,The Florida robotic grove-lab, ASAE Paper No. 88-1578,1988.

[20] Hayashi, Ueda, Orange harvesting robot, Mimeo, KubotaCo., Sakai, Japan 1991.

[21] M. Benady, Y. Edan, A Hetzroni, G.E. Miles, Design ofa "eld crops robotic machine, ASAE Paper No. 91-7028,St. Joseph, MI 49085, 1991.

[22] Y. Edan, G.E. Miles, Design of an agricultural robot forharvesting melons, Trans. ASAE 36 (2) (1993) 593}603.

[23] Y. Edan, Design of an autonomous agricultural robot,Applied Intelligent 5 (1995) 41}50.

[24] V. Stepanov et al., Mobile and manipulation robots foragriculture, in: Proc. 4th Workshop on Robotics in Agri-culture & the Food Industry, IARP, Toulouse, 1995,pp. 137}143.

[25] R. Ceres, J.L. Pons, A.R. JimeH nez, J.M. MartmHn, L. Cal-deroH n, Agribot: a robot for aided fruit harvesting,AGENG96, vol. 2, 1996.

[26] M. Benady, G.E. Miles, Locating melons for robotic har-vesting using structured light, ASAE Paper No. 92-7021,1992.

[27] J.J. Ga!ney, Re#ectance properties of citrus fruit, Trans.ASAE 16 (2) (1969) 310}314.

[28] G. Rabatel, A vision system for Magali, the fruit pickingrobot, Int. Conf. Agricultural Engineering, AGENG88Paper 88293, Paris, 1988.

[29] N. Kawamura, Namikawa, Robots in agriculture, Adv.Robotics 3 (1989) 311}320.

[30] K.C.Cox, G.C. Roman, W.E. Ball, A.F. Laine, Rapidsearch for spherical objects in aerial photographs, IEEEConf. Computer Vision and Pattern Recognition, 1988,pp. 905}909.

[31] D. Slaughter, R.C. Harel, Color vision in robotic fruitharvesting, Trans. ASAE 30 (4) (1987) 1144}1148.

[32] Sites, Dewilche, Computer vision to locate fruit on a tree,Trans. ASAE 31 (1) (1988) 257}263.

[33] D. Slaughter, R.C. Harrel, Discriminating fruit for roboticharvest using color in natural outdoor scenes, Trans.ASAE 32 (2) (1989) 757}763.

[34] R.C. Harrel, D. Slaughter, P.D. Adsit, A fruit-trackingsystem for robotic harvesting, Machine Vision Appl.2 (1989) 69}80.

[35] W. Qiu, S.A. Shearer, Maturity assessment of broccoliusing the discrete Fourier transform, ASAE Paper No.91-7005, St. Joseph, MI, 1991.

[36] M. Cardenas-Weber, A Hetzroni, G.E. Miles, Machinevision to locate melons and guide robotic harvesting,ASAE Paper No. 91-7006, 1991.

[37] Y. Dobrusin, Y. Edan et al., Real-time image processingfor robotic melon harvesting, ASAE Paper No. 92-3515,1992.

[38] F. PlaH , F. Juste, F. Ferri, Feature extraction of sphericalobjects in image analysis: an application to roboticcitrus harvesting. Comput. Electron. Agriculture 8 (1993)57}72.

[39] F. Buemi, M. Massa, G. Sandini, AGROBOT: a roboticsystem for greenhouse operations, in: Proc. 4th Worksopon Robotics in Agriculture & the Food Industry, IARP,Toulouse, 1995, pp. 172}184.

[40] F. PlaH , Recognition of partial circular shapes from seg-mented contours, Comput. Vision Image Understanding63 (2) (1996) 334}343.

[41] O. Trier, A.K. Jain, Goal-directed evaluation of binariz-ation methods, IEEE Trans. PAMI 17 (1995) 1191}1201.

[42] R.O. Duda, P.E. Hart, Use of the Hough transformation todetect lines and curves in pictures, Commun. ACM 15(1972) 11}15.

[43] C. Kimme, D.H. Ballard, J. Sklansky, Finding circlesby an array of accumulator, Commun. ACM 18 (1975)120}122.


[44] D.H. Ballard, Generalizing the Hough Transform to detectarbitrary shapes, in: Computer Vision: Principles, IEEEComputer Society Press, 1991, pp. 183}194.

[45] J. Illingworth, J. Kittler, A survey of the Hough transform,Computer Vision. Graphics Image Process. 44 (1988)87}116.

[46] J. Illingworth, J. Kittler, J. Princen, Shape detection using theadaptive Hough transform, NATO ASI Series, vol. 42, Sen-sors Devices and Systems for Robotics, 1988, pp. 119}142.

[47] D. Nitzan, A.E. Brain, R.O. Duda, The measurement anduse of registered re#ectance and range data in scene analy-sis, Proc. IEEE 65 (1977) 206}220.

About the Author*ANTONIO R. JIMED NEZ graduated in Physics, Computer Science branch (Universidad Complutense of Madrid,June 1991). From 1991 to 1993, he worked in industrial laser applications at CETEMA (Technological Center of Madrid), Spain. From1994, he has been working towards a Ph.D. degree as a research assistant at the Instituto de AutomaH tica Industrial, CSIC, Spain. Hiscurrent research interests include computer vision applications, pattern recognition, range images, shape-based image analysis andautomatic harvesting.

About the Author*ANIL K. JAIN received a BTech degree in 1969 from the Indian Institute of Technology, Kanpur, and the M.S. andPh.D. degrees in Electrical Engineering from Ohio State University, in 1970 and 1973, respectively. He joined the faculty of MichiganState University in 1974, where he currently holds the rank of University Distinguished Professor in the Department of ComputerScience. Dr. Jain served as program director of the Intelligent Systems Program at the National Science Foundations (1980}1981), andhas held visiting appointments at Delft Technical University, Holland, Norwegian Computing Center, Oslo, and Tata ResearchDevelopment and Design Center, Pune, India. Dr. Jain has published a number of papers on the following topics: statistical patternrecognition, exploratory pattern analysis, neural networks, Markov random "elds, texture analysis, interpretation of range images, and3D object recognition. He received the best paper awards in 1987 and 1991, and received certi"cates for outstanding contributions in1976, 1979, and 1992 from the Pattern Recognition Society. Dr. Jain served as the editor-in-chief of the IEEE Transactions on PatternAnalysis and Machine Intelligence (1991}1994), and currently serves on the editorial boards of Pattern Recognition Journal, PatternRecognition Letters, Journal of Mathematics Imaging, Journal of Applied Intelligence, and IEEE Transactions on Neural Networks.

About the Author*RAMOD N CERES graduated in Physics (electronic) from Universidad Complutense of Madrid in 1971 and receivedthe doctoral degree in 1978. After a "rst stay for one year in the LAAS-CNRS in Toulouse (France), he has been working at the Institutode AutomaH tica Industrial (IAI), dependent of the Spanish National Council for Science Research; with a period in 1990}91 ofpermanence in an electronics company (Autelec) as R&D director. Since the beginning, Dr Ceres has developed research activities onsensor systems applied to di!erent "elds such as continuous process control, machine tool, agriculture, robotics and disable people. Onthese topics he has produced more than seventy papers and congress communications, having several patents in industrial exploitation.At present Dr Ceres is the Spanish delegate for the IMT (Brite-Euram) Committee and Deputy Scienti"c Director of the IAI.

About the Author*JOSED L. PONS graduated as Mechanical Engineering (Universidad de Navarra, April 1992). He received a M.Sc.degree in Information Technologies for Production (Universidad PoliteH cnica de Madrid, January 1995). He received the Ph.D. degree inPhysics Sciences from the Complutense University of Madrid, December 1996. Dr. Pons is currently at the Instituto de AutomaH ticaIndustrial, CSIC, where he has been working since 1993. His current research interests include non-traditional sensor-actuationtechnologies, development of new technologies and miniature applications.


Documents

Automatic fruit recognition: a survey and new results ...ben-shahar/ftp/papers/Agrobotics/Fruit... · The sensor used is a laser range-"nder giving range/attenuation data of the sensed