Computers and Electronics in Agriculture 110 (2015) 78–90

Contents lists available at ScienceDirect

Computers and Electronics in Agriculture

journal homepage: www.elsevier .com/locate /compag

Modeling leaf growth of rosette plants using infrared stereo imagesequences

http://dx.doi.org/10.1016/j.compag.2014.10.0200168-1699/� 2014 Elsevier B.V. All rights reserved.

⇑ Corresponding author.E-mail address: eaksoy@gwdg.de (E.E. Aksoy).

Eren Erdal Aksoy a,⇑, Alexey Abramov a, Florentin Wörgötter a, Hanno Scharr b, Andreas Fischbach b,Babette Dellen c

a Georg-August-Universität Göttingen, BCCN, Department for Computational Neuroscience, Inst. Physics-3, Friedrich-Hund Platz 1, D-37077 Göttingen, Germanyb IBG-2: Plant Sciences, Forschungszentrum Jülich, 52425 Jülich, Germanyc Institut de Robòtica i Informàtica Industrial (CSIC-UPC), Llorens i Artigas 4-6, 08028 Barcelona, Spain

a r t i c l e i n f o

Article history:Received 8 November 2013Received in revised form 21 October 2014Accepted 23 October 2014

Keywords:Leaf segmentationLeaf trackingLeaf modelingPlant growthPhenotyping

a b s t r a c t

In this paper, we present a novel multi-level procedure for finding and tracking leaves of a rosette plant,in our case up to 3 weeks old tobacco plants, during early growth from infrared-image sequences. Thisallows measuring important plant parameters, e.g. leaf growth rates, in an automatic and non-invasivemanner. The procedure consists of three main stages: preprocessing, leaf segmentation, and leaf tracking.Leaf-shape models are applied to improve leaf segmentation, and further used for measuring leaf sizesand handling occlusions. Leaves typically grow radially away from the stem, a property that is exploitedin our method, reducing the dimensionality of the tracking task. We successfully tested the method oninfrared image sequences showing the growth of tobacco-plant seedlings up to an age of about 30 days,which allows measuring relevant plant growth parameters such as leaf growth rate. By robustly fitting asuitably modified autocatalytic growth model to all growth curves from plants under the same treatment,average plant growth models could be derived. Future applications of the method include plant-growthmonitoring for optimizing plant production in green houses or plant phenotyping for plant research.

� 2014 Elsevier B.V. All rights reserved.

1. Introduction

With increasing requirements for food due to a growing worldpopulation, optimizing plant production is becoming an importantfactor for the agricultural industry. Plant performance and produc-tivity results from a complex interaction between its genotype andenvironment, resulting in its expressed properties, i.e. its pheno-type. Thus, if one seeks to understand these interdependencies,e.g. to achieve larger yields, plant phenotypes in terms ofexpressed plant structure and function need to be analyzed quan-titatively. For this task automatic, non-invasive methods are highlydesirable, but problems arise from the complex and varyingappearance of plants, making it difficult to detect and recognizerelevant plant organs and growth patterns.

Previously both color and stereo vision have been used to obtainsome relevant plant features, mainly for recognition and classifica-tion purposes (Loch et al., 2005; Moeslund et al., 2005; Quan et al.,2006; Biskup et al., 2007; Song et al., 2007; Jin and Tang, 2009;Alenyà et al., 2011a; Teng et al., 2011; Silva et al., 2013; Wang

et al., 2013), but those procedures are error prone, or require theconcourse of a user to correctly segment and characterize individ-ual leaves. For instance, Quan et al. (2006) modeled plants directlyfrom a set of images for a better supervised leaf segmentation. Jinand Tang (2009) detected corn plants by only using depth imageswithout dealing with the tracking issue. Leaf tracking has, to ourknowledge, so far only been performed with unambiguously iden-tified leaves. For example, Biskup et al. (2007) tracked the leaf ori-entation angles, and Polder et al. (2007) used penalized likelihoodwarping and robust point matching of leaf contours in order todetect emerging damages caused by disease. Alenyà et al.(2011b) showed how a robot arm can track a manually selectedsingle leaf using some geometrical characteristics and color infor-mation. The problem of tracking multiple leaves was not addressedby these works. The work in (De Vylder et al., 2013) uses activecontours to track multiple leaves, but they process time lapse plantimages in batch once the complete sequence is acquired. Their pro-posed segmentation approach is triggered with the last frame ofthe sequence in a semi-supervised manner and the detection phasecan omit new leaves since it goes to the first frame starting fromthe last one. Vylder et al. (2011) combined active contours with aBayesian framework to eliminate parameter tuning steps in the

Fig. 1. Robot gardener used in the European project GARNICS. A black-and-white 5MP camera with infrared filter and required illumination devices were mounted ona lightweight KUKA LBR4 robot arm. For each tobacco plant the robot arm captureda stereo image pair from a top view at every hour.

E.E. Aksoy et al. / Computers and Electronics in Agriculture 110 (2015) 78–90 79

segmentation and tracking phases. However, they need manuallysegmented images to have a good estimate of the probability dis-tribution functions for the calculation of internal and externalprobabilities. Both approaches (De Vylder et al., 2013; Vylderet al., 2011) have also not been tested on plant sequences that lastlonger than 3 days.

Along this line, the European project GARNICS (Gardening with aCognitive System)1 aimed at 3D sensing of plant growth and buildingperceptual representations for learning the links to actions of a robotgardener (see Fig. 1). The project encompassed both the long-termlearning of treatments to achieve specific goals (maximum leafgrowth, homogeneous plant growth) as well as the short-term robotinteraction with plants (for leaf surface measurement, disocclusion,probing), and this study has been conducted in this context.

More precisely, we address the problem of sensing and control-ling plant growth parameters by ways of leaf tracking and modelfitting, using a stereo infrared camera set-up, monitoring tobaccoseedlings during their first three weeks of growth. A major diffi-culty hereby arises from the complex appearance of plants in theimage. Leaves are weakly textured, often overlapping, thusoccluding each other, and their form may be distorted in the 2Dprojection due to steep leaf angles with respect to the camera view.Under these conditions, the automated image segmentation ofindividual leaves is highly challenging, and cannot be guaranteed.In this work, we first over-segment the infrared images and thenemploy a merging procedure using a 2D leaf-shape model, but alsoincorporating 3D information from stereo matching. The maingrowth curves of the plant leaves are extracted and used to analyzeplant development over time. Segmentation failures appear asnoise in the system, and can be handled at least to some degree.Once the main growth curves corresponding to the individualleaves of the plant are found, erroneous segments can be removed,and by using a leaf-shape model, the growth rates for each identi-fied leaf can be computed.

Rosette plants are commonly used in plant research facilities,and the automatic growth analysis of seedlings would come inhandy for many laboratories. Furthermore, growth monitoring ofseedlings can be used in plant production to optimize plant

1 http://www.garnics.eu.

treatments, e.g. with respect to the provision of water and nutri-ents or light requirements. Size and color distribution of plantleaves over time are important cues to monitor the lack of suchrequirements, avoiding plant stress situations.

Note that this study has also been described as a part of a patent(Wörgötter et al., 2013).

2. Plant material

Six tobacco plants (Nicotiana tobacum cv. Samsun) were grownunder constant light conditions (500 lE m�2 s�1) with a 16 h/8 hday/night rhythm. Three of them (Plant IDs 79329, 79335, and79338) received 1.8 ml of water every other hour (‘‘Treatment1’’), the others (Plant IDs 79330, 79336, and 79339) received0.9 ml of water and 0.5 ml of nutrient solution with 1% Hakaphosgreen every other hour (‘‘Treatment 2’’). Water and nutrient solu-tion were applied by the GARNICS robot system, positioning smalltubes, one for water and one for nutrient solution, at predefinedlocations and pumping using an automated flexible-tube pump.

In the GARNICS project, treatments were selected to producetraining data for a cognitive system. The actual amounts of waterand nutrient solution are therefore well adapted to the soil sub-strate such that the sets of plants show distinguishable perfor-mance of generally well growing plants. Finding an optimaltreatment was left for the system. The soil used for the experiment(‘‘Kakteenerde’’) has low nutrient content and dries relatively fastwith an approximately exponential behavior A ¼ A0 expð�t=sÞ,where s � :7 days.

We applied the proposed leaf tracking and modeling algorithmto tobacco-plant sequences showing the growth from germinationwell into the leaf development stage, i.e. we started our observa-tions at growth stage 09 and typically stopped at stage 1006(according to the extended BBCH-scale presented in CORESTACORESTA (2009)), due to size restrictions.

3. Method

3.1. Overview

Our framework for continuous measurement of plant growthparameters consists of three main parts: data acquisition andpreprocessing, segmentation of all frames from a plant videosequence, and consistent leaf tracking and modeling of thesegmented leaves. A schematic showing all steps of the procedureand labeled by numbers is presented in Fig. 2.

As input data we use gray-scale stereo images acquired with aninfrared camera attached on a robot arm. We compared differentillumination options and found that plant structures and bound-aries between tobacco leaves could be detected more easily forinfrared light than for visible light. In addition, plants do not reactto the applied 880 nm IR light, e.g. by photosynthetic activity.Consequently, illumination and acquiring images at night is possi-ble without influencing plant growth, in contrast to visible light. Apair of images (left and right) is captured at each time step by mov-ing the robot head with the infrared camera and light source, pro-viding a stereo baseline (see Fig. 2(A) and (B)). In step 1 of theprocedure, we compute a depth (disparity) map from the stereopair using a block-matching algorithm from the OpenCV libraryBradski (2000) (see Fig. 2(C)). This method gave preferable resultscompared to other methods. We further removed the backgroundfrom the scene to simplify the following computations (seeFig. 2(D)).

Next, in step 2, each preprocessed infrared image of thesequence is segmented independently. Afterwards, each leaf is rep-resented by one or more segments as shown in Fig. 2(E). In step 3,

Fig. 2. Schematic of the multi-level procedure for segmenting and tracking leaves. Numbers mark the different computational steps of the procedure. (A) Acquired left frame.(B) Acquired right frame. (C) Disparity map estimated using block matching. (D) Left frame after background removal. (E) Image segmentation results using the method ofsuperparamagnetic clustering of data (here and further scaled up for a better visibility). (F) Segments after segment merging, and (G) after relabeling. (H) Angle distribution ofcorrected segments for 100 frames. (I) Angle histogram derived from the angular distribution. (J) Tracked segments with reassigned unique labels. (K) Ellipse models fitted tothe tracked segments. (L) Plant growth curves estimated from ellipse models.

80 E.E. Aksoy et al. / Computers and Electronics in Agriculture 110 (2015) 78–90

we employ a merging procedure to group the segments into leafshapes (Fig. 2(F)) by finding the partition that minimizes the over-lap between the convex hulls of the segments. This is a goodenough working assumption as long as the leaves have convexshapes. This merging stage is a necessary improvement, but it stilldoes not guarantee success. Sometimes there are over-segmenta-tions which remain unresolved, as shown in Fig. 2(F). Note thatafter merging, the segments are relabeled (see Fig. 2(G)).

In step 5 of the procedure, the position of the centroid of eachsegment is computed with respect to the plant stem position inpolar coordinates. The plant stem can be found with sufficientaccuracy by computing the centroid of the foreground (containingonly the plant) at an early growth stage. By presenting each seg-ment as a data point in an angle-time plot, growth tracks can bemade visible because the tobacco-plant leaves do hardly changetheir azimuthal angle (Fig. 2(H)). Leaves that are growing in thesame direction can be distinguished based on their depth values.

Hence, when computing the angular histogram of the centroidsover a larger time interval (step 6 of the procedure), the data pointsof the growth tracks accumulate at the angular positions of the cor-responding leaves (see Fig. 2(I)). By first detecting the peaks in thehistogram using a threshold, we can cluster the segments belong-ing to the different tracks and assign them unique, temporally con-sistent labels in step 4 (see Fig. 2(J)). In the final step (9), trackedsegments, corresponding to leaves, are used for fitting appropriateellipse models (see Fig. 2(K)) and estimating growth curves forindividual leaves as shown in Fig. 2(L).

3.2. Image acquisition

For image acquisition a black-and-white 5 MP camera withinfrared pass filter has been used. Images have been taken at reg-ular, hourly time intervals for each plant over a time period of30 days. The camera was mounted on a lightweight KUKA LBR4

E.E. Aksoy et al. / Computers and Electronics in Agriculture 110 (2015) 78–90 81

robot arm (see Fig. 1). For each plant the robot arm captured astereo image pair from a top view every hour by moving a certaindistance (app. 5 mm) along the baseline.

3.3. Preprocessing

Before segmenting the images, we remove the background asshown in the second row of Fig. 3. The table, the plant pot, andthe soil visible in the near infrared (NIR) images can be easily elim-inated by applying a threshold. Furthermore, a disparity map iscomputed with a standard block-matching technique from the ste-reo infrared images.

3.4. Leaf segmentation

For segmenting the images, we use the method of superpara-magnetic clustering of data which runs in real-time on a GraphicsProcessing Unit (GPU). The method of superparamagnetic cluster-ing is inspired by systems of interacting ferromagnets or spins.These systems are characterized by three phases. At low tempera-tures, spins are fully aligned with one another, while at intermedi-ate temperatures, groups of aligned spins coexists. At highertemperatures, the order breaks down into a disordered state. Whenrepresenting pixels by spins and defining spin–spin interactionsdependent on the similarity of adjacent pixels, a natural partitionof the image can be found in the superparamagnetic regime simu-lating the stochastic dynamics of the system with a Metropolisalgorithm.

The method of superparamagnetic clustering has beendescribed in detail elsewhere Abramov et al. (2012). Superparmag-netic clustering has been used previously to segment leaves basedon color and depth acquired with a Kinect camera Wallenberg et al.(2011). However, in this case, plants were fully grown and leavesconsiderably larger. In our experimental set-up, leaves are smaller,and the task of obtaining sufficiently accurate depth informationfor depth-based segmentation would be far more challenging.Typical segmentation results obtained by this technique are shownin the last row of Fig. 3.

Due to varying light conditions and very low intensity differ-ences at the leaf borders, leaves may be wrongly merged by themethod. To avoid this, the segmentation runs in the over-segmen-tation mode (see Fig. 3). This strategy ensures that segments

Fig. 3. Segmentation of near infrared (NIR) images using the method of superparamagnmiddle row indicates frames after the background removal, and the last row shows init

adhere better to leaf borders. Leaves represented by more thanone segment can be recovered later on (see Section 3.5), whilerecovery of two (or more) leaves from one big wrongly mergedsegment (under-segmentation) is more difficult.

3.5. Segment merging

The output given by the image segmentation module manytimes splits one leaf into more than one segment and may containnoisy regions, such as a part of the pot or some areas of high inten-sity compared to the background. Therefore, additional proceduresare required in order to obtain a better segmentation. The firstmajor improvement is achieved by correcting the initial segmentswith a leaf-shape descriptor. For this purpose tobacco plant leavescan be described by their convex hulls with sufficient accuracy.

The segment-merging procedure works as follows. First of all,segments with centroids located far from the plant stem are elim-inated (see the first row in Fig. 4). Noisy speckles are removed aswell (see the second row in Fig. 4). Then a graph is built wherethe centroids of the segments represent the graph nodes. Edgesare drawn between two nodes if the segments are smaller than athreshold apart both in ðx; yÞ distance and depth. Each edge repre-sents a possible merge. Hence, for a total number of s edges, thereare 2s possible merging configurations Mi. Neglecting occlusions,the desired segmentation should more or less preserve the shapeof the leaves, i.e., using the segment’s convex hull as leaf-shapemodel, the total overlap of the convex hulls of all segments shouldbe smallest for this configuration. Let now be Cj the convex hull ofsegment j, then we compute the overlap of a particular mergingconfiguration Mi as Oi ¼

Pelm2Mi

Cl \ Cm þP

kCk \ B, where B is thebackground region. We select the merging configuration with thesmallest overlap. For a small number of edges, we can simply eval-uate all possible configurations. This is the case in our scenario. Forlarge number of edges, approximate methods would have to beemployed to find the minimum.

The depth data (see the third row in Fig. 4) is used to removeedges between neighboring segments that have a large differencein disparity. This also helps to keep s reasonably small. Typicalresults of the segment-merging procedure segmentation areshown in the last row of Fig. 4.

Merging segments that represent a leaf based on shape featuresis a difficult problem for the following reasons: Only a small part of

etic clustering. First row shows left input frames captured with an infrared camera,ial segmentation results (after step 2 of the procedure).

Fig. 4. Segment correction performed by the convex hull approximation with depth information. Top row shows initial segments. Second row indicates clean input segmentswithout noise and borders. Third row represents disparity maps estimated by the block matching technique for a pair of NIR images. Final segments after segment mergingare shown in the last row (after step 3 of the procedure).

82 E.E. Aksoy et al. / Computers and Electronics in Agriculture 110 (2015) 78–90

the boundary of a leaf segment corresponds to the actual leafboundary (the other ones are inner boundaries, i.e, non-leaf bound-aries). Pairwise merging, as employed in standard split-and-mergeapproaches, will thus only be successful for simple cases becausethe shape of the whole leaf will only become apparent when allthe segments have been merged correctly and all inner boundarieshave been removed through the merging process. This is a typicalchicken-egg problem. Occlusion adds further difficulties by alter-ing the visible shape of the leaves. For this reason, given the smallnumber of segments, we opted for the described merging tech-nique which avoids having to apply a standard pairwise mergingprocedure (Horowitz and Pavlidis, 1974; Alenyà et al., 2013) andinstead tests for all possible combinatorial solutions.

3.6. Tracking

Usually leaves grow at an almost constant azimuth angle withrespect to the plant stem during their development, and even iftwo leaves have the same angle, their depth values typically aredifferent. Therefore, we can make use of the natural growth patternof plant leaves for solving the tracking issue.

For each frame, we first calculate coordinates of the plant stemp ¼ fpx; pyg as

px ¼1N

XN

i¼1

sxi; py ¼

1N

XN

i¼1

syi; ð1Þ

where N is the total number of existing segments, whose centers aregiven by fsx; syg.

Each segment center is then represented by r and h defining theradius and angle in polar coordinates as

r ¼ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiðsx � pxÞ

2 þ ðsy � pyÞ2

q; h ¼ arctan 2

sy � py

sx � px

� �: ð2Þ

At each acquired frame, all extracted N segment angles are com-bined into a histogram H representing the distribution of anglesover previous T frames as

H ¼ hi : i 2 1;2; . . . ;360

k

� �� �;

hi ¼XN

n¼1

XT

t¼1

dn;t ; ð3Þ

dn;t ¼1 if i� 1 < hn;t

k < i

0 else

(; ð4Þ

where k is the bin size. In our experiments k and T values are chosenas 10 and 100 unless otherwise stated. Fig. 5 (top row) shows fourplant images. The corresponding segments from the merging proce-dure are shown in the second row. The respective angular distribu-tions of their centroid positions over 100 frames are plotted in thethird row of Fig. 5. The resulting histogram representation for eachplant image is depicted in the fourth row in Fig. 5.

We further continue with calculating local maxima (i.e. peaks)in each histogram and use them to cluster the data. Let mi andmj be the angle positions of two local maxima derived from a givenangle distribution. The maximum at mj is basically ignored ifmi �mj < sd, where sd is a threshold. In our experiments, we usesd ¼ 40�. The extracted local maxima (i.e. all mi) are shown asred circles in Fig. 5. All other local maxima (i.e. all mj) are ignoredsince their distances to their nearest neighbors are below thresh-old. We also ignore those maxima which are smaller than the aver-age histogram value

sm ¼k

360

X360k

i¼1

hi: ð5Þ

The threshold value sm for each histogram is shown as dashed linesin the fourth row of Fig. 5.

The tracking phase is concluded by reassigning a new unique

label li for each maxima mfi at frame number f. The label li is trans-

fered to another local maximum mfþ1j in the next frame f þ 1, if

those maxima are neighbors within a certain threshold sd such that

mfi �mfþ1

j

< sd. In this way, the final label-maxima correspon-

dence map is updated at each frame to track segments continu-ously. In Fig. 5 (fourth row) the assigned labels corresponding tothe extracted local maxima (indicated by red circles) are displayed.The first image shows the plant with three leaves, i.e. the

Fig. 5. Tracking plant leaves with segment angles. Top row shows sample original plant images with corresponding corrected segments depicted in the second row. Segmentsare here scaled up for a better visibility. Respective angular distribution of segments over 100 previous frames are illustrated in the third row. Histogram representation ofeach distribution is depicted in the fourth row. Circles in red indicate calculated final local maxima with assigned unique labels. Dashed lines show the threshold values forlocal maxima. Last row indicates the final tracked leaf segment labels with their unique labels. (For interpretation of the references to color in this figure legend, the reader isreferred to the web version of this article.)

E.E. Aksoy et al. / Computers and Electronics in Agriculture 110 (2015) 78–90 83

cotyledons and first true leaf, then three more leaves appear oneafter the other.

During the tracking phase, the disparity values of corrected seg-ments are used to distinguish leaves overlapping one another asshown in the last column of Fig. 5. Here, a new leaf, assigned withlabel 6, is appearing and occluding the leaf with number 1. In thiscase, these two leaves have almost the same angle, however, due tothe differences in their disparity values, a new label can beassigned to the leaf. The final segmentation result is shown inthe last row of Fig. 5.

3.7. Leaf modeling and extracting leaf-growth curves

Since leaves can occlude each other, the size of the trackedsegments extracted using the methods described in the previoussection cannot be used directly to estimate plant growth parame-ters. To address weak to medium occlusions we fit an ellipse modeldefined as n ¼ fC;H;H;Wg, where C;H;H, and W represent ellipse

center position, tilt angle, and the lengths of the major and minorsemiaxes (height and width), respectively, to each trackedsegment.

In order to calculate these ellipse parameters, we first deter-mine each leaf tip position T , i.e., a segment point with the maxi-mum distance to the plant stem, from N segment edge pointsðex; eyÞ as

T ¼ fT x; T yg ¼ arg maxðdiÞi

;

di ¼ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiðexi� pxÞ

2 þ ðeyi� pyÞ

2q

; i 2 ½1; . . . ;N�; ð6Þ

where px and py are the plant stem coordinates given in Eq. (1). Wecan now calculate the ellipse centroid coordinates C ¼ fCx; Cyg as,

Cx ¼T x þ px

2; Cy ¼

T y þ py

2: ð7Þ

Next, H;H, and W parameters can be approximated as

Fig. 6. Leaf modeling with ellipses. Top row shows sample frames with corrected segments. Second row depicts corresponding tracked segments with reassigned uniquelabels. Here, each segment color represents one unique label. Last row is with final ellipse models estimated for each tracked leaf.

Fig. 7. Tracking results and estimated ellipse models for six different tobacco plants.

84 E.E. Aksoy et al. / Computers and Electronics in Agriculture 110 (2015) 78–90

Fig. 8. Measured leaf size versus time for Plant ID 79336. Top left: raw data. Different colors indicate different growth curves. Same is true for the next 2 plots. Top right:smoothed by median filter and gaps closed by normalized convolution. Bottom left: filtered to ensure monotonic increase. Bottom right: growth curves split into curvesbelonging to a single leaf, horizontal beginnings and ends removed. Blue indicates the first section of a growth curve, green the second and red the third section stemmingfrom one initial growth curve. The vertical and horizontal axes represent leaf size (cm2) and time (h). (For interpretation of the references to color in this figure legend, thereader is referred to the web version of this article.)

Fig. 9. Mean growth curves for the treatment with more nutrients and less water (Treatment 2): single leaf growth curves of all 3 plants sorted by time of emergence, i.e. leafnumber. Thick curves are the measured data, fine curve is the autocatalytic model with constant y-offset robustly fitted to all curves simultaneously. Left to right and top tobottom: cotyledons (i.e. leaf 1 and 2, being 6 curves for 3 plants), leaf 3–7. Cotyledons are temporally aligned such that size 0.04 cm2 corresponds to time 0 h. The vertical andhorizontal axes represent leaf size (cm2) and time (h).

E.E. Aksoy et al. / Computers and Electronics in Agriculture 110 (2015) 78–90 85

Fig. 10. Fitted growth curves and parameters of autocatalytic model with constant offset Aoffset for all leaves. Left plot: Treatment 1. Right plot: Treatment 2. The vertical andhorizontal axes represent leaf size (cm2) and time (h).

86 E.E. Aksoy et al. / Computers and Electronics in Agriculture 110 (2015) 78–90

H ¼ arctan 2T y � py

T x � px

� �;H ¼

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiðT x � pxÞ

2 þ ðT y � pyÞ2

q2

;W

¼ 1N

XN

i¼1

di; ð8Þ

where d is the distance of N segment edge points to the plant stemand is given in Eq. (6). Leaf area is then computed from the respec-tive ellipse size depending on H and W values.

Fig. 6 shows an example how segments are corrected, labelstracked, and ellipses fitted. In the top row, individual segmentationsafter segment merging (step 3) of the method are presented. Thesecond row shows segments with reassigned labels after the track-ing process has been completed (steps 5–7). The last row shows theellipse models fitted to each segment. A movie with derived seg-ments and ellipse models can be found at www.dpi.physik.uni-goettingen.de/~eaksoye/GARNICS. Fig. 7 shows ellipse trackingresults for all six plants.

Our leaf modeling approach is a searching-based method andthere exist similar works in the literature (Song and Wang, 2007;Kaewapichai and Kaewtrakulpong, 2008). Chien et al. (2011) pro-posed an alternative ellipse detection framework which applieselliptical Hough transform to different levels in the image pyramid.Although this approach is robust to noise during the extraction ofmultiple ellipses, it cannot be applied to our plant image sequences

since small leaf segments, observed in the first weeks of the seed-ling, can vanish in the coarsest resolution in the image pyramid.Thus, detection of leaves can be delayed in the temporal scale.

3.8. Resolving total occlusion

In some cases, we observed that disparity and angle cues fromSection 3.6 are not enough to distinguish between leaves. Whena leaf is completely occluded by a subsequently appearing leaf,the first leaf’s growth curve is sometimes continued by the secondleaf. See e.g. Fig. 7 Plant ID: 79330: Cotyledons (segments 1 and 2in red and green, respectively) grow to a small size as expected, butin the 5th and following depicted time instances seem to growstrongly. Same is true e.g. for Plant ID 79336. Fortunately thiscan be easily detected and corrected when plotting growth curvesin terms of ellipse sizes as a function of time (cmp. log-plots of thegrowth curves in Fig. 8).

The raw data (Fig. 8 top left) is median filtered with a filterlength of 24 h in order to suppress diurnal variations. Subsequentlyit is smoothed and small gaps interpolated by normalized convolu-tion Knutsson and Westin (1993) using a Gaussian kernel withstandard deviation 9 h, length 27 h (Fig. 8 top right). The resultingsmooth curves are filtered to be monotonically increasing by pro-cessing them in positive time direction, keeping a vale if its isthe current maximum, else replacing the current value by the so

Fig. 11. Leaf segmentation results obtained using the graph-based approach and the superparamagnetic clustering of data. Left column shows input near infrared (NIR)images for plant number 79339. Middle left and right columns show final segments for the graph-based method with threshold values k ¼ 150 and k ¼ 200, respectively.Segments from the superparamagnetic clustering are shown in the right column. Note that segments are here scaled up for a better visibility.

E.E. Aksoy et al. / Computers and Electronics in Agriculture 110 (2015) 78–90 87

far seen maximum (Fig. 8 bottom left). This enforces the assump-tion that leaves are not shrinking. These smooth monotonic curvesare then cut into separate curves at gaps (cmp. Fig. 8 bottom left,black lines, with the corresponding lines in Fig. 8 bottom right),or when an almost non-growing part is followed by a stronglygrowing one (cmp. Fig. 8 bottom left, e.g. red and green lines, withthe corresponding lines in Fig. 8 bottom right). At each curve, ini-tial or trailing horizontal parts are removed, as they do not reliablyreflect measurements, but extrapolations, only.

Due to the curve cutting process, the natural emergence order,i.e. that growth curve n belongs to leaf n, is no longer given. Ideallycurves should be sorted by the times when leaves have a certain,predefined size. This is not possible here, as some curves start atquite large leaf sizes. As sorting by emergence time of the curveswould lead to wrong ordering, we compensate the later emergenceof a growth curve by fitting a tangent in log-scale (i.e. a purely expo-nential growth curve) to each curve and order by their time offsets.We use a high growth rate of 10%/h for the tangent, being adequatedue to a measurement offset (cmp. Section 3.9). The resultinggrowth curves sorted per leaf number of plants from the treatmentwith more nutrients and less water are depicted in Fig. 9. Therecurves are temporally aligned such that the time point when thefirst Cotyledon reaches size 0.04 cm2 corresponds to time 0 h.

3.9. Leaf growth modeling

To each of the leaf-number-wise sorted growth curve groups(cmp. Fig. 9) a growth model is fitted in a robust way (thin linesin the plots). We use the well known autocatalytic growth model(see e.g. Richards (1959)) with a slight modification

AðtÞ ¼ A1ð1þ expð�grðt � sÞÞÞ�1 � Aoffset; ð9Þ

where AðtÞ is the leaf size at time t;A1 is the final leaf size, gr is thegrowth rate, s is a time offset. Aoffset is an offset compensating anapparent slight underestimation of the true leaf size.

This model is fitted to the data using a robust error norm able toignore outliers at a constant high cost. We use a variant of the trun-cated quadratic (Blake and Zisserman, 1987) where the constantcost after truncation is 10 times higher than the cost at the trunca-tion limit. By this we ensure to have a maximum amount of inliersas e.g. required as optimality condition in random sample consen-sus (RANSAC, see Fischler et al. (1981)).

The time offset s models the leveling off of the growth curveand is not suitable to estimate leaf emergence. Following Tsaiet al. (1997) we use the time point tc when a leaf reaches a smallgiven size AðtcÞ ¼ c. For our autocatalytic model we derive

tc ¼ s� 1gr

logA1

c þ Aoffset� 1

� �: ð10Þ

4. Results

4.1. Fitted leaf growth models

As we are here dealing with a system to measure early plantgrowth, we have investigated and modeled only the first few leaves(counting cotyledons as leaves 1 and 2). When plants are gettingbigger, we observe large and rapid variations in the size estimatesfor some leaves. This is because wrong segment and depth estima-tions occur more often during this phase. Thus measurementsbecome less reliable making leaf sorting ambiguous. In Fig. 10we therefore show results for the first 7 leaves, only. Looking at fit-ted final leaf size A1 for the averaged plant models we observe,that plants under Treatment 1 (see Section 2) grow much largerleaves than under Treatment 2. However, not only growth rates

Fig. 12. Comparison of the estimated number of leaves obtained for plant numbers 79335 (top row), 79336 (middle row), and 79339 (last row) using the graph-basedapproach and the superparamagnetic clustering of data for over-segmentation. The manually measured observable and actual existing number of leaves are used here asground-truth data.

88 E.E. Aksoy et al. / Computers and Electronics in Agriculture 110 (2015) 78–90

gr are higher, but also the time span s� tc between leaf‘‘emergence’’ tc and leveling off time s. For Treatment 1 the averagegrowth duration is 114 h, for Treatment 2 it is 99 h.

The estimated phyllochron, i.e. the time between leaf‘‘emergence’’ time points tc , varies also slightly, average 65 h forTreatment 1 and 61 h for Treatment 2. Leaf 3, the first leaf afterthe cotyledons, emerges after 2–3 days after these. Leaf 4 thenemerges quicker (1.5–2 days) and leaf 5 then takes 5–6 more daysto emerge. Leaves 6 and 7 then again emerge quicker after2–3 days. Thus for our small dataset we observe that there is noconstant time interval between emergence of leaves, but leaf 5emerges with a considerable delay for both treatments.

Table 1The root-mean-square (RMS) error between the estimated and actual observednumber of leaves for three different plants for the graph-based approach and thesuperparamagnetic clustering of data when used in our framework.

Plantnumber

Graph-based(k ¼ 150)

Graph-based(k ¼ 200)

Superparamagnetic

79335 1.2230 1.3519 1.002179336 2.0962 2.2180 1.491379339 0.9883 0.9022 1.0268

4.2. Benchmarking the method

The functioning of the framework presented in this paperstrongly depends on the segmentation process (step 2 of the pro-cedure). The correct perception of plant leaves represents themost critical component of the procedure. In our framework,the superparamagnetic clustering of data has been chosen forthe over-segmentation of leaves due to the following two reasons.First, this method accelerated on the GPU has a very high timeperformance and processes about 10 frames per second for imagesizes of 640� 512 pixels. Second, segments can be better mergedby this algorithm using the convex hull approximation as

compared to segments produced by conventional segmentationtechniques such as the graph-based or mean shift technique fromFelzenszwalb and Huttenlocher (2004); Comaniciu et al. (2002).This is because both of the latter techniques are dense, i.e, seg-ments are forced to grow until all segments are larger than aminimum segment size. As a consequence, segments often growinto the small cavities that exist in the space between other seg-ments, distorting the actual shape of segment, or can get moreeasily merged with other segments, as can be seen in the compar-ative Fig. 11, where corrected segments for plant number 79339using the graph-based segmentation (both middle columns) andsuperparamagnetic clustering of data (right column) within ourframework are shown for selected frames.

In the graph-based approach the number of output segments iscontrolled by the threshold k which should be lower than the

Fig. 13. Under-segmentation errors observed once leaves are getting bigger. Merged segments have the same color.

E.E. Aksoy et al. / Computers and Electronics in Agriculture 110 (2015) 78–90 89

recommended value (k ¼ 500) to achieve the over-segmentationmode. We determined experimentally that k ¼ 150 guaranteesover-segmentation for the majority of input frames (see the middleleft column), while larger k values can produce dramatic merges(see the middle right column). Overall, we obtained better resultswith the superparamagnetic clustering as compared to thegraph-based technique.

We further analyzed how much the estimated number of leavesdeviate from the ground truth provided, and compare the perfor-mance of the superparamagnetic clustering method with the oneof the graph-based method Felzenszwalb and Huttenlocher(2004) when used inside our framework.

Fig. 12 shows the comparison of the estimated number of leavesfor three different tobacco plants in the case of using the super-paramagnetic clustering of data and the graph-based techniquewith the ground-truth data. The ground-truth data is obtainedthrough human visual inspection, counting the number of leaves,including partially occluded ones. Both ground truth and the auto-matically computed number of leaves using our framework areshown for both segmentation approaches as a function of days.We can see that the number of leaves estimated with the super-paramagnetic clustering agrees better with the ground truth thanthe graph-based method. However, both methods cannot handlethe plant number 79336 after 25 days (see the high deviationbetween the estimated and actual observed number of leaves inFig. 12 (middle row). A quantitative evaluation of both methodswith respect to the observable number of leaves based on theroot-mean-square error is presented in Table 1.

5. Discussion

The found average growth models are well in accordance withestablished literature.

Average per leaf growth rates of 2.5% (Treatment 1) or 2.0%(Treatment 2) are in the same range as the growth rates found inWalter and Schurr (1999). There, in Fig. 1D, total leaf growth ratesRGR between 12 and 18%/d, i.e. 0.5 and 0.75%/h, are reportedtogether with the observation, that the biggest leaf contributesapprox. 35% of the overall size and about 30–40% of the growth(Fig. 4B). As non-growing leaves are also taken into account fortotal leaf growth, growth rates for growing leaves need to be signif-icantly higher than the averaging total, well in accordance with ourfindings.

Systematic increase of final leaf size A1 of the first few leaves, asfound for both treatments, are also reported in (Tsai et al., 1997,Fig. 1). Absolute sizes are obviously treatment dependent, see(Walter and Schurr (1999)).

Phyllochron values reported in (Tsai et al., 1997, Fig. 5, p. 911)show a similar behavior as our findings. Leaf 4 emerges earlier thanexpected and leaf 5 somewhat later. The absolute durationbetween leaf emergence of the first 6 leaves lies however higherthan under our treatments, i.e. between 72 h and 144 h with anaverage of approx. 110 h For a treatment with 300 lE m�2 s�1 pho-tons and daily watering. Our treatments feature much higher lightintensities and different watering strategies. Phyllochrons foundhere lie between 32 h and 114 h with averages of approx. 61 h or

65 h, respectively. According to Munns (2002) leaf emergence rateis reduced under drought stress, thus clearly reacts to environmen-tal conditions and thus differences found may be related to treat-ment differences.

The framework has been successfully applied inside a robot per-ception-action loop during experiments that were performed inthe context of the EU project GARNICS. In these experiments, therobot had to make decisions about plant treatment based on sen-sory input, which was being processed with our multi-level pipe-line, and water the plants accordingly. In the final experiments ofthe project the robot succeeded in taking care of the plants overa period of about three weeks, where the treatment found by thesystem resulted in a generally higher growth rate than in any ofthe training data.

6. Conclusion

We presented a novel multi-level procedure for finding andtracking of leaves of growing tobacco plants which allowed us tomeasure automatically important plant parameters, i.e., numberof leaves and leaf size, as a function of time. The main challengeoriginates from the complex appearance of plants, making it diffi-cult to segment plant organs. We used leaf-shape models toimprove leaf segmentation and could successfully segment andtrack tobacco-plant leaves to up to an age of about 25 days. Beyondthis growth stage, leaf segmentation turned out to be increasinglyhard. As leaves grew older, we often observed under-segmentationerrors. Fig. 13 shows examples where such under-segmentationeffects have been observed. These problems can only be resolvedby further improving the segmentation procedure.

The convex-hull approximation works well for tobacco plantsbut might have to be augmented using more sophisticated leafmodels when dealing with other types of plants. The border detec-tion as well as the depth reasoning could be improved in the futureusing e.g. a structured-light imaging system (Geng, 2011). Theaccuracy of the plant models estimated in Section 4.1 can furtherbe improved by simply increasing the number of observed plants.Ellipses are used to estimate the size of the leaves from the seg-ment boundaries in the last step of the algorithm. For tobaccoplants, the ellipse model is an appropriate choice. For other plants,another leaf-shape model could be used instead of the ellipse.Assumptions about the leaf shape are also being made during themerging step (see Section 3.5). It is assumed that leaves have aconvex shape. In some approximation, this holds for many typesof plants, but it is not generally true. For non-convex leaf-shapes,the merging algorithm would have to be modified, and a specificleaf model could be fitted to the boundary of the object insteadof finding its convex hull. Furthermore, we are currently analyzingplant vein structures which can then be used to correct segmentsand fit more accurate ellipses. Initial steps given in Johansson(2010) show promising results along this line.

Acknowledgements

We thank Torge Herber from Forschungszentrum Jülich for theimage acquisition. The research leading to these results has

90 E.E. Aksoy et al. / Computers and Electronics in Agriculture 110 (2015) 78–90

received funding from the European Community’s SeventhFramework Programme FP7/2007-2013 – Challenge 2 – CognitiveSystems, Interaction, Robotics – under grant agreement No247947 – GARNICS. Babette Dellen acknowledges support fromthe Spanish Ministry for Science and Innovation through a Ramony Cajal program.

References

Abramov, A., Pauwels, K., Papon, J., Wörgötter, F., Dellen, B., 2012. Real-timesegmentation of stereo videos on a portable system with a mobile gpu. IEEETrans. Circ. Syst. Video Technol. 9 (22), 1292–1305.

AlenyÃ, Guillem, Dellen, Babette, Foix, Sergi, Torras, Carme, 2013. Robotized plantprobing: leaf segmentation utilizing time-of-flight data. IEEE Robot. Automat.Mag. 20 (3), 50–59.

Alenyà, G., Dellen, B., Torras, C., 2011a. 3d Modelling of leaves from color and tofdata for robotized plant measuring. Proc. IEEE Intl. Conf. Robot. Autom..

Alenyà, G., Moreno-Noguer, F., Ramisa, A., Torras, C., 2011b. Active perception ofdeformable objects using 3d cameras. In: Workshop de Robotica Experimental,Seville, pp. 434–440.

Biskup, B., Scharr, H., Schurr, U., Rascher, U., 2007. A stereo imaging system formeasuring structural parameters of plant canopies. Plant, Cell Environ. 30,1299–1308.

Blake, Andrew, Zisserman, Andrew, 1987. Visual Reconstruction. MIT Press,Cambridge, MA, USA, ISBN 0-262-02271-0.

Bradski, G., 2000. The OpenCV Library. Dr. Dobb’s Journal of Software Tools.Chien, Chung-Fang, Cheng, Yu-Che, Lin, Ta-Te, 2011. Robust ellipse detection based

on hierarchical image pyramid and hough transform. J. Opt. Soc. Am. 28 (4),581–589.

Comaniciu, Dorin, Meer, Peter, Member, Senior, 2002. Mean shift: a robust approachtoward feature space analysis. IEEE Trans. Pattern Anal. Mach. Intell. 24, 603–619.

CORESTA CORESTA. A scale for coding growth stages in tobacco crops, February2009. <http://www.coresta.org/Guides/Guide-No07-Growth-Stages_Feb09.pdf>.

De Vylder, J., Philips, W., Van Der Straeten, D., 2013 Multiple leaf tracking usingcomputer vision methods with shape constraints. In: Proc. of the InternationalConference on Sensing Technologies for Biomaterial, Food, and Agriculture(SPIE: SeTBio2013), Yokohoma, Japan.

De Vylder, Jonas, Donoso, Daniel Ochoa, Philips, Wilfried, Chaerle, Laury, Van DerStraeten, Dominique, 2011. Leaf segmentation and tracking using probabilisticparametric active contours. In: Gagalowizc, A., Philips, Wilfried (Eds.), LectureNotes in Computer Science, 6930. Springer, pp. 75–85.

Felzenszwalb, Pedro F., Huttenlocher, Daniel P., 2004. Efficient graph-based imagesegmentation. Int. J. Comput. Vision 59 (2), 167–181.

Fischler, Martin A., Bolles, Robert C., 1981. Random sample consensus: a paradigmfor model fitting with applications to image analysis and automatedcartography. Commun. ACM 24 (6), 381–395. http://dx.doi.org/10.1145/358669.358692, ISSN 0001-0782.

Geng, Jason, 2011. Structured-light 3d surface imaging: a tutorial. Adv. Opt. Photon.3 (2), 128–160.

Horowitz, S.L., Pavlidis, T., 1974. Picture segmentation by a directed split-and-merge procedure. In: Proceedings of the 2nd International Joint Conference onPattern Recognition, Copenhagen, Denmark, pp. 424–433.

Jin, Jian, Tang, Lie, 2009. Corn plant sensing using real-time stereo vision. J. FieldRobot. 26 (6-7), 591–608.

Johansson, Peter, 2010. Plant condition measurement from spectral reflectancedata. Master’s thesis, Linköping University, Computer Vision.

Kaewapichai, Watcharin, Kaewtrakulpong, Pakorn, 2008. Robust ellipse detectionby fitting randomly selected edge patches. World Acad. Sci., Eng. Technol., 30–33.

Knutsson, H., Westin, C.F.,1993. Normalized and differential convolution: methodsfor interpolation and filtering of incomplete and uncertain data. In CVPR’93,New York City, USA, pp. 515–523.

Loch, B.I., Belward, J.A., Hanan, J.S., 2005. Application of surface fitting techniquesfor the representation of leaf surfaces. MODSIM 2005 Int. Cong. Model. Simul.,1272–1278.

Moeslund, T.B., Aagaard, M., Lerche, D., 2005. 3d Pose estimation of cactus leavesusing an active shape model. In: Application of Computer Vision, 2005. WACV/MOTIONS ’05 Volume 1. Seventh IEEE Workshops on, vol. 1, pp. 468–473.

Munns, R., 2002. Comparative physiology of salt and water stress. Plant, CellEnviron. 25 (2), 239–250.

Polder, G., van der Heijden, G.W.A.M., Jalink, H., Snel, J.F.H., 2007. Correcting andmatching time sequence images of plant leaves using penalized likelihoodwarping and robust point matching. Comput. Electron. Agric. 55 (1), 1–15, ISSN0168-1699.

Quan, L., Tan, P., Zeng, G., Yuan, L., Wang, J., Kang, S.B., 2006. Image-based plantmodelling. ACM Siggraph, 599–604.

Richards, F.J., 1959. A flexible growth function for empirical use. J. Exp. Botany 10(2), 290–301. http://dx.doi.org/10.1093/jxb/10.2.290.

Silva, L.O.L.A., Koga, M.L., Cugnasca, C.E., Costa, A.H.R., 2013. Comparativeassessment of feature selection and classification techniques for visualinspection of pot plant seedlings. Comput. Electron. Agric. 97 (0), 47–55.

Song, Ge, Wang, Hong, 2007. A fast and robust ellipse detection algorithm based onpseudo-random sample consensus. In: Proceedings of the 12th InternationalConference on Computer Analysis of Images and Patterns, pp. 669–676.

Song, Y., Wilson, R., Edmondson, R., Parsons, N., 2007. Surface modelling of plantsfrom stereo images. In: 6th IEEE Intl. Conf. on 3D Digital Imaging and Modelling.

Teng, Ch.-H., Kuo, Y.-T., Chen, Y.-S., 2011. Leaf segmentation, classification, andthree-dimensional recovery from a few images with close viewpoints. Opt. Eng.50 (3). http://dx.doi.org/10.1117/1.3549927.

Tsai, C.H., Miller, A., Spalding, M., Rodermel, S., 1997. Source strength regulates anearly phase transition of tobacco shoot morphogenesis. Plant Physiol. 115 (3),907–914.

Wallenberg, Marcus , Felsberg, Michael, Forssén, Per-Erik, Dellen, Babette, 2011.Leaf segmentation using the kinect. In: Proceedings of SSBA 2011 Symposiumon Image Analysis.

Walter, A., Schurr, U., 1999. The modular character of growth in nicotiana tabacumplants under steady-state nutrition. J. Exp. Botany 50 (336), 1169–1177. http://dx.doi.org/10.1093/jxb/50.336.1169.

Wang, Jianlun, He, Jianlei, Han, Yu, Ouyang, Changqi, Li, Daoliang, 2013. An adaptivethresholding algorithm of field leaf image. Comput. Electron. Agric. 96 (0), 23–39, ISSN 0168-1699.

Wörgötter, F., Abramov, A., Aksoy, E.E., Dellen, B., 2013. Method and device forestimating development parameters of plants. Patent office WO, Patent number2013083146.