PhenoBot: A Robot System for Phenotyping in the Greenhouse Using 3D Light-Field Technology

IEEE TC Agricultural Robotics - AgRA Webinar #20 August 27, 2014. Gerrit Polder, Jan Willem Hofstee, Wageningen UR.

Outline

! Introduction (PhenoBot = SpySee II) ! Autonomous Robot ! 3D Light field Camera System ! Experiments

●  Lab ● Greenhouse

! Discussion

4* IR, Colour, Range (ToF) cameras

SpySee (EU project Spicy)

Song,&Y.,&Glasbey,&C.&A.,&van&der&Heijden,&G.&W.&A.&M.,&Polder,&G.,&&&Dieleman,&J.&A.&(2011).&Combining&stereo&and&EmeFofFflight&images&with&applicaEon&to&automaEc&plant&phenotyping&(Vol.&6688,&pp.&469–478).&Presented&at&the&17th&Scandinavian&Conference&on&Image&Analysis,&SCIA,&Ystad,&Sweden.&&Song,&Y.,&Glasbey,&C.&A.,&horgan,&G.&W.,&Polder,&G.,&Dieleman,&J.&A.,&&&van&der&HEIJDEN,&G.&W.&A.&M.&(2014).&ScienceDirect.&Biosystems)Engineering,&118(C),&203–215.&doi:10.1016/j.biosystemseng.2013.12.008&van&der&Heijden,&G.,&Song,&Y.,&Horgan,&G.,&Polder,&G.,&Dieleman,&A.,&Bink,&M.,&et&al.&(2012).&&&SPICY:&towards&automated&phenotyping&of&large&pepper&plants&in&the&greenhouse.&Func3onal)Plant)Biology,&39(11),&870–&877.&doi:10.1071/FP12019&&Horgan,&G.&W.,&Song,&Y.,&Glasbey,&C.&A.,&van&der&Heijden,&G.&W.&A.&M.,&Polder,&G.,&Dieleman,&J.&A.,&et&al.&(2014).&Automated&esEmaEon&of&leaf&area&development&in&sweet&pepper&plants&from&image&analysis.&Func3onal)Plant)Biology.&doi:10.1071/FP14070&

SpySee – combining stereo and ToF

Stereo&pair&+&ToF&range&image&"&detailed&range&image&Stereo pair + ToF range image Ö detailed range image

SpySee - limitations

! For use in large greenhouses an autonomous platform is needed

! Blue background behind the plants ! Cumbersome and time consuming 3D reconstruction ! The resolution of the ToF camera (64 x 40 px) was to low

for using the depth information for segmentation ! Occlusions hampers accurate estimation of leaf and fruit

features ! Limited depth of focus (DOF) range (difficult to measure

objects close to the camera, or far in the background)

PhenoBot = SpySee II

SpySee (limitations) PhenoBot

Hand driven Autonomous Robot

Colour + Time of Flight camera 3D Light field Camera

Cumbersome sensor fusion for pixel registration of depth and colour image

Camera promises to output direct registered depth and colour image

Small depth of focus Image is in focus over whole depth range

Blue background screen needed in adjacent lane

No background screen needed.

Occlusion due to stereo approach Reduced occlusion due to micro lenses

Robot Trolley

! Adapted spraying robot from Berg Hortimotive

Robot Trolley (adapted spraying robot)

! Spraying boom replaced by camera stand

! Adapted software – start – stop – trigger - distance

3D Light Field Camera

! Placing a micro lens array in front of an image sensor transforms a normal camera into a single lens 3D camera, which also allows the user to change the focus and the point of view after a picture has been taken

! While the concept of such plenoptic cameras is known since 1908, only recently the increased computing power of low-cost hardware and the advances in micro lens array production, have made the application of plenoptic cameras feasible

3D Light Field Camera

! Commercial available cameras:

LYTRO

raytrix

3D reconstruction and extended depth-of-field based on only one snapshot and a single-lens camera - calibration-free monocular camera - robust & space-saving setup - down to micron resolution - extended depth-of-field by software re-focus - captures fast moving objects by single shot

•  US-Pat.-No.: 2012/0050562 A1 , CHIP-Award 2012: „Innovation of the year“ Copyright&©&2013&by&Raytrix&GmbH,&Germany.&All&rights&reserved.&Design,&features,&and&specificaEons&are&subject&to&change&without&noEce.&

3D Light Field Camera Technology*

Micro-lens array (MLA) optics

3D depth map re-focus plane on eye re-focus plane on nose

4D light field raw image data

perspective shift

Light field engine

3D 3D view in total focus

3D Light Field Camera (Raytrix)

!  One camera, one lens, one shot for 3D and 2D data

!  Up to 7MP effective resolution

!  6x extended depth of field

!  Minimized occlusion

!  Passive system

!  Data capture will not interfere with the plants natural behaviour. (no laser scanning or IR pattern projection)

!  Robust

Occlusion

!  Due to the large baseline a stereoscopic system suffers from occlusion. Cylinder occluded in one of the images, no matching possible

1 2

Example of a setup that is prone to occlusion

!  Due to thousands of micro lens images, with about the same perspective through the object lens, occlusion is exceptionally reduced

PhenoBot – First experiments

! Lab experiments: ● Compare hand measurements with 3D imaging ● Fruit and stem size

! Experiments in Enza greenhouse ● Simultaneously manual measurements of

phenotypic characteristics: ●  Stem thickness ●  Leaves – length, width and angle ●  Amount of flowers per bunch ●  Amount of ripe and raw fruits per bunch

PhenoBot – Lab experiments

! Relation between depth value and real distance.

! Relation between real and measured size: ● Cherry tomato ● Beefsteak tomato ● Truss tomato ● Tomato stem

PhenoBot – Lab experiments

! Results:

PhenoBot – Greenhouse measurements

PhenoBot – Greenhouse measurements

PhenoBot – Greenhouse measurements

! Example data

PhenoBot – Greenhouse measurements

PhenoBot – Greenhouse measurements

Find red tomatoes based on colour only

PhenoBot – Greenhouse measurements

Find red tomatoes based on colour and depth

PhenoBot – discussion – advantages

! One camera, one lens, one shot for 3D and 2D data ● No 3D reconstruction needed ● Exposure times similar to standard 2D cameras ●  Image capture with up to 6fps (for R29) ● 3D data calculation from images with up to 6fps ● Up to 7MP effective resolution for R29 camera

! Robust ●  Industrial camera

! Passive system ● No laser scanning or IR pattern projection ● No interference with standard plant behaviour

PhenoBot – discussion – advantages

! Extended depth of field ● No need to change main lens focus for different

plant growth states ! Minimized occlusion

●  find sharp depth discontinuities at leaf borders

! Adaptable system ●  Interchange the main lens to adapt to specific

depth ranges ●  Large range of lenses available for different depth

ranges ( > 40 mm, see cons)

PhenoBot – discussion - disadvantages

! New technique facing a number of teething troubles ● Software (updates, Halcon, 64 bit, …) ● Documentation ● Price

! No calibration for 3D reconstruction, but calibration of the micro lens array (grey reference) appeared to be very sensitive to saturated pixels ● Possible to apply new grey calibration on old data

PhenoBot – discussion - disadvantages

! For main lens only one aperture setting (f/11) ● Flash illumination can only be controlled with

electronic gain ! Wide field of view lenses not possible (> 40 mm)

● SpySee - 75° FOV ● PhenoBot - 50° FOV

! Heavy computational load for calculating depth image. ● Big computer on robot or offline processing

! Large file size (~ 40 Mb – lossless compressed)

PhenoBot – future plans

! Search for optimal viewing angle ! Image stitching ! Illumination / flash ! Data/processing flow

● Offline/online ● Compression

! Feature extraction ! QTL analysis

● Based on features ● Based on direct image information

Questions?

Acknowledgments: •  Dick Lensink (Enza)

•  Bram Veldhuisen and Menno Hutting (students Wageningen UR)

•  Arne Erdmann (Raytrix)

•  Jeroen Gerritse (Berg Hortimotive)

This project was co-financed by

the Consortium for Improving

Plant Yield (CIPY), which is part of

the Netherlands Genomics

Initiative