Phenotype Estimation with Deep Learning

Phenotype Estimation with Deep Learning

Hanqi Sun, Aug 5th

Outline

● Pipeline● Current Result● Future Plan

Pipeline: Part I

3D Point Cloud Reconstruction

from images

Segmented Point Cloud

Phytomer(Single leaf)Point Cloud

SegmentationAlgorithm

ClusteringAlgorithm

Pipeline: Part II (my work)

CenteredPhytomer

Point Cloud

2d GrayscaleProjection Images

Phenotype Estimations

SimulatedPhytomer

Point Cloud

2d GrayscaleProjection Images

Deep neural networks

Centered Phytomer PointCloud

● After we find the stem, we could position the stem along the z axis and the leaf along the y axis

Generate 2d projection images● 128 * 128 greyscale images● One “forward” projection (ignore x coordinate), One “downward” projection

(ignore z coordinate)

Deep neural network

● Alexnet (5 convolutional layers with 3 fully connected layers at the end.)● Built using tensorflow, trained on GPU● About 100 million parameters per network (600 MB+ space)● ~ 2.5 hour training time for one phenotype

Phenotype estimation

● In the end, the pipeline will give a txt file back: (each row is one phytomer)

Result with simulated data (No rotation angle)Name Range RMSE error

Leaf angle* 75.94 3.30 (4.35%)

Leaf radiation angle* 120.66 4.34 (3.60%)

Leaf length* 35.00 0.87 (2.49%)

Leaf width [max] 3.61 0.27 (7.48%)

Leaf width [average] 2.99 0.21 (7.o2%)

Leaf area* 133.45 8.11 (6.08%)

*: projection onto y-z plane

Three rotation angles

X Y Z

With three rotation angles (prediction only)Name Range RMSE error

Leaf angle* 75.94 26.00 (34.2%)

Leaf radiation angle* 120.66 27.75 (22.8%)

Leaf length* 35.00 8.79 (25.1%)

Leaf width [max] 3.61 1.09 (30.2%)

Leaf width [average] 2.99 0.76 (25.4%)

Leaf area* 133.45 27.19 (20.3%)

Error heat map of Leaf angle (34.2%)

X-axis: Z rotation angle

Y-axis: Y rotation angle

Color: error (degrees)

Range: 75.94

Error heat map of Leaf angle (34.2%)

X-axis: Z rotation angle

Y-axis: Y rotation angle

Color: error (degrees)

Range: 75.94

Error heat map of Leaf Length (25.1%)

X-axis: Z rotation angle

Y-axis: Y rotation angle

Color: error (unit)

Range: 35.00

With three rotation angles (With training)Name Range RMSE error

Leaf angle* 75.94 16.53 (21.77%)

Leaf radiation angle* 120.66 18.24 (15.12%)

Leaf length* 35.00 4.55 (13.00%)

Leaf width [max] 3.61 0.46 (12.74%)

Leaf width [average] 2.99 0.36 (12.04%)

Leaf area* 133.45 15.73 (11.78%)

Result with simulated data (No rotation angle)Name Range RMSE error

Leaf angle* 75.94 3.30 (4.35%)

Leaf radiation angle* 120.66 4.34 (3.60%)

Leaf length* 35.00 0.87 (2.49%)

Leaf width [max] 3.61 0.27 (7.48%)

Leaf width [average] 2.99 0.21 (7.o2%)

Leaf area* 133.45 8.11 (6.08%)

*: projection onto y-z plane

Result with Green House Data

● The image of that of green house plants is different from the ones I used to train the network○ The leaves are longer, and the point cloud is complete (no noise/lost points at all)○ So for a little phytomer (leaf), the value does not seem correct (e.g. the leaf angle)

● So for field data, I will look at the phytomer and images, generate data that are similar what we get, and retrain the network.○ It takes 12 hours to generate 60,000 phytomers or 20,000 full plants.○ It takes 2.5 hour to train one phenotype○ Phenotype prediction with pre-trained network is done instantly.○ In total, it takes about 4 days to give the result back.

Good Result 1

Good Result 2

Good Result 3

Bad Result 1 (leaf angle, leaf area too small)

Bad Result 2 (leaf angle too small)

Result for error 1 (double leaves, but prediction OK)

Result for error 2 (double leaves, leaf angle too small)

Result for error 3 (leaf/stem not connected, but prediction OK)

Result for error 4 (no leaf, leaf angle makes no sense, others OK)

Future Effort (minor)

● Retrain the network according to the green house data, see if the result gets better

● Retrain the network with full plant projection. See if other phenotypes (stem height, average leaf angle) can be captured in 2d image as well.

Future Effort (major)

● Try to simplify the pipeline. See if we could get phenotype values directly from 3d plant (without segmentation and projection) or sensor images.

● For this fall, I think we should focus on what we have so far.

Directly train on point cloud (simulated)

Original Point Cloud

Phenotype estimations

Simulated Point Cloud

Name Range RMSE error

Average leaf angle 56.47 17.29 (30.62%)

Stem height 214.89 10.27 (4.78%)

Convex hull volume 66046 4513 (6.83%)

Average leaf radiation angle 41.33 6.86 (16.60%)

Average leaf area 112.75 20.68 (18.34%)

Average max leaf width 3.39 0.87 (25.66%)

Average interligule length 12.67 1.37 (10.83%)

Leaf number [6, 15] 4.40

Thank you!

● hanqis@andrew.cmu.edu any questions! (12 hour time zone difference,)● Simulation code available (MATLAB, 4000+ lines from ground up)

○ https://bitbucket.org/cmu_terra/sorghum_simulation (with permission)

● Deep Learning code to be released.. (python, 1000+ lines)