TraitCapture

NextGen Software and Hardware for Scaling from Seeds to Traits to Ecosystems

Tim Brown, Research Fellow, Borevitz LabARC Centre for Plant Energy Biology, Australian National University

Gareth Dunstone, Sarah Namin, Mohammad Esmaeilzadeh,Justin Borevitz

Borevitz Lab Phenomics

• Lab:

• Measure phenotypes with high precision across large natural populations (Arabidopsis, Brachypodium, Eucalyptus) in varied growth environments

• Identify the genetic basis of traits of interest

• Identify novel, cryptic traits

• Field (forest not crop):

• Monitor phenotype and environment at high precision across scales from plant to ecosystem to identify natural variation on the landscape

Conservation: Measure ecosystem stability; Decide how to spend limited conservation $$

Restoration: Select seeds for building “climate ready” populations and assisted migration in reforestation, etc.

We face challenges of unprecedented scale and complexity

“2 Billion person“yield gap” by 2050

Food shortages

Overpopulation

Biodiversity loss

Climate Change

97% of Great Barrier Reef bleached in 2016

High throughput phenotyping for the masses

Large-scale infrastructure phenotyping systems are great if you got one, but…

• Not affordable for the majority of labs• Not cost effective smaller scale projects• Insufficient capacity to cover all users

globally (thousands of labs)

• Since 2015:

There is a need for open source, low cost, easy-to-useHTP systems for general research

“Pain Points”• Business speak for problem areas that limit our ability to accomplish

what we want to do. They are often things people take for granted but when solved, paradigm shifts can happen

• In research: These are the things that aren’t actual “research” but still required to get our work done. They slow down knowledge discovery.• Zegami is a great example of a solution to this type of problem in

HTP

• My PhD was 6 years of collecting and coding data and <1yr of actual analysis

• What limits adoption of High Throughput Phenotyping (HTP) by the wider research community?• … Pretty much everything!

High Throughput is high complexity

“High Throughput” means we need to do lots of stuff with lots of things and the complexity increases non-linearly

• Hardware: • Explosion of imaging and other hardware, but capturing images isn’t

Data

• Data Management• Essential but time consuming and hard to do well

• Experiment, plant and sample tracking• HTP means thousands of plants, samples and genomes

• Data Visualization: • Thousands of images in time-series and 3D

• Analysis• Complex, Hard to make turn-key

• Data publishing and sharing / Code sharing

Challenges to “consumer” HTPProcessing and managing big data is a huge challenge• We used to be primarily limited by data collection (hardware)

• “The missing heritability is in the field” (Brachi et al, 2011)

• Now we are limited by data processing and curation (software)• “The missing heritability is on your hard drive”

• We need “excel” for big data

Challenges to “consumer” HTPA sample is no longer a data point

• We can now gather huge quantities of “measurements” but it is not “DATA” until we can actually do science with it – NOT a solved problem

• Example: Soil Moisture from mesh network• 10-min intervals @ 20 locations in field site, 6 months of data• The spatial variation is what is interesting... Artifact or signal? Hard to know

without spatial context

Soil Moisture @ 20 sensor locations

Lab vs field phenotypingLab: High precision measurement and control but low realism

youtu.be/d3vUwCbpDk0

Lab vs field phenotyping

Field: Realistic environment but low precision measurements

In the field we have real environments but the complexity (and bad lighting!) reduces our ability to measure things with precision

youtu.be/gFnXXT1d_7s

Borevitz Lab Approach• Create more “natural” Lab conditions in growth chambers

• Measure more precisely in the Field

• My work: open source pipelines for “consumer” HTP

© Suzanne Marselis

enviro-net.org

Lab phenotyping

Normal growth chamber conditions aren’t very “natural”

Kulheim, Agren, and Jansson 2002

Real World

Growth Chamber

Conviron Growth cabinets with dynamic “semi-realistic” environmental & lighting conditions

• Grow plants in simulated regional/seasonal conditions & simulate climate

• Control light intensity, spectra (8/10-band Heliospectra lights), Temp/Humidity @ 5min intervals

• Expose “cryptic” phenotypes

• Repeat environmental conditions between studies and collaborators

Borevitz Lab Solution: SpectralPhenoClimatron (SPC)

Spectral response of Heliospectra LEDs. (L4A s20: 10-band)

Climatron: “Fall” conditions for Wollongong, Australia

Temperature

Daily Light

“Happy” germination conditions

Fall conditions to spreadout phenotypes

TraitCapture: Open-source phenotyping pipeline

• Phenotype 2,000 plants (7 Conviron chambers) in real-time

• 14 DSLR’s (2/chamber) - Controlled by Raspberry Pi computers

• 12 JPG + RAW images/hr during daylight

• (Semi) automated analysis pipeline: phenotype data from 150,000 pot images a day

• Phenotypes• Area

• Diurnal movement

• Color (RGB, Gcc, etc)

• Perimeter, Roundness

• Compactness, Eccentricity

• Upcoming:• Leaf Count

• Leaf tracking

• Leaf length/width/petiole

• Machine learning

Brown, Tim B., et al. (2014). Current opinion in plant biology 18 (2014): 73-79.

Corrected

Segmented

Original

GWAS

Area

TraitCapture Goal: Streamline all the “pain points” associated with HTP

• Hardware

• Data Management

• Experiment, plant and sample tracking

• Data Visualization• Numeric

• New data types

• Analysis

• Data publishing and sharing / Code sharing

PiCam: Hardware controlhttp://traitcapture.org/cameras

• Control all cameras types

• DSLR, USB / pi-cam, IP, PTZ, Gigapixel

• 180° picam (audit lights)Manage Growth conditions• LED Lights (Heliospectra)• Conviron chambers• Measure temp, humidityPush image data to server

TRAITCAPTURE (http://traitcapture.org)* Control * Manage * Visualize

Exif2TimeStream - Turnkey image management

• Bulk processing of 100,000’s of images from almost any source: outputs structured, timestamped timelapse + JSON

• 1 click website: Ex: phenocam.org.au

• TERN – 180,000 images from 25 cameras

• Drag and drop to player• Resize, rotate, color correct, unwarp• Automated image processing for any live

experiment via web services

QA/QC

• Detect image and tray orientation• QR codes for tray tracking• Detect color checker• Detect light changes or failures (via 180°

picam)• SLACK, SMS and Email notifications to

admins/users for chamber,camera, light failure, etc

Experiment Managementhttp://traitcapture.org/experiments

• Schedule chambers, cameras

• Assign Experiment ID’s, plantID’s

• Push control files to PiCam & chambers

TRAITCAPTURE (http://traitcapture.org)* Control * Manage * Visualize

Graphing - Any timestamped datahttps://traitcapture.org/graph

• Drag and drop any timestamped CSV file

• Persistent links to any graph (live-URL or static data)

Experimental outputs:

• Chamber control files & logs

• Phenotypes, sensor streams, etc

• Attach graph objects to any object (experiment, chamber, etc)

PointClouds - https://traitcapture.org/pointcloudsMobile-friendly pointcloud viewer

• Drag and drop Las and PLY files

• View UAV-derived pointclouds or 3D plant scan data

• Time-series point clouds

Segmentation / PhenotypingImages to pixel data

• Color, Area, Growth rate, etc

• Unrandomize experiments based on Accession or any other feature or trait of interest

TimePlayer - Web-based timelapse player

https://traitcapture.org/timestreams• View live and past experiments

• View any image stream via drag and drop JSON

• View segmented data

• View pot layout on chambers

• Annotate and select plants to omit from analysis

• View and annotate timelapse from anyresolution of image

Field Phenotyping…

Emerging tools for HTP in the field1. Gigapixel imaging

2. UAV’s (drones)

3. LiDAR

4. “NextGen” Visualization tools: Virtual and Augmented Reality

5. Other stuff: Hyperspectral, thermal, micrometre-resolution dendrometers; Raspberry Pi-based phenocams

-

Golfer, 7km distant

Monitor daily change in every plant or tree in a field site

Gigapixel imaging

Alpha version player: bit.ly/gigavision-a1

Gigapixel canberra• Usable view area for phenology: ~5,000 Ha

http://bit.ly/CBRFall2014

Gigavision hardware evolution• 2009

• Custom-built system with robotics servos, DSLR’s, hand wired with mini pc, ~0.5-1°accuracy

• 40 minutes per panorama (1.5 gigapixels)

• Sept 2015• Off the shelf Axis PTZ camera (Q6115) with $40 Raspberry Pi

computer running python code (on my desk via 5km wifi link)

• Slack alerts if system offline

• Current camera version: • 8MP resolution PTZ with 700 °/sec rotation

• 2 gigapixel panorama in < 5 min

Alpha version player: bit.ly/gigavision-a1

UAV data to 3D forest modelsPix4D: 20min drone flight + 2hrs processing = 3D model of forest3D Point clouds online: http://traitcapture.org/pointclouds

• ForestUtils - Processing script for extracting tree data (python):• GPS, Height, 3D volume, top-down area, RGB phenology data

• Straight to google maps online

• https://pypi.python.org/pypi/forestutils (Zac Hatfield Dodds)

Kioloa Tech DemosFriday: Drone + Pix4DSaturday: ForestUtils

Putting it all together: National Arboretum Phenomic & Environmental Sensor Array

National Arboretum, Canberra, Australia

Collaboration with Cris Brack, Albert Van Dijk (ANU Fenner school); Borevitz Lab

• Ideal location• 5km from ANU (64 Mbps Wi-Fi) and near many research institutions

• Great site for testing experimental monitoring systems prior to more remote deployments

• Forest is only ~5 yrs. old • Chance to monitor it from birth into the future!

25/20

National Arboretum Sensor Array• 20-node Wireless mesh sensor network (10min sample interval)

• Temp, Humidity

• Sunlight (PAR)

• Soil Temp and moisture @ 20cm depth

• uM resolution dendrometers on 20 trees

• Campbell weather stations (baseline data for verification)

• Two Gigapixel timelapse cameras: • Leaf/growth phenology for > 1,000 trees

• LIDAR: DWEL / Zebedee

• UAV overflights (~monthly) • Georectified image layers

• High resolution DEM

• 3D point cloud of site in time-series

• Sequence tree genomes

Environment

Phenotype

Genetics

2017: Hyperspectral and Thermal PTZ

Arboretum Video

https://www.youtube.com/watch?v=YanOqSlW7yE

The Future – NexGen visualization• Our current tools for data visualization and management

(e.g. Excel, Matlab, R, GIS) aren’t sufficient for Phenomics datasets

• Phenomics / HTP data is• Time-series• Geospatial and 3D• Highly varied: pixels, numeric timeseries, point clouds,

hyperspectral, genetics• Highly dense: 100s’ – 1000s of

data layers / pixel / leaf

• Human insight is amazing but only if your data is in a format that lets youthink about it clearly

• Many of the current challenges in HTPderive from lack of tools that lets usbest use this data

Virtual Reality (VR) and Augmented Reality (AR)

• Virtual Reality (VR) and Augmented Reality (AR) will radically change how we interact with our data

• VR (Oculus, VIVE, etc): View immersive environments (travel to your field site virtually)

• AR (MS Hololens, Magic Leap): Add virtual content to the real world (view your field site on your desk; view tree/crop data on site in the field)

Search: “Magic Leap Wired”

Gaming and animation software for data visualization

• Hollywood and the gaming industry have spent billions of dollars creating tools for generating realistic environments

• We can use these tools for data visualization

• Houdini:

EcoVR: Virtual 3D Ecosystems ProjectGIS for 3D “time-series data”

Goal: Use modern gaming software to explore new methods for visualizing time-series environmental data

• Collaboration with • ANU Computer Science Dept. TechLauncher students

• Stuart Ramsden, ANU VISlab

• Historic and real-time data layers integrated into persistent 3D model of the national arboretum in the Unreal gaming engine

• Visualize any field site with GPS and time-series data

Demo @ Kioloa, Friday

Conclusions

• Consumer HTP and NextGen visualization are design problems – this is a good problem to have because we can fix it

• Inefficiencies in these tools slow down research and reduce efficiency

• We live in exciting times… we are building the future interfaces for how people will interact with their data, and monitor the earth

• Let’s collaborate – share data, software, protocols

This is just the beginning

Atari 2600 “Adventure” circa 1980

“Skyrim” circa 2011

We are at the “ATARI” stage in VR

In 10 years, VR/AR will be indistinguishable from reality.

What will we do with these tools?

How will we create the new interfaces that enable the next

generation of phenomics

Thanks and Contacts

Justin Borevitz – Lab Leader Lab web page: http://borevitzlab.anu.edu.au

• Funding: • APPF / NCRIS• ARC Center of Excellence in Planet Energy Biology | ARC Linkage 2014• Arboretum ANU Major Equipment Grant: 2014, 2016

• TraitCapture: • Gareth Dunstone• Sarah Namin• Mohammad Esmaeilzadeh

• Chuong Nguyen; Joel Granados; Kevin Murray; Jiri Fajkus

• Jordan Braiuka

• Pip Wilson; Keng Rugrat; Borevitz Lab

• Arboretum• http://bit.ly/PESA2014• Cris Brack, Albert VanDijk, Justin Borevitz (PESA Project PI’s)• UAV data: Darrell Burkey, ProUAV• 3D site modelling: Pix4D.com / Zac Hatfield Dodds / ANUVR team• Dendrometers & site infrastructure” Darius Culvenor: Environmental Sensing Systems• Mesh sensors: EnviroStatus, Alberta, CA• Gigavision: Jack Adamson

• ANUVR Team• Yuhao Lui, Zhuoqi Qui, Abishek Kookana, Andrew Kock, Thomas Urwin [2016/17 Team]

• Zena Wolba; Alex Alex Jansons; Isobel Stobo; David Wai [2015/16 Team]

• Contact me:

• tim.brown@anu.edu.au

• http://bit.ly/Tim_ANU

Code: http://github.com/borevitzlab