AI+EMFarron Wallace NOAA Fisheries SEFSC

SEFSC Innovation ProjectIn Collaboration

AFSC, University of Washington, NGO’s and the Fishing Industry

AI?

Proof of Concept

• 4 types of rockfish

Blackspotted Rougheye

Shortraker Thornyhead

Test on Rockfish Dataset

• Training (used previous ~15,000 training dataset)• Blackspotted (59), Rougheye (15), Shortraker (21), Thornyhead (9).

• Testing• Blackspotted (20), Rougheye (5), Shortraker (7), Thornyhead (4).

• Confusion Matrix (Accuracy=91.7%)

Blackspotted Rougheye Shortraker Thornyhead

Blackspotted 18 1 0 0

Rougheye 0 4 0 0

Shortraker 2 0 7 0

Thornyhead 0 0 0 4

Machine Vision Systems (IMS)ØRemote monitoring systems that support the

latest developments in machine learning (AI) to improve system functionality, reliability and timeliness of data.

ØAn automated system that can see, learn, process and transmit data

Cost Categories for Developing Machine Vision?

1. Collect Imagery (EM, Chute, Belt Systems, cameras)

2. Annotate imagery to create a rich training dataset.

3. Feed these data in ML algorithm(s).

4. Hardware development and implementation.

Iterative Development CycleInnovateSystem

Hardware

Deploy into Fishery or

Survey

CollectTraining Datasets

Feed intoMachine Learning

ProgressAutomation

AgileProductManagement

Methods based on iterative and incremental development

and testing

ValidationDatasets

QA/QC

Proof ofConcept

Rich and complexdataset

Guidelines for Developing Automation1. The scale of the data needs to reasonably capture the complexity of the problem.

2. Big data must be accurate.

3. At minimum you need an ML algorithm that captures the discriminating information.

Guidelines for Developing Automation4. The more complex the algorithm the greater the processing power/time required.

5. There is a delicate balance in dedicating resources between Algorithm development and collecting more data.

6. Use multiple sources of information (human versus machine) to validate your system

UW Annotation Software

How many annotations do I need for training?

ML

Algo

rithm

Perf

orm

ance

Number of Annotations (1,000)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0 50 100 150 200 250 300 350 400 450 500

Learning Curve

COMPLEXITY

Large Annotated Datasets Required: Longline Fisheries

Ø35,300,000 Ø50,000 Ø706 Ø14

How do you annotate all those images? Semi-automated annotation tools Strategic sub-sampling of dataQuery Learning/Unsupervised Learning (DL)

# frames collected from 184 hauls# frames processed per week# of weeks to process# of years to process

•

Metric Learning with Multiple Representative Features

• To deal with intra-class variance• For class !, define " representative feature vectors #$,& for intra-class variance:'( )* = max min& 1 )* − #$,& − min

3,&41 )* − #3,&4 + 6, 0

#$,&

#3,&4

)*

)*89

)*:9

class ! class ;

Identifying Catch Events

Red: shipGreen: sea

3D bounding box for size estimation

15

What’s Next?• Distribute the knowledge and design of systems that

collect and analyze imagery.

• Provide access to open source Machine Learning algorithms to promote development of automation in other fisheries (VIAMI, GitHub)

• Provide Access to machine vision Stereo Camera systems to promote automation and development of 3D vision systems to estimate size and volume measurements.

Questions?

17

The NOAA AI Strategy will dramatically expand the application of artificial intelligence (AI) in every NOAA mission area by improving the efficiency, effectiveness and coordination of AI development and usage across the agency.

AI methods will provide transformative advancements in the quality and timeliness of NOAA science, products, and services: are already demonstrating significant improvements in performance and skill at vastly reduced costs and compute time.