1

www.cmore-automotive.com

A Systematic Approach on Automated Annotation

Dr. Matthias Zobel, 2017-07-06

22017-07-06

Types of Annotations

2-D-Bounding-Box

(Objects, Parts)

32017-07-06

Types of Annotations

2-D-Pixelwise

(SemanticSegmentation)

42017-07-06

Types of Annotations

3-D-Bounding-Box

(Objects)

52017-07-06

Annotation – Information Generation

• Analytics

• What is in the data?

• Correlations, Similarities, Clustering, Conclusions, Predictions

• Annotation = Data Enrichment

• Add NEW meaning to the data

• Makes the data valuable!

62017-07-06

Annotation – Why?

• ADAS / AD algorithms simulate human behavior and intelligence

• They mimic

• human cognitive abilities - Senses

• human reasoning abilities - Brain

• human interaction abilities - Body

• Examples

• Object Recognition: Detect cars and pedestrians in a video sequence

• Situation Assessment: Is it safe to change lanes now?

• Behavior Simulation: What steering angle and acceleration to apply?

72017-07-06

Annotation – Why?

• Algorithms need to be taught (programmed) to do so

• Manually by experts

• Automatically by teaching algorithms MACHINE LEARNING (ML)

82017-07-06

Machine Learning - Training

AlgorithmQualifiedexamples

Conclusion

Compare

ADAPT

Input

Label Output

Training Set

92017-07-06

Machine Learning - Testing

AlgorithmQualifiedexamples

Conclusion

Compare

REPORT

Input

Label Output

Testing Set

102017-07-06

Annotation – Why?

• Algorithms need to be taught (programmed) to do so

• Manually by experts

• Automatically by teaching algorithms MACHINE LEARNING (ML)

• Machine Learning = Learning from qualified examples (“labeled data”)

• Statistical approaches A lot of qualified data is necessary

{( , “car”), ...}

112017-07-06

Annotation – Challenges

• Huge amount – Terabytes to Petabytes

• Development and validation of ADAS/AD need to cover all possible varieties of scenarios – „A lot helps a lot!“

• Availability: „Yesterday!“

• Manual labeling is very time consuming

• Depending on task and accuracy demands

• 2-D-BB: approx. 1 min per frame 1 hour @ 60 fps 3600 h ~ 2 MY

• 3-D-BB and 2-D-Semantic even worse

• Manual work is expensive, even if out-sourced

122017-07-06

Annotation – Automation

• Goals

• Faster, Better, Cheaper

• Unfortunately

• THE general solution is not available yet

• No structured approach was available to describe the way from pure manual to fully automated labeling

• CMORE Automated Labeling (CAL) - Levels

AlgorithmsSensorData

Labels

Vision: Fully Automated Labeling

132017-07-06

CAL – CMORE Automated Labeling Levels

CAL 0 CAL 1 CAL 2 CAL 3 CAL 4

Labeler performs all

labeling activities.

Tool proposeslabels without data

knowledge.

Labeler performs most of the labeling activities with

reduced efforts.

Tool proposeslabels with data

knowledge.

Labeler performs some

labeling activities only.

Confirmation of tool results.

Tool performslabeling tasks.

Confirmation of tool results at

random.

Tool performs labeling tasks unsupervised with sufficient

quality.

Manual AssistedPartly

AutomatedHighly

AutomatedFully

Automated

Au

tom

atio

n

M

anu

al

142017-07-06

CAL – CMORE Automated Labeling Levels

CAL 0 CAL 1 CAL 2 CAL 3 CAL 4

Labeler performs all

labeling activities.

Tool proposeslabels without data

knowledge.

Labeler performs most of the labeling activities with

reduced efforts.

Tool proposeslabels with data

knowledge.

Labeler performs some

labeling activities only.

Confirmation of tool results.

Tool performslabeling tasks.

Confirmation of tool results at

random.

Tool performs labeling tasks unsupervised with sufficient

quality.

Manual AssistedPartly

AutomatedHighly

AutomatedFully

Automated

Au

tom

atio

n

M

anu

al

152017-07-06

CAL – CMORE Automated Labeling Levels

CAL 0 CAL 1 CAL 2 CAL 3 CAL 4

Labeler performs all

labeling activities.

Tool proposeslabels without data

knowledge.

Labeler performs most of the labeling activities with

reduced efforts.

Tool proposeslabels with data

knowledge.

Labeler performs some

labeling activities only.

Confirmation of tool results.

Tool performslabeling tasks.

Confirmation of tool results at

random.

Tool performs labeling tasks unsupervised with sufficient

quality.

Manual AssistedPartly

AutomatedHighly

AutomatedFully

Automated

Au

tom

atio

n

M

anu

al

162017-07-06

CAL – CMORE Automated Labeling Levels

CAL 0 CAL 1 CAL 2 CAL 3 CAL 4

Labeler performs all

labeling activities.

Tool proposeslabels without data

knowledge.

Labeler performs most of the labeling activities with

reduced efforts.

Tool proposeslabels with data

knowledge.

Labeler performs some

labeling activities only.

Confirmation of tool results.

Tool performslabeling tasks.

Confirmation of tool results at

random.

Tool performs labeling tasks unsupervised with sufficient

quality.

Manual AssistedPartly

AutomatedHighly

AutomatedFully

Automated

Au

tom

atio

n

M

anu

al

172017-07-06

CAL – CMORE Automated Labeling Levels

CAL 0 CAL 1 CAL 2 CAL 3 CAL 4

Labeler performs all

labeling activities.

Tool proposeslabels without data

knowledge.

Labeler performs most of the labeling activities with

reduced efforts.

Tool proposeslabels with data

knowledge.

Labeler performs some

labeling activities only.

Confirmation of tool results.

Tool performslabeling tasks.

Confirmation of tool results at

random.

Tool performs labeling tasks unsupervised with sufficient

quality.

Manual AssistedPartly

AutomatedHighly

AutomatedFully

Automated

Au

tom

atio

n

M

anu

al

182017-07-06

CAL – CMORE Automated Labeling Levels

CAL 0 CAL 1 CAL 2 CAL 3 CAL 4

Labeler performs all

labeling activities.

Tool proposeslabels without data

knowledge.

Labeler performs most of the labeling activities with

reduced efforts.

Tool proposeslabels with data

knowledge.

Labeler performs some

labeling activities only.

Confirmation of tool results.

Tool performslabeling tasks.

Confirmation of tool results at

random.

Tool performs labeling tasks unsupervised with sufficient

quality.

Manual AssistedPartly

AutomatedHighly

AutomatedFully

Automated

Au

tom

atio

n

M

anu

al

192017-07-06

CAL – CMORE Automated Labeling Levels

• Speak a common language

• Unified view on “Automation”

• Capabilities can be categorized

• Solutions can be compared

• Define goals and roadmaps

• Further development can be planned

202017-07-06

C.LABEL – The CMORE Annotation Solution

• Target for CAL3 with new platform approach

212017-07-06

Automated Annotation - Examples

• Based on our own Deep Learning scheme, fine-tuned for automotive scenarios

• Trained with automatically annotated training set

• Continuous improvement of algorithms with growing amounts of data

• Corrections required are mostly size-related, labeling difficult instances or removing false-positives

222017-07-06

Automated Annotation - Examples

0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0,8

0,9

1

1 2 3 4 5

Correct vs. Automated Labels

Datenreihen1 Datenreihen2

Ave

rage

Pre

cisi

on

232017-07-06

Automated Annotation - Examples

• Used Convolutional Neural Networks (Deep Learning) for proposal of semantic labels

Object

class

Street Sky Building Vegetation Sidewalk Car

Accuracy 96% 91% 88% 78% 92% 64%

CamVid Dataset

242017-07-06

Automated Annotation - Examples

• Manually placing of bounding boxes in 3-D is difficult and requires many viewpoint changes

• Automation can speed up this tedious task

• Two Step Process

• Selection of object hypotheses

• Classification into object types

252017-07-06

Conclusion

• Qualified data is the key to ML based ADAS / AD

• Request for qualified data is tremendous and increasing

• Automation of annotation is necessary to deal with huge amounts of data

• CAL-Levels provide a structured guide for the road aheadFeel free to refer to it