Open Source Multipurpose Multimedia Annotation ToolOpen Source Multipurpose Multimedia Annotation Tool. Joed Lopes da Silva. 1, Alan Naoto Tabata. 1,2, Lucas Cardoso Broto. 1,2, Marta

Open Source Multipurpose Multimedia Annotation Tool

Joed Lopes da Silva1, Alan Naoto Tabata1,2, Lucas Cardoso Broto1,2,Marta Pereira Cocron1, Alessandro Zimmer1,2, and Thomas Brandmeier1

17th International Conference onImage Analysis and Recognition

ICIAR 202024-26 June, 2020 – VIRTUAL CONFERENCE

1 Research and Test Center CARISSMA, Technische Hochschule IngolstadtEsplanade 10, 85049 Ingolstadt, Germany

{Joed.LopesdaSilva, Marta.PereiraCocron}@carissma.eu{alt9707, luc2031, Alessandro.Zimmer, Thomas.Brandmeier}@thi.de

2 Federal University of Parana, XV de Novembro, Curitiba, Paraná 1299, Brazil

Technische Hochschule Ingolstadt | CARISSMA

- Important task

- Models trained with bad quality labeled dataset may not achieve good results

IntroductionData Annotation

2


ICIAR 2020

Bad Quality Labeling

Train

Bad Quality Detection Results

Target: Human Face


- Efficient Data Labeling requires Efficient Tools

- Current Open Source Labeling Tools:

- Supports only few data types: Image or Video or Audio [1,2,3]

- Few customization

- Few/no integrated data analysis

IntroductionData Labeling

3


ICIAR 2020

Raw Data

Labeling Tool

Draw Labels(Editor) Train

Common Data Annotation Workflow

ImportData

ExportData


- Current Demands: Multimedia and Multipurpose

IntroductionData Labeling

4


ICIAR 2020

LiDAR

Data Labeling Toolbox

RADAR

RGB Camera

Near InfraredCamera

IMU

Environment

Video

Video

1D-Signals

Point Cloud

N-DimensionalSignal

Text, Number

Multiple Data Types

Useful Editors Customization

Analysis

Data Storage

Reports

DataFilters

TrainModels

Import and Analyze Results

DatasetApplication 1

DatasetApplication N

DatasetApplication 2

.

.

.

Multiple Applications

ImportData Export

Data

Automotive Scenario


Proposal

5


ICIAR 2020


ProposalMain Objectives

6


ICIAR 2020

Multipurpose

Open Source Project

Multimedia

Data Analysis

The users can collaborate, improve and customize the application according to their requirements.

- Multiple annotation types:- 2D Label: Bounding Boxes, Ellipse, Polygons, Custom.- General Data: Text, Number, Boolean, Arrays, States, Checklist.

- Attributes customization according the project.- Grouped Data.

Support multiple types of data: - Image, Video, Audio, Point Clouds, Signals.

- Statistics: Basic, Descriptive Analysis, Hypothesis Tests, ANOVA.- ML Metrics: Accuracy, Precision, Recall, F1-Score, mAP, etc.- Data Filtering: create subsets based on labeled data.

Export - YOLO Format, Data Frame, CSV, Excel, JSON,.


ProposalSoftware Technologies

7

PyQtGraph


ICIAR 2020

Py


ProposalRecommend Workflow

8


ICIAR 2020


ProposalData Storage Model

9


ICIAR 2020

- Generic Properties- Document Based is Suitable- Functions: Map and Reduce- Remote Labeling


Use Cases

10


ICIAR 2020


- Objective: Driver Distraction Monitoring [4]

- Driver Info: Name, Age, Country, Automobile Licenses

- Computer Vision: Head Pitch, Roll, and Yaw angles

- 2D Label: Custom Deformable Face Model, Eyes, Head Region

- Event Annotation: Texting, Radio, Cellphone use

Use Case 1Driver Distraction Database

11


ICIAR 2020Inertial Sensor

Camera and IlluminationSimulator

Vehicle Data- Velocity- Steering Angle- Position- Break Pressure- etc.

Head Angle


- Objective: Eye Study based on Infrared

Images

- Subject Info:

- Name, Age, Gender, Weight

- Additional Info

- Eye Info:

- State: Blink (Yes, No)

- Yaw and Pitch Angle

- 2D Label:

- Iris (Ellipse)

- Pupil (Ellipse)

Use Case 2Eye Movement Study - Pupillometry

12


ICIAR 2020


- Objective: Label objects in traffic scenario

- Recording Info:

- Day, Night, Sunny, Snow, Rainy

- Environment:

- Highway, Residential Road

- Drawables:

- Bouding Boxes: Car, Truck, Traffic

Signs, Pedestrian, Bycicle,

Advertisement

Use Case 3Multi-Object Traffic Dataset

13


ICIAR 2020


- Objective: Label Image and Point Cloud

- 3D Object Detection and Segmentation

Use Case 4Image + Point Cloud Annotation

14


ICIAR 2020


Project Roadmap

15


ICIAR 2020


- New Features for Implementation:

RoadmapProject Roadmap

16

3D Workflows

NLPWeb Apps

Automatic Labeling


ICIAR 2020

Current Effort

Q4 / 2020

Q2 / 2021Q1 / 2021


E-Mail: [email protected]

www.thi.de/go/thi-labeling-tool

AccessDownload

17


ICIAR 2020

mailto:[email protected]

http://www.thi.de/go/thi-labeling-tool


Thank you!

18


ICIAR 2020


Questions?

19


ICIAR 2020


[1] Ambardekar, A., Nicolescu, M., Dascalu, S.: Ground truth verification tool (GTVT) for video surveillance systems. In: 2009

Second International Conferences on Advances in Computer-Human Interactions, pp. 354–359. IEEE (2009)

[2] Dutta, A., Zisserman, A.: The VIA annotation software for images, audio and video. In: Proceedings of the 27th ACM International

Conference on Multimedia, MM 2019. ACM, New York (2019). https://doi.org/10.1145/3343031.3350535

[3] Jaynes, C., Webb, S., Steele, R., Xiong, Q.: An open development environment for evaluation of video surveillance systems. In:

PETS02, pp. 32–39 (2002)

[4] da Silva, J.L., Thomas Brandmeier, A.Z.: Automatic measurement of automobile drivers attention level via computer

vision. In: XXIV Congresso Brasileiro De Engenharia Biom´edica (2014)

References

20


ICIAR 2020

Documents

Open Source Multipurpose Multimedia Annotation ToolOpen Source Multipurpose Multimedia Annotation Tool. Joed Lopes da Silva. 1, Alan Naoto Tabata. 1,2, Lucas Cardoso Broto. 1,2, Marta