Robert Bosch Centre for Data Science and Artificial IntelligenceDepartment of Computer Science and Engineering

Indian Institute of Technology Madras

Object Detection Over Scientific Plots

1

Dr. Mitesh Khapra Dr. Pratyush KumarNitesh Methani(Research Scholar, IIT Madras) (Assistant Professor, IIT Madras) (Assistant Professor, IIT Madras)

Pritha Ganguly(Research Scholar, IIT Madras)

Introduction

Image Source: Google Images2

Problem Statement

Fast and accurate detection of objects in scientific plots

Image Source: Nitesh Methani, Pritha Ganguly, Mitesh Khapra, Pratyush Kumar, PlotQA: Reasoning over Scientific Plots, WACV 2020.3

Years Bulgaria Cuba

2002 677 593

2003 641 538

2004 604 485

2005 575 440

Introduction

Image Source: Nitesh Methani, Pritha Ganguly, Mitesh Khapra, Pratyush Kumar, PlotQA: Reasoning over Scientific Plots, WACV 2020.4

Q: What is the difference between the number of neonatal deaths in Bulgaria and Cuba in the year 2004?A: 119

Q: What is the average number of neonatal deaths in Cuba across years?A: 514

Q: In which year is the number of neonatal deaths in Bulgaria maximum?A: 2002

Years Bulgaria Cuba

2002 677 593

2003 641 538

2004 604 485

2005 575 440

Introduction

Image Source: Nitesh Methani, Pritha Ganguly, Mitesh Khapra, Pratyush Kumar, PlotQA: Reasoning over Scientific Plots, WACV 2020.5

Q: What is the difference between the number of neonatal deaths in Bulgaria and Cuba in the year 2004?A: 119

Q: What is the average number of neonatal deaths in Cuba across years?A: 514

Q: In which year is the number of neonatal deaths in Cuba maximum?A: 2002

Are existing object detection models good enough?

6

Natural Images v/s Scientific Plots

7Image Source: M. Everingham, L. V. Gool, C. K. I. Williams, J. M. Winn, A. Zisserman, The Pascal VOC Challenge. Int. J. Comput. Vis., 2010Image Source: Nitesh Methani, Pritha Ganguly, Mitesh Khapra, Pratyush Kumar, PlotQA: Reasoning over Scientific Plots, WACV 2020.

Visual elements Visual elements Textual elements+

Natural Images v/s Scientific Plots

8

Small to Large boxes X-Small to X-Large boxes

✔ ✔

Image Source: Google Images

Natural Images v/s Scientific Plots

9

Structural Relationship Structural Relationship

❌

✔ ✔ ✔ ✔

✔

Image Source: Google Images

Natural Images v/s Scientific Plots

10

0.5 IOU✔

✔ ✔

✔

✔ ✔

❌ 0.5 IOU❌

Natural Images v/s Scientific Plots

11

0.5 IOU✔

✔ ✔

✔

✔ ✔

❌ 0.75 IOU❌

Natural Images v/s Scientific Plots

12

0.5 IOU✔

✔ ✔

✔

✔ ✔

0.90 IOU❌ ✔

Natural Images v/s Scientific Plots

13

✔

✔ ✔

✔

✔ ✔

Key Insight: OD over scientific plots has additional challenges as compared to OD over natural images

❌ 0.5 IOU 0.90 IOU✔

Goal 1Investigate whether existing object detection methods are adequate for detecting text and visual elements in scientific plots which are arguably different than the objects found in natural images?

?

14

Twostage

Onestage

Summary of Two Stage Detectors

15

⋯

w

h

w*

h*CNN

Feature Extractor

Warped Image regions

Regression output

Feature Volume

Classification output

InputImage

Flattened vector

⋰

WH

D

⋰

W H D

1R. B. Girshick, J. Donahue, T. Darrell, J. Malik, Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. CVPR14

Summary of Two Stage Detectors

16

⋰CNN

Feature Extractor

Warped Image regions

Networkoutputs

Feature Volume

InputImage

Flattened vector

⋯

⋰CNN

⋰

⋯

CNN

WH

D

⋰

⋯

W H D

ROI Poolfeatures

Summary of SOTA Models: Fast R-CNN2

17

CNN ⋰

⋰

⋰

h

Feature Extractor

Networkoutputs

Feature Volume

InputImage

Fully Connected Layers

x* y* w* h*

⋯

ROI Pool

2Ross B. Girshick, Fast R-CNN. ICCV 2015

18

665

665

665/32 = 20.78800/32 = 25800

800

25

25

20

20 22

20/7 = 2.86

7

7

Coordinate on input image

Coordinate on feature map

CNNVGG16

{

{

Quantization Coordinate on ROI feature

Quantization

Fast R-CNN: ROI Pool

Image Source: https://ardianumam.wordpress.com/2017/12/16/understanding-how-mask-rcnn-works-for-semactic-segmentation/

Summary of SOTA Models: Faster R-CNN3

19

CNN h

Feature Extractor

Feature Volume

InputImage

3S. Ren, K. He, Ross. B. Girshick, J. Sun, Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, NIPS 2015

Summary of SOTA Models: Faster R-CNN

20

CNN

Region Proposal Network

h

Feature Extractor

Feature Volume

InputImage

⋰

⋰

⋰

ROI Poolfeatures

Network outputs

Fully Connected Layers

x* y* w* h*

⋯

Summary of SOTA Models: Mask R-CNN4

21

CNN

Mask Branch

x* y* w* h*

h ⋰

Feature Extractor

ROI Align Network outputs

Feature Volume

InputImage

Fully Connected

⋯

Region Proposal Network

4Kaiming He, Georgia Gkioxari, Piotr Dollár, Ross B. Girshick, Mask R-CNN, ICCV 2017

+ Mask Branch

Summary of SOTA Models: Mask R-CNN

22

CNN h ⋰

Feature Extractor

ROI Align Network outputs

Feature Volume

InputImage

Fully Connected + Mask Branch

Mask Branch

x* y* w* h*

⋯

Region Proposal Network

23

665

665

665/32 = 20.78800/32 = 25800

800

25

25

20.78

20.7

8

2.862.86

20/7 = 2.86

7

7

Coordinate on input image

Coordinate on feature map

CNNVGG16

{

{

No Quantization Coordinate on ROI feature

No Quantization

Mask R-CNN: ROI Align

Image Source: https://ardianumam.wordpress.com/2017/12/16/understanding-how-mask-rcnn-works-for-semactic-segmentation/

24

ROI Pool vs ROI Align

RO

IPoo

lR

OIA

lign

25

Summary of Two Stage Detectors

Model Key Ideas

FRCNN

FrRCNN

MRCNN

ROI Pool

Region Proposal Network

ROI Align

26

c x y w h . . .

P(person)

P(horse)

P(cycle)

⋯

Summary of One Stage Detectors

Input Image Output vector for each grid cell

1 x (1 + 4 + K)

1J. Redmon, S. K. Divvala, R. B. Girshick, A. Farhadi, You Only Look Once: Unified, Real-Time Object Detection. CVPR 2016

c x y w h . . .

27

P(person)

P(horse)

P(cycle)

⋯

Summary of One Stage Detectors

Input Image Output vector for each grid cell

1 x (1 + 4 + K)

c x y w h . . .

28

P(person)

P(horse)

P(cycle)

⋯

Summary of One Stage Detectors

Input Image Output vector for each grid cell

1 x (1 + 4 + K)

c x y w h . . .

29

P(person)

P(horse)

P(cycle)

⋯

Summary of One Stage Detectors

Input Image Output vector for each grid cell

1 x (1 + 4 + K)

c x y w h . . .

30

P(person)

P(horse)

P(cycle)

⋯

Summary of One Stage Detectors

Input Image Output vector for each grid cell

1 x (1 + 4 + K)

c x y w h . . .

31

P(person)

P(horse)

P(cycle)

⋯

Summary of One Stage Detectors

Input Image Output vector for each grid cell

1 x (1 + 4 + K)

32

Summary of SOTA Models: YOLO

7

Input Image

1 + 4 + K7

DarkNetArchitecture

Output volume Feature Extractor

33

Summary of SOTA Models: YOLO-v3

H

Input Image

B x (1 + 4 + K) W

DarkNetArchitecture

Output volume Feature Extractor

⋮ ‘B’ bounding box priors

300x300x3

34

Summary of SOTA Models: YOLO

Input Image

DarkNet

Feature Extractor

Out

put

300x300x3

35

Summary of SOTA Models: SSD2

Input Image Feature Extractor

38x38x512

VGG16

Out

put

conv5 last

2W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. E. Reed, C.Y Fu, A. C. Berg, SSD: Single Shot MultiBox Detector. ECCV 2016

300x300x3

36

Summary of SOTA Models: SSD

Input Image Feature Extractor

38x38x512

VGG16

Out

put

conv4 last

300x300x3 38x38x512 19x19x1024 19x19x1024

37

Out

put

1x1 conv

1x1 conv

Summary of SOTA Models: SSD

Input Image

conv4 last

conv6 (FC6)

conv7 (FC7)

VGG Layers

300x300x3 38x38x512 19x19x1024 19x19x1024 10x10x512 5x5x256 3x3x256 1x1x256

38

1x1 conv

1x1 conv

conv4 last

conv6 (FC6)

conv7 (FC7)

conv8 conv9 conv10 conv11

Summary of SOTA Models: SSD

Input Image VGG Layers Extra Layers

Out

put

c x y w h

39

Summary of SOTA Models: RetinaNet3

3T. Lin, P. Goyal, R. B. Girshick, K. He, and P. Dollár, Focal loss for dense object detection, ICCV 2017

Input Image

1 + 4 + K7

DarkNet

Output volume Feature Extractor

7

k

40

Summary of SOTA Models: RetinaNet

Output volume Feature Pyramid Network

W ✕ H ✕ 4A

W ✕ H ✕ kA

Input Image

⋮ ‘A’ anchor boxes

41

Summary of SOTA Models: RetinaNet

Output volume

W ✕ H ✕ 4A

W ✕ H ✕ kA

Input Image

⋮ ‘A’ anchor boxes

Feature Pyramid Network

42

Summary of SOTA Models: RetinaNet

Output volume

W ✕ H ✕ 4A

W ✕ H ✕ kA

Input Image

⋮ ‘A’ anchor boxes

Feature Pyramid Network

43

Image Conv1 Conv2 Conv3 Conv4 Conv5 FC Softmax

RetinaNet: Feature Pyramid Network4

4T.Y. Lin, P. Dollár, R. B. Girshick, K. He, B. Hariharan, S. J. Belongie, Feature Pyramid Networks for Object Detection. CVPR 2017

44

RetinaNet: Feature Pyramid Network

Image Conv1(last)

Conv2(last)

Conv3(last)

Conv4(last)

Conv5(last)

45

Image C1 C2 C3 C4 C5

RetinaNet: Feature Pyramid Network

46

RetinaNet: Feature Pyramid Network

C1

C2

C3

C4C5

Bottom-Up Pathway

47

1x1

RetinaNet: Feature Pyramid Network

C1

C2

C3

C4C5 M5

Bottom-Up Pathway Top-Down Pathway

48

1x1

2x

RetinaNet: Feature Pyramid Network

C1

C2

C3

C4C5

M4

M5

Bottom-Up Pathway Top-Down Pathway

49

+

1x1

1x1 2x

RetinaNet: Feature Pyramid Network

C1

C2

C3

C4C5

M4

M5

Bottom-Up Pathway Top-Down Pathway

50

+

+

+

1x1

1x1

1x1

1x1

2x

2x

2x

RetinaNet: Feature Pyramid Network

C1

C2

C3

C4C5

M2

M3

M4

M5

Bottom-Up Pathway Top-Down Pathway

51

+

+

+

1x1

1x1

1x1

1x1

3x3

2x

2x

2x

RetinaNet: Feature Pyramid Network

C1

C2

C3

C4C5

M2

P5

M3

M4

M5

Bottom-Up Pathway Top-Down Pathway Pyramid Feature Maps

52

+

+

+

1x1

1x1

1x1

1x1

3x3

3x3

3x3

3x32x

2x

2x

RetinaNet: Feature Pyramid Network

C1

C2

C3

C4C5

M2 P2

P3

P4

P5

M3

M4

M5

Bottom-Up Pathway Top-Down Pathway Pyramid Feature Maps

53

YOLO

SSD

RetinaNet

Summary of One Stage DetectorsModel Key Ideas

Grid based proposal

Featurised Pyramid

Feature Pyramid Network

54

Recap of Goal 1Investigate whether existing object detection methods are adequate for detecting text and visual elements in scientific plots which are arguably different than the objects found in natural images?

?

Twostage

Onestage

Dataset: PlotQA

Image Source: Nitesh Methani, Pritha Ganguly, Mitesh Khapra, Pratyush Kumar, PlotQA: Reasoning over Scientific Plots, WACV 2020.

● Contains over 220,000 scientific plots across three categories:○ Bar (both horizontal and vertical) plots○ Line, and ○ Dot-line plots

55

Evaluation Metric

56

IOU = = 0.54

Image Source: Nitesh Methani, Pritha Ganguly, Mitesh Khapra, Pratyush Kumar, PlotQA: Reasoning over Scientific Plots, WACV 2020.

IOU = = 0.52

Evaluation Metric

57

IOU = = 0.54 IOU = = 0.79 IOU = = 0.98

Image Source: Nitesh Methani, Pritha Ganguly, Mitesh Khapra, Pratyush Kumar, PlotQA: Reasoning over Scientific Plots, WACV 2020.

Evaluation Metric

58

0.54 0.79 0.98

Image Source: Nitesh Methani, Pritha Ganguly, Mitesh Khapra, Pratyush Kumar, PlotQA: Reasoning over Scientific Plots, WACV 2020.

Mark a prediction as correct if it has a 50% IOU with the ground-truth box

Evaluation Metric

59Image Source: Nitesh Methani, Pritha Ganguly, Mitesh Khapra, Pratyush Kumar, PlotQA: Reasoning over Scientific Plots, WACV 2020.

Mark a prediction as correct if it has a 50% IOU with the ground-truth box

0.54 0.79 0.98

Evaluation Metric

60

0.54 0.79 0.98

Image Source: Nitesh Methani, Pritha Ganguly, Mitesh Khapra, Pratyush Kumar, PlotQA: Reasoning over Scientific Plots, WACV 2020.

Mark a prediction as correct if it has a 75% IOU with the ground-truth box

Evaluation Metric

61Image Source: Nitesh Methani, Pritha Ganguly, Mitesh Khapra, Pratyush Kumar, PlotQA: Reasoning over Scientific Plots, WACV 2020.

Mark a prediction as correct if it has a 75% IOU with the ground-truth box

0.54 0.79 0.98

Evaluation Metric

62Image Source: Nitesh Methani, Pritha Ganguly, Mitesh Khapra, Pratyush Kumar, PlotQA: Reasoning over Scientific Plots, WACV 2020.

Mark a prediction as correct if it has a 90% IOU with the ground-truth box

0.54 0.79 0.98

Evaluation of SOTA Models

Models mAP@0.5

SSD 82.33

YOLO-v3 96.27

RetinaNet 90.13

FRCNN 72.83

FrRCNN 88.49

MRCNN 93.72

IOU = 0.53Table: Comparison of existing object detection models on the PlotQA dataset with mAP scores (in %) at IOU of 0.5.

63

Evaluation of SOTA Models

Models mAP@0.5 mAP@0.75

SSD 82.33 39.78

YOLO-v3 96.27 73.31

RetinaNet 90.13 81.13

FRCNN 72.83 63.68

FrRCNN 88.49 50.51

MRCNN 93.72 82.45

IOU = 0.59 IOU = 0.75Table: Comparison of existing object detection models on the PlotQA dataset with mAP scores (in %) at IOUs of 0.5, and 0.75.

64

Evaluation of SOTA Models

Models mAP@0.5 mAP@0.75 mAP@0.9

SSD 82.33 39.78 1.53

YOLO-v3 96.27 73.31 7.43

RetinaNet 90.13 81.13 30.56

FRCNN 72.83 63.68 21.45

FrRCNN 88.49 50.51 4.08

MRCNN 93.72 82.45 35.70

IOU = 0.59 IOU = 0.75 IOU = 0.96Table: Comparison of existing object detection models on the PlotQA dataset with mAP scores (in %) at IOUs of 0.5, 0.75, and 0.9.

65

Evaluation of SOTA Models

IOU 0.9 0.75 0.5

Models\Class

bar dotline

legendlabel

legendpreview

plottitle

x-axislabels

x-axisticks

y-axislabels

y-axisticks

mAP mAP mAP

SSD 1.39 0.60 2.18 0.39 0.04 3.39 0.44 5.14 0.20 1.53 39.78 82.33

YOLO-v3 15.51 8.72 7.15 11.70 0.02 4.39 8.08 9.59 1.70 7.43 73.31 96.27

RetinaNet 16.51 18.50 77.26 29.74 16.58 67.62 28.40 3.14 17.31 30.56 81.13 90.13

FRCNN 53.38 1.68 12.59 14.06 0.03 42.13 25.49 11.68 31.98 21.45 63.68 72.83

FrRCNN 6.92 1.68 1.39 1.45 0.00 4.35 6.10 3.57 5.18 4.08 50.51 88.49

MRCNN 47.54 5.36 50.83 32.43 0.33 40.20 33.72 80.53 30.31 35.70 82.45 93.72

Table: Comparison of existing object detection models on the PlotQA dataset with mAP scores (in %) at IOUs of 0.5, 0.75, and 0.9.

66

Evaluation of SOTA Models

IOU 0.9 0.75 0.5

Models\Class

bar dotline

legendlabel

legendpreview

plottitle

x-axislabels

x-axisticks

y-axislabels

y-axisticks

mAP mAP mAP

SSD 1.39 0.60 2.18 0.39 0.04 3.39 0.44 5.14 0.20 1.53 39.78 82.33

YOLO-v3 15.51 8.72 7.15 11.70 0.02 4.39 8.08 9.59 1.70 7.43 73.31 96.27

RetinaNet 16.51 18.50 77.26 29.74 16.58 67.62 28.40 3.14 17.31 30.56 81.13 90.13

FRCNN 53.38 1.68 12.59 14.06 0.03 42.13 25.49 11.68 31.98 21.45 63.68 72.83

FrRCNN 6.92 1.68 1.39 1.45 0.00 4.35 6.10 3.57 5.18 4.08 50.51 88.49

MRCNN 47.54 5.36 50.83 32.43 0.33 40.20 33.72 80.53 30.31 35.70 82.45 93.72

Table: Comparison of existing object detection models on the PlotQA dataset with mAP scores (in %) at IOUs of 0.5, 0.75, and 0.9.

67

Evaluation of SOTA Models

IOU 0.9 0.75 0.5

Models\Class

bar dotline

legendlabel

legendpreview

plottitle

x-axislabels

x-axisticks

y-axislabels

y-axisticks

mAP mAP mAP

SSD 1.39 0.60 2.18 0.39 0.04 3.39 0.44 5.14 0.20 1.53 39.78 82.33

YOLO-v3 15.51 8.72 7.15 11.70 0.02 4.39 8.08 9.59 1.70 7.43 73.31 96.27

RetinaNet 16.51 18.50 77.26 29.74 16.58 67.62 28.40 3.14 17.31 30.56 81.13 90.13

FRCNN 53.38 1.68 12.59 14.06 0.03 42.13 25.49 11.68 31.98 21.45 63.68 72.83

FrRCNN 6.92 1.68 1.39 1.45 0.00 4.35 6.10 3.57 5.18 4.08 50.51 88.49

MRCNN 47.54 5.36 50.83 32.43 0.33 40.20 33.72 80.53 30.31 35.70 82.45 93.72

Table: Comparison of existing object detection models on the PlotQA dataset with mAP scores (in %) at IOUs of 0.5, 0.75, and 0.9.

68

Qualitative Analysis: SSD

Figure: An example plot from PlotQA dataset.

69

Figure: Detected bounding boxes on an example plot from PlotQA dataset.

Qualitative Analysis: YOLO-v3

Figure: An example plot from PlotQA dataset.

70

Figure: Detected bounding boxes on an example plot from PlotQA dataset.

Qualitative Analysis: RetinaNet

Figure: An example plot from PlotQA dataset.

71

Figure: Detected bounding boxes on an example plot from PlotQA dataset.

Qualitative Analysis: FRCNN

Figure: An example plot from PlotQA dataset.

72

Figure: Detected bounding boxes on an example plot from PlotQA dataset.

Qualitative Analysis: FrRCNN

Figure: An example plot from PlotQA dataset.

73

Figure: Detected bounding boxes on an example plot from PlotQA dataset.

Qualitative Analysis: MRCNN

Figure: An example plot from PlotQA dataset.

74

Figure: Detected bounding boxes on an example plot from PlotQA dataset.

Qualitative Analysis: Summary

7575

SSD YOLO-v3

Retinanet FRCNN

FrRCNN MRCNN

Longer textual objects

Very short objects

Higher IOU settings

❌

❌

❌

Key Observations:

Qualitative Analysis: Summary

7676

Retinanet FRCNN

FrRCNN MRCNN

SSD YOLO-v3Longer textual objects

Very short objects

Higher IOU settings

FPN helps

❌

❌

✔

Key Observations:

❌

Longer textual objects

Very short objects

Higher IOU settings

FPN helps

ROIAlign helps

Qualitative Analysis: Summary

7777

❌

❌

✔

✔

Key Observations:

FrRCNN MRCNN

SSD YOLO-v3

Retinanet FRCNN

❌

Design a deep learning based object detection network that accurately and efficiently detects all the textual and visual objects present in a scientific plot.

Goal 2

Accurate & Efficient

78Image Source: Google Images

Faster R-CNN backbone Feature Pyramid Network (FPN) ROIAlign (RA)

+ +

79

A Hybrid Model

A Hybrid Model: Results

IOU 0.9 0.75 0.5

Class\Models

bar dotline

legendlabel

legendpreview

plottitle

x-axislabels

x-axisticks

y-axislabels

y-axisticks

mAP mAP mAP

FRCNN(RA) 83.25 16.32 62.31 59.10 0.18 65.77 78.72 42.94 78.87 54.16 68.92 72.46

FRCNN(FPN+RA) 87.59 31.62 79.05 66.39 0.22 69.78 88.29 46.63 84.60 61.57 69.82 72.18

FrRCNN(RA) 63.89 14.79 70.95 60.61 0.18 83.89 60.76 93.47 50.87 55.49 89.14 96.80

Hybrid Model

(FrRCNN+FPN+RA)

85.54 27.86 93.68 96.30 0.22 99.09 96.04 99.46 96.80 77.22 94.58 97.76

80

Table 2: Comparison of modified models on the PlotQA dataset with mAP scores (in %) at IOUs of 0.9, 0.75 & 0.5.

A Hybrid Model: Results

IOU 0.9 0.75 0.5

Class\Models

bar dotline

legendlabel

legendpreview

plottitle

x-axislabels

x-axisticks

y-axislabels

y-axisticks

mAP mAP mAP

FRCNN(RA) 83.25 16.32 62.31 59.10 0.18 65.77 78.72 42.94 78.87 54.16 68.92 72.46

FRCNN(FPN+RA) 87.59 31.62 79.05 66.39 0.22 69.78 88.29 46.63 84.60 61.57 69.82 72.18

FrRCNN(RA) 63.89 14.79 70.95 60.61 0.18 83.89 60.76 93.47 50.87 55.49 89.14 96.80

Hybrid Model

(FrRCNN+FPN+RA)

85.54 27.86 93.68 96.30 0.22 99.09 96.04 99.46 96.80 77.22 94.58 97.76

81

Table 2: Comparison of modified models on the PlotQA dataset with mAP scores (in %) at IOUs of 0.9, 0.75 & 0.5.

A Hybrid Model: Results

IOU 0.9 0.75 0.5

Class\Models

bar dotline

legendlabel

legendpreview

plottitle

x-axislabels

x-axisticks

y-axislabels

y-axisticks

mAP mAP mAP

FRCNN(RA) 83.25 16.32 62.31 59.10 0.18 65.77 78.72 42.94 78.87 54.16 68.92 72.46

FRCNN(FPN+RA) 87.59 31.62 79.05 66.39 0.22 69.78 88.29 46.63 84.60 61.57 69.82 72.18

FrRCNN(RA) 63.89 14.79 70.95 60.61 0.18 83.89 60.76 93.47 50.87 55.49 89.14 96.80

Hybrid Model

(FrRCNN+FPN+RA)

85.54 27.86 93.68 96.30 0.22 99.09 96.04 99.46 96.80 77.22 94.58 97.76

82

Table 2: Comparison of modified models on the PlotQA dataset with mAP scores (in %) at IOUs of 0.9, 0.75 & 0.5.

A Hybrid Model: Results

IOU 0.9 0.75 0.5

Class\Models

bar dotline

legendlabel

legendpreview

plottitle

x-axislabels

x-axisticks

y-axislabels

y-axisticks

mAP mAP mAP

FRCNN(RA) 83.25 16.32 62.31 59.10 0.18 65.77 78.72 42.94 78.87 54.16 68.92 72.46

FRCNN(FPN+RA) 87.59 31.62 79.05 66.39 0.22 69.78 88.29 46.63 84.60 61.57 69.82 72.18

FrRCNN(RA) 63.89 14.79 70.95 60.61 0.18 83.89 60.76 93.47 50.87 55.49 89.14 96.80

Hybrid Model

(FrRCNN+FPN+RA)

85.54 27.86 93.68 96.30 0.22 99.09 96.04 99.46 96.80 77.22 94.58 97.76

83

Table 2: Comparison of modified models on the PlotQA dataset with mAP scores (in %) at IOUs of 0.9, 0.75 & 0.5.

A Hybrid Model: Results

IOU 0.9 0.75 0.5

Class\Models

bar dotline

legendlabel

legendpreview

plottitle

x-axislabels

x-axisticks

y-axislabels

y-axisticks

mAP mAP mAP

FRCNN(RA) 83.25 16.32 62.31 59.10 0.18 65.77 78.72 42.94 78.87 54.16 68.92 72.46

FRCNN(FPN+RA) 87.59 31.62 79.05 66.39 0.22 69.78 88.29 46.63 84.60 61.57 69.82 72.18

FrRCNN(RA) 63.89 14.79 70.95 60.61 0.18 83.89 60.76 93.47 50.87 55.49 89.14 96.80

Hybrid Model

(FrRCNN+FPN+RA)

85.54 27.86 93.68 96.30 0.22 99.09 96.04 99.46 96.80 77.22 94.58 97.76

84

Table 2: Comparison of modified models on the PlotQA dataset with mAP scores (in %) at IOUs of 0.9, 0.75 & 0.5.

A Hybrid Model: Qualitative Analysis

FRCNN(RA) FRCNN(FPN+RA)

FrRCNN(RA) Hybrid Model ( FrRCNN+FPN+RA)

Figure: Detected bounding boxes on an example plot from PlotQA dataset for different hybrid models corresponding to Table 2 at an IOU threshold of 0.9.

85

Figure: mAP (in %) v/s Inference Time per image (in ms) for different object detection models on PlotQA at an IOU setting of 0.9. (x, y) represents the tuple (mAP, time). 86

A Hybrid Model: Summary

Existing Models

Figure: mAP (in %) v/s Inference Time per image (in ms) for different object detection models on PlotQA at an IOU setting of 0.9. (x, y) represents the tuple (mAP, time). 87

A Hybrid Model: Summary

Existing Models

Hybrid Model

Figure: mAP (in %) v/s Inference Time per image (in ms) for different object detection models on PlotQA at an IOU setting of 0.9. (x, y) represents the tuple (mAP, time). 88

A Hybrid Model: Summary

Preferred region

Can we do better (faster and more efficient)?

89

1x1 FC

90

Proposed Model: PlotNet

CH

RH

LH

ROI Mask Feature MapRGB Image ROI Aligned Features ROI Volumes Output VectorsFinal Vector

1024

25632

0

1x1 FC

91

Proposed Model: PlotNet

CH

RH

LH

ROI Mask Feature MapRGB Image ROI Aligned Features ROI Volumes Output VectorsFinal Vector

1024

320

256

CV-based proposal

1x1 FC

92

Proposed Model: PlotNet

CH

RH

LH

Feature Map ROI Aligned Features ROI Volumes Output VectorsFinal Vector

1024

320

256

Feature Extractor

ROI Mask RGB Image

1x1 FC

93

Proposed Model: PlotNet

CH

RH

LH

ROI Aligned Features ROI Volumes Output VectorsFinal Vector

1024

320

256

ROI Align

ROI Mask RGB Image Feature Map

1x1 FC

94

Proposed Model: PlotNet

CH

RH

LH

ROI Volumes Output VectorsFinal Vector

1024

320

256

ROI Mask Feature MapRGB Image ROI Aligned Features

AN-ROI Layer

95

Proposed Model: PlotNet

FC

CH

RH

LH

Output VectorsFinal Vector

1024

Class, Regress and Linking Heads

ROI Mask Feature MapRGB Image ROI Aligned Features

1x1

320

256

ROI Volumes

96

Proposed Model: PlotNet

FC

CH

RH

LH

Output VectorsFinal Vector

1024

Class, Regress and Linking Heads

ROI Mask Feature MapRGB Image ROI Aligned Features

1x1

320

256

ROI Volumes

RGB Image Grayscale Image

Preprocess-ing

97

PlotNet: CV-based Region Proposal

RGB Image Laplacian Edges

LaplacianEdge

Detector

98

PlotNet: CV-based Region Proposal

RGB Image Contoured Image

Contour Detection

99

PlotNet: CV-based Region Proposal

RGB Image Proposed ROIs

Fit Bounding

Boxes

100

PlotNet: CV-based Region Proposal

RGB Image Proposed ROIs 1-D ROI Mask

CV techniques

Mask formation

101

PlotNet: CV-based Region Proposal

102

PlotNet: Feature Extractor

ROI Mask Feature MapRGB Image

103

PlotNet: Feature Extractor

ROI Mask Feature MapRGB Image

104

PlotNet: ROI Align Layer

ROI Mask Feature MapRGB Image ROI Aligned Features

105

PlotNet: AN-ROI Layer

ROI Mask Feature MapRGB Image ROI Aligned Features ROI Volumes

Volume size:14 x 14 x 320

106

Volume size:14 x 14 x 256

PlotNet: AN-ROI Layer

ROI Mask Feature MapRGB Image ROI Aligned Features ROI Volumes

1x1

320

1x1 FC

107

PlotNet: Class, Regress, and Linking Heads

CH

RH

LH

ROI Mask Feature MapRGB Image ROI Aligned Features ROI Volumes Output VectorsFinal Vector

1024

25632

0

1x1 FC

108

PlotNet: Class, Regress, and Linking Heads

CH

RH

LH

ROI Mask Feature MapRGB Image ROI Aligned Features ROI Volumes Output VectorsFinal Vector

1024

25632

0

for every ROI

Figure: Comparison of different loss functions at varying IOUs. 109

PlotNet: Loss Function

IOU

Loss

High IOU region

Key Insight

Existing losses give negligible values at high IoUs

Figure: Comparison of different loss functions at varying IOUs. 110

PlotNet: Loss Function

IOU

Loss

● Gives non-negligible values at high IOUs

● Mathematically, the loss is defined:

● 𝛄 determines the rate of the scaling factor

Our Contribution

Figure: Comparison of different loss functions at varying IOUs. 111

PlotNet: Loss Function

IOU

Loss

● Gives non-negligible values at high IOUs

● Mathematically, the loss is defined:

● 𝛄 determines the rate of the scaling factor

Non-negligible values

Our Contribution

Figure: Comparison of different loss functions at varying IOUs. 112

PlotNet: Loss Function

IOU

Loss

● Gives non-negligible values at high IOUs

● Mathematically, the loss is defined:

● 𝛄 determines the rate of the scaling factor

Our Contribution

113

PlotNet: Different Configurations

1)

2)

3)

4)

5)

6)

7)

8)

: AN-ROI Layer : Regression Loss: ROI Features

WACV2021: Additional Experiments

bar dotline

legendlabel

legendpreview

plottitle

x-axislabels

x-axisticks

y-axislabels

y-axisticks

mAP@0.9

91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89

92.16 61.18 98.38 93.46 99.44 97.21 94.21 95.45 94.42 91.77

91.79 41.86 93.74 94.64 98.29 83.11 85.69 89.32 49.36 80.87

91.83 45.78 91.48 94.15 98.95 74.24 87.19 89.34 50.11 80.34

91.88 61.44 96.44 95.58 99.27 97.19 90.64 97.55 87.66 90.88

91.71 49.30 95.99 93.55 98.42 95.03 89.77 94.08 86.06 88.21

91.35 52.22 96.31 93.45 96.82 96.18 89.63 95.46 94.07 89.50

91.15 55.03 97.89 92.99 99.46 96.33 91.30 90.40 95.48 90.00

114

Table 5: Comparison of variants of PlotNet on the PlotQA dataset with mAP score(in %) at IOUs of 0.9.

WACV2021: Additional Experiments

bar dotline

legendlabel

legendpreview

plottitle

x-axislabels

x-axisticks

y-axislabels

y-axisticks

mAP@0.9

92.78 68.26 97.75 95.90 99.04 93.64 92.97 96.24 93.12 92.19

92.74 59.12 94.87 95.58 92.26 94.46 94.12 95.36 76.85 88.39

92.80 70.11 98.47 96.33 99.52 97.31 94.29 97.66 94.48 93.44

92.84 69.19 98.49 95.22 99.42 96.88 94.65 96.51 96.87 93.34

92.96 68.25 98.34 95.96 98.96 97.10 95.08 96.84 97.16 93.41

92.91 58.52 98.40 95.95 95.14 97.16 94.59 97.55 95.63 91.76

115

Table 6: Comparison of variants of PlotNet on the PlotQA dataset with mAP score(in %) at IOUs of 0.9.

WACV2021: Additional Experiments

bar dotline

legendlabel

legendpreview

plottitle

x-axislabels

x-axisticks

y-axislabels

y-axisticks

mAP@0.9

85.30 52.85 29.64 94.30 0.00 10.36 80.77 1.47 81.59 48.48

76.14 61.49 40.19 96.51 0.00 16.49 79.05 1.76 77.68 49.92no linking

no linking

PlotNet: Results

bar dotline

legendlabel

legendpreview

plottitle

x-axislabels

x-axisticks

y-axislabels

y-axisticks

mAP@0.9

91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89

92.16 61.18 98.38 93.46 99.44 97.21 94.21 95.45 94.42 91.77

91.79 41.86 93.74 94.64 98.29 83.11 85.69 89.32 49.36 80.87

91.83 45.78 91.48 94.15 98.95 74.24 87.19 89.34 50.11 80.34

91.88 61.44 96.44 95.58 99.27 97.19 90.64 97.55 87.66 90.88

92.78 68.26 97.75 95.90 99.04 93.64 92.97 96.24 93.12 92.19

92.74 59.12 94.87 95.58 92.26 94.46 94.12 95.36 76.85 88.39

92.80 70.11 98.47 96.33 99.52 97.31 94.29 97.66 94.48 93.44

117

Table 5: Comparison of variants of PlotNet on the PlotQA dataset with mAP score(in %) at IOUs of 0.9.

PlotNet: Results

bar dotline

legendlabel

legendpreview

plottitle

x-axislabels

x-axisticks

y-axislabels

y-axisticks

mAP@0.9

91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89

92.16 61.18 98.38 93.46 99.44 97.21 94.21 95.45 94.42 91.77

91.79 41.86 93.74 94.64 98.29 83.11 85.69 89.32 49.36 80.87

91.83 45.78 91.48 94.15 98.95 74.24 87.19 89.34 50.11 80.34

91.88 61.44 96.44 95.58 99.27 97.19 90.64 97.55 87.66 90.88

92.78 68.26 97.75 95.90 99.04 93.64 92.97 96.24 93.12 92.19

92.74 59.12 94.87 95.58 92.26 94.46 94.12 95.36 76.85 88.39

92.80 70.11 98.47 96.33 99.52 97.31 94.29 97.66 94.48 93.44

118

Table 5: Comparison of variants of PlotNet on the PlotQA dataset with mAP score(in %) at IOUs of 0.9.

PlotNet: Results

bar dotline

legendlabel

legendpreview

plottitle

x-axislabels

x-axisticks

y-axislabels

y-axisticks

mAP@0.9

91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89

92.16 61.18 98.38 93.46 99.44 97.21 94.21 95.45 94.42 91.77

91.79 41.86 93.74 94.64 98.29 83.11 85.69 89.32 49.36 80.87

91.83 45.78 91.48 94.15 98.95 74.24 87.19 89.34 50.11 80.34

91.88 61.44 96.44 95.58 99.27 97.19 90.64 97.55 87.66 90.88

92.78 68.26 97.75 95.90 99.04 93.64 92.97 96.24 93.12 92.19

92.74 59.12 94.87 95.58 92.26 94.46 94.12 95.36 76.85 88.39

92.80 70.11 98.47 96.33 99.52 97.31 94.29 97.66 94.48 93.44

119

Table 5: Comparison of variants of PlotNet on the PlotQA dataset with mAP score(in %) at IOUs of 0.9.

PlotNet: Results

bar dotline

legendlabel

legendpreview

plottitle

x-axislabels

x-axisticks

y-axislabels

y-axisticks

mAP@0.9

91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89

92.16 61.18 98.38 93.46 99.44 97.21 94.21 95.45 94.42 91.77

91.79 41.86 93.74 94.64 98.29 83.11 85.69 89.32 49.36 80.87

91.83 45.78 91.48 94.15 98.95 74.24 87.19 89.34 50.11 80.34

91.88 61.44 96.44 95.58 99.27 97.19 90.64 97.55 87.66 90.88

92.78 68.26 97.75 95.90 99.04 93.64 92.97 96.24 93.12 92.19

92.74 59.12 94.87 95.58 92.26 94.46 94.12 95.36 76.85 88.39

92.80 70.11 98.47 96.33 99.52 97.31 94.29 97.66 94.48 93.44

120

Table 5: Comparison of variants of PlotNet on the PlotQA dataset with mAP score(in %) at IOUs of 0.9.

PlotNet: Results

bar dotline

legendlabel

legendpreview

plottitle

x-axislabels

x-axisticks

y-axislabels

y-axisticks

mAP@0.9

91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89

92.16 61.18 98.38 93.46 99.44 97.21 94.21 95.45 94.42 91.77

91.79 41.86 93.74 94.64 98.29 83.11 85.69 89.32 49.36 80.87

91.83 45.78 91.48 94.15 98.95 74.24 87.19 89.34 50.11 80.34

91.88 61.44 96.44 95.58 99.27 97.19 90.64 97.55 87.66 90.88

92.78 68.26 97.75 95.90 99.04 93.64 92.97 96.24 93.12 92.19

92.74 59.12 94.87 95.58 92.26 94.46 94.12 95.36 76.85 88.39

92.80 70.11 98.47 96.33 99.52 97.31 94.29 97.66 94.48 93.44

121

Table 5: Comparison of variants of PlotNet on the PlotQA dataset with mAP score(in %) at IOUs of 0.9.

● PlotNet performs better than all existing methods at all IOUs.

● At 0.9 IOU threshold, PlotNet improves upon its closest competitor by 16.22 absolute points.

Figure: Detected bounding boxes by PlotNet-v7 on an example plot from PlotQA dataset at an IOU threshold of 0.9.

122

PlotNet: Qualitative Analysis

PlotNet: Comparison to other models

Figure: mAP (in %) v/s Inference Time per image (in ms) for different object detection models on PlotQA at an IOU setting of 0.9. (x, y) represents the tuple (mAP, time).

123

PlotNet: Comparison to other models

Figure: mAP (in %) v/s Inference Time per image (in ms) for different object detection models on PlotQA at an IOU setting of 0.9. (x, y) represents the tuple (mAP, time).

124

PlotNet: Comparison to other models

Figure: mAP (in %) v/s Inference Time per image (in ms) for different object detection models on PlotQA at an IOU setting of 0.9. (x, y) represents the tuple (mAP, time).

125

16.22pts

Figure: mAP v/s IOU threshold for different object detection models.126

PlotNet: Comparison to other models

Use-Case: Plot to Table Converter

(a) Input Image

(c) Ground-truth Table (d) Generated Table

(b) Predicted bounding boxes

Figure: Sample table generation using PlotNet's predictions 127

Use-Case: Plot to Table Converter

(a) Input Image

(c) Ground-truth Table (d) Generated Table

(b) Predicted bounding boxes

Figure: Sample table generation using PlotNet's predictions 128

129

Conclusion

Evaluated existing methods and exemplified the challenges

Proposed PlotNet addressing all the challenges

130

High Recall Proposal Method End2End Training

Future Work

Image Source: Google Images

Communicated:

Pritha Ganguly*, Nitesh Methani*, Mitesh M. Khapra and Pratyush Kumar, A Systematic Evaluation of Object Detection Networks for Scientific Plots., Under review at a Computer Vision Conference.

*the first two authors have contributed equally. 131

Visible Outcome

Dr. Mitesh Khapra Dr. Pratyush KumarNitesh Methani(Research Scholar, IIT Madras) (Assistant Professor, IIT Madras) (Assistant Professor, IIT Madras)

132

Team

Pritha Ganguly(Research Scholar, IIT Madras)

Nikita Suman

Siddhartha Preksha

TarunAnanya

Darwin Aakriti Madhura

JananiHimanshu

Shashank Nikhilesh

Priyesh

Deepak Shweta

133

Thank You!

Image Source: Google Images134

Questions or Suggestions?

135

Extra Slides

PlotNet: Finding the Best Feature Extractor

136

PlotNet: Finding the best feature extractor

IOU 0.9 0.75 0.5

Models\Class

bar dotline

legendlabel

legendpreview

plottitle

x-axislabels

x-axisticks

y-axislabels

y-axisticks

mAP mAP mAP

R-10 89.46 37.63 93.64 78.12 95.22 94.06 88.10 66.95 83.86 80.78 96.74 97.57

R-22 91.37 24.08 97.03 81.01 98.79 90.47 81.99 51.79 47.92 73.83 97.01 98.08

R-50 87.64 15.72 74.57 41.87 98.92 81.60 54.21 43.35 35.67 59.28 93.91 97.67

R-22 FPN 91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89 97.21 98.11

R-50 FPN 90.77 5.12 95.58 80.72 99.16 94.79 76.83 65.56 58.17 74.08 94.09 97.67

137

Table 3: Comparison of different variants of PlotNet on the PlotQA dataset by varying the number of layers in the ResNet(R)-50 architecture with mAP scores (in %) at IOUs of 0.9, 0.75, and 0.5.

PlotNet: Finding the best feature extractor

IOU 0.9 0.75 0.5

Models\Class

bar dotline

legendlabel

legendpreview

plottitle

x-axislabels

x-axisticks

y-axislabels

y-axisticks

mAP mAP mAP

R-10 89.46 37.63 93.64 78.12 95.22 94.06 88.10 66.95 83.86 80.78 96.74 97.57

R-22 91.37 24.08 97.03 81.01 98.79 90.47 81.99 51.79 47.92 73.83 97.01 98.08

R-50 87.64 15.72 74.57 41.87 98.92 81.60 54.21 43.35 35.67 59.28 93.91 97.67

R-22 FPN 91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89 97.21 98.11

R-50 FPN 90.77 5.12 95.58 80.72 99.16 94.79 76.83 65.56 58.17 74.08 94.09 97.67

138

Table 3: Comparison of different variants of PlotNet on the PlotQA dataset by varying the number of layers in the ResNet(R)-50 architecture with mAP scores (in %) at IOUs of 0.9, 0.75, and 0.5.

PlotNet: Finding the best feature extractor

IOU 0.9 0.75 0.5

Models\Class

bar dotline

legendlabel

legendpreview

plottitle

x-axislabels

x-axisticks

y-axislabels

y-axisticks

mAP mAP mAP

R-10 89.46 37.63 93.64 78.12 95.22 94.06 88.10 66.95 83.86 80.78 96.74 97.57

R-22 91.37 24.08 97.03 81.01 98.79 90.47 81.99 51.79 47.92 73.83 97.01 98.08

R-50 87.64 15.72 74.57 41.87 98.92 81.60 54.21 43.35 35.67 59.28 93.91 97.67

R-22 FPN 91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89 97.21 98.11

R-50 FPN 90.77 5.12 95.58 80.72 99.16 94.79 76.83 65.56 58.17 74.08 94.09 97.67

139

Table 3: Comparison of different variants of PlotNet on the PlotQA dataset by varying the number of layers in the ResNet(R)-50 architecture with mAP scores (in %) at IOUs of 0.9, 0.75, and 0.5.

PlotNet: Finding the best feature extractor

IOU 0.9 0.75 0.5

Models\Class

bar dotline

legendlabel

legendpreview

plottitle

x-axislabels

x-axisticks

y-axislabels

y-axisticks

mAP mAP mAP

R-10 89.46 37.63 93.64 78.12 95.22 94.06 88.10 66.95 83.86 80.78 96.74 97.57

R-22 91.37 24.08 97.03 81.01 98.79 90.47 81.99 51.79 47.92 73.83 97.01 98.08

R-50 87.64 15.72 74.57 41.87 98.92 81.60 54.21 43.35 35.67 59.28 93.91 97.67

R-22 FPN 91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89 97.21 98.11

R-50 FPN 90.77 5.12 95.58 80.72 99.16 94.79 76.83 65.56 58.17 74.08 94.09 97.67

140

Table 3: Comparison of different variants of PlotNet on the PlotQA dataset by varying the number of layers in the ResNet(R)-50 architecture with mAP scores (in %) at IOUs of 0.9, 0.75, and 0.5.

PlotNet: Finding the best feature extractor

IOU 0.9 0.75 0.5

Models\Class

bar dotline

legendlabel

legendpreview

plottitle

x-axislabels

x-axisticks

y-axislabels

y-axisticks

mAP mAP mAP

R-10 89.46 37.63 93.64 78.12 95.22 94.06 88.10 66.95 83.86 80.78 96.74 97.57

R-22 91.37 24.08 97.03 81.01 98.79 90.47 81.99 51.79 47.92 73.83 97.01 98.08

R-50 87.64 15.72 74.57 41.87 98.92 81.60 54.21 43.35 35.67 59.28 93.91 97.67

R-22 FPN 91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89 97.21 98.11

R-50 FPN 90.77 5.12 95.58 80.72 99.16 94.79 76.83 65.56 58.17 74.08 94.09 97.67

141

Table 3: Comparison of different variants of PlotNet on the PlotQA dataset by varying the number of layers in the ResNet(R)-50 architecture with mAP scores (in %) at IOUs of 0.9, 0.75, and 0.5.

PlotNet: Qualitative Analysis

IOU 0.9 0.75 0.5

Models\Class

bar dotline

legendlabel

legendpreview

plottitle

x-axislabels

x-axisticks

y-axislabels

y-axisticks

mAP mAP mAP

v0 91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89 97.21 98.11

v1 92.16 61.18 98.38 93.46 99.44 97.21 94.21 95.45 94.42 91.77 97.74 98.24

v2 91.79 41.86 93.74 94.64 98.29 83.11 85.69 89.32 49.36 80.87 96.38 98.20

v3 91.83 45.78 91.48 94.15 98.95 74.24 87.19 89.34 50.11 80.34 96.97 98.26

v4 91.88 61.44 96.44 95.58 99.27 97.19 90.64 97.55 87.66 90.88 97.30 98.31

v5 92.78 68.26 97.75 95.90 99.04 93.64 92.97 96.24 93.12 92.19 97.70 98.18

v6 92.74 59.12 94.87 95.58 92.26 94.46 94.12 95.36 76.85 88.39 96.78 97.82

v7 92.80 70.11 98.47 96.33 99.52 97.31 94.29 97.66 94.48 93.44 97.93 98.32

142

Table 5: Comparison of variants of PlotNet on the PlotQA dataset with mAP score(in %) at IOUs of 0.9, 0.75 & 0.5.

PlotNet: Qualitative Analysis

IOU 0.9 0.75 0.5

Models\Class

bar dotline

legendlabel

legendpreview

plottitle

x-axislabels

x-axisticks

y-axislabels

y-axisticks

mAP mAP mAP

v0 91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89 97.21 98.11

v1 92.16 61.18 98.38 93.46 99.44 97.21 94.21 95.45 94.42 91.77 97.74 98.24

v2 91.79 41.86 93.74 94.64 98.29 83.11 85.69 89.32 49.36 80.87 96.38 98.20

v3 91.83 45.78 91.48 94.15 98.95 74.24 87.19 89.34 50.11 80.34 96.97 98.26

v4 91.88 61.44 96.44 95.58 99.27 97.19 90.64 97.55 87.66 90.88 97.30 98.31

v5 92.78 68.26 97.75 95.90 99.04 93.64 92.97 96.24 93.12 92.19 97.70 98.18

v6 92.74 59.12 94.87 95.58 92.26 94.46 94.12 95.36 76.85 88.39 96.78 97.82

v7 92.80 70.11 98.47 96.33 99.52 97.31 94.29 97.66 94.48 93.44 97.93 98.32

143

Table 5: Comparison of variants of PlotNet on the PlotQA dataset with mAP score(in %) at IOUs of 0.9, 0.75 & 0.5.

PlotNet: Qualitative Analysis

IOU 0.9 0.75 0.5

Models\Class

bar dotline

legendlabel

legendpreview

plottitle

x-axislabels

x-axisticks

y-axislabels

y-axisticks

mAP mAP mAP

v0 91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89 97.21 98.11

v1 92.16 61.18 98.38 93.46 99.44 97.21 94.21 95.45 94.42 91.77 97.74 98.24

v2 91.79 41.86 93.74 94.64 98.29 83.11 85.69 89.32 49.36 80.87 96.38 98.20

v3 91.83 45.78 91.48 94.15 98.95 74.24 87.19 89.34 50.11 80.34 96.97 98.26

v4 91.88 61.44 96.44 95.58 99.27 97.19 90.64 97.55 87.66 90.88 97.30 98.31

v5 92.78 68.26 97.75 95.90 99.04 93.64 92.97 96.24 93.12 92.19 97.70 98.18

v6 92.74 59.12 94.87 95.58 92.26 94.46 94.12 95.36 76.85 88.39 96.78 97.82

v7 92.80 70.11 98.47 96.33 99.52 97.31 94.29 97.66 94.48 93.44 97.93 98.32

144

Table 5: Comparison of variants of PlotNet on the PlotQA dataset with mAP score(in %) at IOUs of 0.9, 0.75 & 0.5.

PlotNet: Qualitative Analysis

IOU 0.9 0.75 0.5

Models\Class

bar dotline

legendlabel

legendpreview

plottitle

x-axislabels

x-axisticks

y-axislabels

y-axisticks

mAP mAP mAP

v0 91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89 97.21 98.11

v1 92.16 61.18 98.38 93.46 99.44 97.21 94.21 95.45 94.42 91.77 97.74 98.24

v2 91.79 41.86 93.74 94.64 98.29 83.11 85.69 89.32 49.36 80.87 96.38 98.20

v3 91.83 45.78 91.48 94.15 98.95 74.24 87.19 89.34 50.11 80.34 96.97 98.26

v4 91.88 61.44 96.44 95.58 99.27 97.19 90.64 97.55 87.66 90.88 97.30 98.31

v5 92.78 68.26 97.75 95.90 99.04 93.64 92.97 96.24 93.12 92.19 97.70 98.18

v6 92.74 59.12 94.87 95.58 92.26 94.46 94.12 95.36 76.85 88.39 96.78 97.82

v7 92.80 70.11 98.47 96.33 99.52 97.31 94.29 97.66 94.48 93.44 97.93 98.32

145

Table 5: Comparison of variants of PlotNet on the PlotQA dataset with mAP score(in %) at IOUs of 0.9, 0.75 & 0.5.

PlotNet: Qualitative Analysis

IOU 0.9 0.75 0.5

Models\Class

bar dotline

legendlabel

legendpreview

plottitle

x-axislabels

x-axisticks

y-axislabels

y-axisticks

mAP mAP mAP

v0 91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89 97.21 98.11

v1 92.16 61.18 98.38 93.46 99.44 97.21 94.21 95.45 94.42 91.77 97.74 98.24

v2 91.79 41.86 93.74 94.64 98.29 83.11 85.69 89.32 49.36 80.87 96.38 98.20

v3 91.83 45.78 91.48 94.15 98.95 74.24 87.19 89.34 50.11 80.34 96.97 98.26

v4 91.88 61.44 96.44 95.58 99.27 97.19 90.64 97.55 87.66 90.88 97.30 98.31

v5 92.78 68.26 97.75 95.90 99.04 93.64 92.97 96.24 93.12 92.19 97.70 98.18

v6 92.74 59.12 94.87 95.58 92.26 94.46 94.12 95.36 76.85 88.39 96.78 97.82

v7 92.80 70.11 98.47 96.33 99.52 97.31 94.29 97.66 94.48 93.44 97.93 98.32

146

Table 5: Comparison of variants of PlotNet on the PlotQA dataset with mAP score(in %) at IOUs of 0.9, 0.75 & 0.5.

PlotNet: Qualitative Analysis

IOU 0.9 0.75 0.5

Models\Class

bar dotline

legendlabel

legendpreview

plottitle

x-axislabels

x-axisticks

y-axislabels

y-axisticks

mAP mAP mAP

v0 91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89 97.21 98.11

v1 92.16 61.18 98.38 93.46 99.44 97.21 94.21 95.45 94.42 91.77 97.74 98.24

v2 91.79 41.86 93.74 94.64 98.29 83.11 85.69 89.32 49.36 80.87 96.38 98.20

v3 91.83 45.78 91.48 94.15 98.95 74.24 87.19 89.34 50.11 80.34 96.97 98.26

v4 91.88 61.44 96.44 95.58 99.27 97.19 90.64 97.55 87.66 90.88 97.30 98.31

v5 92.78 68.26 97.75 95.90 99.04 93.64 92.97 96.24 93.12 92.19 97.70 98.18

v6 92.74 59.12 94.87 95.58 92.26 94.46 94.12 95.36 76.85 88.39 96.78 97.82

v7 92.80 70.11 98.47 96.33 99.52 97.31 94.29 97.66 94.48 93.44 97.93 98.32

147

Table 5: Comparison of variants of PlotNet on the PlotQA dataset with mAP score(in %) at IOUs of 0.9, 0.75 & 0.5.