Crafting and learning for image matching

presented by Dmytro Mishkin

joint work with

Anastasia Mishchuk, Milan Pultar, Filip Radenovic, Daniel Barath, Michal Perdoch, Jiri Matas

12019.07.06, Odesa, EECVC

What is image matching?

• The task is to find the correspondences between pixels in two images and/or find a geometrical relations between camera poses

• It`s special version is also known as “wide baseline stereo”:

• large change in viewpoint, illumination, time & occlusions, modality

2019.07.06, Odesa, EECVC 2

Where is image matching useful? 3D rec & SLAM

R. Mur-Artal, and J. D. Tardós. ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras, arXiv 2016

L. Schonberger and J.-M. Frahm,Structure-from-Motion Revisited, 2016COLMAP

32019.07.06, Odesa, EECVC

Daniel DeTone, Tomasz Malisiewicz, Andrew RabinovichMagicLeap SLAM

Image retrieval 2.0F. Radenovic, series of works

42019.07.06, Odesa, EECVC

Google Landmark Retrieval challenge 2019 winner

Where is image matching useful? Image retrieval

What is NOT topic of my talk

• Semantic correspondences (the object/scene is not the same)

2019.07.06, Odesa, EECVC 5

Images from Aberman et al. “Neural Best-Buddies: Sparse Cross-Domain Correspondence”, SIGGRAPH 2018

What is NOT topic of my talk

• Short baseline stereo (wait for Anastasiia Mishchuk talk at 17:20)

• Optimization methods for stereo• see talks from previous EECVCs:

Alexander Shekhovtsov (2017) and Tolga Birdal (2018)

2019.07.06, Odesa, EECVC 6

Wide baseline stereo pipeline

72019.07.06, Odesa, EECVC

Measurement region selector

Measurement region selector

Matching

Descriptor

DescriptorDetector

Detector

Geometrical verification (RANSAC)

Image credit: Andrea Vedaldi, ICCVW 2017

Single feature visualization

Toy example for illustration: matching with OpenCV SIFT

2019.07.06, Odesa, EECVC 8Try yourself: https://github.com/ducha-aiki/matching-strategies-comparison

Toy example for illustration: matching with OpenCV SIFT

2019.07.06, Odesa, EECVC 9

Recovered 1st to 2nd image projection, ground truth 1st to 2nd image project,inlier correspondences

Geometric verification (RANSAC)

2019.07.06, Odesa, EECVC 10

Measurement region selector

Measurement region selector

Matching

Descriptor

DescriptorDetector

Detector

Geometrical verification (RANSAC)

Homography: planar surface/static camera

2019.07.06, Odesa, EECVC 11

Image credit: forums.fast.ai

Planar surface or static camera → use homographyImage with dominant plane → use homographyNot sure what to use? → try homography first.

Fundamental matrix: general two-view case

2019.07.06, Odesa, EECVC 12

• General two view geometry in static scene. A corresponding point lies somewhere on a line in the other image. Where on the line - depends on the (unknown) depth

• Weaker constraint than homography

• Still rigid (no motion in scene assumed)

Image credit: https://en.wikipedia.org/wiki/Epipolar_geometry

RANSAC: fitting the data with gross outliers

• What is it

2019.07.06, Odesa, EECVC 13

Image credit: https://scipy-cookbook.readthedocs.io/items/RANSAC.html

OpenCV functions:cv2.findHomography()cv2.findFundamentalMatrix()

We will publish soon a python package, which is 2..5 times faster and have an additional tricks inside

https://github.com/ducha-aiki/pyransac (save this link,the repo is private now, will clean-up and open next week)

Pitfails and solutions: homography

• Wrong geometry case because of dirty correspondences

2019.07.06, Odesa, EECVC 14

OpenCV finds 31 wrong inliers in 0.018s.

CMP RANSAC finds 6 wrong inliers in 0.004s.

See the same pattern in img1: 3 corrs in line + group.RANSAC H is prone to such case

Pitfails and solutions: homography

• Solution: 2-way error metric

2019.07.06, Odesa, EECVC 15

CMP RANSAC + transfer check:finds 48 correct inliers in 0.005s.

𝐻2−1 = 𝐻1−2−1

Check #inliers consistent in opposite direction

Python CMP RANSAC package will be available soon

D. Mishkin, J. Matas and M. Perdoch. MODS: Fast and Robust Method for Two-View Matching, CVIU 2015,

Pitfails and solutions: fundamental matrix

• F is too permissive (point to line)

2019.07.06, Odesa, EECVC 16

Pitfails and solutions: fundamental matrix

• LAF-check: remember that local feature is oriented circle or ellipse, not just a point.

• Check if additional points on circle are consistent with geometry

2019.07.06, Odesa, EECVC 17

D. Mishkin, J. Matas and M. Perdoch. MODS: Fast and Robust Method for Two-View Matching, CVIU 2015,

Matching strategies

2019.07.06, Odesa, EECVC 18

Measurement region selector

Measurement region selector

Matching

Descriptor

DescriptorDetector

Detector

Geometrical verification (RANSAC)

Nearest neighbor (1NN) strategy

2019.07.06, Odesa, EECVC 20

Features from img1 are matched to features from img2

You can see, that it is asymmetric and allowing “many-to-one” matches

Nearest neighbor (NN) strategy

2019.07.06, Odesa, EECVC 21

Features from img1 are matched to features from img2

OpenCV RANSAC failed to find a good model with NN matchingFound 1st image projection: blue, ground truth: green,Inlier correspondences: yellow

Mutual nearest neighbor (MNN) strategy

2019.07.06, Odesa, EECVC 22

Features from img1 are matched to features from img2Only cross-consistent (mutual NNs) matches are retained.

Mutual nearest neighbor (MNN) strategy

2019.07.06, Odesa, EECVC 23

OpenCV RANSAC failed to find a good model with MNN matchingNo one-to-many connections, but still badFound 1st image projection: blue, ground truth: green ,inlier correspondences: yellow

Features from img1 are matched to features from img2Only cross-consistent (mutual NNs) matches are retained.

Second nearest neighbor ratio (SNN) strategy

2019.07.06, Odesa, EECVC 24

1stNN

2ndNN

2ndNN

1stNN

2ndNN

1stNN

1stNN/2ndNN > 0.8, drop

1stNN/2ndNN < 0.8, keep

Features from img1 are matched to features from img2- we look for 2 nearest neighbors

- If both are too similar (1stNN/2ndNN ratio > 0.8) →discard

- If 1st NN is much closer (1stNN/2ndNN ratio ≤ 0.8) →keep

D. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, IJCV 2004

2019.07.06, Odesa, EECVC 25

NN

SNN

SNN

NN

SNN

NN

NN/SNN > 0.8, drop

NN/SNN < 0.8, keep

OpenCV RANSAC found a model roughly correctNo one-to-many connections, but still badFound 1st image projection: blue, ground truth: green ,inlier correspondences: yellow

Second nearest neighbor ratio (SNN) strategy

1st geometrically inconsistent nearest neighbor ratio (FGINN) strategy

2019.07.06, Odesa, EECVC 26

SNN ratio is cool, but what about symmetrical, or too closely detected features? Ratio test will kill them.Solution: look for 2nd nearest neighbor, which is far enough from 1st nearest.

Mishkin et al., “MODS: Fast and Robust Method for Two-View Matching”, CVIU 2015

1st geometrically inconsistent nearest neighbor ratio (FGINN) strategy

2019.07.06, Odesa, EECVC 27

SNN ratio is cool, but what about symmetrical, or too closely detected features? Ratio test will kill them.Solution: look for 2nd nearest neighbor, which is far enough from 1st nearest.

Mishkin et al., “MODS: Fast and Robust Method for Two-View Matching”, CVIU 2015

SNN vs FGINN

2019.07.06, Odesa, EECVC 28Mishkin et al., “MODS: Fast and Robust Method for Two-View Matching”, CVIU 2015

SNN: roughly correct

FGINN: more correspondences,better geometry found

Symmetrical FGINN

2019.07.06, Odesa, EECVC 29

Recall, that FGINN is still asymmetric:Matching (Img1 → Img2) ≠ (Img2 → Img1)

We can do both (Img1 → Img2) and (Img2 → Img1)

and keep all FGINNs (union)

or only cross-consistent FGINNs

Learned filtering strategy (CVPR 2018)

2019.07.06, Odesa, EECVC 30Yi et al. Learning to Find Good Correspondences https://arxiv.org/abs/1711.05971

Input: matches (x1, y1, x2, y2) [ N x 4 ]Output: scores [ N x 1 ]

Evaluation on IMW2019 data

2019.07.06, Odesa, EECVC 31

CVPR 2019 competition

https://image-matching-workshop.github.io/

Evaluation

Stereo: features ⇨ matching ⇨

⇨ OpenCV RANSAC ⇨ pose estimation

Participants

organizers

Metric: # of precise enough recovered camera poses (mAP @ 15°)

Evaluation on IMW2019 data

2019.07.06, Odesa, EECVC 32

• NN – are you kidding? Never use it alone• SNN is simple and good• FGINN is always a bit better• Symmetrical FGINN rocks• Learning is not that powerful (yet?)

Descriptor: HardNet (NIPS, 2017)

Mishchuk et.al. Working hard to know your neighbor’s margins: Local descriptor learning loss. NIPS 2017

332019.07.06, Odesa, EECVC

Measurement region selector

Measurement region selector

Matching

Descriptor

DescriptorDetector

Detector

Geometrical verification (RANSAC)

• Q1: How to find correct correspondences?

• Q2: How to filter out features, which do not have a correspondence?

• A1: Nearest neighbor by descriptor distance

• A2: Threshold the second-to-first nearest ratio (SNN)

HardNet: lets use it for training CNNs!

34

Classical way to select good correspondences

2019.07.06, Odesa, EECVC

Architecture: deep, VGGNet style

• Adopted from previous sota L2Net descriptor Tian et al (CVPR 2017).

• Vanilla CNN: Convolution + BatchNorm + ReLU

36

2019.07.06, Odesa, EECVC

37Sampling: positives: random

negatives: hard-in-batch

2019.07.06, Odesa, EECVC

HardNet vs SIFT descriptor

2019.07.06, Odesa, EECVC 38Mishkin et al., “MODS: Fast and Robust Method for Two-View Matching”, CVIU 2015

SIFT: 71 inliers

HardNet: 121 inliers

39

Results: HPatches

1.5 … 2 times better than rootSIFT:

2019.07.06, Odesa, EECVC

GeoDesc: same architecture, special loss and sampling utilizing 3d reconstruction data.

40

HardNet training scales well with the bigger datasets

Luo et.al, ECCV 20182019.07.06, Odesa, EECVC

Descriptor: creating the dataset (CVWW, 2019)Leveraging Outdoor Webcams for Local Descriptor LearningMilan Pultar, Dmytro Mishkin, Jiří Matas

412019.07.06, Odesa, EECVC

Measurement region selector

Measurement region selector

Matching

Descriptor

DescriptorDetector

Detector

Geometrical verification (RANSAC)

● Brown dataset ● HPatches● PhotoSynth● GL3D

Existing datasets for local descriptor learning

42

● 1 128 millions of images● Almost 30K webcams● Continuously growing● Cameras placed across all world● ~10 TB of data

● Each camera in one directory○ Split further into folders by year○ Image timestamp in GMT○ GPS info not always available

Archive of Many Outdoor Scenes (AMOS)

43

Camera 1001

Camera 1002

Images -2011

Images -2013Images -2010

Images -2011

AMOS views

44

Good cameras

Bad cameras

Pipeline of AMOS Patches

45

Camera selection

● Choose randomly 20 images from each camera● Test each image using the criteria

● Keep the camera, if 14/20 images pass46

-> 474 cameras

Sky segmentation in the wild: An empirical study.R. P. Mihail et al, 2012(https://github.com/kuangliu/torchcv)

Appearance clustering

● Solves data redundancy

● Use fc6 layer of ImageNet-pretrained AlexNet

● Run K-means in the AlexNet output space

● Choose K=120 most representative images

(by looking at the corresponding outputs)

47-> 474 cameras, each 120 images

Imagenet classification with deep convolutional neural networks.A. Krizhevsky et al, 2012

Viewpoint reclustering

● Solves switching of cameras between views● Uses MODS (image matching) in greedy algorithm1. Pick a reference image2. Find matching pairs3. Create a new view; exclude images from original sequence4. If original sequence not empty:

Repeat● Keep the biggest view from each camera, 50 images each (if available)

48-> 273 views

Registration

● Results still not satisfactory● Why?

○ MODS often outputs homography matrix only for small part of image

○ Need for final manual check

-> Use GDB-ICP● In each view

○ Run registration on pairs of images○ If a single fail -> remove the whole view

49-> 151 registered views

Manual pruning

● Several problems not detected so far○ Dynamic scenes○ Cloud-dominated scenes○ Views with very similar content

50-> 27 registered views, 50 images each

Patch selection

51

● Apply masks (crop out text etc.)

● Sampling of centers (response function)

● Random rotation (any angle)

● Random scale

Experiments: (de)registration

● Displace each patch randomly○ observe the influence on precision

● Result:○ Precise registration is important

● mAP - mean average precision

52

HPatches matching task, full split

Experiments: #batch composition

● We use hard-in-batch triplet margin loss● Composition of a batch influences precision● Idea: choose a subset of views

as source for patches

● Intuition: Tough pairs often comefrom the same image

-> Improvement

53

HPatches matching task, full split

Evaluation

● New state-of-the-art in matching underillumination changes(to the best of our knowledge)

● Outperforms recently proposedHardNetPS in full split

● We propose AMOS Patches test splitfor evaluation of robustness tolighting and season-related conditions

55

Measurement region selector:orientation

562019.07.06, Odesa, EECVC

Measurement region selector

Measurement region selector

Matching

Descriptor

DescriptorDetector

Detector

Geometrical verification (RANSAC)

Which patch should we describe?

2019.07.06, Odesa, EECVC 57

Detector: x, y, scaleShould we rotate patch?Should we deform patch?

Handcrafted: dominant orientation

Learned orientation: CNN

Yi et al. Learning to Assign Orientations to Feature Points CVPR 2016

D. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, IJCV 2004

If images are upright for sure: don`t detect orientation

2019.07.06, Odesa, EECVC 58

DoG + HardNet matches +FGINN union + RANSAC. Found 1st image projection: blue,ground truth: green ,

inlier correspondences: yellow

Dominant gradient orientation: 123 inliers

Learned orientation:140 inliers

Constant orientation: 181 inliers

AffNet (ECCV 2018)Measurement region selector

592019.07.06, Odesa, EECVC

Measurement region selector

Measurement region selector

Matching

Descriptor

DescriptorDetector

Detector

Geometrical verification (RANSAC)

AffNet: learning measurement region

Mishkin et.al. Repeatability Is Not Enough: Learning Affine Regions via Discriminability. ECCV 2018 602019.07.06, Odesa, EECVC

Do AffNet help? Yes, if the problem is hard

2019.07.06, Odesa, EECVC 61

FGINN union + RANSAC. Found 1st image projection: blue,ground truth: green ,

inlier correspondences: yellow

DoG + HardNet 2.0: 123 inliers

DoG + AffNet + HardNet 2.0: 165 inliers

62

• Find affine shape such that maximizes difference between positive and hardest-in-batch negative examples

• Positive-only learning (Yi et. Al, CVPR2015) leads to degenerated ellipses

• Triplet margin (HardNet) – unstable in training affine shape

AffNet: learning measurement region

2019.07.06, Odesa, EECVC

Local feature detector

2019.07.06, Odesa, EECVC 63

Measurement region selector

Measurement region selector

Matching

Descriptor

DescriptorDetector

Detector

Geometrical verification (RANSAC)

Detector is the often failure point of the whole process

• Yet we still use 10-20 y.o stuff like SIFT or FAST, because nothing significantly better for practical purposed have been proposed

• So let`s stick to the basics

2019.07.06, Odesa, EECVC 64

Stylianou et.al, WACV 2015. Characterizing Feature Matching Performance Over Long Time Periods

SIFT is the DoG detector + SIFT descriptor

• Really, there is not such thing, as SIFT detector.

• But everyone so got used to name DoG as SIFT

2019.07.06, Odesa, EECVC 65

DoG filter is a simple blob templatehttps://docs.opencv.org/3.4.3/da/df5/tutorial_py_sift_intro.html

Gaussian scalespace, “stack of gradually smoothed versions” of original image

Detections on synthetic image

ORB: FAST detector + BRIEF descriptor

2019.07.06, Odesa, EECVC 66

FAST is a corner detector based on segment test

Joint detectors and descriptors

SuperPoint (CVPRW 2017)DELF (ICCV 2017)D2Net (CVPR 2019)

2019.07.06, Odesa, EECVC 67

Measurement region selector

Measurement region selector

Matching

Descriptor

DescriptorDetector

Detector

Geometrical verification (RANSAC)

SuperPoint

2019.07.06, Odesa, EECVC 68DeTone et al., SuperPoint: Self-Supervised Interest Point Detection and Description CVPRW2017

DELF

2019.07.06, Odesa, EECVC 69Noh et al., SuperPoint: Large-Scale Image Retrieval with Attentive Deep Local Features ICCV 2017

“Attention” as weighting for global descriptor

D2Net

2019.07.06, Odesa, EECVC 70Dusmanu et al, D2-Net: A Trainable CNN for Joint Description and Detection of Local Features CVPR 2019

Comparison on toy example

2019.07.06, Odesa, EECVC 71

SuperPoint: 51 inliers

DoG + HardNet: 123 inliers

D2Net: 26 inliers, incorrect geometry

All things together

2019.07.06, Odesa, EECVC 72

SIFT + SNN match + OpenCV RANSAC:27 inliers

SIFT + NoOri + HardNet + FGINN union match + CMP RANSAC:179 inliers

I really need to match this

• View synthesis: MODS

2019.07.06, Odesa, EECVC 73

MODS (controller and preprocessor)

MODS handles angular viewpoint difference up to:• 85° for planar scenes • 30° for structured

D. Mishkin, J. Matas and M. Perdoch. MODS: Fast and Robust Method for Two-View Matching, CVIU 2015,

Affine view synthesis

Images

Det1-Desc1

Det2-Desc2

Match RANSACMatch!

Not match? Try more view synthesis

• If you DO NOT need correspondences & camera pose → DO NOT use local features. Use global descriptor (ResNet101 GeM) + fast search (faiss)

• Step 0: try OpenCV SIFT

• Use proper RANSAC (private now, will clean-up and open next week)

• Matching → use FGINN in two-way mode

• Need to be faster → ORB.

• Need to be more robust → use SIFT + HardNet 2.0

• Custom data → train on your own dataset

• Even more robust → use SIFT + AffNet + HardNet 2.0

• If images are upright, DO NOT DETECT the ORIENTATION

• Landmark data → DELF

2019.07.06, Odesa, EECVC 75

Thank you for your attention

ducha_aiki

ducha-aiki