Change Detection of 3D Scene with 3D and 2D Information for Environment Checking

1

Change Detection of 3D Scene with 3D and 2D Information for

Environment Checking

PhD Candidate: Baowei Lin August 12th, 2013

1. Introduction Research Motivation Change Detection

2. 3D Keypoints Based 3D-2D Matching Background 3D Keypoints Detection Evaluation

3. 3D-2D Based Change Detection Background Image Based Change Detection Evaluation

4. 3D-3D Based Change Detection Background Scale Estimation of a Single Point Cloud Scale Ratio Estimation of Two Point Clouds Evaluation

5. Conclusions

2





5. Conclusions

3

4

Changes should be alerted at these areas.

5

Original configuration

Damaged configuration

wave washing

if changed

dangerous

6

• Impossible to check manually Wide range Huge number of blocks

• Important to check automatically

7

• Impractical to check by fixed cameras

8

• possible to check by hand-held devices

9

Finding potential change area.

Sub-goal 1:

10

Estimating accurate changes.

Sub-goal 2: offline

Finding potential change area. online

Sub-goal 1:





5. Conclusions

11

A change is a difference of objects in the scene at time A and at time B.

12

Time A Time B

13

3D point cloud

Training images (2D images)

1. 2D-2D Method

14

Need Fixed camera input

output

Original image

Change image

Changed area

2D-2D

input

output

2. 3D-2D Method

15

Detection is fast but not accurate

Original point cloud

Change image

Changed area

3D-2D

input

output

3. 3D-3D Method

16

Detection is accurate but slow


Change point cloud

Changed area

3D-3D

17

Potential changed areas 3D information

Camera poses

Online system Offline system

3D-2D based change detection

3D-2D based camera pose estimation 3D-3D based

change detection

Chapter 2 Chapter 3 Chapter 4

3D-3D 3D-2D





5. Conclusions

18

19

3D interesting points

2D interesting points

Camera pose=[R,t]

3D point cloud

2D training images

20

2D-2D

3D-3D

SIFT[Lowe 2004], SURF [Bay 2006], etc.

spin image[Johnson 1998], NARF[Steder 2010], etc.

detector descriptor

detector descriptor

21

3D point cloud

2D training images

3D detector and 2D detector can not be corresponded.

22

Image patch

Point distribution

Can not match

2D image

3D point cloud

23

• Detect keypoints correctly

• Describe keypoints appropriately

2D image

3D point cloud

Introduction ◦ Research Motivation ◦ Change Detection

3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation

3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation

3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation

Conclusions

24

Detected 2D interesting points

25

Feature matching

Obviously, the SIFT features could be used in 3D keypoints detection and description.

Point cloud

26

P1

P2

P3 P4

P5

P6 Camera position

3D keypoint

Projected 3D points

2D images number threshold used for 3D keypoints decision.

the points which can appear on multiple training images

Back face points are not used for computation

3D keypoints

th_v

27

th_v = 1 #3D keypoints ≅10,000

27 training images 105,779 3D points

th_v = 7 #3D keypoints ≅ 1,000

Reconstructed 3D points #3D points ≅30,000

Too many for real time calculating

Smaller number and good distribution

ours

orig

inal

28

2D SIFT keypoints and descriptors

3

3D keypoint& descriptor

Projected 3D points should overlapped to 2D SIFT keypoint

-Keep all 2D descriptors Accurate but slow

Description methods: -Average and Median

SIFT features are different when view directions are different.





Conclusions

29

30

P1 P2 P3 P4 P5 P6

3D point cloud

Ground truth Camera

positions

……

Training images

31

P1 P2 P3 P4 P5 P6

P6’

Camera pose estimation

3D keypoints generation

32

P1

P2

P3 P4

P5

P6

P6’

P6’ =[R ’ |t ’]

P6 =[R | t ]

Tra

nsla

tion e

rror

Rota

tion e

rror[ra

d]

33

1. Our method is accurate 2. th_v does not affect the result

th_v is used for 3D keypoints selection

2 degrees Dataset: 27 training images Image resolution:2256x1504 3D points number:105,779 3D scene size:40x25x5cm Bounding box size:10.6x5.7x1.4

0.24cm

34

3D point cloud Query image

project 3D points

35





Conclusions

36

37


Camera poses



3D-2D based camera pose estimation


38

Our method: 1. Use local feature instead of color 2. Detect any shape of object

Using laser range finder [Goncalves 2010,

Ryle 2011 and Neuman

2011].

Not for wide area targets.

Not applicable for our round shape or natural scenes.

Matching 3D line segments [Eden 2008].

Using color differences [Sato

2006, Pollard 2007 and Taneja 2011]

Not stable for illumination changes.





Conclusions

39

40

36

1. Find the nearest image

Query image

Nearest image

2. Find changed area Nearest image Query image

changed area

3. Visualization

Project 3D points onto changed area

1st Nearest Query image

41

P1 P2 P3 P4 P5

3D keypoints generation

Need fixed camera

Smallest distance

Ground truth

……

Training images 2nd 3rd

P

42

the 1st nearest image

Query image

Points: 2D keypoints

Blue: correspondence

Red: no correspondence

Blue: correspondence

Red: no correspondence

Non-change area

Uncovered area is the changed area

Estimated changed area

3D point cloud

Visualized 3D points 43

change area

projection

Detected result

Projected 3D points





Conclusions

44

3D point cloud Query image

45

Quantitative results visualization

Changed 3D points Changed area

Results for different thresholds 46

Set as:0, 5, 10, 20, 30, 50, 70 and 90 pixels

0 5 10 20

30 50 70 90

Image resolution:2256x1504 The number of Image: 54 The number of 3D points: 190,845

TP rate= True Positive Ground Positive

FP rate= Ground Negative

False Positive

47

Receiver operating characteristic (ROC) plot

threshold = 30

threshold = 30

Ground truth is set manually

We expect the 1st nearest image perform better than others, but the best result is the 2nd nearest image.

Good performance

Bad performance

48 It is the parameter left for users.

1st

2nd

Query image Detection results





Conclusions

49

50


Camera poses





Different size scale because of the character of Structure-from-Motion (SfM)

51

3D-3D registration is actually, the scale registration

3D point cloud 3D point cloud

3D point cloud

Change points

registration

Point clouds of same scene with different size

3D point cloud 3D point cloud

52

Iterative closest point (ICP) based alignment [Besl 1991].

-Need simple scenes -Need initial pose and scale -Not robust to clutters, occlusions and missing part

spin images [Johnson 1998],

NARF [Steder 2010], shape context [Belongie 2002], etc.

Feature based alignment

-Need appropriate neighborhood size

3D SIFT [Scovanner 2007], 3D SURF [Knopp 2010], etc.

-Not robust to clutters, occlusions and missing part

Easy data

Different data

Fixed scale

Adaptive scale

53

1. Scale estimation

2. Scale Ratio estimation

Keyscale1=0.5 Keyscale2=0.1

Scale ratio=5





Conclusions

54

Bunny point cloud

Width=0.001

55

Similar to each other

Different to each other

Width=0.1

Width=1.0

3D keypoints

Spin images

the minimum of similarity between spin images when the width changes.


Keyscale

Robust to clutters, occlusions and missing part

Calculate similarity of collected spin images

56

Decide which set of spin images are different to each other by using Contribution rate.

PCA Robust to order of extracted spin images.

Robust to detail

57 minimum

1 5 10 15

sim

ilarity

sim

ilarity

d

w


Different


minimum is not unique Finding them is not stable

58

minimum minimum

sim

ilarity

w

59

Bunny point cloud

Finding minimum is not stable

sim

ilarity

w





Conclusions

60

Point clouds

61

Register two plots to get scale ratio

Scale ratio ICP

similarity plots

Overlapping parts

Original bunny curves 5 times larger bunny curves

62

Original bunny curves

5 times larger bunny curves

63

Scale ratio t

Displaced Original bunny curves

5 times larger bunny curves

64

65

Similarity estimation

3D registration

Scale

ratio

estim

atio

n

input

Similarity plots

Scale ratio

alignment





Conclusions

66

67


The number of points:207,583 Scene size: 35m x 5m

overlapping area

68

Created 1st point cloud

Created 2nd point cloud

estimate scale ratio


69

The method provides perfect result when the overlap rate is larger than 70%.

Ground truth = 1

Small blocks point clouds

70

Changed block





Conclusions

71

We have proposed three methods for a surveillance system to detect change.

2. Online 3D-2D based change detection

3. Offline 3D-3D based change detection

1. 3D-2D matching

In future: Find a more systematic way for choosing parameters. Improve computation time.

In future: Find the nearest image by considering FOVs. Implement the method on mobile devices.

In future: Accelerate the computation in order to handle much larger number of points.

72


Camera poses





73

Future work:

1. Improve computation speed and detection accuracy for online system. -current computation time: 20 seconds per image

2. Optimize algorithm to operate with huge size data for offline system. -current computation time: 10 minutes for 100,000 points

Education

Change Detection of 3D Scene with 3D and 2D Information for Environment Checking