Self-Calibration and Metric Reconstruction from Single

Images

Self-Calibration and Metric Reconstruction from Single

Images

Ruisheng Wang

Frank P. Ferrie

Centre for Intelligent Machines, McGill University

OutlineOutline

• Contributions

• Existing Methods

• The Idea

• Our Method

• Comparison

• Conclusion

ContributionsContributions

• We developed a direct solution to 3D reconstruction from single images which is model based

– No model-to-image projection and readjustment procedure

• We made it possible to perform accurate 3D measurement using an uncalibrated* camera based on single images only

• We quantitatively evaluated our model-based approach with vanishing point based method, and the results indicate our approach is better than vanishing point based method

* Intrinsic parameters known/estimated

Three vanishing points

Vanishing Points (VPs) Based MethodVanishing Points (VPs) Based Method

• Determine three orthogonal vanishing points

– Manual detection

– Search over Gaussian sphere

– Hough transform

– Projective geometry

• Determine camera focal length and rotation

• Determine camera translation and model dimensions

)/tan( 22 YY yxfa

)/tan( 222222 XXZZ yxfyxfa

)/tan( YY yxak

)( YXYX yyxxf

Problems in VPs Based MethodProblems in VPs Based Method

• Three orthogonal VPs may not be always available

– One or two vanishing point only

• Hard to accurately determine VPs

– Need many lines

• The accuracy of VPs affects the accuracy of the 3D reconstruction

1 Point Perspective 2 Points Perspective

Methods with Ground Control Points/LinesMethods with Ground Control Points/Lines• Point-based methods

– Collinearity Equations

• Line-based methods

– Model-to-Image Fitting

)()()(

)()()(*

)()()(

)()()(*

033032031

023022021

033032031

013012011

ZZmYYmXXm

ZZmYYmXXmfy

ZZmYYmXXm

ZZmYYmXXmfx

2 2 21 1 2 20

( ) ( ) ( )3

l T Ti

lErr h s ds h h h h m A BA m

From Debevec et al.1996

The IdeaThe Idea

• Use model to estimate camera exterior orientation

– Need 6 parameters X3, Y3, L, W, H,

• If an object-centered coordinate system selected– Need three parameters L, W, H

• Divide camera parameters into two groups: rotation and translation

• It’s possible to estimate relative camera exterior orientation without using GCPs and Vanishing Points

L

H

3

6

WX

Y

Z

5

87

4

21

a

H

W

Z

3Y

X

O

L

X3, Y3X3, Y3

a

Overview of Our ApproachOverview of Our Approach

• Self-Calibration

– Recover camera rotation • Initial estimate of camera rotation

• Refinement of camera rotation

– Determine camera translation and building dimensions

• Simultaneous estimates of camera translation and the first building dimensions

• Metric Reconstruction

– Roughly estimate the second building orientation

– Refine the second building orientation

– Determine the second building dimensions and location

• Initial estimate of camera rotation

• Refinement of camera rotation

Recover Camera RotationRecover Camera Rotation

Imaging Geometry Relationship

n

ii

Ti RvmO 2

1 )(

0RvmT

nTnn

Tnn

Tn

TTT

n vRmvRmvRm

vRmvRmvRm

J'''

1'

11'

11'

1

3

• Form objective function

• Solve a constrained

quadratic form minimization

problem

Determine Camera Translation and the First Building DimensionsDetermine Camera Translation and the First Building Dimensions

Imaging Geometry Relationship

0)( tuRmT

n

ii

Ti tuRmO 2

2 ))((

87

6

43

8

7 1

2

Building 2Model edge 56 (v, u)

Camera Coordinate System

Image edge 56{(x1, y1, -f), (x2, y2, -f)}

Object Coordinate System

Building 1Model edge 67(v, u)

5

m (mx, my, mz )C

3( X3, Y3)

4

5

6

12

z

Y

t(X0,Y0,Z0)R

11

• Assuming both buildings lie on the same ground plane

• Initial estimate of the second

building orientation

• Refinement of the second

building orientation

Recover the Second Building OrientationRecover the Second Building Orientation

Imaging Geometry Relationship

n

ii

Ti RvmO 2

1 )(

0cossin 21 WaWa Ti

Ti

Rma Ti

Ti

),,(

x

87

6

43

8

7 1

2

Building 2Model edge 56 (v, u)

Camera Coordinate System

Image edge 56{(x1, y1, -f), (x2, y2, -f)}

Object Coordinate System

Building 1Model edge 67(v, u)

5

m (mx, my, mz )C

3( X3, Y3)

4

5

6

12

z

Y

t(X0,Y0,Z0)R

11

Determine the Second Building Dimensions and LocationsDetermine the Second Building Dimensions and Locations• Unknown parameters

– Building dimensions L, W, H

– Building location X3, Y3

• Known parameters

– Camera pose

– Building orientation

• Solution

– Solve a set of linear equations

Imaging Geometry Relationship

),,(

x

n

ii

Ti tuRmO 2

2 ))((

Comparison with VP based Methods Using Identical Simulation DataComparison with VP based Methods Using Identical Simulation Data• Using same error for two methods

– Additive random noise in endpoints of image segments

– Principle points offsets

• Impact on the outputs from two methods

– Camera pose

– Geometry of the reconstructed buildings

– Topology of the reconstructed buildings

Random Errors in Image SegmentsRandom Errors in Image Segments

Sensi ti vi ty Anal ysi s of Camera Rotati on for Two Methods

05

101520253035

0 5 10 15

Random errors i n endpoi nts of i mage segmentsi n pi xel

Abso

lute

err

ors

inde

gree

Omega-MPhi -MKap-MOmega-VPhi -VKap-V

Sensi t i vi ty Anal ysi s of the Fi rst bui l di ng Di mensi ons f orTwo Methods

0

10

20

30

40

0 5 10 15

Random errors i n endpoi nts of i mage segmentsi n pi xel

Abso

lute

err

ors

inme

ter

Wi dth- MHei ght - MWi dth- VHei ght - V

Sensi t i vi ty Anal ysi s of the Second Bui l di ng Di mensi ons f orTwo Methods

0

10

20

30

40

50

0 5 10 15

Random errors i n endpoi nts of i mage segmentsi n pi xel

Abso

lute

err

ors

inme

ter

Length- MWi dth- MHei ght - MLength- VWi dth- VHei ght - V

Sensi t i vi ty Anal ysi s of Camera Transl at i on f or Two Methods

0

50

100

150

200

250

0 5 10 15

Random errors i n endpoi nts of i mage segments i npi xel

Abso

lute

err

ors

inme

ter

X- MY- MZ- MX- VY- VZ- V

Principle Point OffsetsPrinciple Point Offsets

Sensi t i vi ty Anal ysi s of Camera Transl at i on f or Two Methods

0

1

2

3

4

5

6

0 5 10 15

Pri nci pl e poi nts off sets i n pi xel

Abso

lute

err

ors

inme

ter

X- MY- MZ- MX- VY- VZ- V

Sensi ti vi ty Anal ysi s of Camera Rotati on for Two Methods

0

0. 05

0. 1

0. 15

0. 2

0. 25

0 5 10 15

Pri nci pl e poi nts off sets i n pi xel

Abso

lute

err

ors

inde

gree

Omega-MPhi -MKap-MOmega-VPhi -VKap-V

Sensi t i vi ty Anal ysi s of the Fi rst Bui l di ng Di mensi ons f orTwo Methods

00. 050. 1

0. 150. 2

0. 250. 3

0. 350. 4

0 5 10 15

Pri nci pl e poi nt off sets i n pi xel

Abso

lute

err

ors

inme

ter

Wi dth- MHei ght - MWi dth- VLength- V

Sensi t i vi ty Anal ysi s of the Second Bui l di ng Di mensi ons f orTwo Methods

0

0. 05

0. 1

0. 15

0. 2

0. 25

0 5 10 15

Pri nci pl e poi nt off sets i n pi xel

Abso

lute

err

ors

inme

ter

Length- MWi dth- MHei ght - MLength- VWi dth- VHei ght - V

Comparison Using Identical Real Data Comparison Using Identical Real Data

• Digital Camera: Canon PowerShot SD750

• Image size (3072x 2304 pixels)

Image 2: Burnside Hall, McGill universityImage 1: Two boxes

Results from the Image 1Results from the Image 1

Unit: mm Dimensions measured using a

ruler

Dimensions computed from the image

Absolute errors

Left BoxWidth

V71.1

73.1 2

M 70.9 0.2

HeightV

123.197.9 25.2

M 122.6 0.5

RightBox

LengthV

72.168.8 3.3

M 71.7 0.4

WidthV

50.647.6 3

M 49.6 1

HeightV

15.36.3 9

M 14.8 0.5

Results from the Image 2Results from the Image 2

Unit: meter Dimensions from DWG file Dimensions computed from the image

Length 35.44 35.44

Width 32.42 34.92

Height 50.00 53.33

3D model of Burnside Hall in Google Earth

Visualization of the recovered camera pose and building

ConclusionsConclusions

• Interactive solution to metric reconstruction from single images

– Model-based approach but without model-to-image projection and readjustment procedure

– better than vanishing point based methods

• Using an off-the-shelf camera, taking a picture, one can get object dimensions

Thank You!

Experimental DesignExperimental Design

• Camera Parameters

• Building Parameters

Focal Length

(m)

X0 (m) Y0(m) Z0(m) Omg (degree)

Phi ( degree)

Kap ( degree)

0.0798 -500.672 100.317 650.783 50.346 3.582 2.787

Building 1 Building 2

Length (m) 40 26.413

Width (m) 20 20.927

Height (m) 30 22.315

Orientation along X axis (degree)

0 30.856

Location of a building model vertex (m) X3 100.512

Y3 -200.217