Embed Size (px)
Citation preview
Tomas Svoboda, Daniel Martinec, and Tomas Pajdla. A convenient multi-cameraself-calibration for virtual environments. PRESENCE: Teleoperators and VirtualEnvironments, 14(4):407–422, August 2005.
Software freely available at http://cmp.felk.cvut.cz/~svoboda/SelfCal/
A Software for Complete Calibration ofMulticamera Systems
Tomas Svoboda, [email protected]
with contributions from D. Martinec, T. Pajdla, O. Chum, T. Werner,
and J. Bouguet
Czech Technical University in Prague, Center for Machine Perception
http://cmp.felk.cvut.cz
Last update: November 13, 2006
2/47Outline
Motivation
Problem definition
Proposed solution
Results
Applications
6/47Calibration
many tasks can be accomplished without knowing anything about the
cameras
however, many more when we know . . .
9/47
. . . camera internal parametersfrom geometry to pixels
100 200 300 400 500 600 700 800 900 1000
100
200
300
400
500
600
700
C
x
X
x
y
vu
[u0, v0]
10/47Camera calibration is an old problem
for photogrammetrists (even older problem)
in computer vision
many methods exist
12/47Classical approaches — plate at several positions
http://www.vision.caltech.edu/bouguetj/calib_doc/
13/47Classical methods — revisited
Pros:
many methods (and free codes)
precise, even for complicated camera models
13/47Classical methods — revisited
Pros:
many methods (and free codes)
precise, even for complicated camera models
Cons (for multicamera systems):
many cameras → hand work is not an option
large working volume to fill → big calibration objects/plates
14/47Our solution — overview
We assume at least approximately synchronized multicamera (N ≥ 3) setup.
use 1-point calibration object easily detectable in images
wave the calibration point through the working volume freely
this will create a virtual calibration object (but the 3D position
unknown!)
apply theoretical results from self-calibration field
estimate as complicated camera model as reasonable
validate the results
15/47Virtual calibration points
video
16/47Multiple cameras — Geometry
C1
C2
C3
C4
Xj
u1
ju2
j
u3
j
u4
j
P1
P2
P3
P4
Problem definition:
From uij points, for which λ
iju
ij = P
iXj holds
estimate Euclidean projection matrices Pi
and coordinates of the 3D points Xj
17/47Pinhole camera model
λij
uij
vij
1
= λiju
ij = PiXj , λi
j ∈ R+
j index points
i index camera
λij projective depths
uij point projections (we find them in images)
Xj 3D points (we do not know the positions!)
Pi camera matrices
18/47Multicamera linear model
Ws =
λ1
1
u11
v11
1
· · · λ1n
u1n
v1n
1
... ... ...
λm1
um1
vm1
1
· · · λmn
umn
vmn
1
=
P1
...Pm
3m×4︸ ︷︷ ︸
P
[X1 · · ·Xn]4×n︸ ︷︷ ︸X
Self-calibration (Euclidean stratification)
Ws = PX = PH︸︷︷︸ H−1X︸ ︷︷ ︸ = PX ,
19/47What the software does:
1. Finds the projections uij of the laser pointer in the images.
19/47What the software does:
1. Finds the projections uij of the laser pointer in the images.
2. Discards misdetected points by pairwise RANSAC analysis.
19/47What the software does:
1. Finds the projections uij of the laser pointer in the images.
2. Discards misdetected points by pairwise RANSAC analysis.
3. Estimates projective depths λij and fills the missing points to make
scaled measurement matrix Ws complete.
19/47What the software does:
1. Finds the projections uij of the laser pointer in the images.
2. Discards misdetected points by pairwise RANSAC analysis.
3. Estimates projective depths λij and fills the missing points to make
scaled measurement matrix Ws complete.
4. Performs the rank 4 factorization of the matrix Ws to get projective
shape and motion and upgrades them to Euclidean ones.
19/47What the software does:
1. Finds the projections uij of the laser pointer in the images.
2. Discards misdetected points by pairwise RANSAC analysis.
3. Estimates projective depths λij and fills the missing points to make
scaled measurement matrix Ws complete.
4. Performs the rank 4 factorization of the matrix Ws to get projective
shape and motion and upgrades them to Euclidean ones.
5. Estimates the parameters of the non-linear distortion
19/47What the software does:
1. Finds the projections uij of the laser pointer in the images.
2. Discards misdetected points by pairwise RANSAC analysis.
3. Estimates projective depths λij and fills the missing points to make
scaled measurement matrix Ws complete.
4. Performs the rank 4 factorization of the matrix Ws to get projective
shape and motion and upgrades them to Euclidean ones.
5. Estimates the parameters of the non-linear distortion
6. Optionally, if some true 3D information is known, aligns the computed
Euclidean structures with a world system.
19/47What the software does:
1. Finds the projections uij of the laser pointer in the images.
2. Discards misdetected points by pairwise RANSAC analysis.
3. Estimates projective depths λij and fills the missing points to make
scaled measurement matrix Ws complete.
4. Performs the rank 4 factorization of the matrix Ws to get projective
shape and motion and upgrades them to Euclidean ones.
5. Estimates the parameters of the non-linear distortion
6. Optionally, if some true 3D information is known, aligns the computed
Euclidean structures with a world system.
Many cross-validation steps inside.
20/47Calibration object
A very standard laser pointer with a piece of transparent plastic attached.
21/47Finding points
Needs to be a bit more clever than a simple thresholding
Statistical analysis of the images (almost) solves it.
22/47Finding points
Sub-pixel accuracy is desirable
Interpolated ROI
20 40 60
10
20
30
40
50
60
050
100
0
50
1000
0.02
0.04
Correlation coeffs
020
40
0
20
40100
150
200
250
Active ROI
020
40
0
20
400
0.5
1
1.5
x 10−3PSF approx by 2D Gaussian
/local/MultiCam/Data/20030615H
oengg/arctic1/arctic1.pvi.0001.jpg original
312 314 316 318 320 322 324 326 328
294
296
298
300
302
304
306
308
310
Around 100ms per image.
23/47Calibration input
video
24/47We know
Ws =
λ1
1
u11
v11
1
· · · λ1n
u1n
v1n
1
... ... ...
λm1
um1
vm1
1
· · · λmn
umn
vmn
1
=
P1
...Pm
3m×4
[X1 · · ·Xn]4×n
Ws = PX = PHH−1X = PX ,
However, some [uij, v
ij]> may be missing!
25/47
Estimation of λij
(Sturm & Triggs ECCV96)
uses the epipolar geometry
C1 C2
Xu1 u2l1 l2e1 e2λi
j =(eik × ui
j) · (Fikukj )
‖eik × uij‖2
λkj
26/47We know
Ws =
λ1
1
u11
v11
1
· · · λ1n
u1n
v1n
1
... ... ...
λm1
um1
vm1
1
· · · λmn
umn
vmn
1
=
P1
...Pm
3m×4
[X1 · · ·Xn]4×n
Ws = PX = PHH−1X = PX ,
However, some [uij, v
ij]> and λi
j may be missing!
28/47We know
Ws =
λ1
1
u11
v11
1
· · · λ1n
u1n
v1n
1
... ... ...
λm1
um1
vm1
1
· · · λmn
umn
vmn
1
=
P1
...Pm
3m×4
[X1 · · ·Xn]4×n
Ws = PX = PHH−1X = PX ,
29/47Rank–4 factorization
Ws =
P1
...
Pm
3m×4
[X1 · · ·Xn]4×n
So, matrix Ws should have rank at most 4
Ws = USV> P1
...
Pm
3m×4
[X1 · · ·Xn]4×n = (U√S4)(
√S4V
>)
where S4 is the S with only 4 biggest diagonal values, rest is zeroed.
30/47We know
Ws = PX = PHH−1X = PX ,
We must find a 4× 4 matrix H which upgrades the projective structures P, Xto metric ones, P, X.
31/47
Euclidean stratification(Pollefeys et al, Hartley, . . . )
based on the idea of absolute quadric (conic)
Pi = µi
[KiRi Kiti
]
PiΩ∞Pi> ∼ KiKi>
where
Ω∞ =
1 0 0 00 1 0 00 0 1 00 0 0 0
32/47Euclidean stratification cont.
absolute conic exists also in the projective world!
KiKi> ∼ (PiH−1)(HΩ∞H>)(H−>Pi>)
KiKi> ∼ PiΩ∞Pi>
We know the projective Pi. The projective Ω∞ is 4× 4 symmetric.
Once Ω∞ is known, then we can compute H from
Ω∞ = HΩ∞H>
by eigenvalue decomposition and get the sought Euclidean structures
Pi = PiH and Xj = H−1Xj.
33/47Euclidean stratification — Example of solution
assume everything is known except focal lenghts
Ki =
f i 0 ui0
0 αif i vi0
0 0 1
→ KiKi> =
f i2 0 00 f i2 00 0 1
Remember that KiKi> ∼ PiΩ∞Pi>
(PiΩ∞Pi>)11 − (PiΩ∞Pi>)22 = 0
(PiΩ∞Pi>)12 = 0
(PiΩ∞Pi>)13 = 0
(PiΩ∞Pi>)23 = 0
Each camera contributes by 4 contraints.
34/47We have the metric linear model
Ws = PX
Estimation of non-linear distortion starts from
Xj ↔ uij
correspondences. We use the CalTech package
http://www.vision.caltech.edu/bouguetj/calib_doc/
Then it goes back, adapt parameters and . . .
35/47Aligning the results with the world
−2
0
2
−6
−4
−2
0
2
−6
−4
−2
0
2
4
10
11
13
1
9
2
8
12
reconstructed points/camera setup only inliers are used
3
16
15
14
7
4
6
5
−3−2
−10
12
3
−2
−1
0
1
2
3
4
10
9
8
7
6
11
16
Graphical Output Validation: View from the top camera
12
13
15
14
1
2
3
4
5
User provides some 3D information. Example: “Cameras No. 11,13,15
define the xy plane”.
36/47Results — Filling points
0 100 200 300 400 500 600 700
0
100
200
300
400
500
measured, o, vs reprojected, +, 2D points (camera: 4)
−400 −200 0 200 400 600 800 1000 1200 1400 1600−500
0
500
1000
measured, o, vs reprojected, +, 2D points (camera: 40)
The calibration “point” needs not to be visible in all cameras!
37/47Results — Calibrated setups
−3 −2 −1 0 1 2 3
−2
−1
0
1
2
3
10
9
8
7
6
11
16
Graphical Output Validation: View from the top camera
12
15
13
14
5
3
4
2
1−1 0 1
−2
−1
0
1
2
3 71
70
80
Graphical Output Validation: Aligned data
72
81
82
38/47Results — Linear model
0 100 200 300 400 500 600
0
100
200
300
400
500measured, o, vs reprojected, +, 2D points (camera: 12)
0 2 4 6 8 10 12 14 16 180
1
2
3
4
5
6
7
Id of the camera
2D error: mean (blue), std (red)
pixe
ls
39/47Results — Complete model
0 100 200 300 400 500 600 700
0
100
200
300
400
500
measured, o, vs reprojected, +, 2D points (camera: 12)
0 2 4 6 8 10 12 14 16 180
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
Id of the camera
2D error: mean (blue), std (red)
pixe
ls
Very fine results from (almost) nothing!
40/47Results — Simple setup
−3−2−101
−1
0
1
2
3
4
5
6
4
reconstructed points/camera setup only inliers are used 3
2
2 3 40
0.05
0.1
0.15
0.2
0.25
0.3
0.35
Id of the camera
2D error: mean (blue), std (red)
pixe
ls
41/47Application example — volumetric reconstruction
video
I know, it is just toy example. Still, it shows that the metric is OK.
42/47Application example — mobile multicamera setup
video
43/47Mobile multicamera setup - worker 3D tracking
video
45/47Summary
waving the point object is the only hand work required
no user interaction
complete calibration of 16 camera setup may be done in 60-90 minutes
(95% computation)
Codes, sample data, papers, etc. downloadable from
http://cmp.felk.cvut.cz/~svoboda/SelfCal
46/47References
More details can be found at [1]
[1] Tomas Svoboda, Daniel Martinec, and Tomas Pajdla. A convenient
multi-camera self-calibration for virtual environments. PRESENCE:
Teleoperators and Virtual Environments, 14(4):407–422, August 2005.
100 200 300 400 500 600 700 800 900 1000
100
200
300
400
500
600
700
C
x
X
x
y
vu
[u0, v0]
C1
C2
C3
C4
Xj
u1
ju2
j
u3
j
u4
j
P1
P2
P3
P4
Problem definition:
From uij points, for which λ
iju
ij = P
iXj holds
estimate Euclidean projection matrices Pi
and coordinates of the 3D points Xj
Interpolated ROI
20 40 60
10
20
30
40
50
60
050
100
0
50
1000
0.02
0.04
Correlation coeffs
020
40
0
20
40100
150
200
250
Active ROI
020
40
0
20
400
0.5
1
1.5
x 10−3PSF approx by 2D Gaussian
/local/MultiCam/Data/20030615H
oengg/arctic1/arctic1.pvi.0001.jpg original
312 314 316 318 320 322 324 326 328
294
296
298
300
302
304
306
308
310
C1 C2
Xu1 u2l1 l2e1 e2
−2
0
2
−6
−4
−2
0
2
−6
−4
−2
0
2
4
10
11
13
1
9
2
8
12
reconstructed points/camera setup only inliers are used
3
16
15
14
7
4
6
5
−3−2
−10
12
3
−2
−1
0
1
2
3
4
10
9
8
7
6
11
16
Graphical Output Validation: View from the top camera
12
13
15
14
1
2
3
4
5
0 100 200 300 400 500 600 700
0
100
200
300
400
500
measured, o, vs reprojected, +, 2D points (camera: 4)
−400 −200 0 200 400 600 800 1000 1200 1400 1600−500
0
500
1000
measured, o, vs reprojected, +, 2D points (camera: 40)
−3 −2 −1 0 1 2 3
−2
−1
0
1
2
3
10
9
8
7
6
11
16
Graphical Output Validation: View from the top camera
12
15
13
14
5
3
4
2
1
−1 0 1
−2
−1
0
1
2
3 71
70
80
Graphical Output Validation: Aligned data
72
81
82
0 100 200 300 400 500 600
0
100
200
300
400
500measured, o, vs reprojected, +, 2D points (camera: 12)
0 2 4 6 8 10 12 14 16 180
1
2
3
4
5
6
7
Id of the camera
2D error: mean (blue), std (red)
pixe
ls
0 100 200 300 400 500 600 700
0
100
200
300
400
500
measured, o, vs reprojected, +, 2D points (camera: 12)
0 2 4 6 8 10 12 14 16 180
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
Id of the camera
2D error: mean (blue), std (red)
pixe
ls
−3−2−101
−1
0
1
2
3
4
5
6
4
reconstructed points/camera setup only inliers are used 3
2
2 3 40
0.05
0.1
0.15
0.2
0.25
0.3
0.35
Id of the camera
2D error: mean (blue), std (red)
pixe
ls
![Kanade–Lucas–Tomasi Tracking (KLT tracker)cmp.felk.cvut.cz/cmp/courses/XE33PVR/WS20072008/Lectures/Motion/klt.pdf · [4] Jianbo Shi and Carlo Tomasi. Good features to track. In](https://img.pdfslide.us/doc/110x75/5e5efb19e4419c5b9e1b842c/kanadealucasatomasi-tracking-klt-trackercmpfelkcvutczcmpcoursesxe33pvrws20072008lecturesmotionkltpdf.jpg){kind=tracking}
