Imaging Beyond the Pinhole Camera

Volume 33

Computational Imaging and Vision

Managing Editor

MAX VIERGEVER

Utrecht University, The Netherlands Series Editors

GUNILLA BORGEFORS, Centre for Image Analysis, SLU, Uppsala, Sweden RACHID DERICHE, INRIA, France THOMAS S. HUANG, University of Illinois, Urbana, USA

TIANZI JIANG, Institute of Automation, CAS, Beijing REINHARD KLETTE, University of Auckland, New Zealand ALES LEONARDIS, ViCoS, University of Ljubljana, Slovenia HEINZ-OTTO PEITGEN, CeVis, Bremen, Germany

Imaging Systems and Image Processing

Computer Vision and Image Understanding

Visualization

This comprehensive book series embraces state-of-the-art expository works and advanced

research monographs on any aspect of this interdisciplinary field.

Only monographs or multi-authored books that have a distinct subject area, that is where

series. each chapter has been invited in order to fulfill this purpose, will be considered for the

•••• Applications of Imaging Technologies

Topics covered by the series fall in the following four main categories:

KATSUSHI IKEUCHI, Tokyo University, Japan

Imaging Beyond the Pinhole

Camera

Edited by

Kostas Daniilidis

University of Pennsylvania, Philadelphia, PA, U.S.A.

and

Reinhard Klette

The University of Auckland, New Zealand

A C.I.P. Catalogue record for this book is available from the Library of Congress.

ISBN-10 1-4020-4893-9 (HB)

ISBN-13 978-1-4020-4893-7 (HB)

ISBN-10 1-4020-4894-7 (e-book)

ISBN-13 978-1-4020-4894-4 (e-book)

Published by Springer,

P.O. Box 17, 3300 AA Dordrecht, The Netherlands.

www.springer.com

Printed on acid-free paper

All Rights Reserved

© 2006 Springer

No part of this work may be reproduced, stored in a retrieval system, or transmitted

in any form or by any means, electronic, mechanical, photocopying, microfilming, recording

or otherwise, without written permission from the Publisher, with the exception

of any material supplied specifically for the purpose of being entered

and executed on a computer system, for exclusive use by the purchaser of the work.

Contents

Contributors

Preface

I Sensor Geometry 1

A. Torii, A. Sugimoto, T. Sakai, and A. Imiya/Geometry of aClass of Catadiopric Systems 3

and Dioptric Cameras 21

Caustic Surface of Catadioptric Non-Central Sensors 39

F. Huang, S.-K. Wei, and R. Klette/Calibration of Line-basedPanoramic Cameras 55

II Motion 85

P. Sturm, S. Ramalingam, and S. Lodha/On Calibration,Structure from Motion and Multi-View Geometry for GenericCamera Models 87

R. Molana and Ch. Geyer/Motion Estimation with Essentialand Generalized Essential Matrices 107

R. Vidal/Segmentation of Dynamic Scenes Taken by a MovingCentral Panoramic Camera 125

A. Imiya, A. Torii, and H. Sugaya/Optical Flow Computationof Omni-Directional Images 143

III Mapping 163

R. Reulke, A. Wehr, and D. Griesbach/ Mobile Panoramic

165

vii

xi

S.-H. Ieng and R. Benosman/Geometric Construction of the

J. P. Barreto/ Unifying Image Plane Liftings for Central Catadioptric

Integrated Position and Orientation SystemMapping Using CCD-Line Camera and Laser Scanner with

CONTENTSvi

K. Scheibe and R. Klette/ Multi-Sensor Panorama Fusionand Visualization 185

A. Koschan, J.-C. Ng, and M. Abidi/ Multi-Perspective MosaicsFor Inspection and Visualization 207

IV Navigation 227

K.E. Bekris, A.A. Argyros, and L.E. Kavraki/ ExploitingPanoramic Vision for Bearing-Only Robot Homing 229

A. Makadia/Correspondenceless Visual Navigation UnderConstrained Motion 253

S.S. Beauchemin, M.T. Kotb, and H.O. Hamshari/ Navigationand Gravitation 269

V Sensors and Other Modalities 283

E. Angelopoulou/Beyond Trichromatic Imaging 285

T. Matsuyama/Ubiquitous and Wearable Vision Systems 307

J. Barron/3D Optical Flow in Gated MRI Cardiac Datasets 331

R. Pless/ Imaging Through Time: The advantages of sitting still 345

Index 365

Contributors

Mongi AbidiThe Imaging, Robotics, and Intelligent Systems LaboratoryThe University of Tennessee, Knoxville, 334 Ferris HallKnoxville, TN 37996-2100, USA

Elli AngelopoulouStevens Institute of TechnologyDepartment of Computer ScienceCastle Point on HudsonHoboken, NJ 07030, USA

Antonis A. ArgyrosInstitute of Computer ScienceFORTH Vassilika Vouton, P.O. Box 1385GR-711-10, Heraklion, Crete, Greece

Joao P. BarretoInstitute of Systems and RoboticsDepartment of Electrical and Computer EngineeringFaculty of Sciences and Technology of the University of Coimbra3030 Coimbra, Portugal

John BarronDepartment of Computer ScienceUniversity of Western OntarioLondon, Ontario, Canada, N6A 5B7

Stephen S. BeaucheminDepartment of Computer ScienceUniversity of Western OntarioLondon, Ontario, Canada, N6A 5B7

Kostas E. BekrisComputer Science Department, Rice UniversityHouston, TX, 77005, USA

Ryad BenosmanUniversity of Pierre and Marie Curie4 place Jussieu 75252 Paris cedex 05, France

Kostas DaniilidisGRASP Laboratory, University of PennsylvaniaPhiladelphia, PA 19104, USA

vii

CONTRIBUTORS

Christopher GeyerUniversity of California, Berkeley, USA

D. GriesbachGerman Aerospace Center DLR, Competence CenterBerlin, Germany

H. O. HamshariDepartment of Computer ScienceUniversity of Western OntarioLondon, Ontario, Canada, N6A 5B7

Fay Huang

Atsushi ImiyaInstitute of Media and Information TechnologyChiba University, Chiba 263-8522, Japan

Lydia E. KavrakiComputer Science Department, Rice UniversityHouston, TX, 77005, USA

Reinhard Klette

The University of AucklandAuckland, New Zealand

Andreas KoschanThe Imaging, Robotics, and Intelligent Systems LaboratoryThe University of Tennessee, Knoxville, 334 Ferris HallKnoxville, TN 37996-2100, USA

M. T. KotbDepartment of Computer ScienceUniversity of Western OntarioLondon, Ontario, Canada, N6A 5B7

viii

Department of Computer Science and CITR

Sio-hoi IengUniversity of Pierre and Marie Curie4 place Jussieu 75252, Paris cedex 05, andLab. of Complex Systems Control, Analysis and Comm.E.C.E, 53 rue de Grenelles, 75007 Paris, France

Electronic Engineering DepartmentNational Ilan UniversityI-Lan, Taiwan

CONTRIBUTORS

Suresh LodhaDepartment of Computer ScienceUniversity of California, Santa Cruz, USA

Ameesh MakadiaGRASP LaboratoryDepartment of Computer and Information ScienceUniversity of Pennsylvania

Takashi MatsuyamaGraduate School of Informatics, Kyoto UniversitySakyo, Kyoto, 606-8501, Japan

Rana MolanaUniversity of Pennsylvania, USA

Jin-Choon NgThe Imaging, Robotics, and Intelligent Systems LaboratoryThe University of Tennessee, Knoxville, 334 Ferris HallKnoxville, TN 37996-2100, USA

Robert PlessDepartment of Computer Science and EngineeringWashington University in St. Louis, USA

Srikumar RamalingamDepartment of Computer ScienceUniversity of California, Santa Cruz, USA

Ralf ReulkeHumboldt University BerlinInstitute for Informatics, Computer VisionBerlin, Germany

Tomoya SakaiInstitute of Media and Information TechnologyChiba University, Chiba 263-8522, Japan

Karsten ScheibeOptical Information SystemsGerman Aerospace Center (DLR)Rutherfordstr. 2, D-12489 Berlin, Germany

ix

Peter SturmINRIA Rhone-Alpes655 Avenue de l’Europe, 38330 Montbonnot, France

CONTRIBUTORS

Akihiro SugimotoNational Institute of InformaticsTokyo 101-8430, Japan

Akihiko ToriiSchool of Science and TechnologyChiba UniversityYayoi-cho 1-33, Inage-ku, Chiba 263-8522, Japan

Rene Vidal

Johns Hopkins University308B Clark Hall, 3400 N. Charles StreetBaltimore MD 21218, USA

A. WehrInstitute for Navigation, University of StuttgartStuttgart, Germany

Shou-Kang WeiPresentation and Network Video DivisionAVerMedia Technologies, Inc.Taipei, Taiwan

x

Center for Imaging Science, Department of Biomedical Engineering

Hironobu SugayaSchool of Science and TechnologyChiba University, Chiba 263-8522, Japan

Preface

I hate cameras. They are so much more sure than I am about every-thing.”

John Steinbeck (1902 - 1968)

The world’s first photograph was taken by Joseph Nicephore Niepce(1775–1833) in 1826 on his country estate near Chalon-sur-Saone, France.The photo shows parts of farm buildings and some sky. Exposure time waseight hours. Niepce used a pinhole camera, known as camera obscura, andutilized pewter plates as the support medium for the photographic process.The camera obscura, the basic projection model of pinhole cameras, wasfirst reported by the Chinese philosopher Mo-Ti (5th century BC): lightrays passing through a pinhole into a darkened room create an upside-downimage of the outside world.

Cameras used since Niepce are basically following the pinhole cameraprinciple. The quality of projected images improved due to progress inoptical lenses and silver-based film, the latter one replaced today by digitaltechnologies. Pinhole-type cameras are still the dominating brands, and alsoused in computer vision for understanding 3D scenes based on capturedimages or videos.

However, different applications have pushed for designing alternativearchitectures of cameras. For example, in photogrammetry cameras areinstalled in planes or satellites, and a continuous stream of image data canalso be created by capturing images just line by line, one line at a time. As asecond example, robots require to comprehend a scene in full 360◦ to be ableto react to obstacles or events; a camera looking upward into a parabolic orhyperbolic mirror allows this type of omnidirectional viewing. The devel-opment of alternative camera architectures also requires understanding re-lated projective geometries for the purpose of camera calibration, binocularstereo, or static or dynamic scene comprehension.

This book reports about contributions given at a workshop at the inter-national computer science center in Dagstuhl (Germany) addressing basicsand applications of alternative camera technologies, in particular in thecontext of computer vision, computer graphics, visualisation centers, cam-era producers, or application areas such as remote sensing, surveillance,ambient intelligence, satellite or super-high resolution imaging. Examples

“

xi

PREFACE

of subjects are geometry and image processing on plenoptic modalities,multiperspective image acquisition, panoramic imaging, plenoptic samplingand editing, new camera technologies and related theoretical issues.

The book is structured into five parts, each containing three or fourchapters on (1) sensor geometry for different camera architectures, alsoadressing calibration, (2) applications of non-pinhole cameras for analyzingmotion, (3) mapping of 3D scenes into 3D models, (4) navigation of robotsusing new camera technologies, and (5) on specialized aspects of new sensorsand other modalities.

The success of this workshop at Dagstuhl is also due to the outstandingquality of the provided facilities and services at this centre, supporting arelaxed and focused academic atmosphere.

Kostas DaniilidisReinhard Klette

Philadelphia and Auckland, February 2006

xii

Part I

Sensor Geometry

AKIHIKO TORIISchool of Science and TechnologyChiba University, Chiba 263-8522, Japan

AKIHIRO SUGIMOTONational Institute of InformaticsTokyo 101-8430, Japan

TOMOYA SAKAIInstitute of Media and Information Technology

ATSUSHI IMIYAInstitute of Media and Information TechnologyChiba University, Chiba 263-8522, Japan

as images on a quadric surface which is determined by a mirror of the system. In thispaper, we propose a unified theory for the transformation from images observed bycatadioptric systems to images on a sphere. Images on a sphere are functions on aRiemannian manifold with the positive constant curvature. Mathematically, sphericalimages have similar analytical and geometrical properties with images on a plane. Thismathematical property leads to the conclusion that spherical image analysis providesa unified approach for the analysis of images observed through a catadioptric systemwith a quadric mirror. Therefore, the transformation of images observed by the systemswith a quadric mirror to spherical images is a fundamental tool for image understandingof omnidirectional images. We show that the transformation of omnidirectional imagesto spherical images is mathematically a point-to-point transformation among quadricsurfaces. This geometrical property comes from the fact that the intersection of a doublecone in a four-dimensional Euclidean space and a three-dimensional linear manifold yieldsa surface of revolution employed as a mirror for the catadioptric imaging system with aquadric mirror.

camera model, spherical images

3

Chiba University, Chiba 263-8522, Japan

GEOMETRY OF A CLASS OF CATADIOPRIC SYSTEMS

Abstract. Images observed by a catadioptric system with a quadric mirror are considered

Key words: geometries of catadioptric cameras, central and non-central cameras, spherical

K. Daniilidis and R. Klette (eds.), Imaging Beyond the Pinhole Camera, 3–20.

© 2006 Springer.

4

1.

In this paper, we propose a unified theory for the transformation from im-

tric images, to images on a sphere, say spherical images. The transformedspherical images are functions on a Riemannian manifold with the positiveconstant curvature. Mathematically, spherical images have similar analyti-cal and geometrical properties with images on a plane. For the developmentof new algorithms in the computer vision, we analyze the spherical images.The spherical image analysis provides a unified approach for the analysisof catadioptric images. Therefore, the transformation of images observedby the systems with a quadric mirror to spherical images is a fundamentaltool for image understanding of omnidirectional images.

In the computer-vision communities, traditional algorithms and theirapplications are developed based on the pinhole-camera systems. An idealpinhole camera has no limitation of the region of images. However, theactual camera practically has limitation of the region of images. The pin-hole camera can observe objects in the finite region. Therefore, the estab-lished algorithms employing sequential and multi-view images implicitlyyield the restriction, that is, the observed images share a common re-gion in a space. For the construction of practical systems applying thecomputer vision methods, this implicit restriction yields the geometricalconfiguration among cameras, objects, and scenes. If the camera systemspractically observe the omnidirectional region in a space, this geometricalconfiguration problem are solved. Furthermore, the omnidirectional camerasystems enable us to notate the simple and clear algorithms for the multipleview geometry (Svoboda et al., 1998; Dahmen, 2001), ego-motion analysis

For the generation of the image which practically express the omni-directional scenes in a space, the camera system must project the sceneon a sphere (ellipsoid). The construction of the camera system, which em-ploys the geometrical configuration of CCD sensors and traditional lenses,is still impractical. Consequently, some researchers developed the camerasystem constructed by the combination of a quadric-shaped mirror and ageneral pinhole camera (Nayar, 1997; Baker and Nayar, 1998). Since thiscatadioptric camera system generates the image on a plane collecting thereflected rays from the mirror, the back-projection of this planar imageenables us to transform to the images on the quadric surface as describedin Section 2. Furthermore, all the quadric images are geometrically con-verged to the spherical images as described in Section 3. The applicationof the spherical camera systems enables us to develop unified algorithmsfor the different types of catadioptric camera systems. Moreover, one of

Introduction

A. TORII, et al.

(Dahmen, 2001; Vassallo et al., 2002; Makadia and Daniilidis, 2003), et al.

ages observed by catadioptric systems with a quadric mirror, say catadiop-

GEOMETRY OF A CLASS OF CATADIOPTRIC SYSTEMS 5

the fundamental problems for the omnidirectional camera system is thevisualization of numerical results computed using computer vision and im-age processing techniques such as optical flow and snakes. The transformof sphere is historically well-studied in the field of the map projections(Berger, 1987; Pearson, 1990; Yang et al., 2000). The techniques of themap projections enable us to transform the computational results on asphere preserving the specific features such as angles, areas, distances, andtheir combinations.

It is possible to develop algorithms on the back-projected quadric sur-faces (Daniilidis et al., 2002). However, the algorithms depend on the shapesof the quadric mirror. For the development of unified omnidirectional imageanalysis, a unified notation of the catadioptric and dioptric cameras areproposed (Barreto and Daniilidis, 2004; Ying and Hu, 2004; Corrochanoand Fraco, 2004). In this study, we propose a unified formula for thetransformation of omnidirectional images to spherical images, say quadric-to-spherical image transform. Our unified formulas enable us to transformdifferent kinds of omnidirectional images observed by catadioptric camera

transformation among quadric surfaces. This geometrical property comesfrom the fact that the intersection of a double cone in a four-dimensionalEuclidean space and a three-dimensional linear manifold yields a surface ofrevolution employed as a mirror for the catadioptric imaging system witha quadric mirror.

Furthermore, the traditional computer vision techniques are developedon a planar images where the curvature always equals to zero. The new com-puter vision techniques for the catadioptric camera systems are required todevelop the image analysis methodology on the quadric surfaces (Makadiaand Daniilidis, 2003), where the curvature is not zero, since the combinationof a pin-hole camera and a quadric mirror provides the omnidirectional im-ages. The geometrical analysis of the catadioptric camera system leads thatthe planar omnidirectional image is identically transformed to the imageon the quadric surface. For the first step of our study on omnidirectionalsystems, we develop the algorithms to image analysis on the sphere wherethe curvature is always positive and constant.

2. Spherical Camera Model

As illustrated in Figure 1, the center C of the spherical camera is locatedat the origin of the world coordinate system. The spherical imaging surfaceis expressed as

S : x2 + y2 + z2 = r2, (1)

systems to the spherical images. We show that the transformation of omni-directional images to spherical images is mathematically a point-to-point

6

Figure 1. Spherical-camera model.

where r is the radius of the sphere. The spherical camera projects a pointX = (X, Y, Z)� to the point x = (x, y, z)� on S according to the formula-tion,

x = rX

|X| . (2)

The spherical coordinate system expresses a point x = (x, y, z) on thesphere as ⎛⎝ x

yz

⎞⎠ =

⎛⎝ r cos θ sin ϕr sin θ sin ϕ

r cos ϕ

⎞⎠, (3)

where 0 ≤ θ < 2π and 0 ≤ ϕ < π. Hereafter, we assume r = 1. Therefore,the spherical image is also expressed as I(θ, ϕ).

3. Catadioptric-to-Spherical Transform

As illustrated in Figure 2, a catadioptric camera system generates an imagefollowing the two step. A point X ∈ R3 is transformed to a point x ∈ C2

by nonlinear function f :f : X → x. (4)

The point x ∈ C2 is projected by a pinhole or orthogonal camera to a pointm ∈ R2.

P : x → m. (5)

We assume that the parameter of the catadioptric camera system is known.As illustrated in Figure 3, locating the center of a spherical camera at the

A. TORII, et al.

GEOMETRY OF A CLASS OF CATADIOPTRIC SYSTEMS 7

Figure 2. Transform of a point in space to a point on a quadric mirror.

Figure 3. Transform of a point on a quadric mirror to a point on a unit sphere.

focal point of the quadric surface, a nonlinear function transform g a pointξ ∈ S2 on the unit sphere to the point x ∈ C2:

g : ξ → x. (6)

This nonlinear function is the catadioptric-to-spherical (CTS) transform.

3.1. HYPERBOLIC(PARABOLIC)-TO-SPHERICAL IMAGE TRANSFORM

In this section, we describe the practical image transform. We assume thatall the parameters of catadioptric camera system are known. As illustratedin Figure 4(a), the focus of the hyperboloid (paraboloid) C2 is located

8

at the point F = (0, 0, 0)�. The center of the pinhole camera is locatedat the point C = (0, 0,−2e) (C = (0, 0,−∞)). The hyperbolic(parabolic)-camera axis l is the line which connects C and F . We set the hyperboloid(paraboloid) C2 :

x�Ax = (x, y, z, 1)

⎛⎜⎜⎝1a2 0 0 00 1

a2 0 00 0 − 1

b2− e

b2

0 0 − eb2

− e2

b2+ 1

⎞⎟⎟⎠⎛⎜⎜⎝

xyz1

⎞⎟⎟⎠ = 0. (7)

(x�Ax = (x, y, z, 1)

⎛⎜⎜⎝14c 0 0 00 1

4c 0 00 0 0 −10 0 −1 −1

⎞⎟⎟⎠⎛⎜⎜⎝

xyz1

⎞⎟⎟⎠ = 0). (8)

Figure 4. Transformation among hyperbolic- and spherical-camera systems. (a) illus-trates a hyperbolic-camera system. The camera C generates the omnidirectional image πby the central projection, since all the rays collected to the focal point F are reflected tothe single point. A point X in a space is transformed to the point x on the hyperboloidand x is transformed to the point m on image plane. (b) illustrate the geometrical config-uration of hyperbolic- and spherical-camera systems. In this geometrical configuration, apoint ξ on the spherical image and a point x on the hyperboloid lie on a line connectinga point X in a space and the focal point F of the hyperboloid.

A. TORII, et al.

GEOMETRY OF A CLASS OF CATADIOPTRIC SYSTEMS 9

where e =√

a2 + b2 (c is the parameter of the paraboloid). We set a pointX = (X, Y, Z)� in a space, a point on the hyperboloid (paraboloid) C2, andm = (u, v)� on the image plane π. The nonlinear transform in Equation(4) is expressed as:

x = χX, (9)

where

χ =±a2

b|X| ∓ eZ(χ =

2c

|X| − Z). (10)

The projection in Equation (5) is expressed as:

(m1

)=

1z + 2e

⎛⎝ f 0 0 00 f 0 00 0 1 0

⎞⎠ (x1

)(11)

(m1

)=

⎛⎝ 1 0 0 00 1 0 00 0 0 1

⎞⎠ (x1

). (12)

Accordingly, a point X = (X, Y, Z)� in a space is transformed to the pointm as

u =fa2X

(a2 ∓ 2e2)Z ± 2be|X| (u =2cX

|X| − Z), (13)

v =fa2Y

(a2 ∓ 2e2)Z ± 2be|X| (v =2cY

|X| − Z). (14)

Setting ξ = (ξx, ξy, ξz) to be the point on a sphere, the spherical-cameracenter Cs and the the focal point F of the hyperboloid (paraboloid) C2

are Cs = F = 0. (Therefore, q = 0 in Equation (31).) Furthermore, lsdenotes the axis connecting Cs and north pole of the spherical surface.For the axis ls and the hyperbolic-camera (parabolic-camera) axis l we setls = l = k(0, 0, 1)� for k ∈ R, that is, the directions of ls and l are thedirection of the z axis. For the configuration of the spherical camera andthe hyperbolic (parabolic) camera which share axes ls and l as illustrated

expressed as:x = µξ, (15)

where

µ =±a2

b ∓ eξz(µ =

2c

1 − ξz). (16)

to-spherical) image transform. As illustrated in Figure 4 (b) (Figure 5(b)),we show the hyperbolic-to-spherical (parabolic-step,nexttheFor

in Figure 4(b) (Figure 5(b)), the nonlinear function in Equation (6) is

⎛⎝ ⎞⎠

10

(parabolic) image and the point ξ on the sphere derives the equations:

u =fa2 cos θ sin ϕ

(a2 ∓ 2e2) cos ϕ ± 2be(u = 2c cos θ cot(

ϕ

2)), (17)

v =fa2 sin θ sin ϕ

(a2 ∓ 2e2) cos ϕ ± 2be(v = 2c sin θ cot(

ϕ

2)). (18)

Setting I(u, v) and IS(θ, ϕ) to be the hyperbolic (parabolic) image and thespherical image, respectively, the hyperbolic(parabolic)-to-spherical imagetransform is expressed as follows:

IS(θ, ϕ) = I(fa2 cos θ sin ϕ

(a2 ∓ 2e2) cos ϕ ± 2be,

fa2 sin θ sin ϕ

(a2 ∓ 2e2) cos ϕ ± 2be) (19)

(IS(θ, ϕ) = I(2c cos θ cot(ϕ

2), 2c sin θ cot(

ϕ

2)), (20)

Figure 5. Transformation among parabolic- and spherical-camera systems. (a) illus-trates a parabolic-camera system. The camera C generates the omnidirectional imageπ by the orthogonal projection, since all the rays collected to the focal point F areorthogonally reflected to the imaging plane. A point X in a space is transformed to thepoint x on the paraboloid and x is transformed to the point m on image plane. (b)illustrate the geometrical configuration of parabolic- and spherical-camera systems. Inthis geometrical configuration, a point ξ on the spherical image and a point x on theparaboloid lie on a line connecting a point X in a space and the focal point F of theparaboloid.

A. TORII, et al.

Applying the spherical coordinate systems, the point m on the hyperbolic

for I(u, v) which is the image of the hyperbolic- (parabolic) camera.

GEOMETRY OF A CLASS OF CATADIOPTRIC SYSTEMS 11

3.2. ELLIPTIC-TO-SPHERICAL TRANSFORM

We set that the focus of the ellipsoid C2 is located at the point F =(0, 0, 0)�. The center of the pinhole camera is located at the point C =(0, 0,−2e). The elliptic-camera axis l is the line which connects C and F .We set the hyperboloid S :

x�Ax = (x, y, z, 1)

⎛⎜⎜⎝1a2 0 0 00 1

a2 0 00 0 1

b2− e

b2

0 0 − eb2

e2

b2 − 1

⎞⎟⎟⎠⎛⎜⎜⎝

xyz1

⎞⎟⎟⎠ = 0 (21)

√b2 2

IS(θ, ϕ) satisfy the equation

IS(θ, ϕ) = I(fa2 cos θ sin ϕ

(a2 ± 2e2) cos ϕ ± 2be,

fa2 sin θ sin ϕ

(a2 ± 2e2) cos ϕ ± 2be). (22)

3.3. A UNIFIED FORMULATION OF CTS TRANSFORM

We express quadric surfaces in the homogeneous form as:

x�Ax = 0, (23)

wherex = (x, y, z, 1)�, (24)

andA = {aij}, i, j = 1, 2, 3, 4. (25)

The matrix A satisfies the relation

A� = A. (26)

A quadric surface is also expressed as:

x�A0x + 2b�x + a44 = 0, (27)

wherex = (x, y, z)�, (28)

A0 = {aij}, i, j = 1, 2, 3, (29)

where e = − a . Employing the same strategy for the hyperbolic-to-spherical image transform, the elliptic image I(u, v) and the spherical image

12

andb = (a41, a42, a43)�. (30)

The eigenvalues λm and σn, for m = 1, 2, 3, 4 and n = 1, 2, 3, of the matrixA and A0, respectively.

THEOREM 1. If λm and σn satisfy the two conditions, the quadric surfacerepresent the revolution surface of quadratic curve, that is, a ellipsoid ofrevolution, a hyperboloid of two sheets, a paraboloid of revolution. One isthat the signs of λi are three positives and one negative, and vice versa. Theother is σ1 = σ2 and σ3 ∈ R.

A quadric surfaces, which satisfy Theorem 1, has two focal points. If wecan locate a focal point of quadric mirror at one focal point and a centerof camera at the other focal point, all the rays reflected on the quadricmirror pass through the camera center. (In case of σ3 = 0, a camera centeris the point at infinity. The projection becomes orthogonal.) Furthermore,locating the center of sphere at the focus of quadric mirror, all the rays,which pass through the focus of quadric mirror and the sphere, are identical.Therefore, the nonlinear transform g in Equation (6) is expressed as:

x = µp + q, (31)

where p = ξ and q is the focal point of the quadric mirror, and

µ =−β ±

√β2 − αγ

α, (32)

where

α =4∑

j=1

4∑i=1

pjaijpi,

β =4∑

j=1

4∑i=1

pjaijqi,

γ =4∑

j=1

4∑i=1

qjaijqi,

and p4 = 0 and q4 = 1. The sign of µ depends on the geometrical configu-ration of the surface and ray.

A. TORII, et al.

GEOMETRY OF A CLASS OF CATADIOPTRIC SYSTEMS 13

4. Applications of Spherical Camera Model

4.1. LINE RECONSTRUCTION IN SPACE

As illustrated in Figure 6, we set a spherical camera center C at the originof the world coordinate system. In the spherical camera system, a line L isalways projected to a great circle r by the intersection of the plane π andthe sphere S2. If the normal vector n = (n1, n2, n3)� of π satisfies

n21 + n2

2 + n23 = 1, (33)

n�X = 0 (34)

expresses the great circle r on the sphere S2.The dual space of S2 is S2, we denote the dual of S2 as S2∗. The dual

vector of n ∈ S2 is n∗ ∈ S2∗ such that

n�n∗ = 0. (35)

A vector on S2 defines a great circle corresponded to n, we express thegreat circle as n∗. Therefore, voting the vector n∗ in S2∗, we can estimatethe great circle on S2 as illustrated in Figure 7. Equivalently, voting n∗

ij =ni × nj to S2∗, we can estimate a great circle in S2 due to select the peakin S2∗ as illustrated in Figure 8.

As illustrated in Figure 9, the centers of three spherical cameras arelocated at Ca = 0, Cb = tb and Cc = tc. We assume that the rotation

Figure 6. A line in a space and a spherical image. A line in a space is always projectedto a great circle on a spherical image as the intersection of the plane π and the sphereS2.

14

Figure 7. Estimation of a great circle on a spherical image by Hough Transform.

Rb, and Rc

vectors na, nb, and nc, that is, great circles ra on Sa, rb on Sb, and rc onSc. Simultaneously, we obtain three planes in a space. The intersection ofthe three planes yields the line in a space as follows.

n�a (X) = 0, (36)

(Rbnb)�(X − tb) = 0, (37)(Rcnc)�(X − tc) = 0. (38)

Figure 8. Estimation of a great circle on a spherical image by random Hough Transform.

A. TORII, et al.

among these cameras and the world coordinate system are cali-brated. Employing the random Hough transform, we obtain three normal

GEOMETRY OF A CLASS OF CATADIOPTRIC SYSTEMS 15

Figure 9. Reconstruction of a line in a space using three spherical cameras. If the threeplanes, which are yielded by the great circles, intersect at a single line in a space, then,we have a collect circle-correspondence-triplet.

By employing homogeneous coordinates, these equations are expressed as

MX = 0, (39)

where

M =(

na Rbnb Rcnc

0 −(Rbnb)�tb −(Rcnc)�tc

)�(40)

If the circles corresponds to the line L, the rank of M equals to two. There-fore, these relations are the constraint for a line reconstruction employingthree spherical cameras.

4.2. THREE-DIMENSIONAL RECONSTRUCTION USING FOURSPHERICAL CAMERAS

We proposed the efficient geometrical configurations of panoramic (omni-directional) cameras (Torii et al., 2003) for the reconstruction of points ina space. In this section, we extend the idea for four spherical cameras.

We consider the practical imaging region observed by the transformedtwo spherical cameras which are configurated parallel axially, single axiallyand oblique axially. The parallel-axial and the single-axial stereo cam-eras yield images which have a large feasible region compared with the

16

oblique-axial stereo ones. Therefore, for the geometric configuration of fourpanorama cameras, we assume that the four panorama-camera centers areon the corners of a square vertical to a horizontal plane. Furthermore, allof the camera axes are parallel.

Therefore, the panorama-camera centers are Ca = (tx, ty, tz)�, Cb =(tx, ty,−tz)�, Cc = (−tx,−ty, tz)� and Cd = (−tx,−ty,−tz)� . This config-uration is illustrated in Figure 10. Since the epipoles exist on the panoramaimages and correspond to the camera axes, this camera configuration per-mits us to eliminate the rotation between the camera coordinate and theworld coordinate systems.

For a point X, the projections of the point X to cameras Ca, Cb,Cc and Cd are xa = (cos θ, sin θ, tan a)�, xb = (cos θ, sin θ, tan b)�, xc =(cos ω, sin ω, tan c)� and xd = (cos ω, sin ω, tan d)�, respectively, on thecylindrical-image surfaces. These four points are the corresponding-pointquadruplet. The points xa, xb, xc and xd are transformed to pa = (θ, a)�,pb = (θ, b)�, pc = (ω, c)� and pd = (ω, d)�, respectively, on the rect-angular panoramic images. The corresponding-point quadruplet yields sixepipolar planes. Using homogeneous coordinate systems, we represent X asξ = (X, Y, Z, 1)�. Here, these six epipolar planes are formulated as Mξ = 0,

Figure 10. The four spherical-camera system. A corresponding-point quadruplet yieldssix epipolar plane. It is possible to reconstruct a point in a space using the six epipolarplanes. Furthermore, using the six epipolar planes, we can derive a numerically stableregion for the reconstruction of a point in a space.

A. TORII, et al.

GEOMETRY OF A CLASS OF CATADIOPTRIC SYSTEMS 17

where M = (m1,m2,m3, m4,m5,m6)�,

m1 =

⎛⎜⎜⎝sin θ

− cos θ0

− sin θtx + cos θty

⎞⎟⎟⎠,

m2 =

⎛⎜⎜⎝sin ω

− cos ω0

sin ωtx − cos ωty

⎞⎟⎟⎠,

m3 =

⎛⎜⎜⎝tan c sin θ − tan a sin ωtan a cos ω − tan c cos θ

sin(ω − θ)− sin(ω − θ)tz

⎞⎟⎟⎠,

m4 =

⎛⎜⎜⎝tan d sin θ − tan b sin ωtan b cos ω − tan d cos θ

sin(ω − θ)sin(ω − θ)tz)

⎞⎟⎟⎠,

m5 =

⎛⎜⎜⎝tan d sin θ − tan a sin ωtan a cos ω − tan d cos θ

sin(ω − θ)0

⎞⎟⎟⎠,

and

m6 =

⎛⎜⎜⎝tan c sin θ − tan b sin ωtan b cos ω − tan c cos θ

sin(ω − θ)0

⎞⎟⎟⎠.

Since these six planes intersect at the point X in a space, the rank of thematrix M is three. Therefore, the matrix MR,

MR =

⎛⎝ mi1 mi2 mi3 mi4

mj1 mj2 mj3 mj4

mk1 mk2 mk3 mk4

⎞⎠ =

⎛⎝ m�i

m�j

m�k

⎞⎠, (41)

is constructed from three row vectors of the matrix M. If and only if therank of the matrix MR is three, MR satisfies the equation MRξ = 0. Thepoint X is derived by the equation

X = M−1m4 (42)

18

where

M =

⎛⎝ mi1 mi2 mi3

mj1 mj2 mj3

mk1 mk2 mk3

⎞⎠, m4 =

⎛⎝ −mi4

−mj4

−mk4

⎞⎠. (43)

Equation (42) enable us to reconstruct the point X uniquely from any threerow vectors selected from the matrix M.

5. Discussions and Concluding Remarks

DEFINITION 1. Convex Cone in Rn; Let M to be a closed finite convexbody in Rn−1. We set Ma = M + a for a ∈ Rn. (It is possible to seta = λei.) For x ∈ Ma,

C(M,a) = {x | x = λy, ∀λ ∈ R, y ∈ Ma} (44)

is the convex cone in Rn.

Figure 11 illustrates a convex cone in Rn.

DEFINITION 2. Conic Surface in Rn−1; Let L to be a linear manifold inRn, that is,

L = P + b (45)

for b ∈ Rn and P is a n − 1 dimensional linear subspace in Rn.

L ∩ C(M,a) (46)

is a conic surface in Rn−1.

For n = 3 and M = S, L ∩ C(M,a) is a planar conic. This geometricalproperty derives the following relations.

Figure 11. Definition of a convex cone in Rn.

A. TORII, et al.

GEOMETRY OF A CLASS OF CATADIOPTRIC SYSTEMS 19

Figure 12. Central and non-central catadioptric cameras. It is possible to classifynon-central cameras in two classes. One has a focal line as illustrated in (b) and theother has a focal surface (ruled surface).

1. For n = 4 and M = S2, we have a conic surface of revolution.2. For n = 4 and M = E2(ellipsoid) in R2, we have an ellipsoid of

revolution.

For the cone in the class (ii), it is possible to transform ellipsoid E2 to S2.Therefore, vectors on L ∩ C(M,a) is equivalent to vectors on Sn−1. Thisgeometrical property leads that images observed through a catadioptriccamera system with quadric mirror is equivalent to images on the sphere.

Catadioptric camera systems are classified into central and non-centralcameras depending on the shape of mirrors. Our observation using thecone intersection in Rn leads that it is possible to classify non-centralcatadioptric cameras into two classes. One has a focal line and the otherhas a focal surface (ruled surface).

Acknowledgments

This work is in part supported by Grant-in-Aid for Scientific Researchof the Ministry of Education, Culture, Sports, Science and Technology ofJapan under the contract of 14380161 and 16650040. The final manuscriptprepared while the first author was at CMP at CTU in Prague. He expressesgreat thanks to the hospitality of Prof. V. Hlavac and Dr. T. Pajdla.

References

Svoboda, T., Pajdla, T., and Hlavac, V.: Epipolar geometry of panoramic cameras. InProc. ECCV, Valume A, pages 218–231, 1998.

limits of accuracy and neural matched filters. In Motion vision: computational, neuralDahmen, H. -J., Franz, M.O., and Krapp, H.G.: Extracting egomotion from optic flow: