Spatial augmented reality: merging real and virtual worlds

  • Published on
    27-Dec-2016

  • View
    212

  • Download
    0

Embed Size (px)

Transcript

<ul><li><p>Spatial Augmented Reality</p></li><li><p>Spatial Augmented Reality</p><p>Merging Real and Virtual Worlds</p><p>Oliver BimberBauhaus-University, Weimar</p><p>Ramesh RaskarMitsubishi Electric Research Laboratory,</p><p>Cambridge, MA</p><p>A K PetersWellesley, Massachusetts</p></li><li><p>Editorial, Sales, and Customer Service Office</p><p>A K Peters, Ltd.888 Worcester Street, Suite 230Wellesley, MA 02482www.akpeters.com</p><p>Copyright 2005 by A K Peters, Ltd.</p><p>All rights reserved. No part of the material protected by this copyrightnotice may be reproduced or utilized in any form, electronic or mechani-cal, including photocopying, recording, or by any information storage andretrieval system, without written permission from the copyright owner.</p><p>Library of Congress Cataloging-in-Publication Data</p><p>Bimber, Oliver, 1973Spatial augmented reality : merging real and virtual worlds / OliverBimber, Ramesh Raskar.</p><p>p. cm.Includes bibliographical references and index.ISBN 1-56881-230-21. Computer graphics- 2. Virtual reality. I. Raskar, Ramesh II. Title.</p><p>T385.B5533 2004006.8dc22</p><p>2005043110</p><p>Printed in the United States of America09 08 07 06 05 10 9 8 7 6 5 4 3 2 1</p></li><li><p>To MelO. B.</p><p>To my parentsR. R.</p></li><li><p>Contents</p><p>Preface xi</p><p>1 A Brief Introduction to Augmented Reality 11.1 What is Augmented Reality . . . . . . . . . . . . . . . . . 11.2 Todays Challenges . . . . . . . . . . . . . . . . . . . . . . . 41.3 Spatial Augmented Reality . . . . . . . . . . . . . . . . . . 71.4 Outline of the Book . . . . . . . . . . . . . . . . . . . . . . 8</p><p>2 Fundamentals: From Photons to Pixels 132.1 Light in a Nutshell . . . . . . . . . . . . . . . . . . . . . . . 142.2 Geometric Optics . . . . . . . . . . . . . . . . . . . . . . . 172.3 Visual Depth Perception . . . . . . . . . . . . . . . . . . . . 302.4 Rendering Pipelines . . . . . . . . . . . . . . . . . . . . . . 442.5 Summary and Discussion . . . . . . . . . . . . . . . . . . . 66</p><p>3 Augmented Reality Displays 713.1 Head-Attached Displays . . . . . . . . . . . . . . . . . . . . 723.2 Hand-Held Displays . . . . . . . . . . . . . . . . . . . . . . 793.3 Spatial Displays . . . . . . . . . . . . . . . . . . . . . . . . . 833.4 Summary and Discussion . . . . . . . . . . . . . . . . . . . 90</p><p>4 Geometric Projection Concepts 934.1 Geometric Model . . . . . . . . . . . . . . . . . . . . . . . . 934.2 Rendering Framework . . . . . . . . . . . . . . . . . . . . . 984.3 Calibration Goals . . . . . . . . . . . . . . . . . . . . . . . . 1044.4 Display Environments and Applications . . . . . . . . . . . 1084.5 Summary and Discussion . . . . . . . . . . . . . . . . . . . 108</p><p>vii</p></li><li><p>viii Contents</p><p>5 Creating Images with Spatial Projection Displays 1115.1 Planar Displays . . . . . . . . . . . . . . . . . . . . . . . . . 1115.2 Non-Planar Display . . . . . . . . . . . . . . . . . . . . . . 1265.3 Projector Overlap Intensity Blending . . . . . . . . . . . . . 1295.4 Quadric Curved Displays . . . . . . . . . . . . . . . . . . . 1335.5 Illuminating Objects . . . . . . . . . . . . . . . . . . . . . . 1425.6 Summary and Discussion . . . . . . . . . . . . . . . . . . . 147</p><p>6 Generating Optical Overlays 1496.1 Transparent Screens . . . . . . . . . . . . . . . . . . . . . . 1506.2 Mirror Beam Combiners . . . . . . . . . . . . . . . . . . . . 1526.3 Planar Mirror Beam Combiners . . . . . . . . . . . . . . . . 1526.4 Screen Transformation and Curved Screens . . . . . . . . . 1636.5 Moving Components . . . . . . . . . . . . . . . . . . . . . . 1666.6 Multi-Plane Beam Combiners . . . . . . . . . . . . . . . . . 1666.7 Curved Mirror Beam Combiners . . . . . . . . . . . . . . . 1746.8 Summary and Discussion . . . . . . . . . . . . . . . . . . . 206</p><p>7 Projector-Based Illumination and Augmentation 2137.1 Image-Based Illumination: Changing Surface Appearance . 2147.2 Creating Consistent Occlusion . . . . . . . . . . . . . . . . . 2207.3 Creating Consistent Illumination . . . . . . . . . . . . . . . 2277.4 Augmenting Optical Holograms . . . . . . . . . . . . . . . . 2447.5 Augmenting Flat and Textured Surfaces . . . . . . . . . . . 2557.6 Augmenting Geometrically Non-Trivial Textured Surfaces . 2677.7 Summary and Discussion . . . . . . . . . . . . . . . . . . . 276</p><p>8 Examples of Spatial AR Displays 2798.1 Shader Lamps . . . . . . . . . . . . . . . . . . . . . . . . . . 2808.2 Being There . . . . . . . . . . . . . . . . . . . . . . . . . . . 2868.3 iLamps: Mobile Projectors . . . . . . . . . . . . . . . . . . . 2888.4 The Extended Virtual Table . . . . . . . . . . . . . . . . . . 2918.5 The Virtual Showcase . . . . . . . . . . . . . . . . . . . . . 2978.6 The HoloStation . . . . . . . . . . . . . . . . . . . . . . . . 3028.7 Augmented Paintings . . . . . . . . . . . . . . . . . . . . . . 3088.8 Smart Projectors . . . . . . . . . . . . . . . . . . . . . . . . 3148.9 Summary and Discussion . . . . . . . . . . . . . . . . . . . 320</p><p>9 The Future 3219.1 Displays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3219.2 Supporting Elements . . . . . . . . . . . . . . . . . . . . . . 3269.3 Summary and Discussion . . . . . . . . . . . . . . . . . . . 329</p></li><li><p>Contents ix</p><p>A Calibration of a Projector (or a Camera) 331A.1 Source code for Calibration of a Projector (or a camera) . . 331A.2 Quadric Image Transfer . . . . . . . . . . . . . . . . . . . . 334</p><p>B OpenGLs Transformation Pipeline Partially Re-Implemented 337B.1 General Definitions . . . . . . . . . . . . . . . . . . . . . . . 337B.2 Projection Functions . . . . . . . . . . . . . . . . . . . . . . 337B.3 Transformation Functions . . . . . . . . . . . . . . . . . . . 338B.4 Additional Functions . . . . . . . . . . . . . . . . . . . . . . 340</p><p>Bibliography 343</p><p>Index 363</p></li><li><p>Preface</p><p>Spatial Augmented Reality is a rapidly emerging field which concerns every-one working in digital art and media who uses any aspects of augmentedreality and is interested in cutting-edge technology of display technologiesand the impact of computer graphics. We believe that a rich pallet of dif-ferent display technologies, mobile and non-mobile, must be considered andadapted to fit a given application so that one can choose the most efficienttechnology. While this broader view is common in the very establishedarea of virtual reality, it is becoming more accepted in augmented realitywhich has been dominated by research involving mobile devices.</p><p>This book reflects our research efforts over several years and the materialhas been refined in several courses that we taught at the invitation ofEurographics and ACM SIGGRAPH.</p><p>Who Should Read This Book</p><p>In order for a broad spectrum of readerssystem designers, programmers,artists, etc to profit from the book, we require no particular programmingexperience or mathematical background. However, a general knowledge ofbasic computer graphics techniques, 3D tools, and optics will be useful.</p><p>The reader will learn about techniques involving both hardware andsoftware to implement spatial augmented reality installations. Many Cgand OpenGL code fragments, together with algorithms, formulas, drawings,and photographs will guide the interested readers who want to experimentwith their own spatial augmented reality installations.</p><p>By including a number of exemplary displays examples from differentenvironments, such as museums, edutainment settings, research projects,and industrial settings, we want to stimulate our readers to imagine novel</p><p>xi</p></li><li><p>xii Preface</p><p>AR installations and to implement them. Supplementary material can befound at http://www.spatialar.com/.</p><p>About the Cover</p><p>The images at the top of the front cover show a rainbow hologram of adinosaur (Deinonychus) skull (found in North America). It has been aug-mented with reconstructed soft tissue and artificial shading and occlusioneffects. The soft tissue data, provided by Lawrence M. Witmer of OhioUniversity, were rendered autostereoscopically. A replica of the skull washolographed by Tim Frieb at the Holowood holographic studio in Bamberg,Germany. The hologram was reconstructed by projected digital light thatcould be controlled and synchronized to the rendered graphics. This en-abled a seamless integration of interactive graphical elements into opticalholograms.</p><p>The image at the bottom show an example of Shader Lamps: an aug-mentation of a white wooden model of the Taj Mahal with two projec-tors. The wooden model was built by Linda Welch, George Spindler andMarty Spindler in the late 1970s. In 1999, at the University of North Car-olina, Greg Welch spray painted the wooden model white and Kok-LimLow scanned it with a robotic arm to create a 3D model. The woodenmodel is shown illuminated with images rendered with real time animationof a sunrise.</p><p>The art work on the back cover was created by Matthias Hanzlik. Thesketches show early concepts of SAR prototypes. They have all been real-ized and are described in the book.</p><p>Acknowledgements</p><p>The material presented in this book would not have been realized withoutthe help and dedication of many students and colleagues. We want tothank them first of all: Gordon Wetzstein, Anselm Grundhofer, SebastianKnodel, Franz Coriand, Alexander Kleppe, Erich Bruns, Stefanie Zollmann,Tobias Langlotz, Mathias Mohring, Christian Lessig, Sebastian Derkau,Tim Gollub, Andreas Emmerling, Thomas Klemmer, Uwe Hahne, PaulFockler, Christian Nitschke, Brian Ng, Kok-Lim Low, Wei-Chao Chen,Michael Brown, Aditi Majumder, Matt Cutts, Deepak Bandyopadhyay,Thomas Willwacher, Srinivas Rao, Yao Wang, and Johnny Lee.</p><p>http://www.spatialar.com/</p></li><li><p>Preface xiii</p><p>We also want to thank all colleagues in the field who provided addi-tional image material: Vincent Lepetit, Pascal Fua, Simon Gibson, KiyoshiKiyokawa, Hong Hua, Susumu Tachi, Daniel Wagner, Dieter Schmalstieg,George Stetten, Tetsuro Ogi, Barney Kaelin, Manfred Bogen, Kok-LimLow, Claudio Pinhanez, Masahiko Inami, Eric Foxlin, Michael Schnaider,Bernd Schwald, and Chad Dyner.</p><p>Special thanks go to those institutions and people who made this re-search possible: Bauhaus-University Weimar, Mitsubishi Electric ResearchLaboratories, Jeroen van Baar, Paul Beardsley, Paul Dietz, Cliff Forlines,Joe Marks, Darren Leigh, Bill Yerazunis, Remo Ziegler, Yoshihiro Ashizaki,Masatoshi Kameyama, Keiichi Shiotani, Fraunhofer Institute for ComputerGraphics, Jose L. Encarnacao, Bodo Urban, Erhard Berndt, University ofNorth Carolina at Chapel Hill, Henry Fuchs, Greg Welch, Herman Towles,Fraunhofer Center for Research in Computer Graphics, Miguel Encarnacao,Bert Herzog, David Zeltzer, Deutsche Forschungsgemeinschaft, and Euro-pean Union.</p><p>We thank the Institution of Electronics &amp; Electrical Engineers (IEEE),the Association for Computing Machinery (ACM), Elsevier, MIT Press,Springer-Verlag, and Blackwell Publishing for granting permission to re-publish some of the material used in this book.</p><p>Many thanks to Thomas Zeidler and to Alice and Klaus Peters forhelping us manage the process of writing this book alongside our everydayduties.</p><p>Oliver BimberWeimar April 2005</p><p>Ramesh RaskarCambridge, MA April 2005</p></li><li><p>1A Brief Introduction to</p><p>Augmented Reality</p><p>Like Virtual Reality (VR), Augmented Reality (AR) is becoming an emerg-ing edutainment platform for museums. Many artists have started usingthis technology in semi-permanent exhibitions. Industrial use of augmentedreality is also on the rise. Some of these efforts are, however, limited to us-ing off-the-shelf head-worn displays. New, application-specific alternativedisplay approaches pave the way towards flexibility, higher efficiency, andnew applications for augmented reality in many non-mobile applicationdomains. Novel approaches have taken augmented reality beyond tradi-tional eye-worn or hand-held displays, enabling new application areas formuseums, edutainment, research, industry, and the art community. Thisbook discusses spatial augmented reality (SAR) approaches that exploitlarge optical elements and video-projectors, as well as interactive render-ing algorithms, calibration techniques, and display examples. It provides acomprehensive overview with detailed mathematics equations and formu-las, code fragments, and implementation instructions that enable interestedreaders to realize spatial AR displays by themselves.</p><p>This chapter will give a brief and general introduction into augmentedreality and its current research challenges. It also outlines the remainingchapters of the book.</p><p>1.1 What is Augmented Reality</p><p>The terms virtual reality and cyberspace have become very popular outsidethe research community within the last two decades. Science fiction movies,such as Star Trek, have not only brought this concept to the public, but</p><p>1</p></li><li><p>2 1. A Brief Introduction to Augmented Reality</p><p>have also influenced the research community more than they are willing toadmit. Most of us associate these terms with the technological possibilityto dive into a completely synthetic, computer-generated worldsometimesreferred to as a virtual environment . In a virtual environment our senses,such as vision, hearing, haptics, smell, etc., are controlled by a computerwhile our actions influence the produced stimuli. Star Trek s Holodeck isprobably one of the most popular examples. Although some bits and piecesof the Holodeck have been realized today, most of it is still science fiction.</p><p>So what is augmented reality then? As is the case for virtual reality,several formal definitions and classifications for augmented reality exist(e.g., [109, 110]). Some define AR as a special case of VR; others arguethat AR is a more general concept and see VR as a special case of AR.We do not want to make a formal definition here, but rather leave it tothe reader to philosophize on their own. The fact is that in contrast totraditional VR, in AR the real environment is not completely suppressed;instead it plays a dominant role. Rather than immersing a person into acompletely synthetic world, AR attempts to embed synthetic supplementsinto the real environment (or into a live video of the real environment).This leads to a fundamental problem: a real environment is much moredifficult to control than a completely synthetic one. Figure 1.1 shows someexamples of augmented reality applications.</p><p>As stated previously, augmented reality means to integrate synthetic in-formation into the real environment. With this statement in mind, woulda TV screen playing a cartoon movie, or a radio playing music, then bean AR display? Most of us would say nobut why not? Obviously, thereis more to it. The augmented information has to have a much strongerlink to the real environment. This link is mostly a spatial relation betweenthe augmentations and the real environment. We call this link registra-tion. R2-D2s spatial projection of Princess Leia in Star Wars would bea popular science fiction example for augmented reality. Some technolog-ical approaches that mimic a holographic-like spatial projection, like theHolodeck, do exist today. But once again, the technical implementation asshown in Star Wars still remains a Hollywood illusion.</p><p>Some say that Ivan Sutherland established the theoretical foundationsof virtual reality...</p></li></ul>

Recommended

View more >