View
214
Download
0
Embed Size (px)
Citation preview
5/1/2000 Deepak Bandyopadhyay / UNC Chapel Hill
1
Painting on Movable ObjectsPainting on Movable Objects
Deepak Bandyopadhyay, UNC-CHDeepak Bandyopadhyay, UNC-CH
Ramesh Raskar, MERL Ramesh Raskar, MERL
Henry Fuchs, UNC-CHHenry Fuchs, UNC-CH
5/1/2000 Deepak Bandyopadhyay / UNC Chapel Hill
2
Dynamic Shader LampsDynamic Shader Lamps
• IntroductionIntroduction
• Related WorkRelated Work
• ImplementationImplementation
• ResultsResults
5/1/2000 Deepak Bandyopadhyay / UNC Chapel Hill
3
MotivationMotivation
For some applications, it is easier to work For some applications, it is easier to work with a physical object directly, rather than with a physical object directly, rather than using traditional displays or see-through AR using traditional displays or see-through AR HMDs.HMDs.
» Reasons: Ergonomics, Passive HapticsReasons: Ergonomics, Passive Haptics» Example application: virtual 3D paintingExample application: virtual 3D painting
We show how to implement such systems.We show how to implement such systems.
5/1/2000 Deepak Bandyopadhyay / UNC Chapel Hill
4
OverviewOverview
Projector1 Projector2
Objects
Tracker
PC
5/1/2000 Deepak Bandyopadhyay / UNC Chapel Hill
5
VideoVideo
5/1/2000 Deepak Bandyopadhyay / UNC Chapel Hill
6
ApplicationsApplications
• Testbed for AR Testbed for AR applications applications
– projectors on movable objectsprojectors on movable objects
• AR Interaction medium AR Interaction medium – ArtArt– TeleimmersionTeleimmersion– Architecture Architecture
» annotationannotation» visualizationvisualization
– Others, see paperOthers, see paper
5/1/2000 Deepak Bandyopadhyay / UNC Chapel Hill
7
Dynamic Shader LampsDynamic Shader Lamps
• IntroductionIntroduction
• Related WorkRelated Work
• ImplementationImplementation
• ResultsResults
5/1/2000 Deepak Bandyopadhyay / UNC Chapel Hill
8
Shader Lamps Basic IdeaShader Lamps Basic Idea
Raskar et al 1, 2001
1 Ramesh Raskar, Greg Welch, Kok-Lim Low and Deepak Bandyopadhyay. “Shader Lamps : Animating Real Objectswith Image-based Illumination”, Eurographics Rendering Workshop (EGRW 2001), London.
5/1/2000 Deepak Bandyopadhyay / UNC Chapel Hill
9
Shader Lamps Basic IdeaShader Lamps Basic Idea
Rearranging terms in optical path
5/1/2000 Deepak Bandyopadhyay / UNC Chapel Hill
10
Radiance AdjustmentRadiance Adjustment
Desired Virtual
k(x) cos(p )
d(x)2
L’(x,)= Ip (x, p)
L(x,) = F(x, , i) Li (x, i)
di
Projected Real
d ( x )2
k ( x ) cos( p )Ip (x, p) =
k (x)>0L ( x, )
5/1/2000 Deepak Bandyopadhyay / UNC Chapel Hill
11
Image based IlluminationImage based Illumination
• Basic IdeaBasic Idea» Render images and project on objectsRender images and project on objects» Multiple projectorsMultiple projectors» View and object dependent colorView and object dependent color
5/1/2000 Deepak Bandyopadhyay / UNC Chapel Hill
12
Related Work : 3D PaintingRelated Work : 3D Painting
• Hanrahan & Haeberli, 1990 Hanrahan & Haeberli, 1990 22
– Painting on 3D modelsPainting on 3D models– Screen & mouse interfaceScreen & mouse interface
• Agrawala & Beers, 1995 Agrawala & Beers, 1995 33
– Painting Scanned SurfacesPainting Scanned Surfaces» Object is staticObject is static» Tracked pointer moved on its surfaceTracked pointer moved on its surface» Separate displaySeparate display
2 Pat Hanrahan and Paul Haeberli, “Direct WYSIWYG Painting and Texturing on 3D Shapes”. SIGGRAPH’90.
3 Maneesh Agrawala, Andrew C. Beers and Marc Levoy, “3D Painting on Scanned Surfaces”, Symposium on Interactive 3D Graphics (I3D’95).
5/1/2000 Deepak Bandyopadhyay / UNC Chapel Hill
13
Related Work : 3D PaintingRelated Work : 3D Painting
• Gregory, Ehmann & Lin, 1999 Gregory, Ehmann & Lin, 1999 44
– inTouch haptic paintinginTouch haptic painting» Modeling, painting, force feedbackModeling, painting, force feedback» No real object; purely virtualNo real object; purely virtual
• Keefe, Feliz et al, 2001 Keefe, Feliz et al, 2001 55
– CavePaintingCavePainting» Physical interface objectsPhysical interface objects» Display is purely virtualDisplay is purely virtual
4 Arthur Gregory, Stephen Ehmann and Ming C. Lin, “inTouch : Interactive Multiresolution Modeling and 3D Painting with a Haptic Interface”, Virtual Reality (VR’99).
5 Daniel F. Keefe, Daniel A. Feliz, Tomer Moscovitch, David Laidlaw and Joseph LaViola, Jr., “CavePainting : a Fully Immersive 3D Artistic Medium and Interactive Experience”, I3D 2001.
5/1/2000 Deepak Bandyopadhyay / UNC Chapel Hill
14
Dynamic Shader LampsDynamic Shader Lamps
• IntroductionIntroduction
• Related WorkRelated Work
• ImplementationImplementation
• ResultsResults
5/1/2000 Deepak Bandyopadhyay / UNC Chapel Hill
15
Projector CalibrationProjector Calibration
• Calibrate each projector (5 minutes)Calibrate each projector (5 minutes)– Match 3D points in world Match 3D points in world
space to 2D pixels in space to 2D pixels in projector space.projector space.
– Solve for 3x4 projection Solve for 3x4 projection matrixmatrix
» ignore radial distortionignore radial distortion» minimize least-square error (standard CV method)minimize least-square error (standard CV method)» split into intrinsic (projection) and extrinsic matrixsplit into intrinsic (projection) and extrinsic matrix
• Render 3D points in the world with good Render 3D points in the world with good registrationregistration
2D cross hair
5/1/2000 Deepak Bandyopadhyay / UNC Chapel Hill
16
Model Generation/AuthoringModel Generation/Authoring
• Need: vertices, faces, normals, texture Need: vertices, faces, normals, texture coordinates and color/material scheme per coordinates and color/material scheme per object to be paintedobject to be painted
– Acquire points using magnetic stylusAcquire points using magnetic stylus– Faces, textures by hand / surface reconstructionFaces, textures by hand / surface reconstruction– Texture coordinates non-repeating, fit geometryTexture coordinates non-repeating, fit geometry– Future: automate with Future: automate with
modeling tools.modeling tools.
5/1/2000 Deepak Bandyopadhyay / UNC Chapel Hill
17
Rendering Tracked ObjectRendering Tracked Object
• Algorithm each-frame:
Tracker gives 6DOF reading of sensors.For view i (projector i) do
Load projection and model view matrix iFor object j do
Apply world to sensor j matrixDraw object in sensor coordinatesInteract with object
5/1/2000 Deepak Bandyopadhyay / UNC Chapel Hill
18
3D painting interface3D painting interface
• Objects hand-held or set on table.Objects hand-held or set on table.
• Tracked stylus with spherical tip Tracked stylus with spherical tip – facilitates contact paintingfacilitates contact painting
• Projected touch palette, modeled as Projected touch palette, modeled as a static object with behavior:a static object with behavior:
– choose contact, spray or texture paintchoose contact, spray or texture paint– choose brush colorchoose brush color
5/1/2000 Deepak Bandyopadhyay / UNC Chapel Hill
19
Painting objectsPainting objects
• Determine object intersections (point-polygon)Determine object intersections (point-polygon)– Find set of polygons to modifyFind set of polygons to modify
• Color blending using brush function Color blending using brush function 66
– ContactContact paint uses a function on surface of object paint uses a function on surface of object
B = 1-(r/R)B = 1-(r/R)22, r<R, r<R
– SpraySpray paint uses a similar brush function in 3D paint uses a similar brush function in 3D– Color (or texture) blending:Color (or texture) blending:
CCfinalfinal = C = Cinitinit.(1-B) + C.(1-B) + Cbrushbrush.B.B
rR
6 From Arthur Gregory’s inTouch system
5/1/2000 Deepak Bandyopadhyay / UNC Chapel Hill
20
Painting objectsPainting objects
• Triangle rasterization w/ texture map Triangle rasterization w/ texture map modification modification 66
– Scan 2D and 3D simultaneously.Scan 2D and 3D simultaneously.
q
pu
vconcurrent
(x,y,z)
Polygon in 3D
(u,v)
2D texture map
Update colorLook up brush fn
6 From Arthur Gregory’s inTouch system
5/1/2000 Deepak Bandyopadhyay / UNC Chapel Hill
21
Dynamic Shader LampsDynamic Shader Lamps
• IntroductionIntroduction
• Related WorkRelated Work
• ImplementationImplementation
• ResultsResults
5/1/2000 Deepak Bandyopadhyay / UNC Chapel Hill
22
Performance / numbers ?Performance / numbers ?
• 60fps rendering 3 objects, 500+ polygons 60fps rendering 3 objects, 500+ polygons – ~100 texture maps, mostly 128x128~100 texture maps, mostly 128x128– Running on SGI InfiniteReality2 or 450MHz PC Running on SGI InfiniteReality2 or 450MHz PC
with multi-monitor graphics cardwith multi-monitor graphics card
• Not optimized for larger data setsNot optimized for larger data sets
• 100-110ms latency (high).100-110ms latency (high).– Roughly estimatedRoughly estimated
» 1cm of distance lag when moved at 10cm/s1cm of distance lag when moved at 10cm/s– Later, precisely measuredLater, precisely measured
5/1/2000 Deepak Bandyopadhyay / UNC Chapel Hill
23
Issues and Future WorkIssues and Future Work
• Tracker LimitationsTracker Limitations– Range Range (Polhemus Fastrak, 50-75 cm)(Polhemus Fastrak, 50-75 cm)
» use wide-area trackeruse wide-area tracker– Technology Technology (ferrous objects distort magnetic field)(ferrous objects distort magnetic field)
• Display limitationsDisplay limitations– Dynamic Intensity BlendingDynamic Intensity Blending– Limited dynamic range of projectorLimited dynamic range of projector– Occlusion, overlap, shadowsOcclusion, overlap, shadows
• Scaling it upScaling it up– Room sized environments, multiple projectorsRoom sized environments, multiple projectors
5/1/2000 Deepak Bandyopadhyay / UNC Chapel Hill
24
ConclusionsConclusions
• Dynamic Shader LampsDynamic Shader Lamps– A set of techniques for A set of techniques for interactively interactively changing changing
appearance attributes of neutrally colored physical appearance attributes of neutrally colored physical objects, static or movingobjects, static or moving
• Prototype implementationPrototype implementation– Painting on multiple real objectsPainting on multiple real objects– display devices - projectorsdisplay devices - projectors
» 3D display surfaces (real objects)3D display surfaces (real objects)– 3D input devices - tracker & stylus3D input devices - tracker & stylus
• Extensions Extensions - - how to generalize/scale.how to generalize/scale.
5/1/2000 Deepak Bandyopadhyay / UNC Chapel Hill
25
AcknowledgementsAcknowledgements
• ISAR reviewersISAR reviewers
• Miriam Mintzer Fuchs, age 10Miriam Mintzer Fuchs, age 10
• People at UNC People at UNC (paper, video & demo)(paper, video & demo)– Herman Towles, Greg Welch, Anselmo Lastra, Herman Towles, Greg Welch, Anselmo Lastra,
Russ Taylor, Ming Lin, Scott RussellRuss Taylor, Ming Lin, Scott Russell– Andrei State, Jim Mahaney, Kurtis Keller, John Andrei State, Jim Mahaney, Kurtis Keller, John
Thomas, David Harrison, Arthur GregoryThomas, David Harrison, Arthur Gregory– Kok-Lim Low, Wei-Chao Chen, Ben Lok, Kenny Kok-Lim Low, Wei-Chao Chen, Ben Lok, Kenny
Hoff, Dave McAllister, Miguel Otaduy, Gentaro HirotaHoff, Dave McAllister, Miguel Otaduy, Gentaro Hirota
• STC Tracked Shader Lamps, fundingSTC Tracked Shader Lamps, funding
Deepak Bandyopadhyay / UNC Chapel Hill
Dynamic Shader LampsDynamic Shader Lamps
Deepak Bandyopadhyay, Ramesh Raskar, Henry FuchsDeepak Bandyopadhyay, Ramesh Raskar, Henry Fuchshttp://www.cs.unc.edu/~debug/papers/DSLpaint/http://www.cs.unc.edu/~debug/papers/DSLpaint/
Painting on Movable Objects
Images