Upload
others
View
10
Download
0
Embed Size (px)
Citation preview
1
Image-Based Rendering
Computer VisionCSE576, Spring 2005
Richard Szeliski
CSE 576, Spring 2005 Image-Based Rendering 2
Today’s lectureImage–Based Rendering• Light Fields and Lumigraphs• Panoramas and Concentric Mosaics• Environment Matting• Image-Based models
CSE 576, Spring 2005 Image-Based Rendering 3
Today’s lectureVideo-Based Rendering• Facial animation• Video matting and shadow matting• Video Textures and Animating Stills• Video-based tours
CSE 576, Spring 2005 Image-Based Rendering 4
Readings• S. J. Gortler , R. Grzeszczuk , R. Szeliski and M. F.
Cohen, The Lumigraph, SIGGRAPH'96. • M. Levoy and P. Hanrahan, Light field rendering,
SIGGRAPH'96.• H.-Y. Shum and L.-W. He. Rendering with concentric
mosaics, SIGGRAPH’99.
2
CSE 576, Spring 2005 Image-Based Rendering 5
Readings• D. E. Zongker et al. Environment matting and
compositing, SIGGRAPH'99.• Y.-Y. Chuang et al. Environment matting extensions:
Towards higher accuracy and real-time capture. SIGGRAPH'2000, pp.121-130, 2000.
• P. E. Debevec , C. J. Taylor and J. Malik, Modeling and rendering architecture from photographs:…, SIGGRAPH'96.
CSE 576, Spring 2005 Image-Based Rendering 6
Readings• Y.-Y. Chuang et al. Video matting of complex scenes.
ACM Trans. on Graphics, 21(3):243-248, July 2002• Y.-Y. Chuang et al. Shadow matting. ACM
Transactions on Graphics, 22(3):494-500, July 2003.• A. Schödl et al., Video textures. SIGGRAPH'2000,
pp. 489-498, 2000.• M. Uyttendaele et al. Image-based interactive
exploration of real-world environments. IEEE Comp. Graphics and Applications, 24(3), May/June 2004.
Lightfields and Lumigraphs
(with lots of slides from Michael Cohen)
CSE 576, Spring 2005 Image-Based Rendering 8
Modeling light
How do we generate new scenes and animations from existing ones?
Classic “3D Vision + Graphics”:• take (lots of) pictures• recover camera pose• build 3D model• extract texture maps / BRDFs• synthesize new views
3
CSE 576, Spring 2005 Image-Based Rendering 9
Computer Graphics
Image
Output
ModelSyntheticCamera
CSE 576, Spring 2005 Image-Based Rendering 10
Real Scene
Computer Vision
Real Cameras
Model
Output
CSE 576, Spring 2005 Image-Based Rendering 11
Combined
Model Real Scene
Real Cameras
Image
Output
SyntheticCamera
CSE 576, Spring 2005 Image-Based Rendering 12
But, vision technology falls short
Model Real Scene
Real Cameras
Image
Output
SyntheticCamera
4
CSE 576, Spring 2005 Image-Based Rendering 13
… and so does graphics.
Model Real Scene
Real Cameras
Image
Output
SyntheticCamera
CSE 576, Spring 2005 Image-Based Rendering 14
Image Based Rendering
Real Scene
Real Cameras-or-
Expensive Image Synthesis
Images+Model
Image
Output
SyntheticCamera
CSE 576, Spring 2005 Image-Based Rendering 15
Ray
Constant radiance• time is fixed
5D• 3D position• 2D direction
CSE 576, Spring 2005 Image-Based Rendering 16
All Rays
Plenoptic Function:• all possible images• too much stuff!
5
CSE 576, Spring 2005 Image-Based Rendering 17
Line
Infinite line
4D• 2D direction• 2D position• non-dispersive medium
CSE 576, Spring 2005 Image-Based Rendering 18
Ray
Discretize, then interpolate
Distance between 2 rays• Which is closer together?
CSE 576, Spring 2005 Image-Based Rendering 19
Image
What is an image?
All rays through a point• Panorama
CSE 576, Spring 2005 Image-Based Rendering 20
Panoramic MosaicsConvert panoramic image sequence into a
cylindrical image
+ + … + =
6
CSE 576, Spring 2005 Image-Based Rendering 21
Image
Image plane
2D• position in plane
CSE 576, Spring 2005 Image-Based Rendering 22
Light leaving towards “eye”
2D• just dual of image
Object
CSE 576, Spring 2005 Image-Based Rendering 23
Object
All light leaving object
CSE 576, Spring 2005 Image-Based Rendering 24
Object
4D• 2D position (on surface)• 2D direction
7
CSE 576, Spring 2005 Image-Based Rendering 25
Object
All images
CSE 576, Spring 2005 Image-Based Rendering 26
Lumigraph / Lightfield
Outside convex space
4D
StuffEmpty
CSE 576, Spring 2005 Image-Based Rendering 27
Lumigraph
Inside convex space
4D
EmptyStuff
CSE 576, Spring 2005 Image-Based Rendering 28
Lumigraph
How to ?• organize• capture• render
8
CSE 576, Spring 2005 Image-Based Rendering 29
Lumigraph - Organization
2D position2D direction
sθ
CSE 576, Spring 2005 Image-Based Rendering 30
Lumigraph - Organization
2D position2D position
2 plane parameterization
su
CSE 576, Spring 2005 Image-Based Rendering 31
Lumigraph - Organization
2D position2D position
2 plane parameterization
us
t s,tu,v
v
s,t
u,v
CSE 576, Spring 2005 Image-Based Rendering 32
Lumigraph - Organization
Hold s,t constantLet u,v varyAn image
s,t u,v
9
CSE 576, Spring 2005 Image-Based Rendering 33
Lumigraph - Organization
Discretization• higher res near object
– if diffuse– captures texture
• lower res away– captures directions
s,t u,v CSE 576, Spring 2005 Image-Based Rendering 34
Lumigraph - Capture
Idea 1• Move camera carefully over
s,t plane• Gantry
– see Light Field paper
s,t u,v
CSE 576, Spring 2005 Image-Based Rendering 35
Lumigraph - Capture
Idea 2• Move camera anywhere• Rebinning
– see Lumigraph paper
s,t u,v CSE 576, Spring 2005 Image-Based Rendering 36
Lumigraph - RenderingFor each output pixel
• determine s,t,u,v• either
– find closest discrete RGB– interpolate near values s,t u,v
10
CSE 576, Spring 2005 Image-Based Rendering 37
Lumigraph - Rendering
For each output pixel• determine s,t,u,v
• either• use closest discrete RGB• interpolate near values s u
CSE 576, Spring 2005 Image-Based Rendering 38
Lumigraph - Rendering
Nearest• closest s• closest u• draw it
Blend 16 nearest• quadrilinear interpolation
s u
CSE 576, Spring 2005 Image-Based Rendering 39
Lumigraph - Rendering
Depth Correction• closest s• intersection with “object”• best u• closest u
s u CSE 576, Spring 2005 Image-Based Rendering 40
Lumigraph - Rendering
Depth Correction • quadralinear interpolation• new “closest”• like focus
[DynamicallyReparameterizedLight Fields,Isaksen, SG’2000]
s u
11
CSE 576, Spring 2005 Image-Based Rendering 41
Lumigraph - Rendering
Fast s,t,u,v finding• scanline interpolate• texture mapping• shear warp
s u CSE 576, Spring 2005 Image-Based Rendering 42
Lumigraph - Ray Space
3D space ray spacesurface depth ⇔ slope in ray space
CSE 576, Spring 2005 Image-Based Rendering 43
Lumigraph - Ray Space
Image effects:• parallax• occlusion• transparency• highlights
CSE 576, Spring 2005 Image-Based Rendering 44
Lumigraph - Demo
Lumigraph• Lion, Fruit Bowl
12
CSE 576, Spring 2005 Image-Based Rendering 45
Complex Light Field acquisitionDigital Michelangelo Project
– Marc Levoy, Stanford University– Lightfield (“night”) assembled by Jon Shade
CSE 576, Spring 2005 Image-Based Rendering 46
Unstructured Lumigraph
What if the images aren’t sampled on a regular 2D grid?
• can still re-sample rays
• ray weighting becomes more complex[Buehler et al., SIGGRAPH’2000]
CSE 576, Spring 2005 Image-Based Rendering 47
Surface Light Fields
Turn 4D parameterization around:image @ every surface pt.
Leverage coherence:compress radiance fn(BRDF * illumination)after rotation by n
CSE 576, Spring 2005 Image-Based Rendering 48
Surface Light Fields
[Wood et al, SIGGRAPH 2000]
13
CSE 576, Spring 2005 Image-Based Rendering 49
3D Representations
Image (and panoramas) are 2DLumigraph is 4DWhat happened to 3D?
• 3D Lumigraph subset• Concentric mosaics
CSE 576, Spring 2005 Image-Based Rendering 50
3D Lumigraph
One row of s,t plane• i.e., hold t constant
s,t u,v
CSE 576, Spring 2005 Image-Based Rendering 51
3D Lumigraph
One row of s,t plane• i.e., hold t constant• thus s,u,v• a “row of images”
[Sloan et al., Symp. I3DG 97]
s
u,v CSE 576, Spring 2005 Image-Based Rendering 52
Concentric Mosaics
Replace “row” with “circle” of images[Shum & He, SIGGRAPH’97]
14
CSE 576, Spring 2005 Image-Based Rendering 53
Concentric Mosaics
CSE 576, Spring 2005 Image-Based Rendering 54
Concentric Mosaics
Rendering
( as seenfrom above )
CSE 576, Spring 2005 Image-Based Rendering 55
Concentric Mosaics
Depth correction
CSE 576, Spring 2005 Image-Based Rendering 56
Concentric Mosaics
Demo
15
CSE 576, Spring 2005 Image-Based Rendering 57
2.5D RepresentationsImage is 2DLumigraph is 4D3D
• 3D Lumigraph subset• Concentric mosaics
2.5D• Layered Depth Images• Sprites with Depth (impostors)• View Dependent Surfaces (see Façade)
CSE 576, Spring 2005 Image-Based Rendering 58
Layered Depth Image
ImageDepthLayered
2.5 D ?
CSE 576, Spring 2005 Image-Based Rendering 59
Layered Depth Image
Rendering from LDI[Shade et al., SIGGRAPH’98]
• Incremental in LDI X and Y • Guaranteed to be in back-to-front order
CSE 576, Spring 2005 Image-Based Rendering 60
Layered Depth Image
Rendering from LDI
• Incremental in LDI X and Y • Guaranteed to be in back-to-front order
16
CSE 576, Spring 2005 Image-Based Rendering 61
Sprites with Depth
Represent scene as collection of cutouts with depth (planes + parallax)
Render back to front with fwd/inverse warping [Shade et al., SIGGRAPH’98]
Environment mattingand compositing
D. E. Zongker, D. M. Werner,B. Curless and D. H. Salesin.
SIGGRAPH'99
CSE 576, Spring 2005 Image-Based Rendering 63
Environment Matting
Capture the reflections and refractions of a real-world object
Composite object over a novel background
CSE 576, Spring 2005 Image-Based Rendering 64
Environment Matting - examples
Movie
17
CSE 576, Spring 2005 Image-Based Rendering 65
Environment Matting
Capture the mapping from eachimage pixel to a real-worldray direction(s)
CSE 576, Spring 2005 Image-Based Rendering 66
Acquisition setup
Use several monitors with stripes
CSE 576, Spring 2005 Image-Based Rendering 67
Environment matting equation
Captures foreground colorand background directions
CSE 576, Spring 2005 Image-Based Rendering 68
Environment matting - result
18
CSE 576, Spring 2005 Image-Based Rendering 69
Environment matting extensions
[Chuang et al., SIGGRAPH’2001]• accurate (multiple refractions):
• soft stripes• fast (video rate):
• color ramp
Image-Based Modeling
CSE 576, Spring 2005 Image-Based Rendering 71
Image Based Models
CSE 576, Spring 2005 Image-Based Rendering 72
Image-Based Modeling
Create 3D model (and texture maps) from images
• automated• (structure from motion,
stereo)• interactive
• Façade system
19
CSE 576, Spring 2005 Image-Based Rendering 73
Façade
1. Select building blocks2. Align them in each image3. Solve for camera pose
and block parameters(using constraints)
CSE 576, Spring 2005 Image-Based Rendering 74
View-dependent texture mapping
1. Determine visible cameras for each surface element
2. Blend textures (images) depending on distance between original camera and novel viewpoint
CSE 576, Spring 2005 Image-Based Rendering 75
Model-based stereo
Compute offset from block model
Some more results:
CSE 576, Spring 2005 Image-Based Rendering 76
Image-Based Faces
Estimate shape from imagesMatch metrics to shapeProject video onto shape
• Texture mapAnimate[Z. Liu et al., MSR-TR-2000-11]
20
CSE 576, Spring 2005 Image-Based Rendering 77
Hierarchy of Light Fields [Levoy]8D: Refractive/reflective environment5D: Plenoptic Function (Ray)4D: Lumigraph / Lightfield4D*: Environment Matte (single view)3D: Lumigraph Subset3D: Concentric Mosaics2.5D: Layered Depth Image2.5D: Image Based Models2D: Images and Panoramas
CSE 576, Spring 2005 Image-Based Rendering 78
Graphics/Imaging Continuum
Lumigraph Light field
Geometry centric Image centric
Warping InterpolationPolygon rendering + texture mapping
Fixed geometry
View-dependent geometry
View-dependent texture
Concentric mosaics
Sprites with depth
LDI
What lies beyondImage-Based Rendering?
CSE 576, Spring 2005 Image-Based Rendering 80
Video-Based Rendering
Image-Based Rendering:• render from (real-world) images for efficiency,
quality, and photo-realism
Video-Based Rendering• use video instead of still images for dynamic
elements and source footage• generate computer video instead of
computer graphics
21
CSE 576, Spring 2005 Image-Based Rendering 81
VBR ExamplesFacial animation
• Video Rewrite, …
Layer/matte extraction• Video Matting, …
Dynamic (stochastic) elements• Video Textures, …
3-D world navigation• Image-Based Realities, …
CSE 576, Spring 2005 Image-Based Rendering 82
Facial animation
Modeling from still images• [Pighin et al., SG’98]
Lip-synching from video• Video Rewrite
[Bregler et al., SG’97]• [Ezzat et al., SG’02]
Matting and Compositing
CSE 576, Spring 2005 Image-Based Rendering 84
http://grail.cs.washington.edu/projects/digital-matting/
22
CSE 576, Spring 2005 Image-Based Rendering 85
Video Matting
Pull dynamic α-mattefrom video withcomplex backgrounds
[Chuang et al. @ UW, SIGGRAPH’2002]
CSE 576, Spring 2005 Image-Based Rendering 86
Video Matting
CSE 576, Spring 2005 Image-Based Rendering 87
Shadow Matting
Transfer a shadow from one background to another:
• Extract and model photometry (darkening)• Extract and model geometry (deformation)
CSE 576, Spring 2005 Image-Based Rendering 88
Shadow Matting
23
CSE 576, Spring 2005 Image-Based Rendering 89
Shadow Matting
CSE 576, Spring 2005 Image-Based Rendering 90
Shadow MattingVideo
Video Textures
CSE 576, Spring 2005 Image-Based Rendering 92
Video Textures
How can we turn a short video clipinto an ∞ amount of continuous video?• dynamic elements in 3D games and presentations• alternative to 3D graphics animation?
[Schödl, Szeliski, Salesin, Essa, SG’2000]
24
CSE 576, Spring 2005 Image-Based Rendering 93
Video TexturesFind cyclic structure in the video
(Optional) region-based analysisPlay frames with random shuffleSmooth over discontinuities (morph)
CSE 576, Spring 2005 Image-Based Rendering 94
Region-based analysis
CSE 576, Spring 2005 Image-Based Rendering 95
Crossfading and morphing
Jump Cut Crossfade Morph
CSE 576, Spring 2005 Image-Based Rendering 96
Video portrait
25
CSE 576, Spring 2005 Image-Based Rendering 97
Dynamic scene element
Live waterfall in static panorama
Demo
CSE 576, Spring 2005 Image-Based Rendering 98
Interactive fish
CSE 576, Spring 2005 Image-Based Rendering 99
A complete animation
Video-Based Tours
26
CSE 576, Spring 2005 Image-Based Rendering 101
Video-Based Walkthroughs
Move camera along a rail (“dolly track”) and play back a 360° video
Applications:• Homes and architecture• Outdoor locations
(tourist destinations)
CSE 576, Spring 2005 Image-Based Rendering 102
Surround video acquisition system
OmniCam (six-camera head)
CSE 576, Spring 2005 Image-Based Rendering 103
OmniCamBuilt by Point Grey Research (Ladybug)
Six camera head
Portable hard drives, fiber-optic link
Resolution per image: 1024 x 768
FOV: ~100o x ~80o
Acquisition speed: 15 fps uncompressed
CSE 576, Spring 2005 Image-Based Rendering 104
Acquisition platforms
Robotic cart
Wearable
27
DemoA Virtual Home Tour
CSE 576, Spring 2005 Image-Based Rendering 106
Open issues
How to best sample and interpolate Light Field• (sub-?) pixel accurate stereo• reflections, refractions, …Compositing• how to insert Light Field into new environment• relighting• …?
CSE 576, Spring 2005 Image-Based Rendering 107
SummaryImage–Based Rendering• Light Fields and Lumigraphs• Panoramas and Concentric Mosaics• Matting: natural, environment, and shadows• Image-Based modelsVideo-Based Rendering• Facial animation• Video Textures and Animating Stills• Video-based tours
CSE 576, Spring 2005 Image-Based Rendering 108
SummaryImage–Based Rendering• Light Fields and Lumigraphs• Panoramas and Concentric Mosaics• Environment Matting• Image-Based modelsVideo-Based Rendering• Facial animation• Video matting• Video Textures and Animating Stills• Video-based tours