Surface Signals for GraphicsSurface Signals for Graphics

John SnyderJohn SnyderResearcherResearcher3D Graphics Group3D Graphics GroupMicrosoft ResearchMicrosoft Research

Why Surface Signals?Why Surface Signals?Why Surface Signals?Why Surface Signals?

Many useful types of surface signals:Many useful types of surface signals: texture map texture map [Catmull74, Blinn&Newell76][Catmull74, Blinn&Newell76] (color)(color) bump mapbump map [Max81][Max81] (normal) (normal) displacement map displacement map [Cook 84][Cook 84] (geometric offset) (geometric offset) geometry imagegeometry image (geometry) (geometry) bidirectional texture functionbidirectional texture function (precomputed shading) (precomputed shading) self-transfer textureself-transfer texture (spherical harmonic (spherical harmonic

coefs)coefs) ……

Simplicity of regular 2D imageSimplicity of regular 2D image

Support on current graphics hardware (e.g. pixel shaders)Support on current graphics hardware (e.g. pixel shaders)

Research questions:Research questions: How to generate and manipulate signals?How to generate and manipulate signals? What new graphics architectures?What new graphics architectures?

Many useful types of surface signals:Many useful types of surface signals: texture map texture map [Catmull74, Blinn&Newell76][Catmull74, Blinn&Newell76] (color)(color) bump mapbump map [Max81][Max81] (normal) (normal) displacement map displacement map [Cook 84][Cook 84] (geometric offset) (geometric offset) geometry imagegeometry image (geometry) (geometry) bidirectional texture functionbidirectional texture function (precomputed shading) (precomputed shading) self-transfer textureself-transfer texture (spherical harmonic (spherical harmonic

coefs)coefs) ……

Simplicity of regular 2D imageSimplicity of regular 2D image

Support on current graphics hardware (e.g. pixel shaders)Support on current graphics hardware (e.g. pixel shaders)

Research questions:Research questions: How to generate and manipulate signals?How to generate and manipulate signals? What new graphics architectures?What new graphics architectures?

Surface Signal Research ProjectsSurface Signal Research ProjectsSurface Signal Research ProjectsSurface Signal Research Projects

CreationCreationprecomputed radiance transferprecomputed radiance transfer ParameterizationParameterization

signal-specialized param.signal-specialized param.

RepresentationRepresentationgeometry imagesgeometry images

RenderingRenderingsignal-based signal-based

graphics architecturegraphics architecture

Motivation for Precomputed TransferMotivation for Precomputed TransferMotivation for Precomputed TransferMotivation for Precomputed Transfer

better light integration better light integration and light transportand light transport dynamic, area lights dynamic, area lights shadowing shadowing interreflectionsinterreflections

in real-timein real-time

point lightpoint light area lightarea light

area lighting,area lighting,no shadowsno shadows

area lighting,area lighting,shadowsshadows

Self-Transfer Signal (25D)Self-Transfer Signal (25D)Self-Transfer Signal (25D)Self-Transfer Signal (25D)

Basis Basis 1616

Basis Basis 1717

Basis Basis 1818

illuminateilluminate resultresult

......

......

Reduces shading to a Reduces shading to a 25D 25D dot product dot product (low-frequency lighting)(low-frequency lighting)

Self-Transfer Results (Diffuse)Self-Transfer Results (Diffuse)Self-Transfer Results (Diffuse)Self-Transfer Results (Diffuse)

No Shadows/Inter Shadows Shadows+InterNo Shadows/Inter Shadows Shadows+Inter

Self-Transfer Results (Glossy)Self-Transfer Results (Glossy) Self-Transfer Results (Glossy)Self-Transfer Results (Glossy)

No Shadows/Inter Shadows Shadows+InterNo Shadows/Inter Shadows Shadows+Inter

Self-Transfer DemoSelf-Transfer DemoSelf-Transfer DemoSelf-Transfer Demo

Geometry-basedGeometry-based(know geometry only)(know geometry only)

Signal-specializedSignal-specialized(know geometry+signal)(know geometry+signal)

Parameterization of Surface SignalsParameterization of Surface SignalsParameterization of Surface SignalsParameterization of Surface Signals

Measuring Parameterization QualityMeasuring Parameterization QualityMeasuring Parameterization QualityMeasuring Parameterization Quality

2D texture domain2D texture domain surface in 3Dsurface in 3D

linear maplinear map

singular values: singular values: γγ , , ΓΓ

Geometric Stretch MetricGeometric Stretch MetricGeometric Stretch MetricGeometric Stretch Metric

2D texture domain2D texture domain surface in 3Dsurface in 3D

linear maplinear mapTTTT

singular values: singular values: γγ , , ΓΓ

geometric stretch = geometric stretch = γγ22 + + ΓΓ22 Parameterize = minimize Parameterize = minimize surface integral of surface integral of geometric stretchgeometric stretch

geometric stretch: geometric stretch: γγff2 2 + + ΓΓff

22

signal stretch: signal stretch: γγhh2 2 + + ΓΓhh

22

geometric stretch: geometric stretch: γγff2 2 + + ΓΓff

22

signal stretch: signal stretch: γγhh2 2 + + ΓΓhh

22

Parameterize = minimize Parameterize = minimize surface integral of surface integral of

signalsignal stretch stretch

Signal Stretch MetricSignal Stretch MetricSignal Stretch MetricSignal Stretch Metric

ff

gg

domaindomain surfacesurface

signalsignal

hh = = ggff

Geometric stretch [Sander01]Geometric stretch [Sander01]Geometric stretch [Sander01]Geometric stretch [Sander01]

Conformal [Floater97]Conformal [Floater97]Conformal [Floater97]Conformal [Floater97]

Signal stretch [Sander02]Signal stretch [Sander02]Signal stretch [Sander02]Signal stretch [Sander02]

Geometric stretchGeometric stretch

Signal stretchSignal stretch

(64x64 texture)(64x64 texture)Results: Scanned ColorResults: Scanned ColorResults: Scanned ColorResults: Scanned Color

Results: Normal MapResults: Normal MapResults: Normal MapResults: Normal Map

Geometric stretchGeometric stretch Signal stretchSignal stretch

128x128 texture - multichart128x128 texture - multichart

Results: Precomputed Radiance TransferResults: Precomputed Radiance TransferResults: Precomputed Radiance TransferResults: Precomputed Radiance Transfer

25D signal – 256x256 texture25D signal – 256x256 texture

Geometric stretchGeometric stretch Signal stretchSignal stretch

3D graphics = 2D image processing?3D graphics = 2D image processing?3D graphics = 2D image processing?3D graphics = 2D image processing?

not quite not quite → use → use imagesimages but of but of surface signalssurface signals not not viewsviewsnot quite not quite → use → use imagesimages but of but of surface signalssurface signals not not viewsviews

• synthesize images from synthesize images from 3D surface descriptions3D surface descriptions run-time flexibilityrun-time flexibility – change view, lighting, rendering params – change view, lighting, rendering params

compactnesscompactness – single surface parameterization, not multiple views – single surface parameterization, not multiple views

high qualityhigh quality (global illumination) (global illumination) – resolution independence– resolution independence cheap creationcheap creation – no costly rigs & operator, easy to edit – no costly rigs & operator, easy to edit

• as preprocess, convert 3D descriptions to as preprocess, convert 3D descriptions to 2D image reps2D image reps ( (surface signalssurface signals) to accelerate run-time) to accelerate run-time

signals can be represented as signals can be represented as regular 2D imagesregular 2D images

rendering via rendering via general, programmable image processing opsgeneral, programmable image processing ops

Rendering FactorizationRendering FactorizationRendering FactorizationRendering Factorization

PreprocessPreprocess(slow)(slow)

Run-TimeRun-Time(fast)(fast)

• 3D surfaces (meshes)3D surfaces (meshes)

• 3D graphics: 3D graphics: ray tracing, Monte Carlo ray tracing, Monte Carlo integration, dynamics integration, dynamics simulation, encodingsimulation, encoding

• 2D images, streams2D images, streams

• 2D image processing2D image processingdecoding, interpolation / decoding, interpolation / decimation, programmable decimation, programmable pixel “shaders”, sample pixel “shaders”, sample gather gather

global illumination computation is too expensive from scratchglobal illumination computation is too expensive from scratch

surface surface signalssignalssurface surface signalssignals

EndEndEndEnd

PeoplePeoplePeoplePeople

Microsoft Research 3D Graphics Group:Microsoft Research 3D Graphics Group:Jim Blinn, Conal Elliot, Brian Guenter, Hugues Jim Blinn, Conal Elliot, Brian Guenter, Hugues Hoppe, Charles Loop, Don Mitchell, Kirk Hoppe, Charles Loop, Don Mitchell, Kirk Olynyk, Peter-Pike Sloan, John Snyder, Olynyk, Peter-Pike Sloan, John Snyder, Turner WhittedTurner Whitted

Collaborators:Collaborators:

Steven Gortler, Xianfeng Gu,Steven Gortler, Xianfeng Gu, Ziyad Hakura, Ziyad Hakura, Jesse Hall, Jan Kautz, Leonard McMillan, Jesse Hall, Jan Kautz, Leonard McMillan, Pedro Sander, Zoe WoodPedro Sander, Zoe Wood