Realtime Per Face Texture Mapping (PTEX)

Per-Face Texture Mapping for Per-Face Texture Mapping for Realtime RenderingRealtime Rendering

““Realtime Ptex”Realtime Ptex”

Quick NoteQuick Note

• For CUDA/Compute folks:For CUDA/Compute folks:

– Ptex != PTXPtex != PTX

GoalsGoals

Render native Ptex datasets in real-time on commodity hardware

Remove texture seams from textured modelsRemove texture seams from textured models Remove expensive, manual model unwrap step from art Remove expensive, manual model unwrap step from art

pipelinepipeline Support arbitrary resolutions on a per-face basisSupport arbitrary resolutions on a per-face basis

Video Time!Video Time!

Video RecapVideo Recap

Stored as 8Kx8KRendered as 2Kx2K

Model StatisticsModel Statistics

• ~800 FPS on GTX 460 with no optimization~800 FPS on GTX 460 with no optimization

• 278M of Color Texture Data278M of Color Texture Data

• 5812 Patches5812 Patches

General StepsGeneral Steps

LoadModelLoadModel

RenderRender

Preprocess

Draw Time

Bucket and Sort

Generate Mipmaps

Fill Borders

Pack Texture Arrays

Reorder Index Buffer

Pack Patch

Constants

Pack Patch

Constants

Red: Vertex and Index dataGreen: Patch Constant informationBlue: Texel dataOrange: Adjacency information

Load ModelLoad Model

LoadModelLoadModel

RenderRender

Preprocess

Draw Time

Bucket and Sort

Generate Mipmaps

Fill Borders

Pack Texture Arrays

Pack Patch

Constants

Pack Patch

Constants

Load ModelLoad Model• Vertex DataVertex Data

– Any geometry arranged as a quad-based meshAny geometry arranged as a quad-based mesh

– Example: Wavefront OBJExample: Wavefront OBJ

• Patch TexturePatch Texture

– Power-of-two texture imagesPower-of-two texture images

• Adjacency InformationAdjacency Information

– 4 Neighbors of each quad patch4 Neighbors of each quad patch

• Easily load with library available from Easily load with library available from http://ptex.us/http://ptex.us/

Load Model (cont’d)Load Model (cont’d)

• Texel DataTexel Data

– Per face, load the largest available mipmap level from the Per face, load the largest available mipmap level from the source filessource files

– In memory, place the loaded texel data into a memory In memory, place the loaded texel data into a memory buffer that has a fixed-size texel border regionbuffer that has a fixed-size texel border region

– The borders must be big enough for the largest filter kernel The borders must be big enough for the largest filter kernel you will support (Border size = ½ filter kernel size)you will support (Border size = ½ filter kernel size)

• Bilinear (2x2 kernel): 1 pixel borderBilinear (2x2 kernel): 1 pixel border

• 16x Aniso (16x16 kernel): 8 pixel border16x Aniso (16x16 kernel): 8 pixel border

Load ModelLoad ModelVB: … IB:

Bucket and SortBucket and Sort

LoadModelLoadModel

RenderRender

Preprocess

Draw Time

Bucket and Sort

Generate Mipmaps

Fill Borders

Pack Texture Arrays

Pack Patch

Constants

Pack Patch

Constants

• Bucket ptex surfaces into groups by Aspect RatioBucket ptex surfaces into groups by Aspect Ratio

– Each Aspect Ratio will be one bucketEach Aspect Ratio will be one bucket

– 1:1 bucket, 2:1 bucket, 4:1 bucket, etc1:1 bucket, 2:1 bucket, 4:1 bucket, etc

• Then, within each bucket, sort by decreasing surface Then, within each bucket, sort by decreasing surface size and assign IDs.size and assign IDs.

– This allows us to densely pack texture arrays, avoiding “empty” This allows us to densely pack texture arrays, avoiding “empty” surfaces.surfaces.

Bucket 4:1Bucket 1:1

Reorder Index BufferReorder Index Buffer

LoadModelLoadModel

RenderRender

Preprocess

Draw Time

Bucket and Sort

Generate Mipmaps

Fill Borders

Pack Texture Arrays

Pack Patch

Constants

Pack Patch

Constants

Reorder Index BufferReorder Index Buffer

• OriginalOriginal • Post Sort, we have 2!Post Sort, we have 2!

IB:IB(1:1):

IB(4:1):

Generate MipmapsGenerate Mipmaps

LoadModelLoadModel

RenderRender

Preprocess

Draw Time

Bucket and Sort

Generate Mipmaps

Fill Borders

Pack Texture Arrays

Pack Patch

Constants

Pack Patch

Constants

• Walk through surfaces, and generate a mipmap chain Walk through surfaces, and generate a mipmap chain from native size to (MinFilterSize x MinFilterSize) for from native size to (MinFilterSize x MinFilterSize) for each surface. each surface.

– Bilinear stops at 2x2, 8xAniso stops at 8x8Bilinear stops at 2x2, 8xAniso stops at 8x8

• Ptex data files do not guarantee complete mipmap Ptex data files do not guarantee complete mipmap chains—although the library can generate all levels for chains—although the library can generate all levels for you—with pre-multiplied alpha.you—with pre-multiplied alpha.

• Mipmap chains stop to allow for unique pinning values Mipmap chains stop to allow for unique pinning values in the cornersin the corners

• Done for every surface, but only inside the surface—the Done for every surface, but only inside the surface—the border is not touched.border is not touched.

Fill BordersFill Borders

LoadModelLoadModel

RenderRender

Preprocess

Draw Time

Bucket and Sort

Generate Mipmaps

Fill Borders

Pack Texture Arrays

Pack Patch

Constants

Pack Patch

Constants

• Copy neighbor texels into border area of this surface’s Copy neighbor texels into border area of this surface’s mip levelmip level

– Match source and destination number of pixels when Match source and destination number of pixels when possiblepossible

• Bordered textures are the heart of the logical realtime Bordered textures are the heart of the logical realtime ptex solutionptex solution

• Allows 1-2 texture lookups per ptex sample requestAllows 1-2 texture lookups per ptex sample request

– 1 if not performing tween-mip-level interpolation, 2 otherwise1 if not performing tween-mip-level interpolation, 2 otherwise

Pack Texture ArraysPack Texture Arrays

LoadModelLoadModel

RenderRender

Preprocess

Draw Time

Bucket and Sort

Generate Mipmaps

Fill Borders

Pack Texture Arrays

Pack Patch

Constants

Pack Patch

Constants

Texture ArraysTexture Arrays

• Like 3D / Volume Textures, except:Like 3D / Volume Textures, except:

– No filtering between 2D slicesNo filtering between 2D slices

– Only X and Y decrease with mipmap level (Z doesn’t)Only X and Y decrease with mipmap level (Z doesn’t)

– Z indexed by integer index, not [0,1]Z indexed by integer index, not [0,1]

• E.g. (0.5, 0.5, 4) would be (0.5, 0.5) from the 5E.g. (0.5, 0.5, 4) would be (0.5, 0.5) from the 5 thth slice slice

• API SupportAPI Support

– Direct3D 10+: Texture2DArrayDirect3D 10+: Texture2DArray

– OpenGL 3.0+: GL_TEXTURE_2D_ARRAYOpenGL 3.0+: GL_TEXTURE_2D_ARRAY

• Copy all generated data into Texture2DArrayCopy all generated data into Texture2DArray

• Each Texture2DArray represents a single mipmap levelEach Texture2DArray represents a single mipmap level

– Texture2DArrays present a view of the data that is efficient Texture2DArrays present a view of the data that is efficient for GPU layoutfor GPU layout

– Logical Textures cut across the same page index of every Logical Textures cut across the same page index of every Texture2DArrayTexture2DArray

Logical Texture Layout

10x10x3(1+8+1)x(1+8+1)x3

6x6x4(1+4+1)x(1+4+1)x4

4x4x4(1+2+1)x(1+2+1)x4

GPU Layout Texture2DArray

gColor[0]

gColor[1]

gColor[2]

Pack Patch ConstantsPack Patch Constants

LoadModelLoadModel

RenderRender

Preprocess

Draw Time

Bucket and Sort

Generate Mipmaps

Fill Borders

Pack Texture Arrays

Pack Patch

Constants

Pack Patch

Constants

Pack Patch ConstantsPack Patch Constants

• Each primitive has a “PatchInfo” struct:Each primitive has a “PatchInfo” struct:

– TextureId – which array slice contains our dataTextureId – which array slice contains our data

– TopMipLevel – the index of the top-most mipmap level for this TopMipLevel – the index of the top-most mipmap level for this texturetexture

– FlipUVs – whether or not to flip UVs, allows 1:2 and 2:1 to be FlipUVs – whether or not to flip UVs, allows 1:2 and 2:1 to be grouped into same bucketgrouped into same bucket

– MaxMipLevels – Maximum mipmap level for each edgeMaxMipLevels – Maximum mipmap level for each edge

LoadModelLoadModel

RenderRender

Preprocess

Draw Time

Bucket and Sort

Generate Mipmaps

Fill Borders

Pack Texture Arrays

Pack Patch

Constants

Pack Patch

Constants

RenderRender

• Brief Recap of D3D11 Pipe stagesBrief Recap of D3D11 Pipe stages

New Direct3D 11 StagesNew Direct3D 11 Stages

IAIA VS HS TSTS DS GS RASRAS PS OMOM

Programmable (Shader)

Fixed Function

RenderRender

• In the Hull ShaderIn the Hull Shader

– Store pre-expansion PrimitiveID to output control pointsStore pre-expansion PrimitiveID to output control points

– This is used everywhere to determine which set of Patch This is used everywhere to determine which set of Patch Constants are owned by the currently running thread (in Constants are owned by the currently running thread (in Domain, Geometry or Pixel Shaders)Domain, Geometry or Pixel Shaders)

RenderRender

• In the Domain ShaderIn the Domain Shader

– Vertices belonging to a quad meshes are evaluated with a Vertices belonging to a quad meshes are evaluated with a domain location, which is (0,0)-(1,1) for each patchdomain location, which is (0,0)-(1,1) for each patch

– Use this value to store our UV locationUse this value to store our UV location

RenderRender

• Texture lookups in Domain or Pixel Shader are replaced Texture lookups in Domain or Pixel Shader are replaced with a “ptex” sample function.with a “ptex” sample function.

– Determines which logical texture to work fromDetermines which logical texture to work from

– Compute mipmap level(s) to accessCompute mipmap level(s) to access

– Scale and bias computed (u,v) by mipmap sizeScale and bias computed (u,v) by mipmap size

– Lookup texels, return weighted averageLookup texels, return weighted average

Texture Lookup Shader CodeTexture Lookup Shader Code

• Traditional (D3D11)Traditional (D3D11)

return return

gTxDiffuse.Sample(gTxDiffuse.Sample(

gSampler,gSampler,

I.fTextureUV );I.fTextureUV );

• PtexPtex

return return

ptex( gTxDiffuse,ptex( gTxDiffuse,

gSampler, gSampler,

I.uPrimitiveId,I.uPrimitiveId,

I.fTextureUV );I.fTextureUV );

Complex logic hiddenin single function call

Questions?Questions?

• jmcdonald at nvidia dot comjmcdonald at nvidia dot com

• Brent dot Burley at disneyanimation dot comBrent dot Burley at disneyanimation dot com

• http://ptex.us/http://ptex.us/

• http://groups.google.com/group/ptexhttp://groups.google.com/group/ptex

• Or visit our studio sessionOr visit our studio session

– Monday, 8 August @ 4:30 PM – 5:00 PMMonday, 8 August @ 4:30 PM – 5:00 PM

– The Studio / West Building, Ballroom AThe Studio / West Building, Ballroom A

Additional ConsiderationsAdditional Considerations

• Filtering across edges with differing numbers of pixelsFiltering across edges with differing numbers of pixels

• Filtering across cornersFiltering across corners

Filtering across edgesFiltering across edges

Filtering across cornersFiltering across corners

• Corners with valence > 4 Corners with valence > 4 cannot be exactly cannot be exactly matched with a matched with a bilerp (4 samples)bilerp (4 samples)

• The solution is a bit more involved.The solution is a bit more involved.

– First, walk the mesh and determine which corners are sharedFirst, walk the mesh and determine which corners are shared

– For each shared group, determine the correct value for when For each shared group, determine the correct value for when we’re exactly at that corner. E.g. Simple Average.we’re exactly at that corner. E.g. Simple Average.

– Then, modify every mipmap level of every surface of that Then, modify every mipmap level of every surface of that group s.t. the shared corner has the same valuegroup s.t. the shared corner has the same value

– When you’re in the corner, everyone will perform the same When you’re in the corner, everyone will perform the same lookup—regardless of mipmap level—and continuity prevailslookup—regardless of mipmap level—and continuity prevails

• For Realtime Ptex, we apply pinning to all corners, For Realtime Ptex, we apply pinning to all corners, regardless of valence.regardless of valence.

Pinned CornersPinned Corners

Questions redux?Questions redux?

• jmcdonald at nvidia dot comjmcdonald at nvidia dot com

• Brent dot Burley at disneyanimation dot comBrent dot Burley at disneyanimation dot com

• http://ptex.us/http://ptex.us/

• http://groups.google.com/group/ptexhttp://groups.google.com/group/ptex

• Or visit our studio sessionOr visit our studio session

– Monday, 8 August @ 4:30 PM – 5:00 PMMonday, 8 August @ 4:30 PM – 5:00 PM

– The Studio / West Building, Ballroom AThe Studio / West Building, Ballroom A

Realtime Per Face Texture Mapping (PTEX)

Technology

EKG realtime

Realtime advertising

Realtime search

Realtime webapps

Realtime Overview

Tricks to Making a Realtime SurfaceView Actually Perform in Realtime - Maarten Edger

Ptex: Per-face Texture Mapping for Production Rendering

Goldengate11g Realtime

RealTime Ethernet

Beyond Realtime Marketing: Become A Realtime Brand. -- Chris Lindland, Betabrand

TEXTURE ANALYSIS USING GABOR FILTERSGABOR FILTERSvplab/courses/CV_DIP/PDF/Lect-Gabor_filt.pdf · Texture Types • Definition of Texture Texture Types Definition of Texture • Texture

RealTime Transcriptions

Per-Face Texture Mapping for Realtime Rendering

Project on Ptex Solutions (Autosaved) 2

Realtime web2012

Ptex: Per-face Texture Mapping for Production Rendering Brent Burley and Dylan Lacewell Walt Disney Animation Studios EGSR 2008 (See attached slide notes

ThePlasmodiumtransloconofexportedproteins(PTEX ...fad.macmate.me/McFadden_Lab/Mcfadden_Papers/2013/Matthews_Mol... · ThePlasmodiumtransloconofexportedproteins(PTEX) componentthioredoxin-2isimportantformaintainingnormal

Ptex: Per-face Texture Mapping for Production …disney-animation.s3.amazonaws.com/.../opensource/ptex/ptex-slides.pdfPtex: Per-face Texture Mapping for Production Rendering Brent

Realtime Pcr

Introduction to Realtime Linux Bryan Che · 2020. 4. 15. · Realtime Linux Roadmap Incorporate the realtime patchset into the mainline kernel Current realtime kernel is 2.6.26-based