Tutorial MPEG 3D Graphics

Marius Preda, PhD Chairman of MPEG 3DG

Institut TELECOM

Web3D 2011, June, 20, Paris

  It’s a crazy multimedia world! –  Network is everywhere… but very heterogeneous –  Terminals are “same same”… but different! –  Content must be designed with a priori knowledge of its future use –  Applications are platform-centric instead of user-centric

  Fragmented value chain

Authoring tools

Capturing devices

HW/SW providers

IPR holders

Production

Providers

Networks

Service scenarios

Transmission

Playback devices Terminal

manufacturers

End users

Consumption

INTEROPERABILITY NEEDED!!!

  No common representation –  Heterogeneous kind of data

•  Different types of geometry, appearance and animation models –  Always easier to specify a new data representation format than

learning an existent one

  Very different application domains

FAST TRANSPORT

TROUGHT THE

NETWORK

Size doesn’t matter Size is of main

importance Size doesn’t matter

FAST TRANSPORT

TROUGHT THE

NETWORK

Size doesn’t matter

Size is of main importance

Size doesn’t matter

To define a standard format for compressed 3D synthetic content. In other words to be for graphics what MP3 and AAC are for audio, MPEG-2 and MPEG-4 are for video and JPEG is for still images.

Additionally "MPEG 3D Graphics" aims at providing mechanisms such as APIs to enable easy integration and development of applications using its standard representation tools.

MPEG 3D Graphics within the MPEG standards family

Core technologies in MPEG 3D Graphics

Compressing other standards (COLLADA, X3D, …) with MPEG

Virtual Worlds Interoperability for avatars with MPEG





ISO

JTC1

SC24 Computer graphics and image processing

SC29 Coding of audio, picture, multimedia and hypermedia information

WG1 Coding of still pictures

WG11 Coding of moving pictures and audio

GKS, PHIGS CGM, VRML, X3D

Over 224 technical committees

JPEG

MPEG

Joint Technical Committee with IEC

MPEG has several names •  Common: MPEG = Moving Picture Experts Group •  Official: MPEG = ISO/IEC JTC1/SC29/WG11

MPEG has several subgroups

MPEG

MPEG-1 ISO/IEC 11172 ’92 Video & audio

(CD-ROM)

MPEG-2 ISO/IEC 13818 ’94 Video & audio (DVD & DVB)

MPEG-4 ISO/IEC 14496 ’99

Multimedia & interactive applications


Metadata (description of content)


Terminal & network

specification

Requirements Systems

Video

Audio

3D Graphics

... and produced several successful standards

MPEG-A MPEG-B MPEG-C MPEG-D MPEG-E MPEG-M MPEG-U MPEG-V

Naturals – still images, audio, 2D/3D video

Provide technologies for efficient compression and transmission

Synthetic - audio, 2D/3D objects and scenes

Composition, at end-user side, of natural and synthetic objects, into hybrid and interactive scenes

Compression Compression

Compression

MPEG-4 Features MPEG-4 Objects

MPEG-4 Objectives

MPEG-4 Terminal

Object descriptor

layer

Interactive Scene Description

MPEG-4 stream

Media data layer

Scene layer

MPEG-4 Features System architecture

MPEG-4 3D chronology







  IFS surfaces

Approximation of target surface •  Method: tesselation with planar facets •  Quality: first order (linear) ⇒ no smoothness (C0 continuity)

Mesh definition: IFS (Indexed Face Set) •  Connectivity: list of faces {…, Pn = {in0, in1, in2, …}, …}

⇒ arbitrary topology (non-manifold, open, higher genus, etc.) •  Geometry: list of vertices {…, Vk = (xk, yk, zk), …}

Mesh coding •  Connectivity (lossless): triangle strips, triangle+vertex trees, etc. •  Geometry (lossy): coordinate quantisation + prediction from conn.

LOD concept •  1976: Clark introduced the idea •  Main interest: rendering efficiency

Taxonomy of LOD extraction techniques •  Static vs. dynamic •  Global vs. local •  Progressive vs. hierarchical LODs

Successful simplification techniques •  1996: Hoppe’s edge collapses •  1997: Garland’s quadrics ⇒ qslim

Progressive 3D mesh coding •  1996: Hoppe’s PM (Progressive Mesh) •  1998: IBM’s PFS (Progressive Forest Split)

29519 Triangles: 69451 vs.




MPEG-4 (1999): IFSs •  Based on VRML97 •  Arbitrary topology meshes •  “Properties” (normals, colours and textures)

MPEG-4 Amd.1 (2000): 3DMC (3D Mesh Coding) •  40-50:1 compression of IFSs by IBM’s TS (Topological Surgery) •  Incremental transmission and rendering •  Progressive coding by IBM’s PFS •  Error resilience by SAIT

MPEG-4 (1999): IFSs •  Based on VRML97 •  Arbitrary topology meshes •  “Properties” (normals, colours and textures)

MPEG-4 Amd.1 (2000): 3DMC (3D Mesh Coding) •  40-50:1 compression of IFSs by IBM’s TS (Topological Surgery) •  Incremental transmission and rendering •  Progressive coding by IBM’s PFS •  Error resilience by SAIT

  IFS surfaces

  IFS surfaces   Patches

Approximation of target surface •  Method: tesselation with predefined curved patches

•  Quality: higher order (polynomic/rational) ⇒ Cn continuity

Mesh definition •  Connectivity: regular grid of quads. or triangles

⇒ planar topology •  Geometry: list of control points {…, Pk = (xk, yk, zk), …}

Mesh coding •  Connectivity (lossless): implicit •  Geometry (lossy): coordinate quantisation + prediction from conn.

Tensor product of cubic Bézier curves

Compression

 (3547 polygons; 1215 vertices)

vs.

(86 patches; 212 control points)

Single patch (4x4 control points) vs.

two patches (4x7 control points)

MPEG-4 Part 16 (2003): NURBS •  Based on VRML97 Amd., originally proposed by blaxxun •  Support for NURBS curves and patches

  Specific nodes for Bézier’s curves and patches (for increased efficiency)

•  Support for free-form deformations

  IFS surfaces   Patches   Subdivision surfaces

SS = limit of recursive refinement of base control mesh

NB: refinement affects both connectivity (of abstract graph) and

geometry (of 3D mapping)

z

y x

Geometry smoothing achieved with stencils particular to each scheme

1/8 1/8

½

½

-1/16 -1/16

-1/16 -1/16

-1/16 -1/16 9/16 9/16

Border/sharp vs. interior edge stencils

of “butterfly” scheme

SS inherently define hierarchically nested LODs

Approximation of target surface •  Method: tesselation with curved patches

•  Quality: higher order ⇒ Cn continuity

Mesh definition •  Connectivity: list of triangles/quads., e.g., {…, Tn = {in0, in1, in2}, …}

⇒ arbitrary (manifold) topology •  Geometry: list of control points {…, Pk = (xk, yk, zk), …}

Mesh coding •  Connectivity (lossless): as for polygonal (manifold) mesh •  Geometry (lossy): as for polygonal (manifold) mesh

Polygons + Are the simplest approach (linear approximation) + Can resolve fine details and handle arbitrary topologies – Lead to unstructured, huge meshes

Patches + Are a more powerful approach (higher order approximation) + Are convenient for coarse and smooth models – Need cumbersome trimming and stitching mechanisms

SSs + Connect and unify the two extremes above ++ Provide multi-resolution handles for hierarchical coding/editing

Dyn++’s “butterfly” (1990): triangular, primal, interpolating, C1

Loop’s (1987): triangular, primal, approximating, C2

Catmull-Clark’s (1978): quadrilateral, primal, approximating, C2

  IFS surfaces   Patches   Subdivision surfaces   Wavelet subdivision surfaces

Target surface Base mesh

Subdivide Add details

Price (requirements) •  Base mesh extraction •  Subdivision scheme = predictor •  Details (3D vectors) = prediction errors ⇒ remeshing

Prize (advantages) •  Predictive coding ⇒ immediate (if smooth target mesh)

Target surface Base mesh

Subdivide Add details

ZT ZT ZT ZT …

Spatial partitioning Adding details in appearing parts

Removing details in disappearing parts

by courtesy of FranceTelecom

MPEG-4 Part 16 (2003): “plain” + wavelet SSs •  “Plain” SSs for mesh smoothing

  Considered schemes: Catmull-Clark, [extended] Loop, butterfly   No details are added but…   normal control achievable through edge/vertex tagging of initial control mesh

•  Wavelet/detailed SSs for surface approximation   Possibly tagged base mesh   Details are added after each subdivision step, which are…   wavelet-transformed according to one of several possible schemes   Most suited for multi-resolution editing/animation   Most suited for view-dependent transmission

  IFS surfaces   Patches   Subdivision surfaces   Wavelet subdivision surfaces   Mesh Grid

RG (Reference-Grid)

CW (Connectivity-Wireframe)

Vertex offset is a relative value Update vertex position when grid is deformed or animated

  IFS surfaces   Patches   Subdivision surfaces   Wavelet subdivision surfaces   Mesh Grid   Solids

Solid primitives

Solid models: the “arithmetic of forms”

Implicit equation:

Quadrics (2nd order):

Quartics (4th order):

F1

* 0 1 2

F0 0 0 0 0

1 0 1 2

2 0 2 4

Multiplication of two forms

A cube = multiplication of 3 degenerated quadrics

Examples of solid operations

Virtual models

Architecture Mechanics

Biotechnology

Exact geometry

21 Kb 37 Kb 1.1 Mb 407 Kb

Compactness

  IFS surfaces   Patches   Subdivision surf.   Wavelet SS   Mesh Grid   Solids

  SC-3DMC

MPEG 3DGC Scalable Complexity 3D Mesh Compression

•  Not all 3DG applications have the same needs in compression •  Not all the 3DG applications can afford spending extra CPU/GPU for compression

QBCR

SVA

TFAN

Towards the continuum model: enlarge the application domain where MPEG-4 3DG can be used

SC3DMC

Same Quantization and Binarization blocks

MPEG 3DGC Scalable Complexity 3D Mesh Compression

x, y, z (floats) x, y, z (floats) x, y, z (floats)

i, j, k (integers) i, j, k (integers)

i, j, k (integers)

+ normals (floats) + colors (floats) + … (floats)

Connectivity

Attributes

SC-3DMC General Schema

TFAN

SVA

CABAC

AC

BP

FLB

Parallelogram

Delta Quantization

Barycentre

CABAC

AC

BP

FLB

Delta

Connectivity Analysis

Prediction Binarization Entropy Encoding

Connectivity (lossless)

BAC

Attributes (lossy)

5 different paths with different performances

Prediction Binarization Entropy Encoding

x, y, z

i, j, k

SC-3DMC Connectivity Analysis: Empty

Do Nothing

4 3

2

1

123234

SC-3DMC Connectivity Analysis: SVA

Previous Face Current Face Shared Vertex

4 3

2

1

1

2 3

4

5 6

3

2 4

1 5

1

2 3

1

2 3 OR

Mode 0 Mode 2

Mode 1 Mode 3

4 modes for encoding how consecutive faces share vertices

1, 2, 3, 2, 3, 4

1, 2, 3, 3, 4, 5

1, 2, 3, 4, 5, 6

1, 2, 3, 1, 2, 3

12304 123245

1231 5 1233

SC-3DMC Connectivity Analysis: TFAN

Split the mesh in set of triangle fans and encode each fan

1675 62937 2984

Transformed in local indices

SC-3DMC What we measure?

Encoding performances but also complexity for decoder (and encoder)

3DMC

BIFS Complexity

Performances

NCA

SVA-BP SVA-BAC

TFAN SVA-AC


  SC-3DMC

•  Visual Texture Coding •  Synthesized Textures •  Procedural Textures •  Depth Image-based Representation Less calculation

intensive

Very Compact representation for specific applications

Generic compression tool

2D Texture

3D Mesh

3D object

•  MPEG-4 3DMC •  MPEG-4 MeshGrid •  MPEG-4 Subdivision Surfaces

•  MPEG-4 VTC •  JPEG 2000

3D mipmaps

WT

Zero-Tree

Without mipmapping With mipmapping

Multiple In Place mapping = multi-resolution textures

WT

Zero-Tree

Computational Graceful Degradation: Quality => processing power

Region of Interest with resolution/quality selection

High Resolution

Low Resolution

Visual importance depends on viewing angle

Packet selection by using error-resilience markers


  VTC   Synthesized texture

  SC-3DMC

Line ( LN ), bounded by 2 Terminal Points ( TP ) Line Segment ( LS ), bounded by 2 Line Points ( LP )

The image cannot be displayed. Line marked as control line (Skeleton)

Line Color Profile ( LC ) Area Color Point ( A C )

The image cannot be displayed. Color Patch ( PA )

Sub - Texture ( ST )

Color profile

Vector Graphics Primitives

65x96 pixels 10 seconds animation 1.35 kB


  VTC   Synthesized texture   Procedural texture

  SC-3DMC

DEF Fabric ProceduralTexture { type 2 width 256 height 256 cellWidth 4 cellHeight 4 roughness 1 distortion 0.05 seed 114300 color [ 0.898 0.89418 0.95294, 0.34118 0.29418 0.70196, 0 0 0, 0 0 0 ] aWarpmap [ 0 0, 0.03 1, 0.88 1, 1 0 ] bWarpmap [ 0 0, 0.48 1, 1 0 ] aWeights [ 0, 0, 0, 0, 0, 0, 0.56, 0, 0, 0, 0, 0, 0, 0, 0.20, 0.24 ] bWeights [ 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0 ] }

DEF Marble ProceduralTexture { width 256 height 256 roughness 1 seed 22209 color [ 0.8 0.7098 0.6902, 0.95686 0.8902 0.87451, 0.87451 0.37255 0.23529, 0.95686 0.8902 0.87451 ] aWarpmap [ 0 1, 0.33 0, 1 1 ] bWarpmap [ 0 0, 0.55 0, 0.6 1, 0.65 0, 1 0 ] bWeights [ 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0 ] }


  VTC   Synthesized texture   Procedural texture   DIBR

  SC-3DMC

Reconstruct 3D representation from projections

Depth image

Projection

  VTC   Synthesized texture   Procedural texture   DIBR   Point Texture


  SC-3DMC

  Interpolators   VTC   Synthesized texture   Procedural texture   DIBR   Point Texture


  SC-3DMC

Straightforward animation based on key-frames

Piece-wise linear functions defined by a set of pairs time-value

Standardized by MPEG-4 for: - position - orientation - scale

- coordinate in IFS - normals in IFS

Large amount of data for high quality, smooth animation

Exploit temporal redundancy for the most common interpolators

Re-sampling, sub-sampling: two methods supported by MPEG-4

Path preserving mode Key preserving mode

Three schemes supported by MPEG-4

Coordinate Interpolators Orientation Interpolators

Position Interpolators

Sub-sampling or re-sampling based on minimal distortion

Key(time)

Key value

N-1 key values

Key value

N-2 key values

Key(time)

Key value

N key values

Key(time)

Key value

N-3 key values

Key(time)

Re-sampling, sub-sampling: two methods supported by MPEG-4

Path preserving mode Key preserving mode

Three schemes supported by MPEG-4

Coordinate Interpolators Orientation Interpolators

Position Interpolators

A dedicated elementary stream for IC, multiplexed into an AFX stream

  Interpolators   Bone-Based



  SC-3DMC

Two frameworks: human-like (FBA) and generic skeleton (BBA)

  Face, MPEG-4 V1, 1999

  Skinned Model, MPEG-4 Part 16, 2003

  Body, MPEG-4 Amd1, 2000

Seamless mesh affected by a hierarchical skeleton

  Geometry: Seamless mesh: shapes sharing the same vertices list

  Texture: Image Mapping on vertices sub-set

  Hierarchy: Skeleton layer Muscle layer

Right balance between control parameters and influence volume

  1D controllers: bone & muscle

  for each bone and each muscle - a list of affected vertices - a measure of affectedness are provided

Basic comp.: prediction, freq.transform, quantization and entropy encoding

Frame #n Uncompressed BAP/FAP/BBA

Binary file

Frame P

Prediction

Arithmetic coding

Quantization

Frame I

Arithmetic coding

Quantization

Segment #n Uncompressed BAP/FAP/BBA

DCT

Binary file

Segment P

Prediction

DC Q

Segment I

Huffman coding

DC Q

DC Coeff. AC Coeff.

Huffman coding

AC Q

Very low bit-rate

  Body, MP4 Amd1, 5-30 kbps

  Face, MP4 V1, 2kbps

  Skinned Model, MP4 Part 16, 5-30 kbps for a human like skeleton

  Interpolators   Bone-Based   Morphing



  SC-3DMC

Local and precise control for shape deformation

  Defined as a base mesh and a collection of target meshes

  Animation obtained by updating the weights of the target meshes

  BBA stream updated to include morph data

  Usable for any kind of 3D object

  Interpolators   Bone-Based   Morphing   FAMC



  SC-3DMC

  Cluster the vertices with respect to their motion

  Encode a cluster motion by an affine transform

  Encode the residual error at vertex level by traditional approach (DCT/W, quantization, entropy encoding)


  VTC   Synthesized texture   Procedural texture   DIBR   Point Texture   Remote and

programmatic


  SC-3DMC

Complex application scenario can be built

  Programmatic animation - ECMA Script

part of the scene description - Java Code standardized API for accessing the scene graph

scene Script Node

MPEG-J Stream scene

Bifs-Command scene   Remote animation

BIFS-Anim & BIFS Command

Bifs-Anim


  VTC   Synthesized texture   Procedural texture   DIBR   Point Texture   Remote and

programmatic

  IFS surfaces   Patches   Subdivision surf.   Wavelet SS   Mesh Grid   Solids   SC-3DMC

MPEG 3DG How measuring compression performances?

On-line benchmarking platform: www.MyMultimediaWorld.com

•  Allows easy integration of proprietary algorithms by using an API in C++

•  No need to disclaim the algorithm source code

•  Benchmark automatically updated for new content

•  Restricts and refines the benchmark by means of easy-to-control parameters.


On-line benchmarking : www.MyMultimediaWorld.com

MP4

Extended MP7

Filter and Presentation Engine

Web site 3D compression benchmark Benchmark 2 Benchmark 3

Indexed MDB

3D Compression benchmark manager

Algorithm 1 Algorithm n Algorithm 2 …

AP

I

Ben

chm

ark

man

ager

1 B

enchmark m

anager 2


On-line benchmarking : www.MyMultimediaWorld.com

Proprietary Coder library

Proprietary Decoder library

BitStream (1,n)

n

.Compressed_VB (1,n)

Decoder

DumpCompressedVB

Encoder

GetNbBitstream

DumpBitSream

Vertex Buffer

Benchmarking platform, per-object visualization

- object properties (number of vertices/triangles, number of components and files size, - distortion graph (linear or logarithmic), - compression gain with respect to 3DMC. -  encoding time -  decoding times.


Benchmarking platform, global visualization

Filters

- semantic category, - average distortion, - number of vertices, - number of connected components in the object, - database subset.


Benchmarking platform, global visualization






Geometry compression: -compression ratio 40:1

Animation compression: -compression ratio 100:1

XML Scene Representation Any XML

(Binary) XML XML Binarisation

3D Graphics Compression

Layer 1:

Textual XML representation of the scene




Layer 2:

Binarized XML layer

Contains the unclassified elements of the scene graph




Layer 3:

Compressed layer

Contains specific types of media information (geometry, animation, ...)




Result:

Multiplexed layers 2 and 3




Comp ratio: 40-70:1

MPEG-4 P25 Encoder side: possible implementation (informative)

XMT,

COLLADA,

X3D

MP4

Encoder Side

Lossless for the data structure, lossless or lossy for graphics primitives

MPEG-4 P25 Decoder side (normative)

XMT,

COLLADA,

X3D

MP4

Decoder Side





Standards for Avatars as visualization support

Standard Generic Features Avatar representation

COLLADA 3D objects/ scenes Generic Object

VRML/ X3D 3D objects /scenes

Application behavior

H- Anim

MPEG-4 2D/3D objects/ scenes

Application behavior

Compression

A set of dedicated nodes

A dedicated compressed stream

Standards for Avatars as interaction support

Representation Features HumanML human physical description, emotion

EmotionML emotion, facial expressions, gestures

BML speech, gesture, gaze

MPML speech VHML facial and body animation, emotional representation

CML character attribute and animation definition

Why none of the existing standards is solving the issue of avatar interoperability in Virtual Worlds?

–  The Virtual Worlds are proprietary applications and the 3D assets including the avatars have economical value

–  The Virtual Worlds and in general 3D applications have specific data format allowing rendering optimization

However, while maintaining a strict control for economic and technical reasons, Virtual Worlds allow users to personalize the avatars

Attributes that can be modified by the user

Specify the set of Personalization Parameters (PP) that transforms a template of an arbitrary Virtual World into the user designed avatar

Avatar Template

What avatar feature can be personalized?

Mainly the appearance

Analysis of SecondLife, IMVU, Entropia Universe, SonyPlaystation and HumanML

Very heterogeneous set of personalization parameters

Second Life

Entropia Universe

Nintendo Wii

HumanML

PlayStation

“Appearance” element

<Appearance>

<Body> <BodyHeight value=165/> <BodyFat value=15/> </Body>

<Head> <HeadShape value="oval"/> <EggHead value="true"/> </Head>

</Appearance>

Interoperability at the Animation level

<Animation>

<Greeting> <Salute>salut</Salute> <Cheer>cheer</Cheer>

</Greeting>

<Fighting> <shoot>pousse</shoot> <throw>throw</throw> </Fighting>

</Animation>

Motion retargeting

Avatar Template in VW1

“Walk” animation In VW1

Avatar Template in VW2

No “Walk” animation

defined in VW2

“Control” element

<Control> <BodyFeaturesControl > <UpperBodyBones> <LCalvicle>my_LCalvicle</LCalvicle> <RClavicle>my_RCalvicle</RClavicle> </UpperBodyBones> </BodyFeaturesControl>

<FaceFeaturesControl> <HeadOutline> <Left X=0.23 Y=1.25 Z=7.26/> <Right X=0.25 Y=1.25 Z=7.21/> </HeadOutline> </FaceFeaturesControl>

</Control>

Control

Body Control

Face Control

MPEG-V + MPEG-4 The Avatar in

VW1

The Avatar in VW2

The Avatar in an external player

MPEG-V (Personalization

Parameters)

MPEG 3DGC Ongoing work

MPEG 3DGC Multi-Resolution 3DMC

Current 3DG content in VW

Tomorrow’s 3DG content in VW (100 times denser)

© Second Life © Samsung

© OTOY © OTOY

MPEG 3DGC Reconfigurable Graphics Coding

A framework that allows to set up the decoder at run time

Thank you!

www.MyMultimediaWorld.com

www-artemis.it-sudparis.eu

[email protected]

MPEG 3DGC Multi-resolution 3DMC

Progressive mesh [Hoppe’96] Progressive Forest Split (PFS) [Taubin’98] Patch coloring [Cohen-Or’99]

Valence-based decimation approach [Alliez’01] Octree-based compression [Peng’05]

Spectral coding [Karni’01]

MPEG 3DGC Multi-resolution 3DMC : two proposed methods

1. Progressive TFAN Original

Interpolated with 1% of the vertices


2. KLT encoder

1

1

1

2

2

2


1. Work on -  document the use case scenarios (object examination, navigation, …) -  database -  comparison with other codecs -  comparison method (PSNR, Resolution) -  platform for testing

2. You are invited to contribute

Technology

Tutorial MPEG 3D Graphics