Realistic Real-Time Rendering of Eyes and Teeth

Matt Chiang and Graham Fyffe

ICT Technical Report No. ICT-TR-01-2010. September, 2010.

1 Real-time rendering of faces

We demonstrate realistic real-time renderings of a staticface along with integrated eyeballs and teeth under bothpoint lighting and image based lighting. Example ren-derings are shown in Figure 6. The software prototyperuns on nVidia graphics cards with 1 GB of GPU mem-ory. This includes a Quadro FX 3800 used for develop-ment, as well as the consumer-level GeForce GTX 285used for testing.

1.1 Environment Lighting

The rendering software supports direct point lighting andalso environment lighting. We approximate environmentlighting based on captured lighting environments. Weuse an image-based environment illumination techniqueto light the face with a specified environment map. An ex-ample is shown in Figure 1.The environment map is pre-

filtered with several blur kernels, for diffuse integration,glossy specular integration, and semi-glossy specular in-tegration.

1.2 Diffuse Shading Component

1.2.1 Soft Shadow Terminator

For directional light sources, diffuse shading is com-monly performed using the Lambertian shading model:D = αd max(0,nd · l) where αd is the diffuse albedo, nd

is the diffuse surface normal, and l is the direction to-wards the light. However, this model has a hard shadowterminator. The materials of a human face do not exhibithard shadow terminators, due to their subsurface scatter-ing properties. To approximate this effect, we replace the

Figure 2: Lambertian shading model versus diffuse shad-ing model with soft terminator.

Lambertian shading model with an alternate model havinga soft shadow terminator:D = αd exp

�− 1

2 (cos−1(nd · l))2 − 110 (cos−1(nd · l))6�.

This model is nearly equal to the Lambertian model, butit approaches zero more smoothly (see Figure 2).For environment light sources, diffuse shading is per-formed using a normal-based look-up into the environ-ment map that has been pre-filtered with a diffuse blurkernel: D = αdEd(nd), where Ed is the diffuse-filteredenvironment map.

1.2.2 Hybrid Normal Mapping

The rendering software adapts the ICT Graphics Labora-tory’s hybrid normal map rendering technique [2] to theGPU in order to provide photoreal renderings of humanfaces in real-time. Essentially, a different diffuse normalmap is provided for each color channel, and the diffuse

1

Figure 1: The real-time rendering software demonstrating environment illumination using the Uffizi Gallery environ-ment map from www.debevec.org.

2

lighting component is computed independently for eachcolor channel. Additionally, there is a separate specularnormal map for the specular lighting component. Hy-brid normal maps approximate the effects of ambient oc-clusion and subsurface scattering, which are effectivelybaked in. If hybrid normal maps are not available, theymay be computed by the Virtual Light Stage technique(described later), or by a combination of ambient occlu-sion and subsurface scattering rendering (described next).

1.2.3 Ambient Occlusion

Ambient occlusion is the soft shadowing that occurs onsurfaces by the partial occlusion of ambient illumination.We implemented a real-time screen-space ambient occlu-sion method to compute occlusion on the eyeballs, andin and around the teeth and the surrounding mouth re-gion. The final pixel color is computed by multiplyingthe shaded color of the material by the ambient occlusionterm in a compositing fragment shader.

1.2.4 Subsurface Scattering Approximation

Since hybrid normal maps are not available for eyes andteeth, we investigated and implemented methods to ap-proximate subsurface scattering in the sclera (the white ofthe eye) and the teeth.For eyes, we assume 10% of the light entering through theeye socket is scattered inside the eyeball and re-emittedevenly from any point on the sclera. Thus we blend thegeometry normal with the direction from the front of theface:nd =

0.9ng+0.1n f

|0.9ng+0.1n f |

where ng is the sclera geometry normal and n f is the di-rection pointing out the front of the face. For the iris, weused the Virtual Light Stage technique to compute hybridnormals as observed through the refractive cornea.For teeth, we assume 90% of the light scatters by diffusenormals computed from an approximation of the teethmodel geometry as a cylindrical surface:

nd =0.9nc+0.1ng

|0.9nc+0.1ng|

where ng is the teeth geometry normal and nc is the cylin-der geometry normal. We also computed hybrid normal

maps for the teeth using the Virtual Light Stage approach,however these normals were not used in the software.

1.3 Specular Shading Component

1.3.1 Blinn-Phong with Fresnel term

For directional light sources, specular shading is per-formed using the normalized Blinn-Phong shading modelwith a Fresnel term based on Schlick’s approximation:

S = αs

(k+2)(k+4)8π(2−k/2+k)

max(0,ns ·h)kF ;

F = (i−1)2+4i(1−max(0,ns·v))5

(i+1)2

where αs is the specular albedo, ns is the specular surfacenormal,h = l+v

|l+v| , v is the direction towards the camera, k

is the specular exponent (800 for teeth, 50 for eyes, anda blend of 40 and 100 for skin), and i is the index of re-fraction of the material (1.33 for eyes, 1.4 for teeth andskin).For environment light sources, we use a reflection-direction-based look-up into the environment map thathas been pre-filtered with a blur kernel based on the spec-ular exponent: S = αsEk(2(ns ·v)ns−v)F , where Ek is thefiltered environment map for specular exponent k.

1.4 Skin

We scanned a face using the technique of [2] to obtainthe skin geometry of the face and the diffuse and specu-lar normal and albedo maps. We use the hybrid normalmapping method for diffuse illumination. We do not useambient occlusion for the skin because these are alreadybaked into the hybrid normals, so we render a mask to ex-clude the skin from the ambient illumination compositingstep. Specular illumination is computed with the Blinn-Phong model and Schlick Fresnel term, using the capturedspecular normal map and specular albedo map.

1.5 Eyeball

The eyeball is modeled as generic eye geometry. The tex-ture map of the whole eye (sclera and iris) is based on anumber of photographs of the eye with the subject lookingin different directions, with and without polarization. The

3

Figure 3: Real-time eyeball rendering, incorporating envi-ronment map illumination, subsurface scattering approx-imation in the sclera, initial blood vessels texture, andspecular reflections off of the bulbar conjunctiva.

photographs are loaded into Photoshop, and each one isaligned and warped to place the iris as a circle at the cen-ter of the image, and then the photographs are compositedtogether. The iris texture is cut out from the center of thewhole eye texture.We use ambient occlusion and hybrid normal mappingfor diffuse illumination on the eyeball, using the approx-imated hybrid normals described earlier. Specular illu-mination is computed with the Blinn-Phong model andSchlick Fresnel term, using the eye geometry normals anda specular albedo of 1. Figure 3 shows a close-up of aneyeball rendered with our implementation, under image-based illumination.Eyeballs present additional challenges for photoreal ren-dering, so we implemented the following features:

Control of pupil radius. The radius of the pupil adaptsautomatically to the illumination incident on the pupil.

Corneal refraction. We implemented a shader to ap-proximate the refraction of light at the surface of thecornea. The refraction is computed according to formulaedescribed in [1], using corneal geometry characterized in[3].

Iris caustics approximation. The surface normal of theiris is obtained using the Virtual Light Stage technique.However, to obtain an effect that looks like caustics, thenormal is adjusted half-way towards a concave normal.This effect is based on artistic observation: the concavenormal shifts the brightest part of the diffuse illuminationto the side of the iris that is opposite the direction of thelight source, which is qualitatively similar to the behaviorof caustics. The concave normal is obtained by invertingthe corneal geometry normal in x and y, and scaling downthe corneal geometry normal in z.

Iris darkening from refraction. We apply an artisti-cally designed iris darkening function to darken the irisaround its edges, approximating the “refraction function”in [1].

Color texture for blood vessels in the conjunctiva.We included a color texture map for the blood vesselsin the conjunctiva, modeled from a set of photographsthrough artistic manipulation.

Bump mapping on the conjunctiva. We apply a bumpmap to the specular normals of the conjunctiva, com-bining structured noise with blood vessel relief extractedfrom the sclera color texture. The structured noise mapwas created in Photoshop and the bumps from the scleratexture are computed using a standard method for bumpmapping based on a height image.

1.6 Teeth

The geometry for the teeth was obtained by scanning adental cast of real teeth. We scanned plaster casts of theupper and lower teeth made using standard dental castingtechniques. We scanned the casts with a structured-lightscanning system to produce a 600,000-triangle mesh from16 merged scans for each set. We manually remeshed theteeth scans to have far fewer triangles and better topology.We included color texture maps for the teeth and gums,based on artistically aligning photographs of the real teethto the scanned geometric model in Maya.We use ambient occlusion and hybrid normal mapping fordiffuse illumination on the teeth, using the approximated

4

Figure 4: Real-time teeth rendering, incorporating image-based lighting, specular normals based on dental cast ge-ometry, diffuse normals based on a cylindrical surface,and screen-space ambient occlusion computation.

hybrid normals described earlier. Specular illumination iscomputed with the Blinn-Phong model and Schlick Fres-nel term, using the dental cast geometry normals and aspecular albedo of 1. Figure 4 shows a set of teeth ren-dered with our implementation, under point light illumi-nation.

2 Virtual Light Stage

We developed a technique called Virtual Light Stage forcomputing hybrid normal maps from virtual objects, bysimulating a Light Stage within a physically-based ren-dering application, and performing a virtual Light Stagecapture session. We perform offline renderings (these caninclude sophisticated light transport effects such as oc-clusion, interreflections, and subsurface scattering) undergradient illumination conditions. Then we compute dif-fuse photometric normals from these images in the sameway as [2]. These diffuse normals can then be used to-gether with specular normals from the object geometry toperform the same hybrid normal lighting calculation usedfor the face scan data from real photographs. We appliedthis technique to the teeth and mouth regions in the ren-

dering software.

2.1 Implementation Details

In our virtual light stage rendering development we ex-plored physically-based subsurface scattering simulation.We used the Mental Ray physical subsurface scatteringshader in Maya to apply measured optical properties ofobjects to generate physically-based renderings (Figure5).We also made use of rendering into an object’s texturespace, rather than a typical camera projection space. Oneof the side benefits of the virtual light stage approach isthat it need not be constrained by the limits of what de-vices can be built, but rather any representation we cancompute.We integrated the subsurface scattering shader setup withthe image-based lighting approach for gradient illumina-tion for virtual light stage acquisition in MAYA 2010,rendering into the object’s texture space. We appliedthis to the teeth with surrounding mouth area, to gener-ate normal maps for the hybrid normal map rendering ap-proach. These hybrid normal maps incorporate baked-inlight transport effects such as ambient occlusion, inter-reflection, and subsurface scattering (Figure 6), and incuronly a minimal additional rendering cost in a real-timerendering system.

References

[1] Guillaume Francois, Pascal Gautron, Gaspard Breton,and Kadi Bouatouch. Image-based modeling of thehuman eye. IEEE Transactions on Visualization and

Computer Graphics, 15(5):815–827, 2009.

[2] Wan-Chun Ma, Tim Hawkins, Pieter Peers, Charles-Felix Chabert, Malte Weiss, and Paul Debevec. Rapidacquisition of specular and diffuse normal maps frompolarized spherical gradient illumination. In Render-

ing Techniques (Proceedings of Eurographics Sympo-

sium on Rendering 2007), pages 183–194, 2007.

[3] Ko Nishino and Shree K. Nayar. Eyes for relighting.In SIGGRAPH ’04: ACM SIGGRAPH 2004 Papers,pages 704–711, New York, NY, USA, 2004. ACM.

5

Figure 5: Rendering tests of subsurface scattering using two different sets of physically measured optical properties(left and right).

Figure 6: Example real-time face renderings with eyes and teeth, under image based lighting and point lighting.

6