1

Constructing Physics-Based Facial Models of Individuals

Yueucbeug Leel , Dellletri Terzopoulosl ,3, and Keitb Waters2

I Department of Computer Science, University of Toronto , Toronto, ON, M5S lA4 2Digital Equipment Corporation , Cambridge Research Lab , One Kendall Square, Cambridge, MA 02139

Abstract

This paper develops a highly automated approach to con­structing realistic, working models of human heads for use in animation. Th ese physics-based models are anatomically accurate and may be made to conform closely to specific in­dividuals . We begin by scanning a person with a laser sensor which circles around th e head, acquiring detailed range and reflec tance information . Next, an automatic conformation algorithm adapts a triangulated fa ce m esh of predetermined topological structure to thes e data. Th generic mesh, which is reusable with different individuals , reduces the range data to an effi cient, polygonal approximation of the fa cial geometry and supports a high-resolution texture mapping of th e skin refl ec tivity. Th e conformed polygonal mesh forms th e epi­dermal layer of a new, physics-based model of facial tissu e. A n automatic algorithm constructs the multilayer synthetic skin and estimates an underlying rigid "skull" substructure with a j ointed jaw. Finally, th e algorithm inserts synthetic muscles into the deepest layer of th e fa cial tissue. Th ese co ntractile actuators, which emulate the primary muscles of fa cial expression, generate f orces that deform th e synthetic tissue into m eaningful expressions. To increase realism, we include constraints to emulate tissue incompressibility and to enable th e tissue to slide over the skull substructure with­out penetrating into it. The resulting animate models appear significantly more realistic than our previous physics-based fa cial m odels.

Keywords: Physics-Based Facial Modeling, Fa cial Ani­mation, Cylindrical Facial S canning, Feature-Based Facial Adaptation, Texture Mapping, Discrete Deformable Models.

1 Iutroductiou

The human face has fascinated computer graphics modelers since the early 1970's. An important research goal has been to improve the realism of facial models. Over the years, researchers have progressed from simple, polygonal facial meshes, to parame terized models , to state-of-the-art physical models, each evolutionary step assimilating the best features of the previous generation of facial modeling methods [6] .

We have developed a physics-based approach to modeling the human face which pays serious attention to its anatom­ical s tructure and elaborate biomechanics [8, 12]. Starting from conventional facial meshes , our models incorporate a synthetic facial tissue (a dynamic , multilayer spring-mass lattice with nonlinear elastic properties) and anatomically

3 Fellow , Canadi a n Inst it ute fo r Ad vanced Research E- m a il addresses: { vlee ldt }@vis. t o ro nto .edu . waters@c rl.dec .co m

well-positioned synthetic muscles (a modification of the ear­lier parameterized muscle process of [10]).

Traditionally, facial meshes have been constructed manu­ally from two or more views of the face or by manually digi­tizing plaster casts. This laborious and error prone task has been superseded by active laser scanners [1 , 4] . Scanners , such as the one manufactured by Cyberware, Inc., circles around a person 's head to acquire high resolution , regularly sampled range and reflectance information . To take advil.n­tage of these scanners in human facial modeling and anima­tion , our goal is to develop efficient and highly automated methods for constructing 'working facial models of individu­als from the acquired data [11] .

In this paper we consider the problem of individualiz­ing physics-based facial models. In particular we develop a predominantly automatic method for constructing real­istic, working models of individual's heads for use in ani­mation. The method begins with detailed radial range and reflectance images acquired from an individual using the Cy­berware scanner [1].

We develop an automatic conformation algorithm which adapts a triangulated face mesh of predetermined topological structure to the scanned data. This generic mesh , which is reusable with different individuals, is designed to reduce the range data to an efficient , polygonal approximation of the facial geometry and to support a high-resolution texture mapping of the skin reflectivity. The generic mesh is adapted through a search for salient local minima and maxima of a scalar field function of the range image. The search is directed according to the, known relative positions of nose, eyes, chin , ears , and other facial features with respect to the generic mesh. Facial muscle emergence and attachment points are known relative to the generic mesh , and their positions are adapted as the mesh is conformed to the d a ta.

The conformed facial mesh forms the epidermal layer of a new, physics-based model of facial tissue, a variant of the model in [8, 12] . This synthetic tissue is a dynamic spring­mass lattice composed of triangular prism elements with in­ternal constraints that emulate tissue incompressibility. An automatic algorithm constructs the multil ayer synthetic skin and it also estimates an underlying "skull" structure with a jointed jaw. Finally, the algorithm inserts synthetic muscles into the deepest , muscle layer of the tissue. These contrac­tile actuators, which emulate the primary muscles of facial expression , generate forces that deform the synthetic tissue into meaningful expressions. To increase realism , we include new constraints that enable the deforming facial tissue to slide over an estima ted skull s tructure without penetrating into it as the muscles contract and as the jaw op~ns.

Graphics Interface '93

2

Figure 1: " Heidi" data from Cyberware, Inc. (a) .3D display of range data magnitude. (b) RGB texture map.

Figure 2: Range image (a) and positive Laplacian of range magnitude (b).

2 Cylindrical Laser Range Data

The typical output of the Cyberware color .30 scanner con­sists of 512 x 256 range and refl ec tance (RGS) values . Each range val ue represents the distance from the source to the ob­jec t in cy lindrical coordinates with corresponding longitude (0- .360 degrees) and latitude. The range data from Cyber­ware's "Heidi" data se t are displayed as a three dimensional range ~ urface in Fig. 1 (a) and as a two dimensional image Fig. 2(a) with intensity representing range. Fig . l(b) shows the RGB texture map as a monochrome image.

AI though the head structure in th e cylindrical lase r range data is distorted along the longitudinal airection, important features such as the slope changes of the nose, for ehead , chin, and the contours of the mouth , eyes, and nose are still dis­cerni ble. These facial features become more evident after applying a modified Laplacian operator to the range map Ti .J shown in Fig. 2(a) to produce the fi eld function li .j in Fig. 2(b). The operator is approximated by the following

finite difference formulas:

P(Ti:....l .j - 2Ti .J + Ti+l ,J )/h2

P(Ti .j-l - 2Ti .j + Ti .i+d/h2

l ij + zrJ where F' , IY are the positive second derivatives of depth 1I1

the x, y direc tions , h = 1 is the discrete step size, and

P(x) = { ~ if x > 0 otherwise.

3 G eneric Face Mesh and M esh Adaptation

Motivated by the "adaptive mesh" developed in [7] and em­ployed in [11], we have developed a different approach to adapting the face model to the scanned data by deforming an irregularly triangulated generic face mesh. Fig . 3( a) shows th e planar generic mesh which we obtain through a cylindri­cal projection of the 3D face mesh from [10] (see Fig. 4(a)) . Among the advantages of the generic mesh is that its irreg­ularly triangulated st ructure reduces the number of degrees

Graphics Interface '93

(a)

Figure :3: Adapting the generic face mesh. (a) Unadapted generic mesh. (b) Adapted mesh superimposed on the tex­

ture map. (c) Adapted mesh superimposed on a gradient map.

3

of freedom in the final face model and it has well defined features which form the basis for accurate adaptation and positioning of facial muscl es .

We label all facial featu re nodes in the generic face (Fig. 4(a)) prior to the adaptation step. Feature nodes in­clude eye contours, nose contours, mouth contours , and chin contours .

For any specific range image and its positive Laplacian field function (e.g., Fig. 2), the generic mes h adaptation pro­ced ure performs the following steps to locate feature points

in the range data:

1. Locate the highest range data point in the central area, and globally translate the face mes h to achieve co rre­spondence with the tip of the nose.

2. Locate the chin as the point below the nose with the greatest value of the positive Laplacian of range.

3. Locate the mouth as the point of greatest positive Laplacian between the nose and chin, and scale the po­si tions of all mesh nod es below the nose accordingly.

4. Locate the chin contour as the points whose latitudes li e in between the mouth and chin, and scale all mesh nodes below the chin contour.

5. Locate the ears as the points with a positive Laplacian larger than a threshold value around the longitudinal direction of the nose, and scale all the mesh nod es hor­izontally.

6. Locate the eyes as the points which have the greatest positive Laplacian around the estimated eyes region, and scale all the m es h nod es above the nose.

7. To fin e-tune the mesh nodes after the above adaptation steps, activate spring forces in the planar mesh with the adapted feature nodes treated as immobile bound­

ary conditions (see [7]). As the springs attempt to min­imize their deformations, the nod e positions between features will become regularized according to their orig­inal distances in the generic mesh . Fig. 3(b- c) shows the results of mesh adaptation.

8. Extend the generic m esh geometrically over the rest of the range data to cover the hair.

9. Conform the adapted generic mesh to the 3D geome­try and reflectance of the scanned individual by storing the adapted 20 nodal positions as RC B t exture map coordinates and by sampling the radial range image at 20 nodal positions to compute the 3D positions of the mesh nodes (Fig. 4(b- c)).

The above automatic, multistep adaptation algorithm gives satisfactory results with the relatively clean "Heidi" data set. We have also applied it to more difficult data sets acquired from subjects with nonsalient facial featur es and/or und er unfavorabl e lighting conditions. In such cases it becomes necessary to adjust the positions of some adapted mesh nodes around important facial features such as th e eyes, mouth , and chin. We can typically make the neces­sary manual adj ustmen ts in abou t 15 min u tes.

Graphics Interface '93

.::" " " " .

'. "\ :'.:;

Figure 4: Conforming the generic face mesh to several face geometries Generic mesh (a). Generic mesh conformed to " Heidi" data (b). Generic mesh conformed to another data set (c).

Figure.5: A single faci al tissue element (a). Elements assembled as a skin patch with embedded muscles contacted (b) . A closer look at half of an adapted face mesh with underlying synthetic tissue (c) .

4 Layered Synthetic Tissue Model

From a histologist 's point of view [2], the skull is covered by deformable tissue which has five distinct layers. Four layers-epidermis, dermis, subcutaneous connective tissue, and fascia-comprise the skin, and the fifth consists of the muscles of facial expression. In accordance with the struc­ture of real skin, we have designed a new, synthetic tissue model which is illustrated in Fig . .5 .

The tissue model is composed of triangular prism elements like the one shown in Fig . .5(a). The epidermal surface is defined by nodes 1, 2, and 3, which are connected by epi­dermal springs . The epidermis nodes are also connected by d ermal-fatty layer springs to nodes 4, .5, and 6, which define the' fascia surface. Fascia nodes are interconnected by fasci a springs. They are also connected by muscle layer springs to skull surface nodes 7, 8, 9.

Fig . .5(b) shows 342 such skin elements assembled into an extended "skin patch. " Several "muscles" are embedded into the muscle layer of the skin patch and the figure shows the skin deformation due to muscle contraction . Muscles are fixed in "bone" at their point of emergence and are attached to fascia nodes as they run through several tissue elements . Fig . .5(c) shows a close-up view of the right half of the con­formed "Heidi" face which consists of 432 elements.

Springs in the different layers have different s tress-strain relationships. Furthermore, in order to efficiently approxi­mate the non-linear stress-strain relationship in the dermal­fatty layer tissue (see [.5, 3]) , the dermal-fatty springs have two linear phases with different spring stiffnesses in dermal­fatty phase 1 and dermal-fatty phase 2 (see [8]) . We also include a rapidly growing nonlinear force function to pre­vent the spring from reversing itself when it is compressed.

The spring stiffness of biphasic dermal-fatty springs is de-

Graphics Interface '93

16.00

14.00

12.00

10.00

\ \

"r--- .............. I-------

8.00

6.00

4.00

2.00

0.00

0.00 0.20 0.40 0.60 0.80 1.00

Figure 6: Spring penalty force during compression.

fined as

I ,Kdl ; I,KI/I; I , KI4I(/" I;)II;

for I, > 1;(1 + B;) for I;(I + B;) > I, > I; for I; > I.

and the spring stiffness for other springs is

{ I . K 11;

c, = I,K4I(l" /:) I I; for I, > I; for I; > I,

In the above equations, KI and K2 are the two spring stiffnesses for the biphasic dermal-fatty springs, and K is the spring stiffness for other springs. The actual length of spring 5 is I" while I; is its rest length. The spring's bipha.~ic point B; is given in terms of the deformation ra­tio I,ll;. The multiplicative factor I , has value +1 when spring 5 is not inverted, and -1 when inverted . The func­tion 41(1" I; ) = tan(tT7r(1 -1, 11;)/2) is the spring compres­sion penalty, where 0 < eT < 1 is a scaling factor . Fig. 6

plots 41(1., I;) versus the deformation ratio for the default value eT = 0.98.

The values of the spring stiffness K that we used for springs in the various layers are set as follows :

Derm-fat 1 Derm-fat 2 30 70

See [8] for the remaining details on how the spring forces are applied to the tissue nodes .

5 Muscle Forces

The muscles of facial expression, or the muscular plate, spreads out below the facial tissue . The facial musculature attached to the the skin tissue by short ela.~tic tendons at many places in the fascia, but is fixed to the facial skeleton only at a few points . Contractions of the facial muscles cause movement of the facial tissue.

To better emulate the facial muscle attachments to the fascia layer in our model, a group of fascia nodes situated along the muscle path-i.e., within a predetermined distance from a central muscle vector, as determined by the muscle

5

ll . J

m~ J

P ..... . .

J,1

linear muscle fiber

m· J

Figure 7: Calculation of muscle force ratios.

width-experience forces ' from the contraction of the mus­cle. The face construction a1gori thm determines the nodes affected by each muscle in a precomputation step. Given a muscle j with fixed node positioned at m; and attached node at mt, the algorithm computes the length ratio Pi,i and width ratio Wi ,i of fascia node i positioned at Xi in muscle j as follows (see Fig. 7):

Illj = (m; - mt)/llmf-m111

lli = (11[ + 111)/1111; + u111

Pj,i = (m; - Xi) ' mj

Pi = Xi - m; + Pj,ilIlj

W · . J,' = IIPi - (Pi' nj)njll

where mj is the normalized muscle vector for muscle j , n j is the muscle normal which is determined from the nodal normals ut and u; of the attached and fixed points of the muscle , and Pi is the perpendicular vector from m j to fascia node i.

The muscle force fl acting on fascia node i influenced by muscle j is calculated from Pj ,i and Wj ,i as follows (see Fig. 7):

ej,i = pi,i/llm1- m;1I

6 1 (ej,;) = sin(lrej,;/2) + ~(sin(21rCj, i - 11'/2) + 1)

6 2(wj,i) = ~(cos(1r(Wj,iIWj)1.8) + 1)

ft = 6 1 (ej,i)62 (wj,i)mj

where 0 :::; ej,i :::; 1 is the muscle range ratio in length of fa.~cia node i with respect to muscle fiber j . The exponent t;. of ej,i emphasizes the muscle force. When t;. is increased beyond unity, the influence of the sine weighting function narrows, and it widens as ,.. is decreased below unity. The default value is,.. = 0.48. Function 6 1 scales the muscle force according to the length ratio , while 6 2 scales it according to the width ratio, where Wj is the muscle width for muscle j . Fig. 8 plots the muscle force r! with respect to ej,i. Fig. 5(b) shows an example of the effect of muscle forces applied to a synthetic skin patch.

Graphics Interface '93 ~

1.10

1.00

0.90

0.80

0.70

0.60

0.50

0.40

0.30

0.20

0.10

0.00

/ /

0.00

/ /

/ /

0.20

v-r--/

/ /

/

0.40 0.60 0.80 1.00

Figure 8: M uscle force profile versus muscle length ratio .

Fig . 9 shows th e 26 muscle vectors that we incorporate into our facial model. The number of nodes affected by each muscle-Zygomatic Major , Anguli Depressor , Inner Frontalis, Major Frontalis , Outer Frontalis , Leva tor Labi , Levator Labi Nasi , Late ral Corrugator, Corrugator, Depres­sor Labii , Zygomatic Minor , Buccinator , and Mentalis- is as follows:

Muscle ZygMaj AngDep Inn Fro MajFro

Width 0. 6 0.8 l. 2 l.0 Bundles 12 12 12 13

OutFro LevLab LevLNa LatCor Corrug

1.0 0.3 0.2 0.2 0.4 10 4 4 5 5

DepLab ZygMin Buccin Mental 0.7 0.4 0.8 0.6 16 11 12 21

The a ttachments of these muscl es to skull and fascia are described in [9].

6 Vohune Preservation Forces

Facial tissue is primarily water-based , hence it preserves its volume under deform ation . As in [8], we add a volume CO Il­

st raint to our spring-mass model which minimizes the vol ume change by applying an additional force to each epidermal nod e according to the deformation of the elemen t to which it belongs. For example, in Fig. 5(b), the skin is stretched along the positive x, negative x, and negative y directions. The volume constraint causes it to bulge near the end of the muscles and depress near the central stretching area.

The volume v e and the volume constraint force qi at epidermal node i = 1,2,3 of the prism element defin ed by nodes 1 to 6 in Fig. 5(a), is calculated as foll ows:

(V~2 X V ~3 ) · Vr4

(V~5 x V~4 ) . V~3

(V~4 X V~3) . V.~2

Vie + V2e + V1e

kl (V' - Ve)ui + k2 (pi - pi)

6

Figure 9: Facial expression muscles relative to generic mesh.

where Vie, V2e, and V1e are the volumes of the three tetrahedra

comprising the prism element f, v ij are vectors from node i to node j of e, V e is the volume of e, V· is the volume of e in the rest state, lli is the epidermal normal for epidermal node i, p i is the nodal coordinate for node i with respect to the center of ma.% of f, Pi' is pi in the element rest state, and kl , k2 are force scaling constants.

7 Skull Penetration Constraint Forces

During a facial expression, the facial muscles slide over the skull and the fascia nodes should not penetrate the skull . To prevent penetration, we compute constraint forces based on estimated skull normals. We estimate the skull normal s from the epidermal node normals. In the case of cylindrical data, we directly calculate the normals of the face nodes in cy lindrical coordinates and then transform them to generi c mesh coordin a tes. The array of normals to the topographical range map (Fig. l(a)) is given by

11 ~, 1 = [(Tk+I ,1 - Tk ,!) , 0, Or x [0 , (Tk,l+l - Tk ,!) , or where Tk ,1 is the range value at longitude k and latitude I. In certain places, range data is missing. We therefore estimate missing epidermal range values and nod e normals using interpolation.

The normal vector at node i in the face model is gi ven by

(see, e.g., [3])

[

cos8;

Ui = 0 sin 8i

- 510·n 0; 1 c

Il, cosO;

o

o

where Ili is the topographic normal of the nod e computed by interpolation from the array of normals. The angle 8; in t he

Graphics Interface '93

7

Figure 10: Physi.cs-based head model constructed from "heidi" data set (a). Relaxed face estimated from smiling heidi data (b). Face with contracted zygomatic major, zygomatic minor, depressor labii, and mentalis (c) . Side view (d). Face with contracted inner, major, and outer frontalis, and opened jaw (e). Side view (f) . Face with contracted inner frontalis, lateral corrugator, anguli depressor, and depressor labii (g). Side view (h) . Face in (e) with epidermal superimposed mesh (i).

Graphics Interface '93

direction cosines transformation matrix is the rotation angle of node i ba.~ed on the center of the nose in the positive z direction . For example, for a node with Di = 7r/2 , which is very close to the left ear , and 11i = [0 , 0, If pointing in the positive z direction in the cylindrical coordinate system will yield an 11i = [1 , 0, of, pointing in the positive x direction .

The forces which prevent the tissue from penetrating the skull surface are

{ -(fr· 11i)11i when fr . 11 i < 0

Si = 0 otherwise

where fJ' is the ne t force on fa.~ci a nodes i.

8 Numerical Simulation

The discrete Lagrange equations of motion for each node a re

d2 x i dXi 7n i dt 2 + ii dt + gi + q i + Si + hi = fi

where gi is the net spring force, q i is the net volume restora­tion force, Si is the skull non-penetra tion force for fa.~cia

nodes, hi is a force that restores node i to its rest position , and fi is the net muscle driving force acting on node i. The qu antity ii (dxi /dt) is a velocity-dependent damping force (see [3]) .

As in [8], we use the explicit forward Euler method to perform th e integra tion over time:

t a · . 1 (ft t t t tIt) mi i - 'Yi V i - g i - q i - S i - 1i

vi + ~tai X :+~t = xi + ~tv:+~t

where al , vl , xl are the nodal acceleration , velocity, and position of node i at time t , and ~t is the time step.

9 Animation Examples

Fig. 10(a) illustrat es several examples of animating the physics- based face model after conformation to the "Heidi" data. Note that Heidi wa.~ smiling while she wa.~ being scanned , and Figs. 10(a) shows the texture-mapped model conformed to the scanned da ta. Note that the eyes and teeth a re geometric primitives positioned appropriately relative to the skull (cL [11] in which the eyes and teeth were mapped textures that deformed a.~ the facial tissue deform ed!) . As we adapt the generic mesh to the smiling face, we also es­timate the closed-mouth position so that we can relax the face into the neutral expression shown in 10(b) with relaxed facial muscles.

To genera te the "happiness" expression in Figs. 1 O( c) and 10(d) , we contract the Zygomatic Major , Zygomatic Mi­nor Depressor Labii , and Mentalis.

To generate the "surprise" expression in Figs . 1 O( e) and 10(f) , we contract the Frontalis Major, Frontalis Inner , and Frontalis Outer for the upper half of the face , and we open the j aw by rotating the corresponding j awbone skull­nod es of our face model. Fig. 10(i) shows the face in (e) with the epidermal layer mesh superimposed .

To generate the "worry" expression in Fig. 10(g) and 10(h) , we contract the Lateral Corrugator and Frontalis Inner , for the upper half of the face, and the Anguli Depres­sor and Depressor Labii .

8

10 Conclusion

The human face consists of a biological tissue layer with variable thickness and nonlinear deformation properties, a muscle layer knit together under the skin , and an impene­trable skull structure beneath the muscle layer. This paper has developed a physics-based model of the face which takes all of these structures into account . We have demonstrated a new technique for automatically constructing face models of this sort and conforming them to individuals by exploit­ing high-resolution laser scanned data. The conformation process is carried out by a feature matching algorithm based on a reusable generic mesh. The conformation process also estimates the skull structure over which the new synthetic facial tissue model can slide. Our facial modeling approach demonstrates an unprecedented level of realism and fid elity to any specific individual .

References

[1] Cyberware Laboratory , Inc. 4020/RGB 3 D S canner with Color Digitizer. Monterey, CA , 1990.

[2] Frick and Hans. Human Anatomy, volume 1. Thieme Medical Publishers, Stuttgart , 1991.

[3] Y. Fung. A First Course in Continuum Mechanics. Prentice-Hall , Englewood Cliffs, N J , 1977.

[4] M. Hahas , H. Hutric, M. Rioux, and J . Domey. Facial image synthesis using skin t exture recording. Visual Computer, 6:337- 343, 1990.

[5] W . Larrabee. A finite element model of skin deforma­tion I. biomechanics of skin and soft tissue: a review. Laryngoscope, 96 :399- 405 , 1986.

[6] F. Parke and et al . 'State of the Art In Facial Anima­tion, volume 26, SIGG RAPH '90 Course Notes. AC M SIGGRAPH , 1990.

[7) D. Terzopoulos and M. Vasilescu. Sampling and re­construction with adaptive meshes. In Proceedings of Co mputer Vision and Pattern R ecognition Co nference, pages 70- 75. IEEE, June 1991.

[8) D. Terzopoulos and K. Waters. Physically-based facial modeling, analysis, and animation. Visualization and Computer Animatio'l, 1 :73- 80, 1990.

[9) J . Warfel. Th e Head, Neck, and Trunk. Lea & Febiger, Philadelphia, PA, 1985.

[10) K. Waters. A muscle model for animating three-dimensional facial expression. Computer Graphics, 22(4) :17- 24, 1987.

[11) K. Waters and D. Terzopoulos . Modeling and animat­ing faces using scanned data. Visualization and Com­puter Animation, 2:1.23- 128, 1991.

[1 2) K. Waters and D. Terzopoulps . The computer syn­thesis of expressive faces. Phi/. Trans . R . Soc. Lond., 35.'5 (1 273) :87-93, January 1992.

Graphics Interface '93