- ARTICLE

Image Representations for Visual LearningDavid Beymer and Tomaso Poggio*

Computer vision researchers are developing new approaches to object recognition anddetection that are based almost directly on images and avoid the use of intermediatethree-dimensional models. Many of these techniques depend on a representation ofimages that induces a linear vector space structure and in principle requires densefeature correspondence. This image representation allows the use of learning techniquesfor the analysis of images (for computer vision) as well as for the synthesis of images (forcomputer graphics).

The synthesis problem, the classical prob-lem of computer graphics, can be formulat-ed as the problem of generating novel im-ages corresponding to an appropriate set ofparameters that describe the camera view-point and aspects of the scene. The inverseanalysis problem, that of estimating objectlabels as well as scene parameters from im-ages, is the classical problem of computervision. Since the 1980s, researchers in bothfields have used intermediate, physicallybased models to approach their respectiveproblems of synthesis and analysis. In com-puter graphics, sophisticated three-dimen-sional (3D) modeling and rendering tech-niques have been developed that effectivelysimulate the physics of rigid and nonrigidsolid objects and the physics of imaging (1).Some research in computer vision has fol-lowed a parallel path; most object recog-nition algorithms use 3D object modelsand exploit the properties of geometricaland physical optics to match images to thedatabase of models (2). More recently,researchers in the two key areas of objectrecognition and object detection (3) haveused a rather different approach that hasbeen called image-based or view-based (4-8). In this approach, the image to bearnalyzed is compared directly, possibly af-ter a simple filtering stage, with a set ofexample images.

The Role of Correspondence

The key underlying mathematical assump-tion of the image-based approach is that theimages form a linear vector space (9). How-ever, images are just arrays of numbers orpixels, not vectors. A set of raw images-say, of similar objects-does not have thestructure of a vector space, because opera-

The authors are in the Department of Brain and CognitiveScience, Center for Biological and Computational Leam-ing (CBCL) and Artificial Intelligence Laboratory, Massa-chusetts Institute of Technology, Cambridge, MA 02142,USA.

*To whom correspondence should be addressed.E-mail: tp@ai.mit.edu

tions like addition do not have a well-defined meaning for raw images (10). Inpattern recognition, a standard techniquefor associating a vector to an image is toderive the vector components from an or-dered set of measurements on the image(11). This technique, however, is incom-patible with image-based approaches, wherevector components must correspond to pix-els. A vector space structure implies thatthe ith component of all vectors in the setmust refer to the same type of feature on theimaged surfaces. Strictly speaking, the useof vector space techniques in image-basedapproaches requires the solution of the cor-respondence problem: finding pixels in twoor more images that represent correspond-ing surface features in the scene. Correspon-dence is a difficult problem in computervision that can be solved for sufficientlysimilar images by techniques such as opticalflow algorithms, which find correspondingpixels in two or more images and computetheir displacement vectors (in the imageplane). Correspondence transforms imagesinto vectors by associating to feature point iits color and its x,y position.

In dense correspondence, each pixel cor-responds to a feature point; in this context,it is useful to think of the single vectorcontaining an ordered list of color and x,yvalues as two separate parts. The texturevector is an ordered list of the values (gray-scale or color) of corresponding pixels rela-tive to a reference image. The shape vectoris an ordered list of the disparities, that is,the x,y locations of corresponding featuresrelative to the reference image. We refer tothe correspondence operations that associ-ate the shape and texture vectors to animage as vectorizing the image (12); thisprocess will be explained later (13). Edge-based or contour-based approaches also fitinto this framework; if the images used areedge maps or line drawings, then the shapevector could include only entries for pointsalong an edge. In this case, the texturevector would not be used [(14); see also thecontour-based approach of (15, 16)].

The shape and texture vectors form sep-arate linear vector spaces with specificproperties. The shape vectors resulting fromdifferent orthographic views of a 3D object(in which features are always visible) con-stitute a linear vector subspace of very lowdimensionality [spanned by just two views;see (4, 17)]. For real objects, self-occlusionsand correspondence define different vectorspaces or charts (18) for different points ofview, correspondingly defining different as-pects (19). For a fixed viewpoint, a specificclass of objects (such as faces) with a similar3D structure (20) seems to induce a texturevector space of relatively low dimensional-ity, as shown indirectly by the results ofKirby and Sirovich [(21), see also (22)],who did not use exact correspondence. Us-ing exact correspondence, Vetter and Pog-gio (23) and Beymer and Poggio (24)showed that a good approximation of a newface image can be obtained with as few as50 base faces, which suggests a low dimen-sionality (25) for both the shape and thetexture spaces [see also (16, 26)]. Most ofthe object detection and recognitionschemes mentioned above do not explicitlyacknowledge the need to vectorize imagesand instead use approximate correspon-dence, exploiting anchor points such as theeyes of a face, to provide texture vectors onwhich to run the algorithm (6, 22, 27-29).Others [(4, 15, 30-32); see also (33)] haveassumed or used exact correspondence (re-lying only on shape vectors rather thantexture vectors). Craw and Cameron (18),Lanitis et al. (26), and Beymer (34) haveused shape and texture vectors for recogni-tion. In any case, feature correspondenceand the resulting vector structure underlie,either implicitly or explicitly, many of therecent view-based approaches to recogni-tion and detection.A similar situation is emerging in the

field of computer graphics. In a seminalpaper, Ullman and Basri [(4); see also therank theorem of Tomasi and Kanade (35)]showed that for orthographic projection,new images can be directly synthesized froma small set of images if they are representedas shape vectors. Shashua (36) extendedthis result to the perspective case. Reprojec-tion techniques are now being developedthat synthesize new views of a scene with-out an explicit 3D model [see, for instance,(37)]. Unlike the earlier work (4, 27, 38),the most recent research focuses on visibil-ity issues induced by camera transforma-tions (39, 40).

SCIENCE * VOL. 272 * 28 JUNE 1996 1905

on

Apr

il 14

, 201

1w

ww

.sci

ence

mag

.org

Dow

nloa

ded

from

Learning Networks

We now describe how a vectorized imagerepresentation enables the use of still an-

other powerful framework-leaming fromexamples-to approach both computer vi-sion and computer graphics and to effec-tively bypass physically based 3D models(38). This example-based approach, whichimplies a view-based technique, is related tobut does not directly rely on the linearcombination and reprojection results de-scribed earlier. In the metaphor of this ar-

ticle, we speak of leaming the mapping-from images to pose parameters or from pose

parameters to images-from a set of exam-ples. The examples are input-output pairs ofimages and associated pose parameters. Thelearmed mapping is then used to estimatethe pose for a novel image of the same type,whereas the inverse mapping can synthesizea novel image for new values of the controlparameters. The approach has advantagesand disadvantages, and although it cannotreplace the traditional 3D approach, it maybe quite appropriate in some situations.

Our learning-from-examples approachcan be used for both the analysis and syn-thesis of images of objects such as faces. Forimage analysis, the mapping from novelimages to a vector of pose and expressionparameters is computed by what we call an

analysis network. For synthesis, the inversemapping (from input pose and expressionparameters to output images) can be used tosynthesize new images through a similarlearning network called a synthesis net-

work. Such an example-based leaming ap-proach to vision and graphics trades mem-ory for computation. The more traditional3D approaches use computationally expen-sive rendering techniques, whereas our ap-proach requires the memory needed to storea sufficient number of training examples.

Consider the problem of approximatinga vector field y(z) from a set of sparse data,the examples, which are pairs (Zi,Yi) (i =1,.I. ., N). Several multivariate approxima-tion schemes are possible; here, we use spe-cific instances of regularization networks-a class of networks with one "hidden" layerand linear output units-that encompassmany standard approximation, statistical,and neural network techniques (41, 42).For the case ofN examples, n ' N hiddenunits or "centers," input dimensionality d,and output dimensionality q, the approxi-mation is

n

y(z) = ciG(z - ui) (1)

where the ci are vectors of coefficientsweighting the hidden layer and G is thechosen basis function, which can be a radialbasis function [such as the Gaussian (43)] ora spline such as a tensor-product spline. In

Eq. 1, we have neglected polynomial termsneeded only for G that are not positivedefinite. The ui are d-dimensional centersof the basis functions (44). In the specificanalysis and synthesis networks used here,we have used n = N. The original exampleszi are the centers ui, and the network pa-rameters ci are found by solving a linearsystem of Eqs. 1 over the training data.

As discussed above, the central part ofour approach is to vectorize the images byrepresenting them as separate shape andtexture vectors. For computing correspon-dence, we have found that standard mul-tiresolution optical flow algorithms (31,45), preceded by certain normalizationsteps, can automatically compute dense pix-elwise matches for sufficiently similar imag-es (46). After one image is defined as areference image ref, the (x,y) locations offeature points (here, pixels) of a new imageimg are estimated by computing the opticalflow between the two images (Fig. 1). Theshape vector s contains the relative dis-placements between corresponding featurepoints in images ref and img. The texturevector t contains the image intensity (orcolor) values of img at the set of orderedfeature points. The shape and texture vec-tors can be rendered by using s to warp t.

r3

CA>:

39 X W% - __o v v s v *,,,. * ^s s . _t

w s v vs_-* . .s. % . _so . ._ > . .___r; ;;:---: :tJIw, s * --t3g S::#.'v9

v__,s___*__# s -S_w_0 0__-_S_ w \\;fifi;;

Shape s Texture t

Fig. 1. An (s,t) vectorzed image representation.Pixelwise correspondences are computed be-tween image img and a standard reference imageref(CA, correspondence algorithm). Shape s con-sists of the (x,y) displacements between corre-sponding pixels, and texture t is the texture of imgwarped to the shape of ref. Although t shows theeffect of self-occlusions, it contains all the infor-mation needed for img.

r2 IMg3 -mg4Fig. 2. In this demonstration of the example-based approach to image analysis and synthesis, theexample images img1 through img8 are placed in a 3D rotation-expression parameter space (rj,r2r3),here at the comers of the unit cube. For analysis, the network leams the mapping from images (inputs)to parameter space (output). For synthesis, we synthesize a network that learns the inverse mapping,that is, the mapping from the parameter space to the images. The three axes of the parameter spacecorrespond to smile, left-right rotation, and up-down rotation.

SCIENCE * VOL. 272 * 28 JUNE 1996

L |__+

1906

on

Apr

il 14

, 201

1w

ww

.sci

ence

mag

.org

Dow

nloa

ded

from

The idea underlying the analysis net-work is to synthesize the mapping betweenan imtage and the associated pose parame-ters through the use of a suitable set ofexamples (consisting of images and associ-ated pose parameters). Here, we outline thelearning of a mapping from input shapevectors s to an output pose vector r bymeans of a Gaussian radial basis functionnetwork, Eq. 1 [see also (5, 47)]. Figure 2shows eight example images and their as-signed locations along the corners of a unitcube in 3D pose space r = (r,r2,r), wherer1 is a leftward rotation, r2 is a smile, and r3is an upward rotation. Our choice of theunit cube here is arbitrary; the exampleimages can be irregularly spaced in posespace r. The image representation s is com-puLted by assigning img1 as a reference imageand computing optical flow with respect tothe reference (s, = 0). For analysis, thevector s only contains optical flow in rec-tangular boxes around the eyes, nose, andmouth (found automatically), because theflow computation is robust in these areas.The Gaussian radial basis function networkhas eight fixed centers, ui = s. The learningprocedure estimates the Gaussian widthsand the ci coefficients (48). The trainednetwork can then evaluate the pose r of anew input image of the same type- forinstance, an image of the same person usedin the training set. Figure 3 shows resultsfrom such an analysis network. The net-

work described here exploits only imageshape information s but could be modifiedto also exploit the texture information t.We now discuss an example-based tech-

nique for generating images of nonrigid 3Dobjects as a function of input parameters. Thisimage synthesis task a computer graphicstask is the inverse of the problem of imageanalysis; the input-output roles of images andpose space are reversed. In our approach, theinput pose space is hand-crafted by the user torepresent desired degrees of control over theimages of the 3D object. After example imag-es of the object are assigned to positions ri ininput pose space (Fig. 2), the example imagesare vectorized to create the corresponding im-age representation (sOti). A regularization net-work is then trained on the example pairs(r ,(s,,t-)). Given a new pose vector r, thesynthesis network synthesizes a new image(s,t) by interpolating both the shape and tex-ture of the example images, thus performing akind of multidimensional morphing. Such asynthesis network was used to interpolate theeight images of Fig. 2. The vectorized (s,t)representation for each example image wasobtained automatically by computing opticalflow between the example and the referenceimage imgl; inputs to the network are the poseparameters (r1,r2,r3). The network follows Eq.1, using tensor-product linear splinesG(rl,r2,r3) r I-r21 Ir31 as the basis func-tion. Eight basis functions are used, one foreach example (38). Finally, the network out-put (s,t) is "rendered" as an image through a2D warping algorithm. The shape s representscorrespondences between the texture t at thestandard reference shape and the destinationshape s. Forward warps [see (49)] can be used

to locally remap pixels t Lising the set ofcorrespondences s. Figure 3 shows novel im-ages created by the synthesis network.

The analysis and synthesis networks caneach be used independently in a variety ofapplications. The analysis network is notlimited to face images and can he regardedas a trainable man-machine interface thatcould perform customizable gesture recogni-tion and thereby drive a computer interface.The synthesis network is an alternativegraphics and animation technique that ex-tends to real images an approach alreadydemonstrated by Poggio and Brunelli (27)and Librande (50) for line drawings. Appli-cations of the analysis and synthesis net-works include teleconferencing and verylow bandwidth video e-mail, that is, videoencapsulated in e-mail. Figure 3 shows ademonstration of the basic idea (51).

Large occlusions within the example im-ages can be a problem for the analysis andsynthesis networks described here, not onlyfor the correspondence step but also for therepresentation of actual images. In the syn-thesis case, for instance, the reconstructionstep may have to decide which of two pixelswarped to the same image location is thevisible one. Concepts such as layering (52),common in animation techniques, anddepth ordering from image disparity (40)are ways to endow an image-based represen-tation with qualitative 3D properties.

Discussion

The image analysis and image synthesis net-works we have described are just one exam-ple of the many approaches that a vector

Prototype

748,

*o-

b

c Ic(r1,r2,r3) = (0.450, 0.195, 0.61 1)

Fig. 3. In the boxed image pairs, the novel input realimage (left) is fed into an analysis network to estimateleft-right rotation rl, expression r2, and up-down ro-tation r3. These parameters are then fed into thesynthesis module, which synthesizes the imageshown on the right. In principle, this example-basedapproach can achieve very high compression. Here,each frame is compressed to -3 bytes, but the im-age is very limited in its range of poses and expres-sions. A more realistic system may require the esti-mation and sending of 10 to 20 parameters, plus afew others that define the image plane transforma-tions, for a total of -20 bytes per frame.

Virtual view Virtual view

Fig. 4. One technique for generating virtual views from a single view is parallel deformation, which issimpler than the linear class approach of Vetter and Poggio (23). First, a 2D deformation representing adesired transformation is estimated from two prototype images in correspondence. In this example, thetransformation (a) is a frown, and an optical flow algorithm is used to find feature correspondence at thepixel level. The flow is then mapped onto two novel faces (b), and the novel faces are 2D warped tosynthesize virtual views (c).

SCIENCE * VOL. 272 * 28 JUNE 1996 1907

on

Apr

il 14

, 201

1w

ww

.sci

ence

mag

.org

Dow

nloa

ded

from

representation of images makes possible.The key step in these applications is thecomputation of dense correspondence be-tween images the vectorizing step. Vec-torization allows, for instance, the creationof a flexible model for a class of objects asthe linear combination of prototype imagesand their affine deformations (53). Thisnew type of flexible model can be used as agenerative model to synthesize novel imag-es of the same class. Such a model can alsobe used for image analysis by fitting themodel parameters to an image by means ofan optimization procedure (14, 34). Theidea of this analysis-by-synthesis techniquebased on the linear combination of proto-types (without the use of dense correspon-dence) was first suggested by Taylor andco-workers (16, 26, 30), who have alsodescribed applications in medicine and oth-er areas. Blake and Isard applied a similartechnique to the tracking of dynamicallychanging images (15, 54).

This flexible model and its associatedestimation procedure are very closely relat-ed to the synthesis network described earli-er [see (44), Eq. 3]. The model can be usedas a new example-based technique for com-puting correspondence (14); even thoughstandard optical flow techniques work well(55), more robust approaches are desirable,especially when there are large occlusions(56). Once estimated, the parameters of theflexible model can be used for indexing andrecognition and, more generally, for imageanalysis. They can also be used for thegeneration of "virtual" views and the esti-mation of 3D structure from single images.Consider, for instance, the case in whichonly one example image of an object isavailable; this may occur in object recogni-tion or in analysis and synthesis tasks of thetype described earlier. Vetter and Poggio(23) considered this problem for linear ob-ject classes. An object belongs to a linearclass if its 3D structure can be described asa linear combination of the 3D structure ofa small number of prototypes (32). It ispossible to learn to generate a new virtualview of the object from a single exampleview, represented as a 2D shape vector, ifappropriate prototypical views of other ob-jects in the same class are available (underorthographic projection) (57). In this way(see also Fig. 4), new views of a specific facewith a different pose or expression can beestimated and synthesized from a singleview (23, 24, 58). Similarly, 3D structurecan be estimated from a single image if theimages and structures of a sufficient num-ber of prototypical objects of the sameclass are available (23, 32). Flexible mod-els of this type can underlie the learning ofvisual tasks in a top-down way, specific toobject classes.

An image representation in terms ofshape and texture vectors allows the use oflearning and pattern recognition tech-niques for a variety of computer vision andcomputer graphics tasks (59). The develop-ments described above may have implica-tions for the key role of correspondence inbiological vision (60). Perhaps unsurpris-ingly, some of the neural mechanisms thatunderlie object recognition in the primatecortex seem consistent with the view-basednetworks used in the technical applicationsdescribed here (61) rather than with 3Dmodel-based representations.

REFERENCES AND NOTES

1. Y. Lee, D. Terzopoulos, K. Waters, in SIGGRAPHProceedings, Los Angeles, 6 to 1 1 August 1995 (As-sociation for Computing Machinery, New York,1995), pp. 55-66.

2. W. E. L. Grimson, Object Recognition by Computer:The Role of Geometric Constraints (MIT Press, Cam-bridge, MA, 1990).

3. A. Hurlbert and T. Poggio, Nature 321, 651 (1986).4. S. Ullman and R. Basri, IEEE Trans. Pattern Anal.

Machine Intell. 13, 992 (1991).5. T. Poggio and S. Edelman, Nature 343, 263 (1990).6. H. Murase and S. Nayar, in Proceedings of the Elev-

enth National Conference on Artificial Intelligence,Washington, DC, 1 1 to 15 July 1993 (American As-sociation for Artificial Intelligence Press, Menlo Park,CA, 1993), pp. 836-843.

7. D. Pomerleau, Neural Comput. 3, 88 (1991).8. T. Poggio, in Cold Spring Harbor Symposia on Quanti-

tative Biology (Cold Spring Harbor Laboratory Press,Cold Spring Harbor, NY, 1990), pp. 899-910; T. Breuel,A.l. Technical Report 1374 (Artificial Intelligence Labo-ratory, Massachusetts Institute of Technology, Cam-bridge, 1992); A. Pentland, B. Moghaddam, T. Stamer,in Proceedings of the IEEE Computer Society Confer-ence on Computer Vision and Pattem Recognition, Se-attle, WA, 21 to 23 June 1994 (IEEE Computer SocietyPress, Los Alamitos, CA, 1994), pp. 84-91; B. A.Golomb, D. T. Lawrence, T. J. Sejnowski, in Advancesin Neural Information Processing Systems 3, R. Lipp-mann, J. Moody, D. Touretzky, Eds. (Morgan Kauf-mann, San Mateo, CA, 1991), pp. 572-577.

9. Techniques based on the assumption of a linear vec-tor space include those of Murase and Nayar (6),who used eigenfunctions of object images; Turk andPentland [(22), see also S. Akamatsu, T. Sasaki, H.Fukamachi, N. Masui, Y. Suenaga, InternationalConference on Pattern Recognition, The Hague, 30August to 3 September 1992 (IEEE Computer Soci-ety Press, Los Alamitos, CA, 1992), pp. 217-220],who used eigenfunctions in the space of face imag-es; and Sung and Poggio (28), who used the Mahal-anobis distance in their face detection system [seealso B. Moghaddam and A. Pentland, in (14), pp.786-793; U. Fayyad, N. Weir, S. Djorgovski, in Pro-ceedings of the Tenth International Conference onMachine Leaming, University of Massachusetts, Am-herst, 27 to 29 June 1993 (Morgan Kaufmann, SanMateo, CA, 1993), pp. 1 12-119].

10. A collection of objects V is a vector space if (i) for allu,v in V, au + bv is in V with a,b real numbers and (ii)there is a zero vector 0 such that for all u in V, Ou =0 and u + 0 = u. Other axioms omitted here ensurethat addition and multiplication behave as theyshould.

11. Measurements such as area, perimeter, and statis-tical estimates of color, contrast, and other proper-ties of images are all global features used in the past.Sparse local features can also be used; there areobvious trade-offs between feature complexity andthe difficulty of the correspondence step.

12. In many object recognition tasks, correspondence isnot needed over the full image. Correspondence that

is limited to image patches that correspond to objectcomponents is easier and yields more robust recog-nition and detection schemes. In this article, discus-sions of full image correspondence apply to compo-nents as well.

13. The full texture-shape vector has dimensionality 3N2+ 2N2, where N2 is the number of pixels in the image(because of the three color values per pixel). Ofcourse, it may be possible to define a sparser set ofrelevant feature points in the image. The term shapevector refers to the 2D shape (not 3D) relative to thereference image.

14. M. Jones and T. Poggio, in Proceedings of the FifthInternational Conference on Computer Vision, Cam-bridge, MA, 20 to 23 June 1995 (IEEE ComputerSociety Press, Los Alamitos, CA, 1995), pp. 531-536.

15. A. Blake and M. Isard, in SIGGRAPH Proceedings,Orlando, FL, 24 to 29 July 1994 (Association forComputing Machinery, New York, 1994), pp. 185-192.

16. T. F. Cootes, C. J. Taylor, A. Lanitis, D. H. Cooper, J.Graham, in Proceedings of the Fourth IntemationalConference on Computer Vision, Berlin, 1 1 to 14May 1993 (IEEE Computer Society Press, Los Alami-tos, CA, 1993), pp. 242-246.

17. T. Poggio, Technical Report 9005-03 (Istituto per laRicerca Scientifica e Tecnologica, Povo, Italy, 1990).

18. I. Craw and P. Cameron, in Proceedings British Ma-chine Vision Conference, University of Glasgow, UK,24 to 26 September 1991 (Springer-Verlag, London,1991), pp. 367-370.

19. J. J. Koenderink and A. J. van Doorn, Biol. Cybern.32,211 (1979).

20. T. Vetter, A. Huribert, T. Poggio, Cereb. Cortex 3,261 (1995).

21. M. Kirby and L. Sirovich, IEEE Trans. Pattern Anal.Machine Intell. 12, 103 (1990).

22. M. Turk and A. Pentland, J. Cogn. Neurosci. 3, 71(1991).

23. T. Vetter and T. Poggio, A.l. Memo 1531 (ArtificialIntelligence Laboratory, Massachusetts Institute ofTechnology, Cambridge, 1995).

24. D. Beymer and T. Poggio, in (14), pp. 500-507.25. In this framework, principal components analysis is

just one of the several tools provided by the mathe-matical structure of a vector space, which is theactual key property here.

26. A. Lanitis, C. J. Taylor, T. F. Cootes, in (14), pp.368-373.

27. T. Poggio and R. Brunelli, A.l. Memo 1354 (ArtificialIntelligence Laboratory, Massachusetts Institute ofTechnology, Cambridge, 1992).

28. K. K. Sung and T. Poggio, in Proceedings of theImage Understanding Workshop, Monterey, CA, 13to 16 November 1994 (Morgan Kaufmann, San Ma-teo, CA, 1994), pp. 843-850.

29. H. A. Rowley, S. Baluja, T. Kanade, Technical ReportCMU-CS-95- 158 (Carnegie Mellon University, Pitts-burgh, PA, 1995); H. Murase and S. K. Nayar, Int.J. Comput. Vision 14, 5 (1995).

30. T. F. Cootes and C. J. Taylor, in Proceedings BritishMachine Vision Conference, Leeds, UK, 22 to 24September 1992 (Springer-Verlag, London, 1992),pp. 266-275.

31. A. Shashua, Al. Technical Report 1401 (Artificial In-telligence Laboratory, Massachusetts Institute ofTechnology, Cambridge, 1992); thesis, Massachu-setts Institute of Technology (1992).

32. T. Poggio and T. Vetter, A.l. Memo 1347 (ArtificialIntelligence Laboratory, Massachusetts Institute ofTechnology, Cambridge, 1992).

33. M. Lades et al., IEEE Trans. Comput. 42, 300 (1993).34. D. Beymer, A.l. Memo 1537 (Artificial Intelligence

Laboratory, Massachusetts Institute of Technology,Cambridge, 1995).

35. C. Tomasi and T. Kanade, Int. J. Comput. Vision 9,137 (1992).

36. A. Shashua, IEEE Trans. Pattern Anal. Machine Intell.17, 779 (1995).

37. S. Laveau and 0. Faugeras, Technical Report 2205(Institut National de Recherche en Informatique et enAutomatique, France, 1994); S. Avidan and A.Shashua, Technical Report CIS-9602 (Technion,Haifa, Israel, 1996); T. Evgeniou, thesis, Massachu-

SCIENCE * VOL. 272 * 28 JUNE 1996

MK

1908

on

Apr

il 14

, 201

1w

ww

.sci

ence

mag

.org

Dow

nloa

ded

from

AS

setts Institute of Technology (1996).38. D. Beymer, A. Shashua, T. Poggio, A.l. Memo 1431

(Artificial Intelligence Laboratory, Massachusetts In-stitute of Technology, Cambridge, 1993); T. Ezzat,thesis, Massachusetts Institute of Technology (1996);S. Lines, thesis, Massachusetts Institute of Technol-ogy (1 996).

39. S. Seitz and C. Dyer, in Proc. IEEE Workshop on theRepresentation of Visual Scenes, Cambridge, MA,24 June 1995 (IEEE Computer Society Press, LosAlamitos, CA, 1995), pp. 18-25; S. Chen and L.Williams, in SIGGRAPH Proceedings, Anaheim, CA,1 to 6 August 1993 (Association for Computing Ma-chinery, New York, 1993), pp. 279-288; L. McMillanand G. Bishop, in (1), pp. 39-46.

40. T. Werner, R. Hersch, V. Hlavac, in (14), pp. 957-962.

41. T. Poggio and F. Girosi, A.l. Memo 1140 (ArtificialIntelligence Laboratory, Massachusetts Institute ofTechnology, Cambridge, 1989). See also T. Poggioand F. Girosi, Science 247, 978 (1990).

42. F. Girosi, M. Jones, T. Poggio, Neural Comput. 7,219 (1995).

43. D. S. Broomhead and D. Lowe, Complex Systems 2,321 (1988).

44. The equation can be rewritten in matrix notation as

y(z) = Cg(z) (2)where g is the vector with elements g, = G(z - z,). If Gis defined as the matrix with elements Gj, = G(z, - z),then the "weights" c are "learned" from the exam-ples by solving Gcm = Ym. If Y is defined as thematrix in which column e is the example y, and C asthe matrix in which row m is the vector cm, then theset of weights C is given by C = YG+ (G' is thepseudoinverse). The vector field y is approximatedby the network as the linear combination of the ex-ample fields y,, that is,

N

y(z) = YG+g(z) = b, (z)y, (3)

which is the dual representation of Eq. 1. The b,depend on the chosen G, according to b(z) =YG+g(z). Thus, for any choice of the regularizationnetwork, the estimated output (vector) image is al-ways a linear combination of example (vector) imag-es with coefficients b that depend (nonlinearly) on thedesired input value. The result is valid for all networkswith one hidden layer and linear output units, provid-ed that a L2 criterion is used for training (42).

45. J. R. Bergen, P. Anandan, K. J. Hanna, R. Hingo-rani, in Proceedings of the Second European Con-ference on Computer Vision, Santa MargheritaLigure, Italy, 19 to 22 May 1992 (Springer-Verlag,Berlin, 1992), pp. 237-252. See also B. Lucas andT. Kanade, in Proceedings IJCAI, Vancouver, Brit-ish Columbia, 24 to 28 August 1981 (AmericanAssociation for Artificial Intelligence, Menlo Park,CA, 1981), pp. 674-679.

46. Dense correspondence can be computed at videorate by machines such as the Carnegie Mellon Uni-versity video-rate stereo machine [T. Kanade, H.Kano, S. Kimura, A. Yoshida, K. Oda, in Proceedingsof the International Conference on Intelligent Robot-ics and Systems, Pittsburgh, PA, 5 to 9 August 1995(IEEE Computer Society Press, Los Alamitos, CA,1995), pp. 95-100] or the David Sarnoff Vision FrontEnd [P. Anandan, P. Burt, J. Pearson, in Proceed-ings of the Image Understanding Workshop, PalmSprings, CA, 12 to 15 February 1996 (Morgan Kauf-mann, San Mateo, CA, 1996)].

47. S. Mukherjee and S. K. Nayar, in (14), pp. 794-800.48. The width for the ith Gaussian, au, is estimated from

the average distance from s, to its neighbors, andthen the c, are computed by solving the linear systemof Eqs. 1 over the training data.

49. G. Wolberg, Digital Image Warping (IEEE ComputerSociety Press, Los Alamitos, CA, 1990).

50. S. Librande, thesis, Massachusetts Institute of Tech-nology (1992). See also the Cartoon-o-Matic Webpage, http://www.nfx.com.

51. A different approach to rendering a face with a focuson the text-to-speech component can be found in K.Waters and T. M. Levergood, Technical Report CRL93/4 (Cambridge Research Laboratory, DigitalEquipment Corporation, Cambridge, MA, 1993).

52. J. Y. A. Yang and E. H. Adelson, IEEE Trans. Imag.Proc. 3, 625 (1994).

53. Such a model constitutes a special case of the gen-eral concept of flexible templates parameterized bythe coefficients of the prototypes [for different typesof flexible templates, see D. J. Burr, IEEE Trans.Pattern Anal. Machine Intell. 3, 708 (1981); M. A.Fischler and R. A. Eschlager, IEEE Trans. Comput.C-22, 67 (1973); U. Grenander, Y. Chow, D. M.Keenan, Hands: A Pattern Theoretic Study ofBiolog-ical Shapes (Springer-Verlag, New York, 1991); G. E.Hinton, C. K. I. Williams, M. D. Revow, inAdvances inNeural Information Processing Systems 4, J. Moody,S. Hanson, R. Lippmann, Eds. (Morgan Kaufmann,San Mateo, CA, 1992), pp. 512-519; A. Yuille, Neu-ral Comput. 2, 1 (1990); P. W. Hallinan, thesis, Har-vard University (1995); M. Kass, A. Witkin, D. Terzo-poulos, in Proceedings of the First Intemational Con-ference on Computer Vision, London, 8 to 11 June1987 (IEEE Computer Society Press, Washington,DC, 1987)]. The flexible model of Jones and Poggio[M. Jones and T. Poggio, in preparation; see also(14)] is learned from a small number of vectorizedprototype images. Given prototype images /0, . . ..IN and their shape vectors s; and texture vectors tjrelative to /0, where si = (Axi,Ay,) and t, =I[x +Ax,(x,y), y + Ay,(x,y)], the model is

N 1

Imodel(R,y) =Ebitj (4)

withR = x' + POX' + ply' + P2 (5)

Y=' + P3X' + P4Y' + P5 (6)N 1

x' = x + cAx,j(x,y) (7)

N 1Ni1

y =y + c,Ay,(x,y) (8)

The two linear combinations (for shape and texture,respectively) use the same set of prototype imagesbut two different sets of coefficients. The parametersof the model are the ci's, bj's and Pk'S of Eqs. 4 to 8.Given a novel image, the parameters that corre-spond to the best fit of the model are estimated byminimizing an error measure between the novel im-age and the flexible model after rendering it as animage. For instance, parameters can be found thatminimize the function

E(c,p, b) = E [ lnovel( -odel 2 (9)2x.y

54. A. Blake, R. Curwen, A. Zisserman, Int. J. Comput.Vision 11, 127 (1993).

55. Other matching techniques for object recognition areclosely related to optical flow algorithms, even if theconnection is not always made explicit. For instance,the functional minimized in the technique of von derMarlsburg and co-workers (33) is closely related tothe regularization functional [V. Poggio, V. Torre, C.Koch, Nature 317, 314 (1985)] used in many opticalflow algorithms.

56. Optical flow techniques work between images withsufficiently low disparities. When correspondence isdesired between images that are quite different, oth-er techniques are required [P. Lipson, thesis, Mas-sachusetts Institute of Technology (1993); I. Bach-elder, thesis, Massachusetts Institute of Technology(1991); A. Witkin, D. Terzopoulos, M. Kass, Int.J. Comput. Vision 1, 133 (1987)]. We do not implythat dense correspondence is always needed in vi-sion and graphics tasks. Symbolic techniquesshould be used in general to guide the low-level cor-respondence algorithms we have used in this work.

57. This procedure is similar to training a network withinput-output pairs of prototypical views representingeach prototype in the initial and in the desired pose.Then, for a new input image, the network synthesiz-es a virtual view in the desired pose. If the network istrained with pairs of prototype images as inputs (rep-resented as 2D shape vectors) and their 3D shape asoutput, it will effectively compute 3D shape for novelimages of the same class [see (23, 32) and comparethe approach of J. Atick, P. Griffin, N. Reid-lich, Network: Comput. NeuralSyst. 7,1 (1996)]. Thelinear class assumption induces a linear combina-tion of the 2D shape vectors but not of the corre-sponding texture vectors, not even for Lambertianreflectance and uniform albedo (A. Yuille, personalcommunication).

58. An approach in the same spirit was used earlier byPomerleau to increase the number of training exam-ples in his system that learns to steer a vehicle fromimages of the road (7).

59. It is not surprising that a small set of correspondingimages constitutes a powerful representation forrecognition and graphics, because complete 3Dstructure can be recovered from a small number ofviews in correspondence (three orthographic andtwo perspective views are sufficient).

60. T. Poggio and S. Ullman, in preparation.61. H. H. Bulthoff and S. Edelman, Proc. Natl. Acad. Sci.

U.S.A. 89, 60 (1992); N. Logothetis, J. Pauls, T.Poggio, Curr. Biol. 5, 552 (1995).

62. We thank H. Bulthoff, F. Girosi, P. Dayan, M. Jor-dan, T. Vetter, T. Sejnowski, D. Glaser, A. Shashua,E. Grimson, M. Jones, R. Romano, and especiallyS. Ullman, C. Tomasi, C. Koch, A. Blake, D. Terzo-poulos, and T. Kanade for reading the manuscriptand for many insightful and constructive com-ments. Sponsored by grants from the Office ofNaval Research and the Advanced ResearchProjects Agency under contracts N00014-93-1 -0385 and N00014-92-J-1879. Support for CBCLis provided in part by grants from NSF under con-tract ASC-9217041, by the Multidisciplinary Uni-versity Research Initiative under contract N00014-95-1-0600, and by Siemens, Daimler-Benz, East-man Kodak, and ATR. T.P. is supported by theUncas and Helen Whitaker Chair at Whitaker Col-lege, Massachusetts Institute of Technology. D.B.was supported in part by a Howard Hughes DoctoralFellowship from the Hughes Aircraft Company.

SCIENCE * VOL. 272 * 28 JUNE 1996

M.- ..x11113 MMMIMERVIVIAMM. IN, .:,.. .v

1909

on

Apr

il 14

, 201

1w

ww

.sci

ence

mag

.org

Dow

nloa

ded

from