Bernardini etal-cga02

0272-1716/02/$17.00 © 2002 IEEE

Building a DigitalModel ofMichelangelo’sFlorentine Pietà

IEEE Computer Graphics and Applications 59

Because scanning devices are less expen-sive and easier to use than they were just

a few years ago, a wide range of applications employ3D scanning technology. As a result, various organiza-tions can now produce models of cultural artifacts andworks of art.

Members of the National Research Council of Cana-da, developers of high-accuracy scanning equipment,

have applied their technology toscanning paintings, sculptures, andarchaeological sites. Recent workemphasizes the importance ofportable, reliable equipment thatresearchers can easily deploy at thescanning site.1 Zheng and Zhongscanned archaeological relics incooperation with the Museum ofTerra Cotta Warriors and Horses,China.2 Their goals included creat-ing a database of information aboutthe excavation site and testing andemploying virtual restoration tech-niques. Recently, Marc Levoy and ateam from Stanford Universityscanned many of Michelangelo’ssculptures,3 including the 5-m tall

David in the Galleria dell’Accademia. They used sever-al types of scanners, including a high-resolution lasertriangulation system mounted on a custom-mademechanical gantry and a time-of-flight long-range sen-sor. The large quantity of data collected should have amajor impact in future development of shape recon-struction algorithms. Numerous other projects havebeen conducted or are currently underway.

In this article, we describe a recent project to acquireand build a 3D model of the Michelangelo’s FlorentinePietà. Figure 1 (next page) shows a photograph of thePietà and an image of our model. The work we describehere is unique in that an art historian, not a technologist,conceived and specified the project. Our goal wasn’t sim-ply to produce the statue’s model but also to provide theart historian with material and tools to enable him toanswer his own research questions. The project gave us

the opportunity to explore the value of 3D scanning andvisualization in a nontechnical discipline, art history. Theproject’s second goal was to develop scanning technolo-gy accessible to other cultural heritage projects both interms of cost and usability. Such technology could poten-tially be used in widespread commercial applications,such as e-commerce, where equipment cost must be min-imal.

OverviewJack Wasserman, professor emeritus of art history at

Temple University in Philadelphia, had been studyingMichelangelo’s Florentine Pietà for several years. Heintended primarily to document all aspects of this impor-tant work and its history for future researchers, andsecondarily to investigate his own theories on Michelan-gelo’s composition. He had used high-quality tradition-al photography, x-ray, and ultraviolet light studies, aswell as researching the complex history and symbolismof the statue and its analysis by past art historians.

Although it’s not clear that a 3D model would be use-ful in studying every sculpture, Wasserman felt that thisnew technology was especially well suited to studyingthe Pietà.4

Accounts from Michelangelo’s contemporaries tell usthat the artist intended the Florentine Pietà to serve as hisown tomb monument. Beginning late in his life, in the1550s, he carved four larger-than-life figures from a sin-gle block of marble. The Christ figure in the center restsacross the lap of the Virgin Mary, supported on the leftby Mary Magdalene. Behind and above, supporting thestatue of Christ, is a figure believed to represent Nicode-mus and to bear Michelangelo’s face. At some point,Michelangelo decided to break off parts of the statue.He then abandoned it. Shortly before his death he per-mitted one of his students, Tiberio Calcagni, to repairthe statue. Calcagni reattached several pieces to the stat-ue and partially finished the figure of the Mary Magda-lene. Thus, what we see today is, in a sense, a compositeof Michelangelo’s work and his student’s: the originaldesign damaged, repaired, and overlaid by later work.

The unique aspects of the history of this statue make

We present the

methodology we used to

acquire the data and

construct a computer model

of Michelangelo’s Florentine

Pietà with enough detail and

accuracy to make it useful in

scientific studies.

Fausto Bernardini, Holly Rushmeier, Ioana M.Martin, Joshua Mittleman, and Gabriel TaubinIBM T. J. Watson Research Center

Feature Article

it a promising candidate for using 3D scanning tech-nology. It’s important for art historians to view the stat-ue in the environment Michelangelo intended, examineit without the pieces Michelangelo removed, and ana-lyze the detailed toolmarks in the unfinished portion ofthe work. Furthermore, the statue’s complex geometrylimits what can be done with traditional techniques. Acamera can’t capture certain views of the statue becausethe statue itself or the walls of the room where it standsinterfere with proper camera placement.

Scanning system and methodologyThree-dimensional scanning technology is evolving

rapidly. Current sensors use a number of techniquessuch as laser triangulation, laser time-of-flight, passivestereo vision, and structured light projection to samplesurfaces. Typical considerations in choosing the mostappropriate scanning technology include target accu-racy, surface reflectance characteristics, and cost.

Design considerationsScanning a large statue in a museum poses a number

of constraints in designing the scanning system andprocess. In our case, the small size of the room in whichthe statue is displayed limited scanner size and the dis-tance from the scanner to the statue. We didn’t have per-

mission to work or leave any equipment visible aroundthe statue when the museum was open to the public.Therefore, we needed a system that we could easily setup and remove at the beginning and end of each eveningscanning session. The irreplaceable nature of the piecerestricted contact to a minimum and required us toensure that we operated the scanner system safely. Thecomplex shape of the group of figures in the Pietàrequired the ability to freely position the sensor to accessthe marble surface’s recessed parts.

We had a limited budget for buying noncomputerequipment and a limited amount of time for design andcustomization. These constraints led us to consider asmall, portable structured-light system rather than amore expensive laser triangulation scanner. By thischoice, we sacrificed geometric resolution, which wewould have had to recover with a supplementary system.

Our main technical requirements were dictated by thenature, resolution, and accuracy of the data needed toaddress Wasserman’s needs. The goal was to obtain datato allow realistic rendering of the synthetic model. Thestatue is 2.25 meters tall, and we wanted to capture itsshape and surface details, such as cracks and toolmarks,on the scale of 1 to 2 mm. Besides geometry, we wereinterested in capturing the surface’s reflectance proper-ties. We therefore needed to achieve submillimeter accu-racy in measurements.

Capturing a large object at such fine resolution entailsa number of difficulties, especially under the less-than-ideal conditions outside a lab. Issues of repeatability andprecision make scanners based on moving parts expen-sive to build and difficult to transport and operate. Sub-surface scattering of laser light in marble limits theaccuracy that laser triangulation systems can achieve.We decided to use a system that could capture a smallportion of the surface from a single position, acquire alarge number of overlapping scans, and rely on softwareregistration to integrate the results.

The amount of data we needed to represent the sur-face at such a fine level of detail presents additionalproblems. For example, we can’t store, process, and visu-alize a triangle mesh of hundreds of millions or billionsof triangles on current PCs or midrange workstations.Because we aimed to make the results accessible to a

wide audience, we decided to rep-resent shape as a triangle mesh witha resolution of a few millimeters andto store additional fine geometricdetails as a normals map. Plus, wecould store reflectance values asRGB image maps. Having thus cho-sen the final representation of ourmodel, we avoided a great deal ofintermediate computation bydesigning a system that capturesdata directly in that format.

ScanningFigure 2 shows a schematic of our

3D capture methodology. Our scan-ner is based on a multibaselinestereo system, supplemented by a

Feature Article

60 January/February 2002

1 A photo-graph ofMichelangelo’sFlorentine Pietà(left). A synthet-ic picture fromour 3D comput-er model(right).

2 Our 3D capture methodology. (a) Take multiple digital photos. (b) Compute each scan’ssurface shape, color, and details. (c) Align and merge the scans into a single model.

(a) (b) (c)

photometric system. The scanner,visible in Figures 3a and 3b, is a cus-tomized version of the VirtuosoShapeCamera. A photographic flashprojects a pattern of vertical stripeson the subject. At the same time, sixblack-and-white digital camerasphotograph the illuminated areafrom different angles. Figure 3cshows a sample stripe image. Anadditional digital color camera pro-vides a texture image. A multiviewstereo algorithm,5 part of the soft-ware suite provided with the Virtu-oso system, computes a trianglemesh approximating the scannedarea. In our scanning conditions,each scan typically covered a 20 cmby 20 cm area and comprised onaverage about 10,000 measuredpoints. The typical intersample dis-tance for these scans is about 2 mm.In tests we conducted on referenceobjects, we were able to accuratelymeasure a depth of 0.1 mm for a sin-gle scan.

The photometric system (Figures3a and 3b) consists of five lightsources and the built-in color cam-era, plus some control electronics.For each camera pose, we took fiveadditional color pictures, each withone of the five light sources, whileall other lights were turned off. We also used low-powerlaser sources to project red dots onto the statue (shownmounted on light stands in Figure 3b). The laser pro-jectors that we used each generated an 11 × 11 grid ofrays. From a distance of about 1 meter, they produced anirregular pattern of red dots on the statue, with an aver-age spacing of 2 to 4 cm. For each pose, we took a pictureof the dots (with all other light sources turned off) tohelp align overlapping meshes (see Figure 3d). The colorpictures have a resolution of 1280 × 960 pixels, with 24bits of RGB color per pixel. Typically we had a 0.5-mmintersample distance in the color images. As a result, wecomputed reflectance and surface normals from thesepictures at a resolution about four times greater thanthe underlying geometry.

Our initial design included a magnetic tracker torecord an approximate estimate of the camera positionand orientation with respect to a global frame of refer-ence. We hoped to use this pose estimate to provide astarting point for our software registration process. Weused a Polhemus system, fitted with the long-rangesource to provide an electromagnetic field large enoughto cover our work volume. We attached a sensor at thetip of a 40-cm plastic rod, rigidly secured to the camerabody. Unfortunately, we quickly discovered that metal-lic material in the room, including our own equipment,distorted the field to the point of making measurementsuseless. We also had initially planned to use additionalhardware and software to remotely control the scanner

and facilitate data transfer operations. However, to keepsetup and teardown time to a minimum, we simplifiedour system considerably.

Our streamlined procedure consisted of the followingsteps. We positioned the large photographic tripod andsecured the scanner to it. Then, we placed the five laserprojectors on three light stands to cover the area to bescanned in one session with a grid of laser dots. We cap-tured the data by placing the scanner at one area to becovered and shot one set of pictures, then we moved thescanner across the target area to take successive overlap-ping picture sets, covering the region with a regular pat-tern of scanned tiles. We kept track of the approximatearea covered by each picture set on paper diagrams of thestatue. We estimated that we had enough overlap by com-paring previews on the scanner display. We moved thescanner conservatively, about 10 cm between shots, toensure that we had enough data. We processed the stripepictures during the day, before the next evening’s scan-ning session, to make sure that we had acquired enoughdata and not left any holes. One person operated the scan-ner, while another acted as supervisor to ensure that theproper standoff distance was respected and safety rulesfollowed. Wasserman was present during the entireprocess to provide input on his priorities.

We did a preliminary scan of the statue (without thephotometric system) in February 1998, spending fivesix-hour evenings and four full days in the museum. Werepeated the scan using the photometric system during


3 (a) The scanner used in our project. We added the five-light photometricsystem to a Virtuoso ShapeCamera. (b) The scanner in use in the museum.The stands visible in the picture held the laser projectors. (c) Detail of oneof the six stripe images simultaneously taken by the scanner. (d) Detail ofthe laser dots projected on the statue, as captured by the color cameramounted on the scanner.

(a) (b)

(c) (d)

two one-week visits, one in June 1998 and one in July1999. The total time spent doing the final scanning wasabout 90 hours over 14 days, including the equipmentsetup and teardown each day. It took about 800 scansto cover the whole statue. The raw data consisted of4,800 640 × 480 pixel, 8-bit gray-scale stripe picturesand 4,800 coregistered 1280 × 960 pixel, 24-bit RGBcolor images. Stored in lossless compressed format, theraw data occupies 3 Gbytes of storage.

In retrospect, we believe that we chose the right scan-ning technology, although with additional planning anddesign we could have built a more efficient system. Themain bottlenecks in the process were the scanner’s rel-atively long cycle time, the small area covered by eachscan, and the offline processing of data. The timerequired to complete the acquisition and local storageof one set of images was about 2 minutes, and werequired about the same amount of time to process oneset of striped images to obtain a triangle mesh. Track-ing the camera pose would have saved us from doingthe pairwise manual alignment of the scans that pro-vided a starting point for our registration algorithms.

The main advantage of our scanning system, besidesmeeting the requirements of our original design, is thatit potentially provides a starting point for future devel-opment of a low-cost system built out of commoditycomponents. If it’s augmented with reliable tracking,fast capture, and high-resolution cameras, it could leadto a system for real-time scanning of large objects.

Reconstruction pipelineThe acquired raw data consisted of roughly 800

scans, each consisting of six black-and-white stripe

images and six color images. Weused the Virtuoso Developer soft-ware to compute triangle meshesfor each single scan from the sixstripe images. From this point on,we applied several algorithms tothe data to build the final model.We registered the individual scanstogether based on matching geo-metric and image features. Weremeshed the resulting point cloudto obtain a seamless geometricmodel. We extracted the color anddetail information from five of thecolor images and reorganized themin the form of normals andreflectance maps. Figure 4 illus-trates the sequence of steps.

RegistrationWe start with a pairwise, manu-

al, approximate alignment of themeshes, obtained interactively byselecting three or more matchingfeatures on overlapping pairs ofscans. We used the diagramsrecorded during scanning to identi-fy sequences of overlapping scansand constructed a tree of pairwise

alignments that spans the whole set of scans. We need-ed to do this initial manual alignment because the track-ing hardware we intended to use didn’t performsatisfactorily in the museum. We progressively refinedthe alignment using several registration algorithms thatused geometric and image information at increasing res-olution levels.

For each scan, we found the red dots in the imagetaken with the laser projectors on and mapped theseimage points back onto the scan geometry. Given theinitial manual alignment, we searched in the neighbor-hood of each laser point for matching points in over-lapping scans, adding additional consistency constraintsto prune false matches. We improved the registrationby minimizing the sum of square distances betweenmatching points using Besl and McKay’s method.6 Wethen ran several iterations of an n-scan iterated closestpoint (ICP) algorithm7 to further reduce the registra-tion error. Additional details of our geometry-basedalignment appear elsewhere.8

To refine the geometry-based alignment obtainedwith the ICP algorithm, we applied an image-based reg-istration method that considers additional informationcontained in the high-resolution reflectance maps com-puted for each scan (see the following section). We useda combination of smoothing, edge-detection, andthresholding operations for selecting candidate pointsin feature-rich areas of each image. We conducted a cor-relation-based search in images associated with over-lapping scans to find corresponding points. Wesubsequently back projected the resulting pairs onto thescans and derived a rigid transformation that minimizesdistances in a least-squares sense. See Bernardini, Mar-

Feature Article


Manualpairwise

alignment

Laser dotsalignment

ICP

Image-basedregistration

Conformancesmoothing

Registrationmatrices



BPA

Simplify

Meshsegmentation

Photometric

Coloralignment

Patches

Mesh

Mesh

Pointcloud

Albedo,normals map(per scan)

Texturesynthesis

Viewer

Albedo,normals map(per patch)

Correctedalbedo

4 The reconstruction pipeline.

tin, and Rushmeier9 for additional details of the image-based registration phase.

We also attempted to reduce scanner line-of-sighterrors by computing more accurate estimates of truesurface points from multiple overlapping scans whilefiltering out small high-frequency components (whichthe photometric system can better capture). We call thisprocess conformance smoothing. Figure 5 shows anexample of the successive alignment steps on three testscans of Nicodemus’ face. The finer grain of the colorvariations after the conform step indicates that theshape of the overlapping scans are nearly the same.Experiments showed we could improve the registrationerror if we accounted for the line-of-sight error duringthe alignment. We’re experimenting with alternatingiterations of registration and conformance smoothingto obtain a better final alignment.

We didn’t have equipment to accurately measure largedistances between points on the statue (for example,from the top of Nicodemus’ head to a point on the base).Therefore, we were unable to quantitatively state theglobal accuracy of the alignment. We discuss the valida-tion of our results using 2D projections and photographsin the section “Validating and using the model.”

MeshingThe result of the alignment and conformance pro-

cessing is a large set of estimated surface samples. Thispoint cloud has nonuniform density because the num-ber of overlapping scans varies from one part of the sur-face to another and because the density within eachscan varies locally depending on the angle at which thescanner saw the surface. However, except for areas thatthe scanner couldn’t reach, the sampling density is usu-ally larger than necessary to recover the surface’s shapeto a reasonable approximation. We designed our sys-tem to acquire geometry with an average intersampledistance of 2 mm. Note that this spatial resolution isindependent of the accuracy in measuring point posi-tion. The scanner we used has a precision of 0.1 mm incomputing depth for each sample point.

The ball-pivoting algorithm (BPA)10 computes a tri-angle mesh interpolating the point cloud, using a region-growing approach. Our implementation of the BPAhandles large data sets in a memory-efficient way, byprocessing input data in slices.

The Pietà data consist of 800 scans containing a totalof 7.2 million points. We processed the data in slices of10 cm, using ball radii of 1.5, 3, and 6 mm. The BPA ranin 30 minutes on a Pentium II PC using 180 Mbytes ofmemory and output a 14-million triangle mesh.

We applied a mesh simplification algorithm to gen-erate a hierarchy of models at different resolutions. Wefound that conventional, in-core, simplification algo-rithms couldn’t handle the large mesh generated fromour data. We computed simplified models by breakingup the mesh into smaller, manageable pieces. We thenapplied a traditional, high-quality simplification algo-rithm,11 leaving the boundary of each piece untouchedand stitched the resulting simplified pieces together. Ina successive pass, we broke up the mesh along differentedges, so that we could simplify the previous bound-

aries. We can repeat the process as many times as need-ed. Eventually, the simplified mesh is small enough tobe further processed in a single pass by the in-core algo-rithm. We expect memory-efficient simplification algo-rithms to become a hot topic of research as capturemethods improve and large models become widespread.

Details and colorThe mesh produced using the Virtuoso camera has a

spatial resolution of approximately 2 mm, which is ade-quate for studying the statue’s proportions from variousviewpoints. However, it’s inadequate for studying thesmall-scale tool marks. To capture data at a higher spa-tial resolution, we exploited the fact that the Virtuosoincludes a color camera that produces images with a res-olution of about 0.5 mm per pixel. We computed detailat this pixel resolution using a photometric stereo sys-tem built around the Virtuoso.

Figure 3 shows our photometric system. Given threeimages of a surface lit by three different light sources inknown positions, we can solve a set of simultaneous equa-tions for the surface normals corresponding to the pointsvisible at each pixel in the image. Given the normal ateach pixel, we can compute the relative reflectance, oralbedo, at each pixel for the red, green, and blue bands.We used five light sources rather than three because inany given image a point may be in a shadow or a specu-lar highlight. Figure 6a shows four typical imagesobtained from a single camera position. Figure 6b showsthe resulting normals (lit from the side). For further detailof the physical system design, see Rushmeier et al.12

To compensate for possible errors in the photometricnormals calculations, we used data from the 2-mm res-olution mesh to compute the direction and relative dis-tance to each point visible in each image and to estimatethe relative light source intensity in the neighborhood ofeach pixel from each of the five lights. To compensatefor scan-to-scan color variations, we performed a colorregistration analogous to the geometric registration ofscans. We found corresponding points in all overlappingcolor albedo maps and then found a least-squares solu-


5 Subsequentsteps of theregistration ofthree test scans.Each scan showsa differentcolor. In theconform step,the scans havebeen slightlydeformed(shown with achanged color)to compensatefor residualregistration andmeasurementerror.

tion for scaling factors for each of the color channels ineach of the images to obtain the best color match on cor-responding points. Rushmeier and Bernardini13 discussadditional details of adjustments made using the under-lying mesh and color registration.

Texture synthesisWe partitioned the triangle mesh into height-field

patches with a simple region-growing heuristic. For eachpatch, an orthogonal projection in the direction thatmaximizes the projected patch area defines a mappingbetween geometry and corresponding textures.

The texture synthesis process computes surface nor-mals and reflectance maps as weighted combinations ofcorresponding values in all the overlapping images.Weights are assigned to take into account the degree ofconfidence in each pixel value, based on distance to thecamera and viewing angle. Because weight maps cor-respond to scans and not to patches, transitions acrosspatch boundaries aren’t visible. Also, since the weightsfor each scan decrease with distance to the scan border,scan-to-scan boundaries aren’t visible.

In our implementation, we streamlined computationsby presampling the patch geometry and loading valuesfrom all maps simultaneously. Occlusions are handledelegantly by comparing depth values in precomputeddepth buffers. Image and geometric information isloaded on demand to allow for processing of large datasets that don’t fit in memory. We provide additionaldetails regarding our image-based registration and tex-ture synthesis algorithms elsewhere.9

Validating and using the modelA large digital model isn’t useful to art historians. As

a result, we needed to derive an assortment of presen-tations of the data suited to Wasserman’s needs, whichin some cases required new techniques. Before devel-oping other results from our model we needed to vali-date its accuracy to Wasserman’s satisfaction. His testwas that images derived from our model must correlatewell with the high-quality photographs he had com-missioned from a professional photographer.

To perform the validation, we selected features indigitized versions of Wasserman’s photographs andfound the corresponding 3D coordinates of thosepoints on our model. We then used Tsai’s calibrationmethodology14 to compute camera parameters to gen-

erate a synthetic image from thesame viewpoint. We couldn’t esti-mate the lighting conditions in thecommissioned photographs. Toaddress the effect of lighting, wealso matched camera viewpointsfor images we took with a digitalcamera for which we knew the flashlocation. Initially, we computedimages with geometry alone andfound that including the surfacealbedo was essential to perceivingthe proportions in the syntheticimage.

Overview of resultsWasserman’s research questions defined our primary

goals for the Pietà project. We shaped our presentationof the results to fit his needs. We developed a plan to ful-fill his requirements by delivering a variety of results,including

� precisely defined views,� impossible views,� embedding the statue in virtual environments,� providing precise measurements,� modifying the statue, and� providing an interactive viewer.

To answer certain questions about Michelangelo’scomposition, Wasserman wanted to see the statue fromphysically or practically impossible points of view. Theseincluded views from directly above the statue to revealdetails of the composition not normally visible (Figure7c, next page) and from various angles at a height belowthe base of the statue to illustrate it as it would haveappeared in the context Michelangelo originally intend-ed. We also recreated some of the settings in which thePietà stood during its history, using 3D models and ani-mations to illustrate the statue’s visual impact in thesevarious environments (Figures 7e and 7f). To reconstructthe tomb and garden settings shown in these figures,Wasserman provided drawings and images of similarenvironments and some crude dimensions. Accuratelymodeling the environments required a number of vari-ations of each environment that Wasserman evaluatedagainst his understanding of the historical record.

Editing the modelThe ability to modify our model of the statue provid-

ed Wasserman with opportunities to study it in waysotherwise impossible. Using the 3D model, we recon-structed the statue with Christ’s missing left leg, approx-imating its appearance before Michelangelo broke it.We removed the pieces that Calcagni reattached, illus-trating the statue as it may have appeared without hisefforts. Figure 7g shows this second modification. In thefigure, some of the surfaces that would be occluded bythe limbs removed are now visible. Internal areas thatrevealed where the marble was broken are colored a flatgray. We also separated the four figures that make upthe statue so that they can be examined in isolation.

Feature Article


6 (a) Colorimages takenwith four of thefive lightsources. (b)Synthetic pic-ture computedusing the sur-face normalsobtained withthe photometricsystem.

(a) (b)

Identifying the pieces that Michelangelo removedis itself a problem. Four major pieces were removedfrom the statue (three arms and a portion of a leg thatwas never replaced). The three pieces that were reat-tached were each composed of a set of smaller frag-ments, so it’s not obvious what was removed. Basedon his own close study of the physical work and thex-rays he had commissioned, Wasserman sketchedon electronic images of the statue the various lineswhere he believed the breaks were made.

Directly editing a large model with millions of ver-tices isn’t feasible, particularly because our trianglemesh doesn’t have sufficient resolution to model thebreakage exactly as Wasserman wanted. We tried twomethods of editing the model. First, we tried paintingeach of the color images associated with the individualscans to precisely mark which parts belonged to theremoved sections. This approach had some problems,since hand marking didn’t give pixelwise identical loca-tions for all of the breaks across the various scans. How-ever, given the painted images, we could automaticallysegment the statue by simply removing vertices thatwere painted with the broken color. This simple com-putation was useful while we were producing early ver-sions of the model (before all the data was acquired,added, and tightly aligned) to give Wasserman an indi-cation of what results to expect.

For the final model, we crudely segmented the modelby defining bounding boxes enclosing the broken seg-ments. We then identified individual patches contain-ing portions of the cracks for editing. While thisapproach would be tedious to repeat many times (thecracks extend over many different patches), it was themost reliable approach for the final model.

We also used the painting to separate the four figuresin the model. This task is more sensitive to the problemof ambiguous identification across scans because no pre-cise lines on the statue define the figures. Wassermanwanted the figures separated because they revealedshapes and relationships, like the relative position ofMary Magdalene’s hands, that we can’t observe from thesolid statue. While we were able to achieve the segmen-tation of the statue we needed for this study, our experi-ence indicates that detailed editing of high-resolutionmodels is an area that requires additional research.

Interactive viewerTo enable Wasserman to study the statue on his own

computer, we designed a viewer that could run on a PC

(Figure 7d). The combination of a large data set and aslow computer required special attention to the trade-offs between speed and quality and between usabilityand flexibility. Our target audience consisted of unso-phisticated computer users unaccustomed to the navi-gation paradigms common to interactive 3D graphics.We found that we needed to radically simplify the con-trols to provide a fast learning curve and then adaptthem to our users’ abilities and interests. Maintaininginteractivity was essential, so that users could easilygrasp the function of the navigation controls and com-pensate for their inevitable limitations.

We had two objectives in designing an intuitive inter-face: maintaining a frame of reference when zoomingin on detail and providing clear, separate controls foraltering views and lighting. Our viewer presents userswith a simplified model of the statue around which theycan navigate interactively and lets them render a full-resolution image of a selected detail. The simplifiedmodel, only 1 percent the complexity of the full model,acts as a map to the more detailed model. Users canselect an area of interest, using simple 3D navigationcontrols to reach the desired view. We chose a naviga-tion paradigm in which the camera orbits around a user-selected center at a fixed distance and zooms in and outalong a radius. Users can select a new center by pickingor dragging the image parallel to the view plane.

To examine a region in more detail, users frame a sec-tion of the scene and render the desired image from a data-base of the full-detail model. This step currently can takea few minutes on a laptop computer. We enhanced theresulting 2D image to let users interactively vary the light-


(a) (b) (c) (d)

7 (a) Black-and-white rendering of the model. (b) Close-up view of themodel. (c) Bird’s eye view of the model. (d) Interactive viewer. (e) Syn-thetic image in a niche above the tomb. (f) Synthetic image in a garden.(g) The statue without the pieces Michelangelo removed.

(e) (f)

(g)

ing by dragging the mouse across the window. Manydetails invisible with the light in one position appear whenthe light moves. Users can thus understand fine-scalestructures more clearly. We designed this technique tomodel a method that we observed Wasserman using onthe statue. He moved a small flashlight slowly back andforth across the statue to highlight small surface irregu-larities. The virtual light editor produces similar effects.

The viewer was useful for isolating portions of themodel and rendering high-resolution close-ups of sec-tions of interest. We had hoped that the viewer wouldalso be helpful to Wasserman in evaluating the appear-ance of the statue from various views, to develop a the-ory of exactly how Michelangelo intended the statuteto be viewed. The simplified model in the viewer provedinadequate for this interactive study. When we simpli-fied the model to the extent that allowed interactiveviewing, it still looked like a good representation to thecasual observer. However, for Wasserman’s in-depthstudy of specific proportions, the simplified modelwasn’t accurate enough.

The current viewer still needs improvement. It’s farfrom satisfactory as a tool for art historians. We need todo a great deal of work to find intuitive methods for non-technical users to interact with 3D data, especially whenviewing large data sets that we must simplify to allowinteractive display. Part of the problem is rendering speed.Incorporating ideas such as point-based rendering15 ormultipass texturing now available on inexpensive graph-ics cards would improve this aspect of our system.

ConclusionsThis project has demonstrated that 3D scanning and

graphics can be useful tools for art historians, and byextension for similar studies in archaeology, architec-ture, and other disciplines where detailed examinationand manipulation of artifacts is necessary but not feasi-ble in reality. Wasserman noted three aspects of the vir-tual model that were especially useful to art historians:

� Having a virtual model of the statue lets researchersexamine the statue at their leisure, to discover detailsthey hadn’t noticed in the time they spent with thestatue and to verify their recollections and theories.

� The ability to control lighting precisely letsresearchers see the statue as it might have appearedin environments outside the museum and to highlightsmall details not easily visible.

� Measuring the statue freely and precisely lets histo-rians factor out subjective perception and betterunderstand the artist’s use of perspective and com-position.

Our technical experience shows that it’s possible tobuild a detailed computer model of a large object froma large collection of small individual scans, which wecan acquire with a relatively inexpensive device. Ourplans for future work include studying improved algo-rithms for accurately registering multiple scans anddeveloping a handheld, real-time scanning system. Formore information about our work, visit http://www.research.ibm.com/pieta.

The final conclusions Wasserman draws from the dig-ital model will be in his book published by PrincetonUniversity Press, which will include more of our resultson a CD-ROM. Kiosks are now (or have been) present-ing our work at a number of museums throughout theworld (including sites in Asia, Australia, South America,Europe, and the US). �

AcknowledgmentsWe express our appreciation to Claudio Silva and

André Gueziec for their contributions to this project,the Museo dell’Opera del Duomo in Florence for theirassistance in our study of the statue and Jack Wasser-man for his inspiration and for creating the opportuni-ty for our work.

References1. J.-A. Beraldin et al., “Portable Digital 3D Imaging System

for Remote Sites,” Proc. 1998 IEEE Int’l Symp. Circuits andSystems, IEEE Press, Piscataway, N.J., 1998, pp. 488-493.

2. J. Zheng and L. Zhong, “Virtual Recovery of ExcavatedRelics,” IEEE Computer Graphics and Applications, vol. 19,no. 3, May/June 1999, pp. 6-11.

3. M. Levoy, “The Digital Michelangelo Project,” Proc. 2ndInt’l Conf. 3D Digital Imaging and Modeling, IEEE Press, Pis-cataway, N.J., 1999, pp. 2-13.

4. J. Abouaf, “The Florentine Pietà: Can Visualization Solvethe 450-Year-Old Mystery?,” IEEE Computer Graphics andApplications, vol. 19, no. 1, Jan./Feb. 1999, pp. 6-10.

5. C.L. Zitnick and J.A. Webb, Multi-Baseline Stereo Using Sur-face Extraction, tech. rep. CMU-CS-96-196, School of Com-puter Science, Carnegie Mellon Univ., Pittsburgh, Penn.,Nov. 1996.

6. P.J. Besl and N.D. McKay, “A Method for Registration of 3DShapes,” IEEE Trans. Pattern Analysis and Machine Intelli-gence, vol. 14, no. 2, Feb. 1992, pp. 239-256.

7. R. Bergevin et al., “Towards a General Multi-View Regis-tration Technique,” IEEE Trans. Pattern Analysis andMachine Intelligence, vol. 18, no. 5, May 1996, pp. 540-547.

8. F. Bernardini and H. Rushmeier, “Strategies for RegisteringRange Images from Unknown Camera Positions,” Three-Dimensional Image Capture and Applications III (Proc.SPIE), vol. 3958, Jan. 2000, pp. 200-206.

9. F. Bernardini, I.M. Martin, and H. Rushmeier, “High-Quality Texture Reconstruction from Multiple Scans,” IEEETrans. Visualization and Computer Graphics, Oct.–Dec.2001, vol. 7, no. 4, pp. 318-332.

10. F. Bernardini et al., “The Ball-Pivoting Algorithm for Sur-face Reconstruction,” IEEE Trans. Visualization and Com-puter Graphics, vol. 5, no. 4, Oct.–Dec. 1999, pp. 349-359.

11. A. Guéziec, “Locally Toleranced Surface Simplification,”IEEE Trans. Visualization and Computer Graphics, vol. 5,no. 2, Apr.–June 1999, pp. 168-189.

12. H. Rushmeier et al., “Acquiring Input for Rendering atAppropriate Level of Detail: Digitizing a Pietà,” Proc. 9thEurographics Workshop on Rendering, Eurographics Assoc.,Aire-la-Ville, Switzerland, 1998, pp. 81-92.

13. H. Rushmeier and F. Bernardini, “Computing ConsistentNormals and Colors from Photometric Data,” Proc. 2nd Int’lConf. 3D Digital Imaging and Modeling, IEEE Press, Piscat-away, N.J., 1999, pp. 99-108.

Feature Article


14. R.Y. Tsai, “An Efficient and Accurate Camera CalibrationTechnique for 3D Machine Vision,” Proc. IEEE ComputerVision and Pattern Recognition (CVPR 86), IEEE Press, Pis-cataway, N.J., 1986, pp. 364-374.

15. S. Rusinkiewicz and M. Levoy, “Qsplat: A MultiresolutionPoint Rendering System for Large Meshes,” ComputerGraphics (Proc. Siggraph 2000), ACM Press, New York,2000, pp. 343-252.

Fausto Bernardini is a researchstaff member and manager of theVisual and Geometric Computinggroup at the IBM T.J. WatsonResearch Center. His research inter-ests include interactive visualizationof large models, 3D scanning and

reconstruction, and geometric modeling. He has been oneof the principal investigators in a research effort to build adigital model of Michelangelo’s Florentine Pietà. Hereceived an electrical engineering degree from the Univer-sity of La Sapienza, Rome, Italy, in 1990, and a PhD incomputer science from Purdue University in 1996.

Holly Rushmeier is a researchstaff member at the IBM T.J. WatsonResearch Center. Her research inter-ests include data visualization, ren-dering algorithms, and acquisitionof input data for computer graphicsimage synthesis. She received a BS,

MS, and PhD in mechanical engineering from Cornell Uni-versity in 1977, 1986, and 1988, respectively. In 1990, shewas selected as a US National Science Foundation Presi-dential Young Investigator. She has served as papers chairor cochair for the ACM Siggraph conference, the IEEE Visu-alization conference, and the Eurographics RenderingWorkshop. From 1996 to 1999, she was editor in chief ofACM Transactions on Graphics.

Ioana M. Martin is a researchstaff member in the Visual Technolo-gies Department at the IBM T.J. Wat-son Research Center. Her researchinterests include interactive comput-er graphics, visualization, and imageprocessing. She has developed and

implemented methods for processing and rendering largedata sets, including progressive refinement and multires-olution techniques with applications to structural biolo-gy. She has worked on various projects including modelingwith subdivision surfaces, adaptive delivery of 3D modelsover networks, 3D data compression, and model recon-struction. She received a PhD in computer science fromPurdue University in 1996.

Joshua Mittleman is a program-mer at the IBM T.J. Watson ResearchCenter, specializing in simplification,level-of-detail control, and otheralgorithms applied to rendering com-plex models. He has an MS in com-puter science from Rutgers University.

Gabriel Taubin is a research staffmember and former manager of theVisual and Geometric ComputingGroup at the IBM T.J. WatsonResearch Center. He is known for hiswork on 3D geometry compressionand progressive transmission of

polygonal models, the signal processing approach forpolygonal meshes, algebraic curve and surface rasteriza-tion, and algebraic curves and surfaces in computer vision.He received the IBM Research 1998 Computer Science BestPaper Award for his paper on geometry compressionthrough topological surgery. He earned a PhD in electri-cal engineering from Brown University and an MS in math-ematics from the University of Buenos Aires, Argentina.

Readers may contact Fausto Bernadini at the IBM T.J.Watson Research Center, PO Box 704, Yorktown Heights,NY 10598, email [email protected].

For further information on this or any other computingtopic, please visit our Digital Library at http://computer.org/publications/dlib.


Technology

Bernardini etal-cga02