Reporting Status or Progress

3D PhotographySensors

Ioannis StamosPerspective projectionIoannis Stamos CSCI 493.69 F08

Pinhole & the Perspective ProjectionSCREENSCENE

(x,y)Is there an image being formed on the screen?3Ioannis Stamos CSCI 493.69 F08

PinholeObjectPinhole cameraPinhole CameraCamera obscura known since antiquityImageImage plane4Ioannis Stamos CSCI 493.69 F08

Perspective Camera

(X,Y,Z)(x,y,z)Center ofProjectionrrr =[x,y,z]T

r=[X,Y,Z]Tr/f=r/Zf: effective focal length:distance of image plane from O.x=f * X/Zy=f * Y/Zz=fFrom Trucco & Verri5Ioannis Stamos CSCI 493.69 F08

Magnification

(X,Y,Z)(x,y)Center ofProjection(X+dX,Y+dY,Z)(x+dx,y+dy)x/f=X/Zy/f=Y/Z(x+dx)/f=(X+dX)/Z(y+dy)/z=(Y+dY)/Zdx/f=dX/Zdy/f=dY/Z=>dd From Trucco & Verri6Ioannis Stamos CSCI 493.69 F08

Magnification

(X,Y,Z)(x,y)Center ofProjection(X+dX,Y+dY,Z)(x+dx,y+dy)ddMagnification: |m|=||d||/||d||=|f/Z|or m=f/Zm is negative when image is invertedFrom Trucco & Verri7Ioannis Stamos CSCI 493.69 F08

Implications For Perception** A Cartoon Epistemology: http://cns-alumni.bu.edu/~slehar/cartoonepist/cartoonepist.html

Same size things get smaller, we hardly notice

Parallel lines meet at a point8Ioannis Stamos CSCI 493.69 F08

Vanishing Points

(from NALWA)9Ioannis Stamos CSCI 493.69 F08

Vanishing points

VPLVPRH

VP1VP2VP3Different directions correspond to different vanishing pointsMarc Pollefeys103D is different3D Data Types: Volumetric DataRegularly-spaced grid in (x,y,z): voxelsFor each grid cell, storeOccupancy (binary: occupied / empty)DensityOther propertiesPopular in medical imagingCAT scansMRI3D Data Types: Surface DataPolyhedralPiecewise planarPolygons connected togetherMost popular: triangle meshesSmoothHigher-order (quadratic, cubic, etc.) curvesBzier patches, splines, NURBS, subdivision surfaces, etc.2-D DataImage: stores an intensity / color alongeach of a set of regularly-spaced rays in spaceRange image: stores a depth alongeach of a set of regularly-spaced rays in spaceNot a complete 3D description: does notstore objects occluded (from some viewpoint)View-dependent scene description2-D DataThis is what most sensors / algorithmsreally returnAdvantagesUniform parameterizationAdjacency / connectivity informationDisadvantagesDoes not represent entire objectView dependent2-D DataRange imagesRange surfacesDepth imagesDepth mapsHeight fields2-D imagesSurface profilesxyz mapsRange Acquisition TaxonomyRangeacquisitionContactTransmissiveReflectiveNon-opticalOpticalIndustrial CTMechanical (CMM, jointed arm)RadarSonarUltrasoundMRIUltrasonic trackersMagnetic trackersInertial (gyroscope, accelerometer)Range Acquisition TaxonomyOpticalmethodsPassiveActiveShape from X:stereomotionshadingtexturefocusdefocusActive variants of passive methodsStereo w. projected textureActive depth from defocusPhotometric stereoTime of flightTriangulationOptical Range Acquisition MethodsAdvantages:Non-contactSafeUsually inexpensiveUsually fastDisadvantages:Sensitive to transparencyConfused by specularity and interreflectionTexture (helps some methods, hurts others)StereoFind feature in one image, search along epipolar line in other image for correspondenceStereo Vision

depth mapTriangulationfTZPplprOlOrLeft CameraRight CameraX-axisZ-axisclcrFixationPoint:Infinity.Paralleloptical axes.Calibrated Cameras

Similar triangles:

d:disparity (difference in retinal positions).T:baseline.Depth (Z) is inversely proportional to d (fixation at infinity)Traditional StereoInherent problems of stereo:Need textured surfacesMatching problemBaseline trade-offUnstructured point setHoweverCheap (price, weight, size)Mobile Depth plus Color Sparse estimatesUnreliable results23Why More Than 2 Views?BaselineToo short low accuracyToo long matching becomes hardMultibaseline Stereo

[Okutami & Kanade]Shape from MotionLimiting case of multibaseline stereoTrack a feature in a video sequenceFor n frames and f features, have2nf knowns, 6n+3f unknownsPhotometric Stereo Setup

[Rushmeier et al., 1997]Photometric Stereo

Use multiple light sources to resolve ambiguityIn surface orientation.

Note: Scene does not move Correspondence between points in different images is easy!

Notation: Direction of source i: or Image intensity produced by source i:

28Lambertian Surfaces (special case)

orientationalbedoUse THREE sources in directionsImage Intensities measured at point (x,y):

29Photometric Stereo: RESULT

orientationalbedoINPUT30Data Acquisition Example

Color Image of Thomas HunterBuilding, New York City.Range Image of same building. One million 3D points.Pseudocolor corresponds to laserintensity.Time-of-flight scannersData Acquisition Spot laser scanner. Time of flight. Max Range: 100 meters. Scanning time: 16 minutes for one million points. Accuracy: ~6mm per range point

Data Acquisition Leica ScanStation 2 Spherical field of view. Registered color camera. Max Range: 300 meters. Scanning time: 2-3 times faster Accuracy: ~5mm per range point

34Data Acquisition, Leica Scan Station 2, Park Avenue and 70th Street, NY

Cyclone view and Cyrax Video

Pulsed Time of FlightSend out pulse of light (usually laser),time how long it takes to return

Pulsed Time of FlightAdvantages:Large working volume (more than 100 m.)Disadvantages:Not-so-great accuracy (at best ~5 mm.)Requires getting timing to ~30 picosecondsDoes not scale with working volumeOften used for scanning buildings, rooms, archeological sites, etc.Optical Triangulation

LensSheet oflightCCD arraySources of error: 1) grazing angle, 2) object boundaries.39Optical Triangulation

LensSheet oflightCCD arraySources of error: 1) grazing angle, 2) object boundaries.Active Optical Triangulation

XcYcZcImageLight Stripe System.

Light Plane:AX+BY+CZ+D=0 (in camera frame)Image Point:x=f X/Z, y=f Y/Z (perspective)Triangulation: Z=-D f/(A x + B y + C f)

Move light stripe or object.Triangulation

42Triangulation: Moving theCamera and IlluminationMoving independently leads to problems with focus, resolutionMost scanners mount camera and light source rigidly, move them as a unitTriangulation: Moving theCamera and Illumination

Triangulation: Moving theCamera and Illumination

The Digital Michelangelo Project

Marc Levoy (Stanford)calibrated motionspitch (yellow)pan (blue)horizontal translation (orange)uncalibrated motionsvertical translationremounting the scan headmoving the entire gantry

Scanner design

4 motorized axeslaser, range camera,white light, and color cameratruss extensionsfor tall statues480:30Scanning the Davidheight of gantry: 7.5 metersweight of gantry: 800 kilograms

492:30 through closeup of eye0:40So here we are scanning our piece de resistancethe gantry is extended to nearly its full heightIf youve looked at our web siteyou know the little story about the gantry being too short initiallybecause we relied on art history books for the height of the Davidand they're all wrong by 3 feet !!so at the last minute we had to add a piece to the gantryhereand load another few hundred pounds of ballast to the basehere to keep the gantry from tipping over on the statueBut as you can see from the photo on the right,we did finally reach the top of the statue0:454:00 through architectural repsTriangulation Scanner IssuesAccuracy proportional to working volume(typical is ~1000:1)Scales down to small working volume(e.g. 5 cm. working volume, 50 m. accuracy)Does not scale up (baseline too large)Two-line-of-sight problem (shadowing from either camera or laser)Triangulation angle: non-uniform resolution if too small, shadowing if too big (useful range: 15-30)Triangulation Scanner IssuesAccuracy proportional to working volume(typical is ~1000:1)Scales down to small working volume(e.g. 5 cm. working volume, 50 m. accuracy)Does not scale up (baseline too large)Two-line-of-sight problem (shadowing from either camera or laser)Triangulation angle: non-uniform resolution if too small, shadowing if too big (useful range: 15-30)Multi-Stripe TriangulationTo go faster, project multiple stripesBut which stripe is which?Answer #1: assume surface continuity

Multi-Stripe TriangulationTo go faster, project multiple stripesBut which stripe is which?Answer #2: colored stripes (or dots)

Multi-Stripe TriangulationTo go faster, project multiple stripesBut which stripe is which?Answer #3: time-coded stripes

Time-Coded Light PatternsAssign each stripe a unique illumination codeover time [Posdamer 82]SpaceTimeLaser-based SensingProvides solutionsDense & reliableRegular gridButComplex & large point setsRedundant point setsNo color information (explain this)Expensive, non-mobileAdjacency infoMesh56Major IssuesRegistration of point sets Global and coherent geometryremove redundancyhandle holeshandle all types of geometries Handle complexityFast Rendering57Representations of 3D Scenes Geometry and Material Geometry and Images Images with DepthColored VoxelsLight-Field/LumigraphPanorama with DepthComplete descriptionTraditional texture mapping

Scene as a light sourceGlobalGeometryLocalGeometryFalseGeometryPanoramaNo/Approx.GeometryFacade58

Head of Michelangelos Davidphotograph1.0 mm computer model

590:20We havent assembled the David at full-resolution yetHeres a 1 mm model of the headwatertightwith colorThe photograph at the left was taken with uncalibrated lightingso the two images dont match exactlybut hopefully you agree that weve basically captured the statues appearance

Davids left eyeholes from Michelangelos drillartifacts from space carvingnoise from laser scatter

0.25 mm modelphotograph600:30Heres a closeup of Davids left eyeat the maximum resolution of our model mmwe can see some interesting thingsThese bumpsare holes from Michelangelos drillso thats real geometryThese bumpsare artifacts from space carving, the hole-filling part of our range image merging algorithmsmoothing out those artifacts, while preserving the observed surfaces, is future workFinally, these bumpsare noise from laser subsurface scatteringLets zoom in some moreand look at the triangle mesh0:45IBMs Piet ProjectMichelangelos Florentine PietLate work (1550s)Partially destroyed by Michelangelo, recreated by his studentCurrently in the Museo dellOpera del Duomoin Florence

ScannerVisual Interface VirtuosoActive multibaseline stereoProjector (stripe pattern),6 B&W cameras, 1 color cameraAugmented with 5 extrapoint light sources forphotometric stereo(active shape from shading)

DataRange data has 2 mm spacing, 0.1mm noiseEach range image: 10,000 points, 2020 cmColor data: 5 images with controlled lighting, 1280960, 0.5 mm resolutionTotal of 770 scans, 7.2 million pointsScanningFinal scan June 1998, completed July 1999Total scanning time: 90 hours over 14 days(includes equipment setup time)

Results

HUNTER COLLEGE (our work):

Segmented Planar Area.Exterior and Interior borders shown.Interiorborder.Range sensingdirection.Each SegmentedPlanar Area isshown with differentcolor for clarity.Exterior border. Intersection line.HUNTER COLLEGE

HUNTER COLLEGE: Final result 24 scan pairs~600 lines per scan

HUNTER COLLEGE: Matching 3D with 2D imagery

3D parallelepipeds2D rectanglesMatches shown in blueHUNTER COLLEGE: Virtual (synthetic) scene views

Virtual (synthetic) scene views

Virtual (synthetic) scene views