United States Patent [19]

Adelson[11] Patent Number: 5, 07 6,6 87 [45] Date of Patent: Dec. 31, 1991

[54] OPTICAL RANGING APPARATUS

[75] Inventor: Edward H. Adelson, Cambridge,Mass.

[73] Assignee: Massachusetts Institue ofTechnology, Cambridge, Mass.

[21] Appl. No.: 574,417

[22] Filed: Aug. 28, 1990

[51] Int. Cl.5 ...................................... G01C 3/08[52] US. Cl. ............................... 356/4 ; 250/201.7; 250/201.8; 354/402[58] Field of Search ........... 356/4; 250/201.7, 201.8; 354/402[56] References Cited

U.S. PATENT DOCUMENTS

4,185,191 1/1980 Stauffer .4,230,941 10/1980 Stauffer .4,373,791 2/1983 Araki .4,414,470 11/1983 Nakaoka .4,644,148 2/1987 Kusaka et al. .............................. 250/201.84,945,407 7/1990 Winnek ........................................... 358/88

OTHER PUBLICATIONS

“Fast and Reliable Passive Trinocular Steriovision”, byNicholas Ayache & Francis Lustman, 1987, pp. 422-427.“Towards Real-Time Trinocular Sterio”, by Charles Hansen,Nicholas Ayache & Francis Lustman, 1988, pp. 129-133.“An Iterative Image Registration Technique with anApplication to Stereo Vision”, by Bruce D. Lucas and TakeoKanade, 1981, pp. 121-130.“Three-Dimensional Imaging Techniques”, by TakanoriOkoshi, 1976, pp. 1-42 & 60-123.“The Trinocular General Support Algorithm: A Three-Cam-

era Sterio Algorithm for Overcoming Binocular MatchingErrors”, by Charles V. Steward & Charles R. Dyer, 1988,pp. 134-138.

Primary Examiner—Stephen C. BuczinskiAttorney, Agent, or Firm—Hamilton, Brook, Smith &Reynolds

[57] ABSTRACTAn optical ranging apparatus is provided which resolves thedepth of objects in an object field of a main lens. Light i sdirected by the main lens to a lenticular array consisting ofan array of lenticules each of which generates an image ofthe lens surface. Each of the generated images is directed to aphotodetector array divided into macropixels each of whichreceives one of the lenticule images. Each macropixel i smade up of a number of subpixels, each of which is a discretephotodetector element and generates an electrical signalproportional to the intensity of light upon it. A dataprocessor receives the signals from the subpixels andassembles subimages, each of which consists of onesubpixel from each macropixel. Each subimage represents aview of the entire object field through one portion of themain lens. The data processor determines depth of objects inthe object field by comparing the parallax between sub-images. A diffuser may be provided between the object fieldand the lenticular array to low pass filter light from theobject field and reduce aliasing. A field lens may also beprovided between the main lens and the lenticular array toredirect light from the main lens and prevent skewing of thelight reaching the macropixels. The images from thelenticular array may be relayed to a photodetector array setback from the lenticular array by using a relay lens.

44 Claims, 6 Drawing Sheets

U.S. Patent Dec. 31, 1991 Sheet 1 of 6 5,076,687

U.S. Patent Dec. 31, 1991 Sheet 2 of 6 5,076,687

U.S. Patent Dec. 31, 1991 Sheet 3 of 6 5,076,687

U.S. Patent Dec. 31, 1991 Sheet 4 of 6 5,076,687

U.S. Patent Dec. 31, 1991 Sheet 5 of 6 5,076,687

U.S. Patent Dec. 31, 1991 Sheet 6 of 6 5,076,687

5 ,076 ,6871

OPTICAL RANGING APPARATUS

BACKGROUND OF THE INVENTION

For performing an optical measurement of depth, it i sfrequently desirable to derive a range image, whichrepresents the distances between a viewing point andvarious surfaces in a scene. Range images may be derivedby active techniques in which some form of energy i sdirected toward an object and then measured on its return.Popular methods of active ranging include sonar, laserrangefinding, and structured light. Range images may alsobe derived by passive techniques, in which the light froma normally illuminated scene is captured and analyzed byone or more television cameras. Of the passive tech-niques, binocular stereo is the most popular.

There are many situations in which active techniques areinconvenient or ineffective in acquiring the desired rangeimagery. The most popular passive technique, binocularstereo, has a number of disadvantages as well. It requiresthe use of two cameras that are accurately positioned andcalibrated. Analyzing the data involves solving the corre-spondence problem, which is the problem of determiningthe matches between corresponding image points in thetwo views obtained from the two cameras. The correspon-dence problem is known to be difficult and demandingfrom a computational standpoint, and existing techniquesfor solving it often lead to ambiguities of interpretation.The problems can be ameliorated to some extent by theaddition of a third camera (i.e. trinocular stereopsis), butmany difficulties remain.

The correspondence problem can also be avoided if oneacquires a series of images from a series of closely spacedviewpoints, as is done in epipolar analysis. However, theprocedure in this case is quite cumbersome, since a cameramust move along a trajectory over an extended period oftime in order to gather a sequence of images.

SUMMARY OF THE INVENTION

An optical depth resolver according to the presentinvention has a converging lens which receives lightfrom objects in an object field. The lens directs the lightto an image field of the converging lens where it i sreceived by a plurality of imaging elements distributedabout an image field of the converging lens. Each of theimaging elements receives light from the converging lensand forms an image from the light it receives.

The images from the imaging elements are received by aphotodetector array which is divided into a plurality ofmacropixels, each of which is made up of a plurality ofsubpixels. Each macropixel of the photodetector arrayreceives the image from one of the imaging elements, andeach subpixel generates an electrical signal indicative ofthe intensity of light incident upon it.

Also provided with a preferred embodiment of thepresent invention is a data processor which receives the electrical signals from the subpixels and compares the relative intensities of dififerent subpixels to establishthe depth of objects in the object field relative to a focalplane of the converging lens. In an alternativeembodiment of the present invention, the imagingelements are in the form of an array of pinhole cameras,each of which images fight from the entire surface of theconverging lens. However, the preferred embodiment usesa lenticular array. The lenticular array may havecylindrical lenses, but a two dimensional array usingspherical lenses arrayed in a rectangular or hexagonal lat-

5

10

15

20

25

30

35

40

45

50

55

60

65

2tice is preferred.

A diffuser positioned between an object being imagedand the imaging elements is also provided in thepreferred embodiment. The diffuser acts as a low passfilter to reduce aliasing. A field lens is also used, and i spositioned between the converging lens and theimaging elements. The field lens redirects the lightfrom the converging lens such that it appears to befocused from infinity. A relay lens may also be usedbetween the imaging elements and the photodetectorarray. The relay lens forms an image of the light imagedby the imaging elements onto a screen. The relay lensrelays the image from the screen to the photodetectorarray which can be set back from the imagmg elements.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an optical ranging apparatusaccording to the present invention.

FIG. 2 shows a CCD array and a preferred form ofpartitioning the array for the present invention.

FIGS. 3A-3C illustrate the imaging of a point objectby the present invention, when the object is in differentregions of the object field.

FIGS. 4A-4C demonstrate typical CCD sensoroutputs for the imaging tasks of FIGS. 3A-3C , respec-tively.

FIG. 5A is an illustration of a first subimage formedby the present invention.

FIG. 5B is an illustration of a second subimageformed by the present invention.

FIG. 5C is an illustration of the subimage of FIG. 5Aoverlayed on the image of FIG. 5B.

FIG. 6A shows the geometry for a main lens focusedon infinity as used with the present invention.

FIG. 6B shows the geometry for a main lens focusedat a distance D as used with the present invention.

FIG. 7 shows a preferred embodiment of the rangingapparatus of the present invention.

FIG. 8 an alternative embodiment of the rangingapparatus of FIG. 6 having a relay lens.

FIG. 9 is a flowchart showing the image processingsteps of the present invention.

DETAILED DESCRIPTION OF THEPREFERRED EMBODIMENTS

Shown in FIG. 1 is an optical ranging apparatus 1 0having a main converging lens 12 receiving light froman object 14 in an object field 1 6 , and focusing thereceived light toward a lenticular array 1 8 . The lenticu-lar array is an array of lenses (lenticules 20) alignedadjacent one another in am image field 22 of the mainlens 1 2 . Each lenticule 20 of the lenticular array 1 8forms an image from light received from the entire mainlens 12 as seen from the location of that particularlenticule 2 0 . Since each lenticule 20 is in a differentregion of image field 22 of the main lens 1 2 , eachlenticule images light from the entire surface of themain lens 12, but the light received by each lenticule i sfocused from a different portion of the object field 1 6 .The object field 16 of the main lens 12 is an area fromwhich light is received by the main lens 12 and directedtoward the lenticular array 1 8 . Similarly, the imagefield 22 is the area to which light received from theobject field 16 is passed by the main lens 12.

Positioned behind the lenticular array 18 is an arrayof charge coupled devices (CCD array 23) which func-

5 ,076 ,6873

tion as photodetectors. FIG. 2 depicts a portion of CCDarray 23 and demonstrates how the CCD array is arrangedfor use in the present embodiment. The CCD array 2 3consists of subpixels 24 each of which is a discretephotodetector and produces an output voltage proportion-al to the intensity of light incident upon it. The subpixels24 are grouped in macropixels, each macropixelcontaining a finite number of subpixels 2 4 . The macro-pixel regions of the CCD array are defined by the regionsof the CCD array upon which the lenticule 10 images areincident. The pixels of the CCD array which areencompassed by one lenticule image make up all thesubpixels of one macropixel. In FIG. 2 , the double-lineborders in the figure signify the separation betweendifferent macropixels. Therefore, in the position of array23 shown in FIG. 2 , each macropixel has nine differentsubpixels 2 4 . It will be understood that the double linesshown in FIG. 2 may not exist on the surface of the CCDarray 23 itself, but serve to illustrate how groups ofsubpixels 24 are preferably arranged for processing. Infact, the lenticular and CCD arrays may not be properlyaligned, but compensation for such misalignments may beprovided by the data processing.

Since each macropixel of the CCD array is positionedbehind one of the lenticules 20 of the lenticular array 1 8 ,each of the macropixels receives an image which containslight passed by the entire main lens 1 2 . However, thelight received by each macropixel is received from adifferent angular position relative to the optic axis of themain lens 12. In the embodiment of FIG. 1, the object 14is shown as being perpendicular to the optic axis of themain lens 1 2 , and in a plane of focus. Thus, any pointalong the surface of the object 14 is imaged by the mainlens to be in focus at the lenticular array 1 8 . Because ofthis focusing, the light directed from any point on theobject 14 is received by only one of the lenticules 20 ofthe lenticular array 1 8 , and there is no overlap of lightbetween lenticules. If, however, objects in the object field16 of the main lens 12 are displaced relative to the focalplane of the lens 12, these objects do not come to a focusat the lenticular array 1 8 , and light from the sameposition of the object field may be received by more thanone adjoining lenticule 20.

FIG. 3A shows main lens 12 having a point object 2 6in a focal plane of the lens 1 2 . As shown, the light fromthe object 26 comes to a focus at the lenticular array 1 8 ,and is therefore contained in only one lenticule 2 0 . Thebar graphs along the bottom of each of FIGS. 3A , 3 B ,and 3C show the relative intensities of light received byeach of three subpixels across one dimension of the somacropixels of the CCD array 2 3 . This dimension is inthe direction shown perpendicular to the optic axes of thelenses of the lenticular array 1 8 . As shown in FIG. 3 A ,the focusing of the light from the point object at thelenticular array 18 causes all the light received from theobject to enter the center lenticule shown in the figure.Within the center lenticule, the light is evenly dispenedacross all three horizontal subpixel locations of the centermacropixel.

The object 26 is shown again in FIG. 3B , but i sbeyond the plane of focus in the object field. This causesthe light from object 26 to come to a focus beforereaching the lenticular array 1 8 . Thus, the light divergespast the focal point and "spills over" into the lenticules20 adjacent the center lenticule. This "spillover" causesintensity distributions to be recorded in the macropixelsadjacent the macropixel of the center lenticule. As can beseen from the bar graph of FIG. 3B, each of the intensity

5

10

15

20

25

30

35

40

45

50

55

60

65

4distributions of the macropixels receiving light has anintensity peak in one of its subpixels. The locations ofthe intensity peaks in the macropixels of the array 2 3and the overall intensity distributions give informationas to the location of the object 26 in the object field ofthe main lens.

In FIG. 3C, the point object 26 is closer to the mainlens 12 than the focal plane of the lens 1 2 . This causesthe focus of the lens 12 in the image plane to be beyondthe lenticular array 1 8 , and the light from object 2 6does not come to a focus before reaching the array 1 8 .Like the FIG. 3B example, this causes overflow of lightinto the lenticules adjacent the center lenticule.However, since the converging light has not yet come toa focus upon reaching the lenticular array 1 8 , thedistribution of the light from different subpixels i sdifferent than the distribution resulting from the objectbeing farher from the main lens 12 than the plane offocus (as in FIG. 3B). Specifically, for objects near theoptic axis of the main lens 12, the intensity tends to begreatest at the subpixels closest to the central lenticulein FIG. 3C , and tends to be greatest at the subpixelsaway from the central lenticule in FIG. 3B . Further fromthe optic axis there is a shift in the distribution unless afield lens is used as described below.

Each of the subpixels across the dimension shown inFIGS. 3A-3C is labelled x, y, or z according to itslocation within a particular macropixel. Each of thesubpixels x, y, and z corresponds to the light receivedthrough a different portion of main lens 1 2 . Theparticular macropixel in which the subpixels x, y, and zare located determines the region of the object field fromwhich the light originates. Thus the number of lenticulesdetermines the spatial resolution of the received images.The number of subpixels per macropixel determines thenumber of different "views" through the main lens 12, orthe "motion resolution" or "parallax resolution" of theimage at the CCD anay. Each view through the lens i sfrom a different angle, each subpixel group correspon-ding to one angle.

To determine the position of an object in the objectfield of main lens 12, the subpixels of the CCD array arecompared. FIGS. 4A-4C show the bar graphs from FIGS.3A, 3C, respectively. Underneath the bar graph, in eachof the figures, the intensity output for each of thesubpixels x, y, and z is shown separately. If each of thesubpixels of one type (e.g. x, y, or z) is assembled into asubimage, the result is three separate subimages, eachrepresenting a view through a different portion of mainlens 1 2 . Thus, each subimage is a view of the eritireobject field as seen through a different "viewingaperture" of the main lens 1 2 . By selecting all the xsubpixels, a virtual aperture through the right side of thelens is selected. By selecting all the z subpixels, a vir-tual aperture through the left side of the lens is selected.

As shown in FIG. 4A, a point object in the focal planeof the main lens results in light being received in thesubpixel groups x, y, and z in equal portions and only inthe center macropixel. Thus, if one were to display thesubimages of that point in sequence as an animation, thepoint would remain stationary. In FIG. 4 B(corresponding to FIG. 3B), it is shown that theintensity peaks output by subpixels of the subimagesvary as the viewing aperture is changed from the xsubimage to the y subimage, and vary further as theaperture is changed from the y subimage to the zsubimage. Peaks to the left of center (in the orienta- tion shown in FIG. 4A) decrease as the viewing aperture

5 ,076 ,6875

is changed from x to y to z (i.e. from left to right).However, the peaks to the right of center increase as theviewing aperture changes from the x subimage to the ysubimage to the z subimage. Described in a differentmanner, the location of the highest peak in the subimagesshifts from left to right as the viewing apertures arechanged from x to y to z. If viewed sequentially, a pointsomewhat out of focus (spreading across pluralmacropixels) would move from left to right with changingangle of view. In FIG. 4C (corresponding to FIG. 3C ) ,left-of-center peaks increase in intensity when the view-ing aperture changes from left to right (x subimage to zsubimage). Similarly, the right-of center peaks decrease inintensity when the viewing aperture changes from left toright. Thus, the location of the highest peak in thesubimages shifts from right to left as the viewing aper-tures are changed from x to y to z.

From the above it can be seen that the changing of peakintensities is opposite depending on whether the object26 is behind or in front of the focal plane of the main lens1 2 . Therefore, from detecting how these intensitieschange across the surface of the CCD array, an indicationof the depth of the object in the object field can beobtained.

Within a single macropixel receiving an image from alenticule 2 0 , each subpixel 24 detects light from adifferent position of the main lens 1 2 . The subimages,when assembled, therefore appear as views of objects inthe object field from different angles. The parallaxbetween each of the subimages gives information on thedepth of objects in the object field. When the subimagesare viewed sequentially, the intensity differences appear asrelative position changes of objects depicted in theimages.

Because of the change in viewing apertures, the depth ofother objects in the object field may be determined by howmuch parallax exists relative to the focal plane of themain lens 12. The more an object appears to rearranging,change position from one image to the next, the farther i tis from the focal plane. Objects farther from the main lens12 than the focal plane appear to change position towardthe left when viewing the images in the order of leftsubpixel image through right subpixel image (x subimagethrough z subimage). Conversely, objects closer to themain lens 12 than the focal plane appear to therefore,change position toward the right if the same viewingorder is used.

The multiple image example discussed above i sillustrative of the capabilities of the present invention toestablish depths of objects in an object field. The numberof subpixels across one dimension of a macropixel givesthe "parallax resolution" of the apparatus (how manydifferent viewing angles are obtained). The number ofmacropixels across one dimension of the array determinesthe "spatial resolution" of the apparatus in that direction(the number of different components into which theimage plane is broken across that dimension). Althoughthe system may be used to generate a sequence of images for the purpose of presenting a scan of the objectfield, as suggested above, the data acquired from may alsobe used directly for the ranging of an object in the objectfield.

FIGS. 5A and 5B show two different views of an object27 representative of two different subimages. The view ofFIG. 5A might be an "x" subimage received by the systemof FIG. 1 , while the view of FIG. 5B might be a "z"subimage received by the system of FIG. 1 . The subimage of FIG. 5A is an image as seen through the

5

10

15

20

25

30

35

40

45

50

55

60

65

6right side of the main lens, while the image of FIG. 5 Bis an image as seen through the left side of the main lens.The focal plane of the main lens is somewhere near thecenter of the object 2 7 , and one view, relative to theother, therefore appears as a rotation of the object 2 7about an axis passing through the center of the object27. FIG. 5C shows the image of FIG. 5A overlayed onthe image of FIG. 5B , demonstrating the relativedisplacement changes from one view to another ofportions of the depicted object 2 7 . These displacementchanges are identified during processing of the images,and are used in establishing measurements of relativedepth.

One example of a technique for determining the i sdepth of an object in the object field of main lens 12 canbe described using FIG. 6A. Main lens 12 is focused oninfinity, and has a focal length F. An object in the objectfield of the lens 12 is at a distance d from a plane of thelens. An image of the object forms at a distance g, where:

1/d = 1/F – 1/g (1)

By similar triangles,

e/h = g/v, (2)

where v is the known aperture displacement determinedby the number of subpixels in a macropixel, h is themeasured image displacement of the object, and e equalsthe distance (g–F). Therefore,

(g – F)/h = g/v (3)

or

g – F = gh/v (4)

rearranging,

F = g – gh/v (5)

or,

F = g(1 – h/v) (6)

therefore,

1/g = (1/F)(1 – h/v) (7)

substituting into equation (1),

d = Fv/h (8)

Therefore, by knowing F and v, and meaxuring h, thedistance d can be calculated.

In the case of a lens focused on a plane at a distance D,a distance calculation is made using the diagram of FIG.6B . The lens of FIG. 6B has the followingcharacteristics:

F=focal lengthf=distance to lenticular arrayD=distance at which lens is focusedd=distance of object planeg=distance of image planee=distance of image plane beyond lenticular arrayv=viewing position of apenureh=displacement of object ray

By similar triangles,

5 ,076 ,6877

h/e = v/g (9)

Since e = g – f,

h/(g – f) = v/g (10)

or

g – f = gh/v (11)rearranging,

f = g – gh/v (12)

or

f = g(1 – h/v) (13)

therefore,

g = g/(1– h/v) (14)

or

1/g = (1/f)(1 – h/v) (15)

By the lens equation,

1/F = 1/g + 1/d (16)

Substituting eq. (15) in eq. (16),

1/d = 1/F – 1/f(1 – h/v) (17)

Using the relation

f = Dh/v (18)

eq. (17) may be put in terms of D, such that it reduces tothe following:

1/d = h/Fv – (1/D)(1 – h/v) (19)

or

1/d 5(h/v)(1/F – 1/D) + 1/D (20)

Thus, knowing F, v, and D, and measuring h, the distanced to a point on an object may be found.

In a preferred embodiment of the invention, lenticulararray is used which has a two dimensional array of lensesto provide image data in each of two different dimensions.A CCD sensor such as that of FIG. 2 is used, and the imagefrom one of the lenses of the lenticular array is distributedacross all of the subpixels of a particular macropixel.Thus, each macropixel is given a number of different visu-al apertures of the lens in each of the two dimensions ofthe array. In the array shown in FIG 2, all of the subpixelslabelled xx are assembled into one subimage, all thesubpixels labelled xy are arranged into a subimage, etc. Inthis manner, nine separate subimages are assembled, eachof which consists of light having passed through adifferent region of main lens 12.

A preferred embodiment, such as mentioned above, i sshown graphicafiy in FlG. 7 . Main lens 12 is receivinglight from an object 30 in the object field of the lens 1 2 ,and passing that light through to the image field of thelens 12. Located in the image field between the main lens12 and lenticular array 32 is a field lens 34 which adjuststhe focused light from the main lens such that it appearsto have been focused from infinity. The absence of such a field lens 34 causes the light received by the CCD

5

10

15

20

25

30

35

40

45

50

55

60

65

8array 23 of FIGS. 3A-3C to be skewed across the surfaceof the subpixels in each macropixel. Therefore the centerof the lenticule image (and therefore the center of themacropixel) are displaced relative to the CCD array 2 3 .Thus, subpixels on one side of a macropixel may beoverly illuminated due to the light being converging ordiverging. The effects of such skewing are predictableand may be compensated for with processing tech-niques. However, the field lens removes the need for the additional processing by providing an optical solu-tion.

The lenticular array 32 of FIG. 7 is a two dimensionallens array of spherical lenses, and therefore allows twodimensional motion analysis from the acquiredsubimages. In this embodiment, the lenses of the arrayare arranged in a rectangular lattice. However, ahexagonal or other type of lattice arrangement may beused as an alternative. The CCD array 33 is somewhatlike the CCD array 23 of FIG. 1 , and serves as an inputto data processor 3 6 , shown in block diagram form.Also included with the embodiment of FIG. 7 is a weakdiffuser element 38 which diffuses the light from theobject prior to its reaching the lenticular array 3 2 .Without the diffuser, problems with aliasing might arise.Since the spatial resolution of the apparatus i sdetermined by the number of lenticules, so is thesampling "frequency". If light from depth changes in theobject field falls within an area which is smaller than thearea of the object field from which light is received by asingle lenticule, accurate resolution of the depth changecan not be accomplished. In other words, not enoughsamples exist to properly resolve the depth change andaliasing results (i.e. no useful information is providedabout depth in that region).

Diffuser 38 however, helps control aliasing bydispersing the light received from the object field,thereby acting as a "low pass filter". Some commonexamples of diffusers are glass elements with randomsurface variations or diffracting optics. A weak, len-ticular array may also be used. If a one dimensionallenticular array 32 (cylindrical lenses) is used, a onedimensional diffuser 38 may also be used, since onlyone dimension of depth changes need be resolved. In anyevent, the diffuser should ideally be in the aperture plane,and make a point spread function (PSF) of about 1lenticule width.

As mentioned above, the lenticular array may beselected to be one dimensional (cylindrical) or twodimensional. A one dimensional array has betterresolution, but only one direction of parallax. A twodimensional array may be used which consists ofspherical lenses instead of cylindrical lenses. Such anarray is preferably arranged in a rectangular or hexagonallattice. Such two dimensional screens are commerciallyavailable, or may be constructed by crossing two cylin-drical screens. On any lenticular array, a filter arraymeans better spatial resolution, but fewer viewpoints(subimages).

One alternative embodiment of the present inventionuses a relay lens to reimage the lenticular image onto adistant sensor array. Such an arrangement is shown inFIG. 8. Object 30 is shown being imaged by main lens12 through diffuser 38. Light from the main lens 12 i spassed through field lens 34 and onto lenticular array3 2 . However, instead of imaging immediately onto aCCD array, the image formed on the rear surface of thelenticular array is viewed by relay lens 40 which focus-es the lenticular image onto photodetector array 42. An

5 ,076 ,6879

optional piece of ground glass may be provided on theback of the lenticular array in on which a clear image maybe formed. The combinition of relay lens and photo-detector array 42 might be in the form of a televisioncamera assembly. This system allows more flexibility inhow the images from the lenticular array are dispersedthrough the photodetector array 4 2 . However, additionaloptics are required and the relative positioning of the ele-ments make the system less stable.

The present invention is unique in that it records asingle monocular image and uses that single image tocalcuLate the depth of objects in the object field. Once theimage is detected by the CCD array, the processing of theinformation is performed by a data processor. The dataproccessor uses image processing techniques, many ofwhich are standard in the art of image processing. Thefollowing is a list of steps which may be followed in apreferred embodiment of the present invention to accom-plish the accurate processing of the acquired image infor-mation.

PROCESSING OF THE IMAGE DATA

A number of different processing methods may be usedto extract depth information from the recorded image data.The preferred embodiment uses a sequence of general stepswhich are shown in the blocks of the flowchart of FIG. 9 .First the image received by the CCD array is digitized(block 50 ). The intensity of the light received by eachpixel of the CCD array results in the generation of ananalog voltage which is converted to a digital code by ananalog-to-digital (A/D) converter. The number of quantiza-tion levels available with the A/D converter depends onhow many bits are used to represent each pixel. This inturn determines the number of distinct gray leve1s whichare recognized by the data processor.

After digitization the subimages corresponding todifferent viewpoints are extracted (block 52 ). For thisstep the image is multiplied by a weighting mask to selectindividual subimages. Preferably the lenticuiar array i saligned with the CCD array such that the image from eachlenticule is evenly distributed across one macropixeltherefore encompassing an even number of subpixels. Insuch a case, a mask is defined which for each macropixelmultiplies all the subpixels by zero except for thesubpixel corresponding to the desired subimage. For eachdifferent subimage a different subpixel for each micro-pixel is selected. Thus for each position the mask blocksout all but one set of subpixels, such that each subimageconsists of one subpixel from each macropixel. Eachsubimage theretore represents a different virtual apertureposition (viewing light from a different portion of thelens).

In some cases, the lenticular array may not be exactlyaligned with the CCD array and a weighting and averagingtechnique is used to generate the diffirent subimages. Inthis case, it is not possible to generate the subimage byselecting individual pixel values because the effectiveposition of the desired pixel may lie at a fractiona1position between CCD e1ements. Therefore, it is neces-sary to estimate the value of the virtual pixel at a fraction-al position. This can be achieved by taking a weightedaverage of the observed pixels surrounding the virtualpixel and using wel1 known interpolation techniques suchas bilinear interpolation.

Once the desired subimages are assembled, eachsubimage is enhanced through well known imageprocessing techniques (block 54). In the preferred em-

5

10

15

20

25

30

35

40

45

50

55

60

65

1 0bodiment this enhancing includes preprocessing thesubimages to remove low-spatial frequency variationthat may result due to non-uniformity in the opticalsystem or sensor sensitivity. This may be accomplishedwith a broadband high-pass filter which for each pixel of a subimage subtracts a local average of imageintensity taken from surrounding pixels. Alternatively, alocal automatic gain control process may be used which divides regions of the subimage by a local aver-age.

Diplacement analysis is then performed (block 56) tomeasure object displacement between image pairs. Thisanalysis uses techniques typically used for motionanalysis or stereo analysis. One possible technique is toperform a least-squares match as commonly understoodin the art of image processing. Such a least-squaresmatching technique is demonstrated by Lucas and Kanadein An Iterative Image Registration Technique with anApplication to Stereo Vision, Proceedings of ImageUnderstanding Workshop, April 1981, pp. 121-130.Using this technique the displacement estimate is givenby:

∑(IxIµ)/∑[(Ix)2].

where is is the spatial derivative of the image intensityin the x direction and Iµ is the derivative of intensity asthe viewing position is moved in the x direction of theaperture. Summation is taken over a small spatial patchwithin the image. Larger patches lead to estimates withreduced noise but tend to smooth out sharp changes indisplacement. The output using this technique is animage with a displacement estimatc at each position.Following the above least-squares method a confidencefactor of ∑[(Ix)

2] may also be assigned to each local esti-mate.

In processing the image data in two dimensions thepreviously mentioned least squares technique dcscribedfor subimages in one dimension. can similarly be usedfor subimage data in a second dimension. Once this i scomplete the final confidence-weighted estimates arecomputed (block 58 ). Displacement estimates areextracted for a11 adjacent image pairs. On the assumtionthat the magnitude of the displacements in one dimen-sion should equal the magnitude of the displacements inthe second dimension, all of the displacement estimatesin both dimensions are combined into a single displace-ment estimate. This is accomplished by weighting eachdisplacement image by its correponding confidenceimage.

Once a final displacement estimate is acquired thedisplacement estimate is converted to a depth estimate(block 63) using the geometry shown in FIGS. 6A and6B . This depth estimate output is then applied to theparticular application for which the invention is beingused.

One processing consideration which arises inpracticing the present invention is the alignment of thedesired macropixels with the actual subpixels of thephotodetector array. In the aforementioned case of thelenticular array not lining up evenly with the subpixelsof the CCD array it is necessary to generate a weightingmask to be used during processing. The following is apreferred method for generating such a mask.

A uniformly illuminated white card is first placed infront of the main lens in the object field. The aperture ofthe main lens is then reduced so that the macro- pixels include only a few sensor subpixels. The image is

5 ,076 ,6871 1

digitized and the locations of the peaks in the macro-pixels are determined. A weighting mask is then defined inwhich the smooth weighting functions underneath eachmacropixel are centered on the correct center of themacropixel.

While the invention has been particularly shown anddescribed with reference to a preferred embodimentthereof, it will be understood by those skilled in the artthat various changes in form and details may be madetherein without departing from the spirit and scope of theinvention as defined by the appended claims. In particular,the specific processing techniques may vary depending onthe application of the invention. The least-squares methoddescribed above is but one image processing techniqueunderstood by those skilled in the art. Furthermore, thelenticular array may be replaced by another form ofimaging device such as a pinhole array. The photodetectorarray may also be a type of photodector other than a CCDdevice.

The data which is gathered by the optical system may beprocessed for various purposes. For example, the data canbe applied to a three dimensional display. The subimagesmay be viewed individually or sequentially to provide twodimensional projections from different angles. The rangedata obtained from processing the subimases may be usedto identify objects in the object field. A system using sucha ranging technique may also be used for such applicationsas setting off security alarms or for automatic navigation(obstacle avoidance).

I claim:1. An optical depth resolver comprising:a converging lens receiving light from objects in an

object field and directing the received light to animage field of the converging lens;

a plurality of imaging elements distributed about theimage field of the converging lens in the image fieldof the lens such that each imaging element receiveslight from the converging lens and forms an imagefrom the light it receives;

a photodetector array divided into a plurality ofmacropixels each of which is made up of a plurality ofsubpixels, each macropixe1 receiving an imageformed by one of the imaging elements such that eachsubpixe1 of a macropirxe1 receives light from oneportion of the lens as imaged by one of the imagingelements, each subixel generating an electrical signalindicative of the intensity of 1ight incident upon it;and

a data processor receiving the electrica1 signals fromthe subpixels and processing the signals fromdifferent subpixels, the data processor identifyingsubimages of subpixels having 1ike spatialpositions within different macropixels and perfor-ming, across the image field, a plurality of localdisplacement estimates of image displacementbetween subimages to create a range image of depthsof objects across the object field.

2 . An optical depth resolver according to claim 1wherein the plurality of imaging elements comprises alenticular array.

3 . An optical depth resolver according to claim 2wherein the lenticular array comprises an array of cylin-drical lenses.

4 . An optical depth resolver according to claim 2wherein the lenticular array comprises an array ofspherical lenses.

5 . An optical depth resolver according to claim 4wherein the spherica1 lenses are arrayed in a rectan-

5

10

15

20

25

30

35

40

45

50

55

60

65

1 2gular lattice.

6 . An optical depth resolver according to claim 1wherein each of the imaging elements receives lightfrom the entire surface of the converging lens.

7 . An optical depth resolver accordirg to claim 1wherein the photodetector array is a CCD array.

8 . An optical depth resolver according to claim lwherein each subimage is represensative of lightpassing through a different region of the converginglens.

9 . An optical depth resolver according to claim 8wherein the data processcr assembles the subimages andcompares the parallax from one subimage to another indetermining depth of objects in the object field.

1 0 . An optical depth resolver according to claim 8wherein the shift of intensity peaks from one subimageto another is detected by the data processor and useddetermining depth of objects in the object field

1 1 . An optical depth resolver acccording to claim lwherein there is one macropixel for each imagegenerated by an imaging element.

1 2 . An optical depth resolver according to claim 1wherein the data procossor runs a least squaresprocessing method in making determinations of depthof objects in the object field.

1 3 . An optical depth resolver according to claim 1further comprising a weak diffuser which diffuses thelight from objects in the object field prior to the lightbeing received by the imaging elements.

14. An optictal depth resolver according to claim 1 3wherein the diffuser is a piece of glass with random sur-face variations.

1 5 . An optical depth resolver according to claim 1further comprising a field lens positioned between theconverging lens and the imaging elements. and redirec-ting the light from the converging lens such that thelight from the converging lens appears to have beenfocused from infinity.

1 6 . An optical depth resolver according to claim lfurther comprising a relay lens between the imagingelements and the photodetector anay the relay lensredirecting the images from the imaging elements to thephotodetector array.

17. An optica1 depth resolver according to claim 1 6wherein the relay lens refocuses the images generatedby the imaging elements.

18. An optical depth resolver according to claim 1 7wherein the distribution of the images from the imagingelements across the surface of the photodetector array i scontrolled by the focusing of the relay lens.

19. An optical depth resolver according to claim 1 6wherein the relay lens is part of a television camera.

20. An optical depth resolver as claimed in claim 1wherein the data processor estimates time based onderivatives of local intensity between subimages forpatches of macropixels.

21. An optical depth resolver as claimed in claim 2 0wherein the estimates are based on derivatives in twodimensions.

22. An optical depth resolver as claimed in claim 1wherein the data processor applies a weighting mask tothe electrical signals generated by the photodetectorarray in order to adjust the signals for misalignment ofthe imaging elements with the photodetector array.

23. An optical depth resolver as claimed in claim 1wherein the data processor processes subpixels of atleast three subimages in performing each local displace-ment estimate.

5 ,076 ,6871 3

24. An optical depth resolver comprising:a converging lens receiving light from objects in an

object field and directing the received light to animage field of the converging lens;

a lenticular array having a two dimensional array oflenticules distributed about the image field of theconverging lens such that each lenticule receiveslight directed from a different portion of the objectfield and receives the directed light at a differentangle relative to an optic axis of the main lens, each lenticule forming an image from the light i treceives;

a photodetector array divided into a plurality ofmacropixels each of which is made up of a plurality ofsubpixels, each macropixel receiving the imageformed by one of the lenticules such that eachsubpixel of a macropixel receives light from oneportion of the converging lens as imaged by onelenticule each subpixel generating an electricalsignal indicative of the intensity of light incidentupon it;

a data processor receiving the electrical signals fromthe subpixels and processing the subpixel signals todetect variations in intensity of light directed from aplurality of locations in the object field relative tothe portion of the converging lens through which thelight passes the data processor identifying subimagesof subpixels having like spatial positions withindifferent macropixels and performing, across theimage field, estimates of depths of objects across theobject field based on local derivatives of intensitybetween subimages to create a range image.

2 5 . An optical depth resolver according to claim 2 4wherein the photodetector array is a CCD array.

2 6 . An optical depth resolver according to claim 2 4further comprising a weak diffuser diffusing light betweenthe object field and the lenticular array.

2 7 . An optical depth resolver according to claim 2 4further comprising a field lens between the converginglens and the lenticular array, the field lens redirecting thelight from the converging lens such that it appears tohave been focused from infinity.

2 8 . An optical depth resolver according to claim 2 4further comprising a relay lens between the lenticulararray and the photodetector array, the relay lens redirec-ting the images from the lenticular array to the photo-detector array.

2 9 . An optical depth resolver as claimed in claim 2 4wherein the estimates are based on derivatives in twodimensions.

3 0 . An optical depth resolver as claimed in claim 2 4wherein the data processor applies a weighting mask to the electrical signals degenerated by the photodetec- tor array in order to adjust the signals for misalign- ment of the lenticular array with the photodetector array.

3 1 . An optical depth resolver as claimed in claim 2 4wherein the data processor processes at least threesubimages to perform each estimate.

3 2 . A method of making depth measurements,comprising:

providing a converging lens receiving light fromobjects in an object field and directing the receivedlight to an image field of the converging lens;

receiving said light from the converging lens with a plurality of imaging elements distributed about the image field of the converging lens, each imaging

5

10

15

20

25

30

35

40

45

50

55

60

65

1 4element forming an image from light it receives;

receiving the images formed by the imaging elementswith a photodetector array divided into a plurality ofmacropixels each of which is made up of a pluralityof subpixels, each macropixel receiving an imageformed by one of the imaging elements such thateach subpixel of a macropixel receives light fromone portion of the lens as imaged by one of theimaging elements each subpixel generating anelectrical signal indicative of the intensity of lightincident upon it; and

receiving the electrical signals from the subpixelswith a data processor and processing the signal fromdifferent subpixels to create a range image ofestimates of depths of objects across the objectfield.

3 3 . A method according to claim 32 whereinreceiving said light from the converging lens with aplurality of imaging elements comprises receiving saidlight from the converging lens with a lenticular array.

3 4 . A method according to claim 32 whereinreceiving said light from the converging lens with aplurality of imaging elements comprises receiving saidlight with the imaging elements such that each imagingelement receives light from the entire surface of theconverging lens.

3 5 . A method according to claim 32 whereinreceiving the images formed by the imaging elementsfurther comprises arranging the subpixels of eachmacropixel of the photodetector array in a similarpositional distribution such that a group of subpixels,each from a similar position in a different macropixel,together form a subimage representative of light passingthrough one region of the converging lens.

3 6 . A method according to claim 35 whereinprocessing the electrical signals from different subpixelsfurher comprises assembling said subimages andcomparing the parallax from one subimage to another indetermining depth of objects in the object field.

37. A method according to claim 35 wherein proces-sing the electrical signals from different subpixels furhercomprises detecting the shift of intensity peaks fromone subimage to another in determining depth of objectin the object field.

3 8 . A method according to claim 32 furthercomprising diffusing said light from object in the objectfield with a weak diffuser.

3 9 . A method according to claim 32 furthercomprising redirecting the light from the converginglens to the imaging elements with a field lens.

4 0 . A method according to claim 32 furthercomprising redirecting the images from the imagingelements to the photodetector array with a relay lens.

41. A method as claimed in claim 32 wherein the dataprocessor estimates are based on derivatives of localintensity between subimages for patches of macropixels

4 2 . A method as claimed in claim 41 wherein theestimates are based on derivatives in two dimensions.

43. A method as claimed in claim 32 wherein the dataprocessor applies a weighting mask to the electricalsignals generated by the photodetector array in order toadjust the signals for misalignment of the lenticulararray with the photodetector array.

44. A method as claimed in claim 32 wherein the dataprocessor processes at least three subimages to deter-mine each depth of an object.

S S S S S