3
A microscale camera using direct Fourier-domain scene capture Patrick Robert Gill, 1,2, * Changhyuk Lee, 1 Dhon-Gue Lee, 1 Albert Wang, 1 and Alyosha Molnar 1 1 School of Electrical and Computer Engineering, Cornell University, 223 Phillips Hall, Ithaca, New York 14853, USA 2 Department of Psychology, Cornell University, 211 Uris Hall, Ithaca, New York 14853, USA *Corresponding author: [email protected] Received May 13, 2011; revised June 23, 2011; accepted July 3, 2011; posted July 7, 2011 (Doc. ID 147442); published August 1, 2011 We demonstrate a chip-scale (<1 mm 2 ) sensor, the Planar Fourier Capture Array (PFCA), capable of imaging the far field without any off-chip optics. The PFCA consists of an array of angle-sensitive pixels manufactured in a standard semiconductor process, each of which reports one component of a spatial two-dimensional (2D) Fourier transform of the local light field. Thus, the sensor directly captures 2D Fourier transforms of scenes. The effective resolution of our prototype is approximately 400 pixels. © 2011 Optical Society of America OCIS codes: 040.1240, 050.1970, 070.6760, 110.5200, 230.4000, 280.4788. Electronic image capture permits digital photography, as well as a variety of automatic sensor [1,2] and robotic [3,4] applications. Far-field imaging is traditionally ac- complished through focusing optics (lenses or mirrors), which map incoming light based on its incident angle to a sensor plane made up of photosensitive pixels. In gener- al, image resolution is proportional to the 2 3 power of instrument volume (see Fig. 1), but technology relying on focusing permits only a limited range of this trade- off. Moreover, focusing optics require precision manufac- turing and occupy significant volume, limiting the degree to which even low resolution imaging systems can be miniaturized. To minimize cost and size, an ideal micro- camera would be manufacturable in a standard planar semiconductor process and be capable of imaging with- out any off-chip optics. Here we introduce such a device: the Planar Fourier Capture Array (PFCA). Our approach is to recover far- field information from an array of diverse angle-sensitive pixels (ASPs) [5,6]. ASPs are composed of a photodiode under two metal gratings formed by interconnected metal layers intrinsic to modern planar semiconductor processes (Fig. 2). The top grating generates an inter- ference pattern at the depth of the second grating; the second grating blocks or passes light depending on the relative phase between the interference pattern and the second grating, which is in turn dependent on the in- cident angle of incoming light. Individually, ASPs have a light sensitivity that varies sinusoidally with incident angle as R ¼ I 0 ð1 m cosðbθ þ αÞÞF ðθÞð1 þ ηÞ; ð1Þ where R is the readout of the ASP, I 0 is proportional to the light flux at the ASP, θ is the incident angle along the sensitive axis, b is the angular sensitivity [designed to range from 7 to 39see Eq. (2)], m is the modulation depth of the ASP [see Eq. (3)], α is a designable phase offset, F ðθÞ is a slowly varying aperture function, and η is multiplicative noise. Both the grating pitch p and the depth between the gratings d influence b, as follows: b ¼ 2π d pn ; ð2Þ where n is the refractive index of the medium, in this case SiO 2 . Modulation depth m is at a maximum when a self-image of the top grating is formed at the depth of the second grating; optimal depths d are therefore in- teger multiples of half the Talbot depth [7,8]: m maximal when d ¼ a p 2 λ ; a I: ð3Þ Additionally, the angle-sensitive axis is set by the orien- tation of the metal gratings. This oriented, sinusoidal response is similar to the individual components of a two-dimensional (2D) Fourier Transform, which is com- monly used in image analysis [9]. In general, the complete 2D Fourier transform of a far-field image can be computed by taking inner pro- ducts of the image with oriented sinusoids of different periodicity (Fig. 3). Therefore, an array of appropriately chosen ASPs can report Fourier-complete information about the far field. To accomplish Fourier completeness, we selected an ensemble of 18 combinations of p and d available in a 180 nm complementary metaloxidesemiconductor (CMOS) process [10] chosen to yield a collection of closely spaced values of b. Given minimum p constraints and a discrete number of manufacturable d values, there are a finite number of possible devices that satisfy Eq. (3). The solid circles in Fig. 4 plot b and p for Fig. 1. (Color online) Trade-off between imager size and image resolution. The PFCA fills in a gap left in the available resolutionsize trade-off between a naked single photodiode and the smallest commercial focusing cameras [14]. August 1, 2011 / Vol. 36, No. 15 / OPTICS LETTERS 2949 0146-9592/11/152949-03$15.00/0 © 2011 Optical Society of America

A microscale camera using direct Fourier-domain scene capture

  • Upload
    alyosha

  • View
    223

  • Download
    6

Embed Size (px)

Citation preview

Page 1: A microscale camera using direct Fourier-domain scene capture

A microscale camera using directFourier-domain scene capture

Patrick Robert Gill,1,2,* Changhyuk Lee,1 Dhon-Gue Lee,1 Albert Wang,1 and Alyosha Molnar11School of Electrical and Computer Engineering, Cornell University, 223 Phillips Hall, Ithaca, New York 14853, USA

2Department of Psychology, Cornell University, 211 Uris Hall, Ithaca, New York 14853, USA*Corresponding author: [email protected]

Received May 13, 2011; revised June 23, 2011; accepted July 3, 2011;posted July 7, 2011 (Doc. ID 147442); published August 1, 2011

We demonstrate a chip-scale (<1mm2) sensor, the Planar Fourier Capture Array (PFCA), capable of imaging the farfield without any off-chip optics. The PFCA consists of an array of angle-sensitive pixels manufactured in a standardsemiconductor process, each of which reports one component of a spatial two-dimensional (2D) Fourier transform ofthe local light field. Thus, the sensor directly captures 2D Fourier transforms of scenes. The effective resolution ofour prototype is approximately 400 pixels. © 2011 Optical Society of AmericaOCIS codes: 040.1240, 050.1970, 070.6760, 110.5200, 230.4000, 280.4788.

Electronic image capture permits digital photography,as well as a variety of automatic sensor [1,2] and robotic[3,4] applications. Far-field imaging is traditionally ac-complished through focusing optics (lenses or mirrors),which map incoming light based on its incident angle to asensor plane made up of photosensitive pixels. In gener-al, image resolution is proportional to the 2

3 power ofinstrument volume (see Fig. 1), but technology relyingon focusing permits only a limited range of this trade-off. Moreover, focusing optics require precision manufac-turing and occupy significant volume, limiting the degreeto which even low resolution imaging systems can beminiaturized. To minimize cost and size, an ideal micro-camera would be manufacturable in a standard planarsemiconductor process and be capable of imaging with-out any off-chip optics.Here we introduce such a device: the Planar Fourier

Capture Array (PFCA). Our approach is to recover far-field information from an array of diverse angle-sensitivepixels (ASPs) [5,6]. ASPs are composed of a photodiodeunder two metal gratings formed by interconnectedmetal layers intrinsic to modern planar semiconductorprocesses (Fig. 2). The top grating generates an inter-ference pattern at the depth of the second grating; thesecond grating blocks or passes light depending on therelative phase between the interference pattern andthe second grating, which is in turn dependent on the in-cident angle of incoming light.Individually, ASPs have a light sensitivity that varies

sinusoidally with incident angle as

R ¼ I0ð1 −m cosðbθ þ αÞÞFðθÞð1þ ηÞ; ð1Þ

where R is the readout of the ASP, I0 is proportionalto the light flux at the ASP, θ is the incident angle alongthe sensitive axis, b is the angular sensitivity [designed torange from 7 to 39—see Eq. (2)], m is the modulationdepth of the ASP [see Eq. (3)], α is a designable phaseoffset, FðθÞ is a slowly varying aperture function, andη is multiplicative noise.Both the grating pitch p and the depth between the

gratings d influence b, as follows:

b ¼ 2π dpn

; ð2Þ

where n is the refractive index of the medium, in thiscase SiO2. Modulation depth m is at a maximum whena self-image of the top grating is formed at the depthof the second grating; optimal depths d are therefore in-teger multiples of half the Talbot depth [7,8]:

m maximal when d ¼ a

�p2

λ

�; a ∈ I: ð3Þ

Additionally, the angle-sensitive axis is set by the orien-tation of the metal gratings. This oriented, sinusoidalresponse is similar to the individual components of atwo-dimensional (2D) Fourier Transform, which is com-monly used in image analysis [9].

In general, the complete 2D Fourier transform of afar-field image can be computed by taking inner pro-ducts of the image with oriented sinusoids of differentperiodicity (Fig. 3). Therefore, an array of appropriatelychosen ASPs can report Fourier-complete informationabout the far field. To accomplish Fourier completeness,we selected an ensemble of 18 combinations of p andd available in a 180 nm complementary metal–oxide–semiconductor (CMOS) process [10] chosen to yield acollection of closely spaced values of b. Given minimump constraints and a discrete number of manufacturable dvalues, there are a finite number of possible devices thatsatisfy Eq. (3). The solid circles in Fig. 4 plot b and p for

Fig. 1. (Color online) Trade-off between imager size andimage resolution. The PFCA fills in a gap left in the availableresolution–size trade-off between a naked single photodiodeand the smallest commercial focusing cameras [14].

August 1, 2011 / Vol. 36, No. 15 / OPTICS LETTERS 2949

0146-9592/11/152949-03$15.00/0 © 2011 Optical Society of America

Page 2: A microscale camera using direct Fourier-domain scene capture

devices satisfying Eq. (3) exactly for λ ¼ 520 nm (invacuum) light. An array using only devices optimal for520 nm light would leave large gaps in the tiling of b.To better span these gaps, our ensemble of devices in-cludes designs optimal for different wavelengths, butall fairly close to optimal devices for 520 nm light (seered open circles of Fig. 4). We laid out 10 μm × 10μm de-vices in concentric rings around four central devicessimilar to those reported in [11] to capture low frequencyinformation. Each design constitutes one ring of ASPs,with larger rings containing devices of higher b (Fig. 5).The greater number of ASPs for devices with higher bpermits more grating orientations. This is desirable, sincethe number of independent Fourier components a givendesign must observe is proportional to b. Devices oppo-site each other have α values [see Eq. (1)] 90° apart tocapture magnitude and phase information of each Four-ier component. We built two complementary arrays(with α values differing by 180°; otherwise identical) of38 × 38 unique ASPs (2,888 sensors total) in an unmodi-fied 180 nm CMOS process (Fig. 5).

Fourier completeness relies on our measurements til-ing Fourier space up to our Nyquist limit. As the range ofallowed incident angles increases, so does the frequencyresolution needed to cover all Fourier space. Specifically,the relationship between the maximum allowable eccen-tricity of incident light (relative to the normal to thedevice) h and the maximum difference in b between con-secutive designs is

h ¼ 180°ffiffiffi2

pΔb

: ð4Þ

The largest gap between implemented devices (open cir-cles of Fig. 4) gives a Δb of 2.62, such that the ensembletransfer functions of either of the 1,444-sensor arraysyield overcomplete coverage of Fourier space for imageswith h < 48:6°.

To characterize the array, we presented a series of ran-dom calibration images [Fig. 6(a)] to the sensor using asquare CRT area 20 cm on a side, 22:86 cm from the PFCA

Photodiode

Response vs. Angle

Fig. 2. (Color online) Angle-sensitive pixels. Incident lightinteracting with a grating composed of an upper metal layerproduces an interference pattern at the depth of the second-layer grating. Light is either passed or blocked depending onthe alignment of the interference pattern and the second layer.This alignment, in turn, is sensitive to changes in incident angle;the net effect is that the light passed by an ASP depends sinu-soidally on incident angle.

Fig. 3. Decomposition of natural images into Fourier compo-nents. (a) All images can be expressed as a sum of 2D Fourierbasis functions [e.g., (b)] by taking the sum over all values in(c) the basis-scaled image.

5 10 15 20 25 30 35 400.5

1

1.5Optimal for 520nm lightConfigurations Used

Gra

ting

Pitc

h p

(µm

)

Effective b

Fig. 4. (Color online) Selecting devices for the PFCA. Filledcircles indicate manufacturable devices with maximal mfor 520nm light; open circles indicate the suite of devices wemanufactured.

570 Microns

Fig. 5. (Color online) Manufactured PFCA. Concentric ringsof ASPs with increasingly high sinusoidal periodicity yield acomplete Fourier description of the light intensity from thefar field. Slowly varying orientation is evident from the enlargedsection, where schematics show different metal layers in differ-ent colors.

Fig. 6. (Color online) PFCA Calibration. Transfer functions ofeach pixel are found by presenting (a) calibration images on aCRT screen to (b) the array and performing reverse correlationbetween the observed photocurrent of each sensor and the im-age presented. (c) The kernels of three ASPs are shown; theseresemble Fourier components.

2950 OPTICS LETTERS / Vol. 36, No. 15 / August 1, 2011

Page 3: A microscale camera using direct Fourier-domain scene capture

[12] for an h of 31:7° at the square’s corners. With thismaximum eccentricity well under the limit of Fouriercompleteness [48:6°; see Eq. (4)], the PFCA’s outputsare somewhat redundant. We then used linear systemidentification tools [13] to reconstruct the kernel of eachASP [Fig. 6(c)]. Afterward, we presented various testimages on the CRT [Fig. 7(a)], captured the ASP re-sponses with an accumulation time of 16:7ms, and suc-cessfully reconstructed the images [Fig. 7(b)] up to theNyquist limit of our sensor, set by the highest-b designin our array ðbmax ¼ 39Þ. Our effective resolution onthe active square was approximately 20 × 20 pixels; thenumber of resolvable pixels scales with b2max.We have demonstrated a PFCA: an ASP array that re-

lates complete Fourier information (up to a maximumspatial frequency) about the far field without usingfocusing optics. The device is manufactured in an unmo-dified semiconductor process and requires no externaloptics or alignment. Its construction cost and resolu-tion specifications fill gap between the smallest miniatur-ized cameras and single photodiodes (Fig. 1), makingit a suitable choice for a large range of cost- andsize-sensitive applications that cannot be served withfocusing optical systems.

We would like to thank the Defense Advanced Re-search Projects Agency (DARPA), which supported thisresearch via a Young Faculty Award to A. Molnar, and theNational Institutes of Health (NIH), who helped fund thiswork under R21 grant EB 009841-01.

References and Notes

1. B. Batchelor, D. Hill, and H. Hodgson, Automated Visual

Inspection (Elsevier Science, 1985).2. T. Lillesand, R. Kiefer, and J. Chipman, Remote Sensing and

Image Interpretation, 5th ed. (Wiley, 2004).3. C. Urmson, J. Anhalt, D. Bagnell, C. Baker, R. Bittner,

M. Clark, J. Dolan, D. Duggins, T. Galatali, C. Geyer,M. Gittleman, S. Harbaugh, M. Hebert, T. M. Howard,S. Kolski, A. Kelly, M. Likhachev, M. McNaughton, N. Miller,K. Peterson, B. Pilnick, R. Rajkumar, P. Rybski, B. Salesky,Y.-W. Seo, S. Singh, J. Snider, A. Stentz, W. R. Whittaker, Z.Wolkowicki, J. Ziglar, H. Bae, T. Brown, D. Demitrish, B.Litkouhi, J. Nickolaou, V. Sadekar, W. Zhang, J. Struble,M. Taylor, M. Darms, and D. Ferguson, J. Field Robot.25, 425 (2008).

4. N. Bergström, J. Bohg, and D. Kragic, in Proceedings of

the 7th International Conference on Computer Vision

Systems, Vol. 5815 of Lecture Notes in Computer Science(Springer-Verlag, 2009), pp. 245–254.

5. A. Wang, P. Gill, and A. Molnar, Appl. Opt. 48, 5897 (2009).6. A. Wang, P. R. Gill, and A. Molnar, in Proceedings of IEEE

Sensors, 2010 (IEEE, 2010), pp. 1706–1709.7. H. Talbot, Phil. Mag. Series 3 9, 401 (1836).8. S. Teng, Y. Tan, and C. Cheng, J. Opt. Soc. Am. A 25, 2945

(2008).9. J. Escofet, M. Millán, and M. Ralló, Appl. Opt. 40, 6170

(2001).10. p is constrained by the minimum manufacturable linewidth

of the metal, and dmust correspond to one of four availableinter-layer depths of the five metal layers suitable for mak-ing gratings.

11. C. Koch, J. Oehm, J. Emde, and W. Budde, IEEE J. Solid-State Circuits 43, 1588 (2008).

12. As 23 cm ≫ 570 μm (the PFCA’s size), images presentedare in the far-field regime where the light field at each pointin the PFCA is essentially identical.

13. N. Wiener, Extrapolation, Interpolation, and Smoothing of

Stationary Time Series: with Engineering Applications

(MIT Press, 1949).14. L. Paulson, Computer 43, 19 (2010).

Fig. 7. Image reconstructions. Using the basis functions ob-tained in the calibration phase (Fig. 6), (a) the image presentedwas (b) reconstructed up to the Nyquist limit of our array. Nooff-chip optics were used.

August 1, 2011 / Vol. 36, No. 15 / OPTICS LETTERS 2951