A 180nm CMOS image sensor with on-chipoptoelectronic image compression

Albert Wang, Sriram Sivaramakrishnan, Alyosha MolnarCornell University Ithaca, New York 14853

Abstract— We demonstrate an image sensor which directly ac-quires images in a compressed format. The sensor uses diffractiveoptical structures integrated in the CMOS back-end layer stackto compute a low-order 2D spatial Gabor transform on visualscenes. As this computation occurs in the optical domain, thereadout back-end uses the transform outputs to implement thesubsequent image digitization and compression simultaneously.Implemented in a 180nm logic CMOS process, the image sensoruses a 384x384 array of complementary angle-sensitive pixelpairs, consumes 2mW at a frame rate of 15fps, and achievesa 10:1 compression ratio on test images.

I. INTRODUCTION

Imaging systems typically combine an image sensor todigitize visual scenes with a subsequent transformation of theimage into a sparser format for further processing. For ex-ample, image compression eases data storage or transmissionrequirements by applying an image transform, such as thediscrete cosine transform (DCT), and rounding the resultingoutputs. Many applications employ dedicated digital signalprocessors to compute 2D image transforms. Recent workhas demonstrated image sensors which integrate transformcalculations in the analog [1], [2] or digital [3] domain on asingle chip. The image sensor presented here computes spatialGabor transforms on chip in the optical domain, enablingefficient image processing in the backend circuits. Fabricatedin 180nm digital CMOS, the chip requires only ambient lightand a single camera objective to capture compressed images.

II. BACKGROUND

Spatial Gabor transforms are a family of oriented, periodic2D sinusoids with a Gaussian envelope, each of which acts asa bandpass filter in 2D frequency space. Figure 1 illustratesthe convolution of the standard Lena test image with a singleGabor bandpass filter. Gabor transforms are frequently used ina diverse variety of applications from edge detection to imagecompression and object recognition [4] and also accuratelymodel pathways in the vertebrate visual cortex [5].

Unlike the DCT employed in JPEG compression, the Gabortransform is not separable and cannot be computed using twosequential 1D filtering operations. This is a significant draw-back, as computing a full 2D convolution with a number ofdifferent filter kernels to obtain a transformed image requiresconsiderable processing power. Our image sensor circumventsthese operations by using the oriented, periodic response ofangle-sensitive pixels [6] to perform the required spatial imagetransform in the optical domain.

Image capture

Convolution

Transform coefficients

Fig. 1. Gabor transforms involve a 2D convolution with a set of oriented,periodic filter kernels to generate a set of transform coefficients.

A B

Diffraction gratings

CMOS Active Pixels

Fig. 2. Angle-sensitive pixels combine diffractive optics with conventionalphotosensors and respond to incident angle.

III. SYSTEM DESIGN

A. Optical design

Illustrated in Fig. 2, angle-sensitive pixels (ASPs) use a pairof local diffraction gratings with conventional active imagerpixels and possess a periodic sensitivity to incident angle.Because the pixel output changes in response to both incidentangle and intensity, we rely on pairs of complementary pixelsto distinguish changes in incident angle from changes inintensity. These complementary pixel pairs, labeled A and Bin Fig. 2, have different relative shifts between the diffractiongratings, and thus different angular responses. We treat thesignals from these two pixels as a differential signal, wherethe differential mode encodes incident angle, and the commonmode encodes light intensity.

To remove biases in this differential signal which resultfrom the physical separation of complementary angle-sensitivepixels, we have utilized the common centroid layout of Fig.3(a) to create an “optical differential pair.” Measuring theoutput differential signal from this pair of ASPs, we obtaina windowed sinusoidal response like that shown in Fig. 3(b)to changes in incident angle along the axis perpendicular tothe direction of the diffraction gratings. The geometry of thediffraction gratings controls the periodicity of the response,while the finite aperture of image-forming optics sets the width

978-1-4673-1556-2/12/$31.00@2012 IEEE

Pixel A

A B

B A Σ –

+

Pixel B Difference

(a)

Dif

fere

nce

(V

)

Incident angle (deg.)

0

0 -25 -50 50 25

(b)

Fig. 3. The differential response of a) one pair of complementary angle-sensitive pixels is b) a periodic function of angle.

Image sensor Lens Spatial

offset

(a)

Measured impulse response

(b)

Fig. 4. A lens in a) an optical system maps spatial displacement to incidentangle so that ASPs act as b) 2D Gabor filters in space.

of the windowing function. Along the axis parallel to thediffraction gratings, only the windowing function appears inthe angular response. This response can be described as a 2DGabor filter operating on incident angle distributions [7].

When an image sensor is placed away from the focal planeof the imaging optics, as in Fig. 4(a), different spatial offsetsof objects in a visual scene map to distinct incident angles.Consequently, the 2D Gabor filter response in angle generatedby one optical differential pair projects to a 2D Gabor filter inthe spatial domain (Fig. 4(b)). The geometry of the diffractiongratings utilized in the component angle-sensitive pixels setsthe characteristic frequency, orientation, and phase of the filterresponse, and the aperture of the imaging optics control theenvelope of the filter response. By using a variety of differentcomplementary ASP pairs, each with a distinct geometry toprovide a distinct transform coefficient, we obtain a full,low-order spatial 2D Gabor filter bank which implements acomplete Gabor transform.

Figure 5 demonstrates the operation of this optically com-puted 2D spatial Gabor filter bank. Our implemented imagesensor employs a tiled set of 24 optical differential pairs whosedifferential signals generate a set of 24 2D spatial Gabor filterswith 3 frequencies, each of which has 4 orientations and 2phases (sine and cosine). A micrograph of the pixel layoutis shown in Fig 5(a), as well as a set of measured impulseresponses in Fig. 5(b) illustrating the response of each filterto a small white dot on a black background.

B. Electrical design

Because Gabor filters efficiently compact the informationfound in a visual scene [4], they provide a good basis for imagecompression. Comparing a histogram of pixel-by-pixel valuesin the Lena test image with a histogram of the coefficientsgenerated by convolving the image with a Gabor filter as inFig. 6, we observe that many of the coefficients resulting fromthe filter operation are zero or near zero. By approximatingthe coefficients near zero as zero, and rounding the remaining

60μ

m

(a)

Low frequency Mid frequency High frequency

(b)

Fig. 5. Implemented Gabor filter bank: a) 24 distinct angle-sensitivepixel designs generate spatial filters, with b) measured Gabor-like impulseresponses.

nonzero coefficients to reduced precision, we reduce thenumber of bits required to encode an image relative to simplyencoding each pixel-by-pixel brightness value at a fixed bitprecision. The result of these two operations is a compressedimage. Therefore, the readout back-end of our image sensor

Co

un

t

Pixel value 0 255

Pixel Histogram

Coefficient value

Filter Histogram

0

Bitmap image Transform image

Fig. 6. Gabor filter coefficients of an image are concentrated near zero.

simply needs to round and approximate differential pixel sig-nals to perform image compression. These circuits for imagecompression are simple and rely on the characteristics of ouron-chip optical image transformations during the digitizationprocess. By initially performing the image transform andfiltering in the optical domain, we eliminate both the powerconsumed by analog or digital image transform circuits and thesilicon area dedicated to implementing these computationallycostly matrix operations. For image compression, these gainsare particularly significant as a large number of transformoutputs are simply discarded and never transmitted, wastingthe resources dedicated to computing these results.

Each distinct Gabor filter resulting from the output of oneselected differential optical pair uses one dedicated back-end readout channel, resulting in 24 Gabor filter outputson the image sensor. The back-end channels incorporatesa programmable gain differential amplifier, followed by avariable-resolution successive approximation ADC which hasselectable “dead zone” which rounds near-zero inputs to zero.

Schematically shown in Fig. 7, the PGA has a differentialarchitecture, taking as inputs the outputs of a pair of comple-mentary angle-sensitive pixels. Programmable load resistors(R3) provide 4 gain settings to account for the different signal

Pixel B Pixel A R1 R2 R3 R3

AB Boost

to ADC

CM Tap

Fig. 7. Schematic of programmable gain amplifier: resistors R1/R2 setimbalanced degeneration to correct common mode and R3 sets gain.

gains of different filter designs and ensure effective utilizationof the available ADC dynamic range. To cancel the poor com-mon mode rejection of the optical differential pixel pair, wehave employed programmable, imbalanced degeneration of theinput transistors using resistors R1 and R2. The output of theamplifier is buffered with a class-AB source follower capableof driving a large voltage onto the ADC when necessary, whilesaving power for small signal swings.

While the differential mode signal from pairs of angle-sensitive pixels provide us with Gabor filter coefficients, thecommon mode signal encodes local scene brightness. To ac-count for this additional information, each PGA has a current-based common mode output, shown as the common mode tapof Fig. 7. Since we rely on each block of 24 optical differentialpairs of Fig. 5(a) to provide the necessary set of filter outputs,we sum the common-mode current across all 24 amplifierchannels and digitize the result with its own dedicated ADCfor a 25th, spatial low-pass filter output providing a measureof scene brightness seen by the full optical filter block.

The variable-resolution differential charge-redistributionsuccessive-approximation ADC of Fig. 8 performs N+1 com-parisons for each N-bit conversion. The first two comparisonsbracket a “dead zone” around differential zero. If the filteroutput falls in this zone, the conversion terminates, and theoutput is rounded to zero. This operation has been shown toefficiently quantize image information [8]. If the input fallsoutside this “dead zone” the MSB is set and the conversioncontinues, generating a digital value with a resolution between5 and 8 bits. Measured transfer curves of the ADC confirma programmable, symmetric dead zone range of 0 to 650mV.Offset corrections of up to the full range of the ADC can alsobe programmed dynamically into each ADC to compensate forinput offsets from the converter or from the preceding pixeland amplifier circuitry.

A complete block diagram of the image sensor is shownin Fig. 9. The pixel array of the image sensor is a 96x64array of the 24 filter blocks shown in Fig. 5(a), correspond-ing to an array of 384x384 filters, for a total of 295,000angle-sensitive pixels arranged in common-centroid opticaldifferential pairs. The 25 readout paths operate in parallel,amplifying and digitizing the spatial bandpass filter and localintensity outputs simultaneously. Compared to the common-mode intensity and low spatial frequency filters, the higherfrequency filters are intentionally designed with a lower max-imum resolution of 7 bits, reflecting the reduced high spatial

Differential capacitor bank

9-bit SAR Logic

2 MSBs

Cycle Sample 1 2 3 ∙∙∙∙ 9

Mux Offset Window 1 Window 2 SAR ∙∙∙∙ SAR

4:1 Mux

8

Window 1 8

Window 2 8

Offset 8

STOP

Output Code

8 7 LSBs

Inside window, stop

Input

Sample

Fig. 8. Block diagram of programmable, variable-resolution SAR ADC.

frequency content seen in natural scenes [9]. The fabricatedchip (Fig. 10) is manufactured in TSMC’s 180nm logic CMOSprocess, measures 25mm2 in area and consumes 2 mW froma 1V(digital)/1.8V(analog)/3.3V(pixel) supply at a frame rateof 15fps.

5-8

bit

A

DC

Intensity

384x384 Angle-sensitive pixel filter bank

5-8

bit

A

DC

8x

Low frequency

5-7

bit

A

DC

8x

Mid-frequency High frequency

5-7

bit

A

DC

8x

Tim

ing

con

tro

l

Fig. 9. Top-level block diagram of imaging system.

Pixel Array Ti

min

g/A

dd

ress

ing

Programmable Amplifiers

ADCs

Fig. 10. Die microphotograph.

IV. RESULTS

We demonstrated the capability of this transform basedimage sensor in Figure 11. A copy of the Lena test image wascaptured using a single Nikon camera lens (f=50mm, F#=1.8)under ambient white light scene illumination. Each of the 24Gabor filter channels captures unique features from the scene,with three shown on the left in Fig. 11(a). The 25th lowpassfilter channel, generated from the common mode output ofthe angle-sensitive pixels, captures local scene brightness ason the right in Fig. 11(a). Inverting these spatial transforms

Low frequency filter Mid frequency filter High frequency filter Common mode filter

(a)

Bitmap image

1.2Mbits (150Kpixel, 8 bit)

Full reconstruction

120Kbits (10:1 reduction)

90Kbits data across all 24 Gabor filter channels

Invert spatial transform and sum for image recovery

(b)

Fig. 11. Demonstration of image sensor performing image compression. a) Representative outputs of image sensor when shown the Lena test image. Threeof the 24 Gabor filter outputs, corresponding to different spatial frequencies, and the 25th common mode output, corresponding to local image brightness, areshown. b) Combining the 24 filter outputs (digitized using 90Kbits) and the common mode output (digitized with 30Kbits) generates a reconstructed imageat a 10:1 data reduction compared to a raw bitmap image, while preserving significant image detail.

and summing generates an outlined version of the test image,shown second from left in Fig. 11(b). At a 5-bit resolutionfor the filter outputs, 90Kbits are required to digitize thenonzero coefficients of all 24 channels, with independentlyprogrammed “dead zones” for each channel.

When combined with the common mode channel, whichrequires 30Kbits at a resolution of 5 bits, we obtain thereconstructed image shown second from right in Fig. 11(b).The compressed image requires a total of 120Kbits of imagedata. In contrast, the same image, put in focus and capturedby the same image sensor requires 1.2 Mbit to digitize thecommon mode of each individual angle-sensitive pixel pairto an accuracy of 8 bits. Comparing the image reconstructedfrom the compressed image data with the bitmap image, imagedata has been reduced by a factor of 10 without significantdegradation in image quality.

V. DISCUSSION

We have presented a CMOS image sensor which computes2D spatial Gabor transforms in the optical domain and utilizesthe resulting transform coefficients for image compression.Optical transform computation has the advantages of zeropower consumption and drastically simplified system design,as we have eliminated the power and cost of analog or digitalcircuits which must store filter coefficients and perform therequisite 2D convolution. Although we have demonstratedan image sensor back-end dedicated to image compression,other possible design choices which optimize the back-endprocessing for other image processing tasks, such as edge

identification and feature recognition, are possible and willresult in optoelectronic systems for a variety of imaging tasks.

VI. ACKNOWLEDGEMENTS

This work was supported by the DARPA YFA Program,under Grant #66001-10-1-4028

REFERENCES

[1] A. Bandyopadhyay, J. Lee, R. Robucci, and P. Hasler, “MATIA: aprogrammable 80 µw/frame CMOS block matrix transform imager ar-chitecture,” IEEE J. Solid-State Circuits, vol. 41, no. 3, pp. 663–672,Mar. 2006.

[2] A. Nilchi, J. Aziz, and R. Genov, “Focal-plane algorithmically-multiplying CMOS computational image sensor,” IEEE J. Solid-StateCircuits, vol. 44, no. 6, pp. 1829–1839, Jun. 2009.

[3] Y. Nishikawa, S. Kawahito, M. Furuta, and T. Tamura, “A high-speedCMOS image sensor with on-chip parallel image compression circuits,”in Proc. IEEE Custom Integrated Circuits Conf. (CICC), Sep. 2007, pp.833–836.

[4] J. Daugman, “Complete discrete 2-D Gabor transforms by neural net-works for image analysis and compression,” IEEE Trans. Acoust., Speech,Signal Processing, vol. 36, no. 7, pp. 1169–1179, Jul. 1988.

[5] J. Touryan, G. Felsen, and Y. Dan, “Spatial structure of complex cellreceptive fields measured with natural images,” Neuron, vol. 45, no. 5,pp. 781–791, Mar. 2006.

[6] A. Wang, P. R. Gill, and A. Molnar, “An angle-sensitive CMOS imagerfor single-sensor 3D photography,” in IEEE Int. Solid-State Circuits Conf.(ISSCC) Dig., Feb. 2011, pp. 412–414.

[7] A. Wang, S. Hemami, and A. Molnar, “Angle-sensitive pixels: a newparadigm for low-power, low-cost 2D and 3D sensing,” in Proc. SPIEElectronic Imaging, vol. 8288, no. 828805, 2012.

[8] G. Sullivan, “Efficient scalar quantization of exponential and Laplacianrandom variables,” IEEE Trans. Inform. Theory, vol. 42, no. 5, pp. 1365–1374, sep 1996.

[9] G. Burton and I. Moorehead, “Color and spatial structure in naturalscenes,” Applied Optics, vol. 26, no. 1, pp. 157–70, 1987.