Eugenio Culurciello

Department of Electrical Engineering

Yale University – October 5th 2004

Conventional Image Sensors andAddress-Event Image Sensors:

Taking hints from nature:

• How does nature solve everyday problems• Can we implement nature’s solutions?

… in Silicon?

Biomimetic Circuits

Human Eye: a wonderful machine

• Small and light: 1 inch, 7 grams• Retina: neural sensor network, rods and cones• Optic nerve carries ‘digital’ signals to the brain

Biomimetic Circuits

http://webvision.med.utah.edu/anatomy.html

• Dynamic Range: 10+ orders of magnitude• Bandwidth: 100M sensors, 1M fibers in optic nerve• Specialization:

– Cones in color, high resolution - fovea– Rods in the dark / motion

Biomimetic Circuits

http://webvision.med.utah.edu/anatomy.html

Everyone wants silicon eyes!

• Small• Light• Acute: now > 1Mpixel• Must work in:

– Dim restaurant– Outside BBQ

Long life = Low power

Almost like a human eye!

Digital Cameras and Si Eye

http://www.panasonic.com.au/product_pdf/EB-X70.pdf

What would it take to reproduce the human eye in Si?

http://www.nips.cc/Web/Groups/NIPS/NIPS2000/00papers-pub-on-web/KurinoNakagawaLeeNakamuraYamadaParkKoyanagi.pdf

3D Fabrication Process

High Connectivity

Availability?

SOI?

Digital Cameras and Si Eye

IMAGERS TYPES

• IMAGERS: Analog or Digital output

• Analog → pixel output an analog voltage– Analog Pixel (been around for a while…)

• Digital → pixel output digital bit(s)– Oversampled pixel (Octopus, 2001)– Pixel ADC (Kleinfeld, 2001)

Conventional Image Sensors

• Integrate light on a capacitor for a fixed time• Sample the analog capacitor voltage• Pixels are synchronously scanned

ANALOG IMAGERS ARCHITECTURE

Pixel, scanning and output circuitry:

PHOTO-TRANSDUCTION

Photo-transduction power budget:

Iphoto = 100pA MAX

Cphoto = 5fF (follower gate)

Reset power =

Cphoto Vph^2 fscan

ANALOG IMAGER

• Vertical and horizontal scanners

Horizontal selection: every √N pixels

Vertical selection: Every pixel

Vertical S

canner

Horizontal Scanner

N pixels

ANALOG IMAGER

• Vertical and horizontal scanners power:

Nfscan dVcol^2Ccwsr 2/1

+N)fscan Vdd^2(Ccwsr 1/sqrt(N))+(1 =srpwr

• Cost of addressing + cost of outputting data

• Cost of reset (row-wise)

N)fscan Vdd^2Ccwsr 1/sqrt(N)(

ANALOG IMAGER

• Pads power consumption:

Cload = 20pF

padpow = 4 alpha Cload fscan N dVout Vdd

• ADC power

ANALOG IMAGER

• Cost of outputting data: voltage follower

DIGITAL IMAGER

• Two kinds:

– Oversampled

– Pixel ADC Ic

event

reset

Vdd_r

DIGITAL IMAGER

• Vertical and horizontal scanners power:

N)fscan Vdd^21/4(Ccwsr

+N)fscan Vdd^2(Ccwsr 1/sqrt(N))+(1 =srpwr

• Cost of addressing + cost of outputting data HIGHER than analog

CCD IMAGER

• Vertical and horizontal scanners:

• Measure the time to integrate to a fixed voltage• Light triggers a digital event • Integrate (to threshold) and fire

Time-Domain Image Sensors

Ic

event

reset

Vdd_r

Event Driven!

Time

Events: digital pulses

• We can use an inverter to generate an event, Right? …

Time-Domain Image Sensors

Ic

event

reset

Vdd_r

Power consumption!

Slew rate gain?

Our Pixel

• Photocurrent is integrated on a 0.1pF capacitor. Slew Rate of 0.1V/ms in typical indoor light of 0.1mW/cm2

• Pixel is reset to ‘Vdd_r’

• While integrating light, the voltage on the capacitor will decrease down to the threshold of the inverter

•The switching current of the inverter is fed back by a current mirror to sharpen the transition. The integrating capacitor is disconnected to minimize power consumption during reset.

• Reduced power consumption when compared to an inverter

• Slew rate gain

Our Pixel

0 1 2 30

1

2

3

time [ms]

Vin

[V

]

0 1 2 3

1

2

3

time [ms]

Vo

ut

[V]

0 1 2 30

1

2

3

time [ms]

Vc

[V

]

0.9670.9675

-3

-2

-1

x 10-6

time [ms]

I [A

]

1.98 1.99 2

-4

-2

0

2

x 10-5

time [ms]

Pixel Operation

0,2254

, ln

QQQQ

phswitchin

IL

W

W

L

L

W

I

q

nKTV

C

ItVV ph

rddin

_

• Equation of the switching point (voltage):

• In time domain:

Pixel Operation

Accessing Pixel Data

• How do we extract data from a large pixel array operating in Time-Domain?• Use a Neuromorphic approach:

the Address-Event Representation

Address-Event• Address-Event Representation: asynchronous protocol for communication between large arrays

,...,,...,,,,...,,...,,,' 11001100 gigiggggACii txtxtxftxtxtx

Address-Event Architecture

Address-Event Architecture

Inter-Event Image Histogram Image

t1/Ti

t

N/Th

Image Reconstruction

How do we reconstruct an image from a stream of events?

We can use two techniques:

Inter-Event Image Histogram Image

t1/Ti

t

N/Th

Sample Images from Sensor

100k samples10k samples

Chip layout

E. Culurciello, R. Etienne-Cummings, K. A. Boahen, ``A Biomorphic Digital Image Sensor'‘, IEEE Journal of Solid-State Circuits, Vol. 38, No. 2, February 2003.

Sensor PerformanceTechnology 0.6μm 3M CMOS

Array Size 80 (H) x 60(V)

Pixel Size 32μm x 30μm

Fill Factor 14%

Dynamic Range 200dB(Pix),120dB(Array)

Bandwidth 8mHz – 40MHz (Pix.)

40Hz-40MHz (Array)

Sensitivity [Hz/mW/cm2] 2x106 (Array), 42 (Pix)

FPN (STD/Mean pixel-pixel) 0.5% @ 0.1 mW/cm2

Max. FPS 8.3K (effective)

Digital Power (@ 0.1mW/cm2) 3.4mW @ 2.9V Supply

Analog Power (@ 0.1mW/cm2) < 10μW @ 2.7V Supply

Our

Arr

ay (

1/8

VG

A)

100

101

102

103

103

104

105

106

107

108

Power @ 16bits

Power @ 12bits

Power @ 10bits

Power @ 8bits

Pow

er [

mW

]

Sensor size [# Pixel]

VG

A

Scaling: Power Consumption

100

101

102

103

104

105

1

10

103 104 105 106 107 108

Effective FPSDynamic Range

Eff

ecti

ve F

ram

e R

ate

[Hz]

Dynam

ic Range [D

ecades]

Sensor Size [# Pixels]

Our

Arr

ay (

1/8

VG

A)

VG

A

Scaling: Dynamic Properties

Imager Statistical Data

0 100 200 300 400 50010

-1

100

101

102

103

Seconds

Co

un

t

Poisson distributed output spikes

Pixels act independently

The probability of an address from a certain region is proportional to the light intensity in that neighborhood

Image Sensor Linearity

Sensor linearity versus incident light intensity

Data is for event rate produced by entire array

10-6

10-5

10-4

10-3

10-2

105

106

107

Incident Light Power [W/cm2]

Ima

ge

r S

pik

ing

Fre

qu

en

cy

Outline

• Address-event image sensors

• Second generation AE sensors

• The SOS fabrication process

• SOS Image Sensors

• Future research

Channel Access

• ALOHA

• Arbitration

• Scanning

Scanning Registers Access

for j = 1:size(Y)

for i = 1:size(X)

V= Array (i, j);

Transmit(V, i, j);

end

end

Algorithm

Cell Array

X Scanning Register

Y S

canning Register

ALOHA Access

for-ever

while (!Event) wait;

E = Event;

V = Array (E.x, E.y);

Transmit(V, E.x, E.y);

end

Algorithm

Cell Array

X Request Detector

Y R

equest Detector

Arbitrated Access

for-ever

while (!Event);

Y = Arbiter(Events.y);

E = Arbiter(Event(Y))

V = Array (E.x, E.y);

Transmit(V, E.x, E.y);

end

Algorithm

Cell Array

X Arbiter Tree

Y A

rbiter Tree

The ALOHA protocol

• Invented at University of Hawaii, Abramson 1970

• The ALOHA protocol is the foundation of Ethernet

• Unfettered: pixel access channel at will

• Simple to implement

• Reduced transmission delay

• Low throughput, only 18% of output bandwidth

• ALOHA access technique• Simple and efficient• Address-event asynchronous circuits• Automatically detects collisions on the output bus

ALOHA Image Sensor

International Conference IEEE ISCAS (Circuits and Systems) 2004, Vancouver.

Technology 0.6µm 3M CMOS

Array Size 32 x 32

Pixel Size 32.7µm x 29.7µm

Fill Factor 6.5%

Sensor Core Size 1.2 x 1.2mm

Dynamic Range 241dB (Pixel)181dB (Array)

Bandwidth 8.13μHz - 10MHz (Pixel)8.33mHz – 10MHz (Array)

Sensitivity [Hz/W/m2]

1.7x103 (Array)2.8 x109 (Pixel)

Max. FPS 4.88K (effective)

Digital Power 115μW at 3.30V

Analog Power 680μW at 3.30V

Image Sensor Performance

Channel Access

• Comparing various channel access techniques:

• ALOHA

• Arbitration

• Scanning

E. Culurciello, A. G. Andreou, "Access Topologies For Address-Event Communication Channels", IEEE Transaction on Neural Networks, Special Issue on Neural Networks Hardware Implementations, September 2003.

Channel Access

• Comparative index:

• Throughput S

• Integrity I

• Power P

• Delay δ

),(),(

),(),(max

ACACc

ACAC

f fGfGP

fGIfGSAC

¼ VGA

• Sensors network – low power smart dust

eventuser

Zzz

Zzz Zzz Zzz

• Mote @ UCBerkeley

Network of Eyes:

We connected the ALOHA imager to the Berkeley sensor network mote

But it can be a lot smaller!

Network of Eyes:

And we obtained the first Network of Eyes!

Radio: CC1000 19kbps, processor: Atmega 8Mhz

And it can be a lot faster!

Network of Eyes: