Neuromorphic Image sensors Eugenio Culurciello Yale University
EENG427 Lesson 8
Slide 2
Taking hints from nature: How does nature solve everyday
problems Can we implement natures solutions? in Silicon? Biomimetic
Circuits
Slide 3
Human Eye: a wonderful machine Small and light: 1 inch, 7 grams
Retina: neural sensor network, rods and cones Optic nerve carries
digital signals to the brain Biomimetic Circuits
http://webvision.med.utah.edu/anatomy.html
Slide 4
Structure of the Eye
Slide 5
The Retina
Slide 6
Dynamic Range: 10+ orders of magnitude Bandwidth: 100M sensors,
1M fibers in optic nerve Specialization: Cones in color, high
resolution - fovea Rods in the dark / motion Biomimetic Circuits
http://webvision.med.utah.edu/anatomy.html
Slide 7
Everyone wants silicon eyes! Small Light Acute: now >
1Mpixel Must work in: Dim restaurant Outside BBQ Long life = Low
power Almost like a human eye! Digital Cameras and Si Eye
http://www.panasonic.com.au/product_pdf/EB-X70.pdf
Slide 8
Smart-sensors are those devices in which the sensors and
circuits co-exist, and their relationship with each other and with
higher-level processing layers goes beyond the meaning of
transduction. Smart-sensors are information sensors, not
transducers and signal processing elements. Smart sensors are not
general purpose devices. Everything in a smart sensor is
specificaly designed for the application targeted for. Neuromorphic
Image Sensors
Slide 9
When compared to a vision processing system consisting of a
camera and a digital processor, a vision chip provides many system
level advantages. Speed: The processing speed achievable using
vision chips exceeds that of the camera-processor combination. A
main reason is the information transfer bottleneck between the
imager and the processor. In vision chips information between
various levels of processing is processed and transferred in
parallel. Large dynamic range: Many vision chips use photodetectors
and photocircuits which have a large dynamic range over at least 7
decades of light intensity. Many also have local and global
adaptation capabilities which further enhances their dynamic range.
Conventional cameras are at best able to perform global automatic
gain control. Neuromorphic Image Sensors
Slide 10
Size: Using single chip implementation of vision processing
algorithms, very compact systems can be realized. The only parts of
the system that may not be scalable are the mechanical parts (like
the optical interface). Power dissipation: Vision chips often use
analog circuits which operate in subthreshold region. There is also
no energy spent for transferring information from one level of
processing to another level. System integration: Vision chips may
comprise most modules, such as image acquisition, and low level and
high level analog/digital image processing, necessary for designing
a vision system. From a system design perspective this is a great
advantage over camera-processor option. Neuromorphic Image
Sensors
Slide 11
Although designing single-chip vision systems is an attractive
idea, it faces several limitations: Reliability of processing:
Vision chips are designed based on the concept that analog VLSI
systems with low precision are sufficient for implementing many low
level vision algorithms. The precision in analog VLSI systems is
affected by many factors, which are not usually controllable. As a
result, if the algorithm does not account for these inaccuracies,
the processing reliability may be severely affected. Vision chips
also use unconventional analog circuits which may not be well
characterized and understood. Resolution: In vision chips each
pixel includes a photocircuit which occupies a large proportion of
the pixel area. Therefore, vision chips have a low fill-factor and
a low resolution. The largest vision chip reported has only 210 230
pixels, for a photocircuit consisting of six transistors only.
Neuromorphic Image Sensors
Slide 12
Difficulty of the design: Vision chips implement a specific
algorithm in a limited silicon area. Therefore, often off-the-
shelf circuits cannot be used in the implementation. This involves
designing many new analog circuits. Vision chips are always full
custom designed, and full custom design is known to be time
consuming and error-prone. Programming: None of the vision chips
are general purpose. In other words, many vision chips are not
programmable to perform different vision tasks. This inflexibility
is particularly undesirable during the development of a vision
system. Neuromorphic Image Sensors
Slide 13
Mahowald's silicon retina chip is among the first vision chips
which implemented a biological facet of vision on silicon. The
computation performed by Mahowald's silicon retina is based on
models of computation in distal layers of the vertebrate retina,
which include the cones, the horizontal cells, and the bipolar
cells. The cones are the light detectors. The horizontal cells
average the outputs of the cones spatially and temporally. Bipolar
cells detect the difference between the averaged output of the
horizontal cells and the input. Mahowald, Mead's silicon
retina
Slide 14
PASIC sensor from Linkping University The ``Processor ADC and
Sensor Integrated Circuit'' (PASIC) as the name suggests consists
of a sensor array, A/D converters, and processors. Each column has
its own ADC and processor.
Slide 15
This silicon retina is an implementation of the outer-plexiform
of retinal processing layers. The design has a distinctive feature
that separates it from all other silicon retinas. The
implementation uses a very compact circuit, which has enabled the
realization of a 210 x 230 array of image sensors and processing
elements with about 590,000 transistors, which is the largest among
all reported vision chips. This silicon retina uses a diffusive
smoothing network. The function of this one-dimensional network can
be written as dQn/dt is the current supplied by the network to node
n, and D is the diffusion constant of the network, which depends on
the transistor parameters, and the voltage. Andreou and Boahen's
silicon retina
Slide 16
The function of the network can be approximated by the
biharmonic equation where g and h are proportional to the the
diffusivity of the upper and lower smoothing layers, respectively.
More details about the function of the circuit can be found in
relevant references. All the 2D chips use a hexagonal network with
six neighborhood connection. The largest chip occupies an area of
9.5x9.3mm, in a 1.2um CMOS process with two layers of metal and
poly. A cell size of about 40x40um has been achieved for this
implementation. Under typical conditions the chip dissipates 50mW.
Andreou and Boahen's silicon retina
Slide 17
Andreou and Boahen have encapsulated the model of the retina in
a neat and small circuit (below). This circuit includes two layers
of the diffusive network. The upper layer corresponds to horizontal
cells in retina and the lower layer to cones. Horizontal N-channel
transistors model chemical synapses. Andreou and Boahen's silicon
retina
Slide 18
What would it take to reproduce the human eye in Si? Biomimetic
Circuits
http://www.nips.cc/Web/Groups/NIPS/NIPS2000/00papers-pub-on-web/KurinoNakagawaLeeNakamuraYamadaParkKoyanagi.pdf
3D Fabrication Process High Connectivity
Slide 19
And with a conventional process? NEURONS: Advantage IN SPACE
Neurons in the human brain make up to 10 5 connections with their
neighbors CIRCUITS: Advantage IN TIME Integrated circuits handle
communication cycles six orders of magnitude smaller than the
inter-event interval for a single neuron or cell Biomimetic
Circuits
Slide 20
Conventional Image Sensors Integrate light on a capacitor for a
fixed time Sample the analog capacitor voltage Pixels are
synchronously scanned
Slide 21
Measure the time to integrate to a fixed voltage Light triggers
a digital event Integrate (to threshold) and fire Address-Event
Image Sensors Event Driven! Time Events: digital pulses
Slide 22
Pixel Operation Photocurrent is integrated on a 0.1pF
capacitor. Slew Rate of 0.1V/ms in typical indoor light of 0.1mW/cm
2 Pixel is reset to Vdd_r While integrating light, the voltage on
the capacitor will decrease down to the threshold of the
inverter
Slide 23
The switching current of the inverter is fed back by a current
mirror to sharpen the transition. The integrating capacitor is
disconnected to minimize power consumption during reset. Reduced
power consumption when compared to an inverter Slew rate gain Pixel
Operation
Slide 24
Slide 25
Equation of the switching point (voltage): In time domain:
Pixel Operation
Slide 26
Address-Event Address-Event Representation: asynchronous
protocol for communication between large arrays The AER model
trades complexity in wiring of the biological systems for
processing speed of integrated circuits
Slide 27
Address-Event Architecture
Slide 28
Slide 29
Inter-Event ImageHistogram Image t 1/T i t N/T h Sample Images
from Sensor 100k samples10k samples
Slide 30
Chip layout E. Culurciello, R. Etienne-Cummings, K. A. Boahen,
``A Biomorphic Digital Image Sensor', IEEE Journal of Solid-State
Circuits, Vol. 38, No. 2, February 2003.
Slide 31
Sensor Performance Technology0.6m 3M CMOS Array Size80 (H) x
60(V) Pixel Size32m x 30m Fill Factor14% Dynamic
Range200dB(Pix),120dB(Array) Bandwidth 8mHz 40MHz (Pix.) 40Hz-40MHz
(Array) Pixel Frequency Jitter (STD/Max)6x10 -4 % Sensitivity
[Hz/mW/cm 2 ]2x10 6 (Array), 42 (Pix) FPN (STD/Mean
pixel-pixel)0.5% @ 0.1 mW/cm 2 Max. FPS8.3K (effective) Digital
Power (@ 0.1mW/cm 2 )3.4mW @ 2.9V Supply Analog Power (@ 0.1mW/cm 2
)< 10W @ 2.7V Supply