Dynamic Vision Sensor - Robotics and Perception...

June 2, 2017

S.LSI ICRA'17 Workshop on "Event-based Vision"

The Road to Market

Dynamic Vision Sensor

Yoel Yaffe*, Nathan Levy, Evgeny Soloveichik, Sebastien Derhy, Ayal Keisar, Elad Rozin, Liron Artsi

Jun-Seok Kim†, Keunju Park, Bongki Son, Yunjae Suh, Heejae Jung, Changwoo Shin, Jooyeon Woo, Yohan Roh, Hyunku Lee, Hyunsurk (Eric) Ryu

*Samsung Israel Research Center, Israel †System LSI , South Korea

Samsung Electronics

Contact: yoel.yaffe@samsung.com

2 S.LSI ICRA'17 Workshop on "Event-based Vision"

The Road Less Traveled

image by aosleading

Market Driven Requirements VS Early DVS Prototypes

Requirements Early DVS Prototypes

Low Cost Large Pixel No standard for Quality / Die sorting

Minimal Module Size Large, due to large Pixel -> Large optical format

Ultra Low Power Low power

Good Event Quality Redundant events (especially at low light) Motion artifacts, Timing accuracy Noise, Flicker

Low Data Rate Scene dependent; In some cases may exceed CIS typical throughput

Vision processing at Edge Not available

The Road to Market - Overview

oReducing Cost and Module Size

oReducing Power

oDelivering Good Event Quality

oData Throughput Reduction

oEvent Processing Acceleration

oSummary

oReducing Power

oSummary

Reducing Pixel Size and Optical format

18.5 x 18.5 µm2 → 6 x 6 µm2 ~90% pixel area reduction

Pixel Shrink

o BSI process

o Downsizing MIMCAP Without a decrease in minimum contrast sensitivity

o Stacked Cu-Cu Technology

12 µm

18.5 µm

by Stacked Pixel

Technology

PD + LOGA+C1

C2+CFA+Comp.

Reducing Pixel Size and Optical format

3μm x 3μm Photodiode / 9μm x 9μm Pixel

5μm x 5μm Photodiode / 6 μm x 6 μm Pixel / Wafer-stacking

o Photo Enhancement:

Delivering Product to Market - Overview

oReducing Power

oSummary

Reducing Power

oUse Samsung’s advanced process

o Subsampling modes

oDynamic wakeup, window of interest

Reducing Power: Statistics for Dynamic Decisions

o Spatial Histogram & Blocking Area:

ISSCC’17

(0,0) (0,1) (0,2) (0,3) (0,16) (0,17) (0,18) (0,19)

(1,0) (1,1) (1,2) (1,3) (1,16) (1,17) (1,18) (1,19)

(14,0) (14,1) (14,2) (14,3) (14,16) (14,17) (14,18) (14,19)

640x480 Image Plane

Reducing Power: Statistics for Dynamic Decisions

ISSCC’17

Spatial Histogram Data

Reducing Power: Blocking, Subsampling

ISSCC’17

APS+ANALOG

Cropper

AER Packetizer

AER Packet

MIPI I/F

USB I/F

Parallel I/F

OUTPUT

Spatial Histogram

IIC I/F

PAD Out

Histogram

INT I/FPAD Out

IIC Bus

Wakeup

Histogram

Interrupt

Histogram

Digital AER

AER Gen

Analog Test Control

Time Stamper

Test Pattern

Subsampler

Blocking Area

Reducing Power: Statistics Collection

ISSCC’17

APS+ANALOG

Cropper

AER Packetizer

AER Packet

MIPI I/F

USB I/F

Parallel I/F

OUTPUT

Spatial Histogram

IIC I/F

PAD Out

Histogram

INT I/FPAD Out

IIC Bus

Wakeup

Histogram

Interrupt

Histogram

Digital AER

AER Gen

Analog Test Control

Time Stamper

Test Pattern

Subsampler

Blocking Area

Reducing Power: Wakeup

ISSCC’17

APS+ANALOG

Cropper

AER Packetizer

AER Packet

MIPI I/F

USB I/F

Parallel I/F

OUTPUT

Spatial Histogram

IIC I/F

PAD Out

Histogram

INT I/FPAD Out

IIC Bus

Wakeup

Histogram

Interrupt

Histogram

Digital AER

AER Gen

Analog Test Control

Time Stamper

Test Pattern

Subsampler

Blocking Area

oReducing Power

oSummary

Good Event Quality: Objective

Clean and perfect edge map Obtained by subtracting frames on a High Frame-Rate CIS camera

Good Event Quality

Generated Events

“Meaningful” Events

Generated Events

Good Event Quality

Redundant Events o Falsely Generated o Multiple detections o Not applicable

Examples: o Noise; Bad Pixels o Response to high

contrast change o Light source flicker;

Background motion ; Too many events

Missing Events o Not/badly detected o Discarded

Examples: o Overflow,

Timestamp errors o Filtered (mitigating

redundant events)

Generated Events

Good Event Quality

Redundant Events o Falsely Generated o Multiple detections o Not applicable

Examples: o Noise; Bad Pixels o Response to high

contrast change o Light source flicker;

Background motion ; Too many events

Missing Events o Not/badly detected o Discarded

Examples: o Overflow,

Timestamp errors o Filtered (mitigating

redundant events)

Redundant Events: Bad Pixels; Noise

Basic model:

o Bad Pixel: Specific Pixels that generate events at high frequency (regardless of the scene)

o Noise: Random pixels that generate events at low frequency (regardless of the scene)

Actual behavior:

Arbitrary separation between bad pixels and noise 10% of max number of events

Number of pixels per event rate

Events/sec

Event Histogram: DVS filming a white wall

Bad Pixel Suppression Example:

Noise Suppression Example:

Redundant Events: Multiple detections

LogAMP of DVS Gen2 (ISSCC’17)

MLOG2 MSFVLOG

ΔVTH_ON

step response

o Overall pixel BW (speed) is limited by LogAMP BW

o LogAMP BW is mainly affected by 1) Its structure; 2) The amount of photocurrent

o Redundant events generated at low light (IPD) on @ large contrast change

Redundant Events: Multiple detections

ΔVTH_ON

step response

o Overall pixel BW (speed) is limited by LogAMP BW

o LogAMP BW is mainly affected by 1) Its structure; 2) The amount of photocurrent

o Redundant events generated at low light (IPD) on @ large contrast change

Solutions:

o Structure: Increase ~x5-x10 LogAMP BW by new structure

o Limit photocurrent: Introduce “HOLD” signal to pixel

“Hold” function

Redundant Events: Light Source Flicker

o Light Source Flicker: Events that are not generated by motion but by flickering lights (with different frequencies)

o Note: This has to be differentiated from periodic motion (for example a waving hand)

Redundant Events: Light Source Flicker

Flicker Suppression Example:

Missing Events: Overflow, Timestamp Errors

o Overflow: Number of events describing the scene change, exceeds the circuit maximal throughput.

25% active area, 1ms refresh rate:

90% active area, 1ms refresh rate:

#1040 #1041 #1042

#109 #110 #110

Missing Events: Overflow, Timestamp Errors

o Timestamp Errors: Events grouped according to incorrect timestamp

609556500 609557000 609557500 609558000 609558500 609559000 609559500

Frame_9 Frame_10column

Solution: Pixel Readout Revision

o Solution: Number

… Pixel Array

… Prev.: Unfair arbitration, no frame context

Column selection

Column Arbiter

Readout controls Gen1 & Gen2 Gen3

Column selection Unfair arbitration scheme Scanning scheme

In-pixel memory Free running (no event hold) Free running (no event memory)

Global hold (all pixels in pixel array)

Pixel reset

Pixels with an event in a column Pixels with an event in a column

- All pixels in a column

- Global reset (all pixels in pixel array)

New: Scanning scheme, frame context options

… Pixel Array

Column selection

Scanning order : 0 to 639

Pixel Readout: Timestamp Error Comparison

Average Event rate

Input: Poisson random events

100M 1G

1 msec

10 usec

100 usec

10M 1M

1 usec

Selective scan without frame buffer

Selective scan with frame buffer

Full scan without frame buffer

Full scan with frame buffer

Full scan with In-Pixel memory without frame buffer

oReducing Power

oSummary

Data Throughput: DVS Low Event Rate Example

Less Events Easier to process

CIS 100fps CIS 30fps Moving DVS Static DVS + Strongmotion

Static DVS + weakmotion

Static DVS, no motion

Event rate per second

Data Bandwidth @ 32 bit/event Encoding

DVS AER 32bit per event

CIS 100fps CIS 30fps Moving DVS Static DVS + Strongmotion

Bits per seconds

Group Address Event Representation (GAER)

Y event

group_event

… …

AER (32 bits/event)

Address Event Representation (TS, X, Y, event) o P: packet encoding (2bit) o TS : timestamp (10bit) o X : column address : 0 ~ 639 (10bit) o Y : row address : 0 ~ 479 (9bit) o event : 1-bit. on/off

GAER (~64-4 bits/event – content dependent) Group AER (TS, X, G, group_event)

o P: packet encoding (10bit group, 12bit per col) o TS: timestamp (10bit) o X : column address : 1 ~ 640 (10bit) o G : group_address : 1 ~ 60 (6bit) o group_event : a bundle of 8 events. 16-bits

o on_evnet : 8-bits o off_event : 8-bits

Data throughput: AER 32 bit/event

DVS AER 32bit per event

CIS 100fps CIS 30fps Moving DVS, 1000 fps Static DVS + Strongmotion

Bits per seconds

Data throughput: GAER ~12 bit / event

DVS GAER 12bit per event (average)

Bits per seconds

Data throughput: Improved GAER ~8 bit / event

DVS GAER with offset based column 8 bit per event (average)

Bits per seconds

oReducing Power

oSummary

Event Processing Acceleration

o GPU/DSP are not optimized for event-based computer vision:

o Considering HW accelerator/processor with a DVS-specific architecture

GPU/DSP Requirements for Event Processing

Optimized for High arithmetic density and high throughput

Low latency

Memory access Privilege coalesced data Optimized for random access

Concept: Less but More Informative Events

o Programmable feature

o A feature event is sent by the sensor when the feature appears strongly

o For example: direction detection (can be used to produce Histogram of Gradients in the Host for human detection)

Horizontal (yellow) and vertical (cyan): Slash (red) and Backslash (green):

Applied on Human captured is a DVS movie:

oReducing Power

oSummary

Summary: Market Driven Requirements

Requirements Samsung Early Prototypes Samsung DVS Product Roadmap (2017,2018)

Low Cost Large Pixel No standard for Quality / Die sorting

Similar size to 16-8MP mobile camera

Minimal Module Size Large, due to large Pixel -> Large optical format

Smaller, Optical size reduced to 1/3” (VGA)

Ultra Low Power Low power Lower power

Minimal Optical Format Large, due to large Pixel Reduced to 1/3” (VGA)

Good Event Quality

Redundant events (especially at low light) Motion artifacts, Timing accuracy Noise, Flicker

Improved pixel circuit. On chip Bad Pixel, Noise and flicker suppression. External Triggered Scan, Global reset & hold capable

Low Data Rate Scene dependent Might exceed CIS typical throughput

Compressed format in the bounds of slow CIS typical throughput

Edge device vision processing Not available Considering low level features

Thank You!

Dynamic Vision Sensor - Robotics and Perception...

Documents

Automatic Calibration of Hybrid Dynamic Vision …cdn.intechopen.com/pdfs/5783.pdfAutomatic Calibration of Hybrid Dynamic Vision System for High Resolution Object Tracking 123 The

Improved Cooperative Stereo Matching for Dynamic Vision ... · Improved Cooperative Stereo Matching for Dynamic Vision Sensors with Ground Truth Evaluation Ewa Piatkowska1, Juergen

Vision-based Tracking with Dynamic Structured Light for Video See-through … · 1998-11-20 · iii MARK ALAN LIVINGSTON. Vision-based Tracking with Dynamic Structured Light for Video

Cisco Vision Dynamic Signage Solution · solution. The Cisco Vision Dynamic Signage solution enables the integration and automated delivery of customized and dynamic content from

Dynamic and static hand gesture recognition in computer vision

Vision-Based Retrieval of Dynamic Hand Gestures

Dynamic Textures - UCLA Vision Labvision.ucla.edu/papers/dorettoCWS03.pdf · Dynamic Textures GIANFRANCO DORETTO ... Keywords: textures, dynamic scene analysis, 3D textures, ... main

Dynamic Integration of Motion and Neural Data to Capture ... · Research Vision and Goals •DDDAS approach to Catalyze the development of Computational Dynamic Integration of Motion

The Vision SSM Models Use Cases Activity Models Dynamic Models Object Models Programs Databases Business Computing

Dynamic Vision Training Module for i-Learning in ...spectronet.de/story_docs/vortraege_2005/050921_imeko/session_5/501...DYNAMIC VISION TRAINING MODULE FOR i-LEARNING IN INDUSTRIAL

Cognitive Radio for Dynamic Spectrum Access Vision … Radio for Dynamic Spectrum Access ... Joseph Mitola III: ... What is the meaning of “Dynamic”? Nobody knows

DYNAMIC STEREO VISION FOR INTERSECTION ASSISTANCE … · DYNAMIC STEREO VISION FOR INTERSECTION ASSISTANCE 1Franke, Uwe*, 2Rabe, Clemens, 1Gehrig, Stefan, 3Badino, Hernan, …

CHIME LEAD Forum DC 2015 - "HIT Leader 3.0 Cornerstone: Setting Vision and Strategy in Dynamic Times"

Vision and SLAM on a highly dynamic mobile two-wheeled robot · Vision and SLAM on a highly dynamic mobile two-wheeled robot Patrick Suhm Course of Study: Technische Kybernetik Examiner:

Dynamic Business 2 0 Vision Paper

Image Thresholding and Contour Detection with Dynamic ... · Image Thresholding and Contour Detection with Dynamic Background Selection for Inspection Tasks in Machine Vision K. Židek,

Cisco Vision Dynamic Signage Director …...Under Cisco Vision Dynamic Signage Director Server Support, page 10, revised Note to strongly recommend virtual environments for future

Dynamic Vision Brochure

Vision Lecture 6 Notes: Color vision and adaptation · Central questions about color vision and adaptation: ... Dynamic computer test (City University Dynamic Color Vision Test) 40

Event-Based SLAM at Slamcore - Davide Scaramuzzarpg.ifi.uzh.ch/docs/ICRA17workshop/Slamcore.pdf · 2017-06-12 · Enabling Robotic, AR and VR Systems Slamcore Limited | contact@slamcore.com