LATTICE CAMERA SOLUTIONS - T3LAB | Technology … Pixel Correction with-2D Noise Reduction 5x5 HQ DeBayer Right Hand Screen Sensor Port with -Adaptive Black Level Correction Linearization

LATTICE CAMERA SOLUTIONSVito Ruocco, Regional Sales Manager

[email protected]

Marco Boschini, Field Application Engineer [email protected]

Bologna - Consorzio T3LAB, 28 giugno 2012

Page: 2 Lattice Semiconductor Confidential

AGENDA

• Introduction to Cameras

• HDR-60 Video Camera Development KIT

• HiSPi Sensor Interfacing

• MACHXO2 Dual Sensor Interface Board (DSIB)

• @XO2 Speed Campaign


AGENDA







MARKET & TECHNOLOGY OVERVIEWSURVEILLANCE MARKET REVENUE

Overall Video Surveillance Revenue,World Market, Forecast: 2008 to 2015

Video Surveillance Camera Shipments by Type

World Market, Forecast: 2010 to 2015

Type Shipments 2010 2011 2012 2013 2014 2015

CAGR

(08-15)

CCTV (Millions) 29 27 25 24 22 21 -4%

IP (Millions) 3 6 13 24 38 52 63%

Total (Millions) 32 33 38 48 60 73 14%

Source: ABI Research

Video Surveillance Camera Revenue by Region

World Market, Forecast: 2010 to 2015

Region Revenue 2010 2011 2012 2013 2014 2015

CAGR

(08-15)

United States ($ Millions) 1,922 2,100 2,314 2,845 3,430 3,928 12%

Canada ($ Millions) 306 333 373 469 581 682 10%

Europe ($ Millions) 2,636 2,635 2,566 2,875 3,146 3,559 4%

Asia-Pacific ($ Millions) 1,409 1,581 1,785 2,316 2,988 3,600 16%

Rest of World ($ Millions) 1,079 1,148 1,300 1,592 2,032 2,357 15%

Total ($ Millions) 7,351 7,797 8,337 10,098 12,177 14,126 10%

Source: ABI Research

Video Surveillance Storage Revenue by Type, World Market, Forecast: 2008 to 2015

nearly doubling in 3 years!

Very strong in all regions!


MARKET & TECHNOLOGY OVERVIEWTRENDS

TRENDS/DRIVERS

• Wireless Video: reduces infra structure cost, sometimes the only option, best suited for city surveillance

• Cloud-Based Video Surveillance (Video Surveillance as a Service (VSaaS))

• Video Analytics: Video Content Analysis (VCA) beyond Security (automotive, defense, medical, consumer, digital signage)

• From Analog to IP: Tipping Point for Network Video: Global in 2015, 2013 in Americas, 2012 in EMEA, Middle-East and Russia already tipped.

• From HD to 3D: 3D offers depth perception

• “Affordable” Thermal Imaging/Surveillance: New phenomenon. Commercial IR market is heating up.

• HDcctv gaining momentum: An open standard for transmission of HD video using coax-cable. Built upon HD-SDI standard. Not expected to impact the adoption of network cameras in the short term.

• From SD IP to HD IP (IMS Research: 30% HD/MP in 2011, 60% HD/MP by 2015)

• HD the resolution sweat spot: HD offers increased resolution but not to the extreme level where the cost of storage and bandwidth become unmanageable.

• ISP: From single sensor to multi-sensor processing (Dual and Quad)


ISP

Flash SDRAM

SDI Tx

SDI Camera

SD/HD/3GSDI SDI Camera (HDcctv)

• Highest Quality• Uncompressed• Coax cable based

SensorI/FLens

ISP

Flash SDRAM

DACLens

Analog Camera

CVBSVGA/RGB

Analog Camera (cctv)• Highest volume• Lowest cost• Coax cable based

SensorI/F

ISP

Flash SDRAM

NetworkProcessor

IP Camera

RJ-4510/100MBps

IP Camera (SD/MP)• Video, Control, (& Power)

over one cable (CAT5)• Higher maintenance• Requires network / skillset

SensorI/FLens

MARKET & TECHNOLOGY OVERVIEWCAMERA ARCHITECTURES

Lattice


AGENDA







Price $399

Industry Leading Auto

Exposure,High Dynamic

Range &Auto White

Balance

Full HD 1080p60

Capable HDR ImageSignal

Processing (ISP) Pipeline

HDR-60 VIDEO CAMERA DEVELOPMENT KITOVERVIEW

Other Applications

: Traffic, Automotive,

Video Conferencin

g

Major Application: Security &

Surveillance Cameras

Fully Production-Ready HD

Video Camera

with HDR


HDR-60 VIDEO CAMERA DEVELOPMENT KITTRI-PARTITE PARTNERSHIP

Lattice Semiconductor

Corp.

• Lattice ECP3-70 & HDMI PHY

Helion GmbH

• ‘IONOS’ Image Signal Processing IP

Aptina LLC

• Aptina A-1000 HDR Sensor


HELION AT A GLANCE

More than

110

Image Processing IP cores available

• Founded 2003 as a spin-off company from the Fraunhofer Institute

• More than 15 years experience in HDR/WDR and image pre-processing

• Engineering Services in customized digital cameras and display systems

• Complete image processing solution from sensor to display, called IONOS™

• Consulting in image sensor configuration and FPGA programming

• Solutions in the field of Automotive, Medical, Security, Industrial

• Inventor of the WDR/HDR-BLENDFEST™ technology and the IONOS™ IP suite

• Exclusive Lattice IP core partner for image pre-processing

• Supported image sensors, like: Aptina, Panasonic, Sony, OmniVision, Cypress/OnSemi, and more...


HDR-60 VIDEO CAMERA DEVELOPMENT KITKIT CONTENTS

Lattice HDR-60 Video Camera Development Kit

• Lattice HDR-60 Video Camera Main Board

• Lattice Nanovesta Sensor Board

• 1 x HDMI Cable

• 1x HDMI – DVI Adapter

• 2 x Standard USB Programming Cables

• Universal Power Supply

• Quick Start Guide


HDR-60 VIDEO CAMERA DEVELOPMENT KITDEMO SETUP PLUG-AND-PLAY DEMO

HDMI / DVI cable

HDMI / DVI Monitor


COLOR IMAGE SIGNAL PROCESSING (ISP)PIPELINE FROM HELION


HDR-60 VIDEO CAMERA DEVELOPMENT KITHIGH DYNAMIC RANGE (HDR)

CAR?Person?

Under Exposed Over ExposedHDR

The Power of High Dynamic Range (HDR) High Dynamic Range (HDR) Examples:

HDR (also called WDR) enables the capture and display of both bright and dark areas in the same frame, in a way that there

are details in both areas, i.e. bright areas are not saturated, and dark areas are not too dark.


HDR-60 VIDEO CAMERA DEVELOPMENT KITDEMO SPLIT SCREEN DIFFERENCES

Left Hand Screen

�Sensor Port with- Adaptive Black Level Correction

�Linearization 12 > 20

�Defective Pixel Correction with- 2D Noise Reduction

�5x5 HQ DeBayer

Right Hand Screen

�Sensor Port with- Adaptive Black Level Correction

�Linearization 12 > 20

�Defective Pixel Correction with- 2D Noise Reduction

�5x5 HQ DeBayer

�Auto Exposure

�AWB

�CCM

�Gamma

�HDR Tone Mapping

�Overlay


HDR-60 VIDEO CAMERA DEVELOPMENT KIT32-BIT PIPELINE LOWERS LATENCY, ELIMINATES NOISE, REDUCES COST

Competition 18 bit PipeLattice-Helion 32-bit Pipe

De Bayer

CCM

AWB

FRAME BUFFER (adds cost)

18-bit ISP Pipeline MUST do 3D NR; else too much noise. But still can’t dare to throw away the last couple of noisy bits.Significant noise still present in output

3D Noise Reduction

LEGEND:

= Latency= Clean bit= Noisy bit

Internal PipelineWidth is 18 bit.Lost 2 bits already!.Low light performance now impacted.

(Each operation introduces noise)

32-bit ISP Pipeline nevereven maps the noisy bits to final RGB. Negligibleor no noise present in output.

(Each operation introduces noise)

ISPISP

Inside the

SENSOR

Internal PipelineWidth is 32 bit.

Pipeline is now very noisy, so,

can’t use it. Must do 3D NR.

Back and forth between buffer memory adds latency


HDR-60 VIDEO CAMERA DEVELOPMENT KITFPGA RESOURCE USAGE

IP Block LUT EBR 18Kbit 18x18 Multiplier

Sensor Port with Auto Black Level Correction 1915

Linearization 12>20 100

Defective Pixel Correction & 2D Noise Reduction (2DNR) 1539 12

De-Bayer 5 x 5 HQ 1959 12

Color Correction Matrix (CCM) 1712 18

Statistics Engine 2000

Fast Auto Exposure* (AE) - - -

Auto White Balance & Saturation Control (AWB) 2167 2 8

High Dynamic Range (HDR) 1612 12 16

Gamma Correction 650 1 2

Overlay Helion Logo 336 5 3

Lattice Mico 32** 1800 22 2

I2C Master 132

Lattice HDMI PHY 276

TOTAL 16198 66 49

Percentage ECP3-35*** Resources Used 49.08% 91.67% 76.56%

* Fast AE is done by LM32 with data from Statistics Engine

** EBR for LM32 depends on code size. 22 used in demo*** Device on board is ECP3-70. Percentage usage of ECP3-35 shown since design fits an ECP3-35


HELION’S IONOS IP SUITE

18

Helion’s IONOS-IP-CORE OVERVIEW

Defect Pixel CorrectionDPC

Color Filter Interpolation

Color Correction Matrix

Gamma Correction

Adaptive Black Level Correction

High Dynamic Range Imaging

Statistics

Overlay

Color Space Conversion

Automatic White Balance

BLC

CFI

CCM

Υ

CSC

AWB

HDRI

STAT

OSD

TG

AE

FRC

LIN

NR

AC

Test Pattern

Auto Exposure

Frame Rate Converter

Linearisation

Noise Reduction

Aperture Correction

LSC Lens Shading Correction

OI Output Interface

MI Memory Interface

SI Sensor Interface

COMPREHENSIVE IP LIBRARY:

Helion is providing a comprehensive intellectual property (IP) library. Containing more than 90 individual pieces of IP, the library is available for licensing either entirely or in parts. Developed by Helion, the library provides complete sensor to display ISP support.

All licensing is handled through Lattice!

Two Step licensing process:1. IP Evaluation License2. Full IP License


HELION IP LICENSING MODEL

Customer wants to evaluate (Helion IP Evaluation Licensing Agreement)

Customer signs IP evaluation license agreement with Lattice:⇒ Customer will be given access to Helion FTP siteFTP site includes:• Complete ISP pipeline example projects• IP cores (time-bombed NGO format)• Documentation

Customer can evaluate and start designing a project!

No commitment required! No cost involved!

Customer wants to license IP (Helion IP Licensing Agreement)

Customer signs IP license agreement with Lattice:⇒Customer will be given license file for defusing IP time-bomb⇒Customer commits to ONLY use special part number devices with Helion ISP!

• special part numbers allow Lattice to track number of devices used with Helion IP and compensate Helion for IP usage (per device royalties)

Customer does not pay any Royalties, or NRE’s! No MOQ’s required!

⇒No Risk!

• Available part numbers for Helion IP usage: LFE3-35EA-8FTN256IAEJ ; LFE3-35EA-8FN484IAEJ ; LFE3-70EA-8FN484IAEJ ; LFE3-95EA-8FN484IAEJ ; LFE3-150EA-8FN672IAEJ


Features Benefits

Fully Production Ready. Free Demo Bitstream, Schematics & Layout.Fits Commercially Available Camera Housings.

Accelerated Time to Market

Lowest Cost BOM compared to both competition FPGA and DSP, for 1080p60 HDR Camera. Needs No External Frame Buffer.

Full HD* Functionality and HDR at Lowest System Cost

Industry Leading Sensor Roadmap Support: up to 16MP Future-proof Kit Investment

Industry Leading:- Auto Exposure- Auto White Balance- HDR Range

Differentiation at Low System Cost

Available Comprehensive Wishbone Compliant

ISP LibraryVersatile Product Offering with

Single Source, fully-compatible IP

Broadcom BroadReach™ PHY and BNC ConnectorLegacy Cable Infrastructure Support.

Coax-ready for up to 700m Signal Travel.

ECP3-70 On Board.FTDI Chip On Board.RJ45 Ethernet Connector On Board.

Room to add H.264 Codec, Video Analytics IP and Customer’s own IP.Easy Programming via regular USB Cable

On Board Twin-Sensor Support Connectors Video Stitching Applications Support

HDR-60 VIDEO CAMERA DEVELOPMENT KITFEATURES & CUSTOMER BENEFITS

* Kit is 1080p60 capable. Current sensor on board is 720p.


HDR-60 VIDEO CAMERA DEVELOPMENT KITSUMMARY

Fully production ready design

HDR 1080p60 capable FPGA stream processing

Industry-leading features:

- Fast auto exposure

- 120dB+ High Dynamic Range

- Support for up to 16MP sensors

Comprehensive Image Signal Processing library

Low cost BOM


AGENDA







SENSOR INTERFACINGHIGH SPEED SERIAL PIXEL (HISPI)

Background:

• Traditional parallel sensor interface can no longer handle increased sensor resolutions and frame rates. New sensor interfaces have been introduced by sensor manufacturers.

• A majority of ISP devices support only parallel sensor interface and require a bridge device converting high-speed serial data to a parallel format

• Lattice XO2 and XP2 FPGA’s provide an efficient and cost-effective solution for converting HiSPi-to-parallel and subLVDS-to-parallel bridging.

Available Reference Designs:

• Aptina HiSPi sensor I/F to Parallel bridge:

• MT9024 , AR0331

• Panasonic subLVDS sensor I/F to Parallel bridge:

• MN34081, MN34041

• Sony subLVDS sensor I/F to Parallel bridge:

• IMX036, IMX136

• Dual-Sensor Bridges:

• 2x Aptina MT9024 HiSPi to Parallel

http://www.latticesemi.com/products/intellectualproperty/referencedesigns/aptinahispitoparallelsens/index.cfm

Coming soon: MIPI Interface/Bridges


APTINA HISPI TO TI DSP BRIDGEREFERENCE DESIGN FEATURES

Complete reference design

Designed to emulate parallel sensor output: output bus widths 8, 10, 12 or 14 bits

Supports HiSPi formats Packetized and Streaming SP: from 1-4 lanes running up to 700Mbps each

Bridge device offered in a small 132csBGA (8x8mm). TQFP packages also available

Requires no external PROM

Tested with TI TMS320DM3X5 and Aptina A1000 sensor

Parallel interface can be configured for 1.8v, 2.5v or 3.3v LVCMOS levels

Local Tone Mapping available as optional enhancement


AGENDA







• MachXO2-4000 PLD for sensor interfacing and driving a parallel ISP bus

• LP SDRAM: 16-bit data over a 32Mb address space

• Built-in USB 2.0 download to MachXO2

• Can be configured for a flywire ispDOWNLOAD® cable connection

• HiSPi and other serial sensor interfaces can be supported

• 36-pin flat ribbon cable connector for TI IPNC camera connections

• 3.3V, 2.5V and 1.8V voltages are possible for various MachXO2 I/O banks

MACHXO2 DSIBDUAL SENSOR INTERFACE BOARD


Some common applications and uses for the MachXO2 DSIB

• 3D stereoscopic video cameras

• Aftermarket automotive black box DVR

• 3D camera for analytics in security/surveillance applications

• Traffic cameras where one sensor records video and the other sensor takes the photo

• Interfacing to the Texas Instruments (TI) IPNC DM812x, DM385 or DM385 camera base boards

• Evaluation of MachXO2 device interfacing to sub-LVDS signals

• Sensor bridges using a MachXO2 device to convert from serial sensors to parallel buses

MACHXO2 DSIBCOMMON APPLICATIONS


MACHXO2 DSIBDEMO

• Targeting dual-sensor & stereoscopic video applications

• Demo supports the merging and synchronizing of two image sensors and outputting one bus for an ISP to process.

• DSIB plugs into HDR-60 mainboard

• DSIB can be used with TI DSP board or Helion ISP

Picture: HDR-60 mainboard w/ Dual-Sensor Headboard

Lattice’ XO2 based Dual Sensor Board (DSIB):TI DSP Interface

NanoVestaSensor #1(Right Eye)

NanoVestaSensor #1(Right Eye)

MachXO2 Dual-Sensor Interface Board (DSIB)

HDR-60Baseboard


AGENDA







@MACHXO2 SPEED CAMPAIGN

1. www.latticesemi.com

2. Home>Products>CPLD>MachXO2

3. Click on following picture


@MACHXO2 SPEED CAMPAIGN

FunctionRequired

LUTs (MachXO)

Required LUTs

(MachXO2)

SPI Master 115 0

I2C Master 234 0

Timer/Counter 50-80 0

I2C Controllers

• Two controllers• Configurable master or slave• Programmable address• Up to 400KHz• Clock stretching supported

SPI Controller

• Configurable master or slave• Four wire interface• Up to 50Mbps

Timer/Counter

• 16-bit counter• Clock scalar (8, 64, 256, 1024)• Normal, clear on match,

Fast PWM, phase correct PWM, watchdog mode


@MACHXO2 SPEED CAMPAIGNEMBEDDED FUNCTION BLOCK


WHAT’S NEW WITH MACHXO2REFERENCE & DEMO DESIGNS, APP NOTES

Topic Function Type Doc # Board(s) Code Title

XO2 I2C Master Design Demo Design UG54 Pico CMachXO2 Master I2C Demo Using 'C' &

LM8

Reference Design RD1124 - RTLI2C Slave Peripheral Implementation

Using Lattice EFB

Demo Design UG55 Pico (x2) RTLMachXO2 Hardened I2C Master/Slave

Demo

XO2 SPI Master Design Demo Design UG54 Pico CMachXO2 Master SPI Demo Using 'C' &

LM8

Reference Design RD1125 - RTLSPI Slave Peripheral Implementation

Using Lattice EFB

Demo Design UG56 Pico (x2) RTLMachXO2 Hardened SPI Master/Slave

Demo

UFM Access via WB Reference Design RD1126 - RTL RAM-Type Interface for Embedded UFM

CFG Access via WB Demo Design UG57 - RTL MachXO2 Programming Via Wishbone

Embedded I2C

UFM & CFG AccessReference Design RD1129 Versa, Pico C

MachXO2 I2C Embedded Programming

Access Firmware

Accessing Power

Controller Through I2C,

SPI & User Logic

Demo Design UG58 Pico RTL MachXO2 Low Power Control Demo

- Usage Guide TN1204 - -Mach XO2 Programming

and Configuration

- Usage Guide TN1205 - -User Flash Memory & Hardened

Control Functions in Mach XO2

-Reference

GuideTN1246 - -

User Flash Memory & Hardened

Control Functions in Mach XO2

Application

Notes

I2C

XO2 I2C Slave Design

XO2 SPI Slave Design

SPI

Wishbone

Other


LOW-COST HARDWARE KITSINCLUDING $29 PICO DEVELOPMENT KIT

MachXO2 Family

• Low Cost

• Low Power

• High System Integration

Pico Development Kit

• Discounted from $49 to $29

Embedded Function Block

• I2C, SPI, user Flash memory

• Pre-designed, pre-verified hard IP

• Saves design time & up to 429 LUTs

• New reference designs

• New re-usable demonstration designs

EFB

Download reference designs and Diamond design software today at www.latticesemi.com

THANK YOU


HELION ISP IP BLOCK DESCRIPTIONS

Lattice


IP BLOCK DESCRIPTIONSENSOR PORT

SensorPort

Basic Description:• Configures Image Sensor for Detection• Captures Image Data

• Detects the image sensor by scanning the I2C-bus. • Reads out the version and subversion register of the

image sensor. • Resets the sensor, loads the individual setup settings table

into the sensor

Advanced Information:• Maps sensor physical registers to IP block’s internal logical

registers • (e.g. integration time to the physical register

number, e.g. R0x3012)• Configures after each readout of a frame the following

registers: • Integration Times• Gains• Readout Coordinates

• Stores (FIFO) one line of incoming pixel data with pixel clock speed and reads outs this FIFO with a lower internal speed.

Integration Time

Gain

Coordinates

I2C

SENSORPHYSICAL

REGISTERS

LOGICALREGISTER

S

SENSOR IP BLOCK


IP BLOCK DESCRIPTION LINEARIZATION

Linear10 > 12

Basic Description:• Decompresses Compressed Sensor Data

Advanced Information:• Linearization 10 – 12 bit• For MT9V022/V023/V024/V032/V033/V034 use only

• In case of the use of the 12Bit mode:• Decompands the companded image data in the

datastream • Applies an offset correction• At the output of this IP the datastream behavior is linear.

• Linearization 12 - 20 bit • For A-1000 HDR-Mode only• Selects the necessary decompanding scheme

(similar to Linear 10->12), • Depending on the selected HDR companding modes

Linear12 > 20


IP BLOCK DESCRIPTION DEFECTIVE PIXEL CORRECTION

Def-2D

Basic Description:• Determines defective pixels• Corrects defective pixels by examining neighboring

pixels and modifying defective pixel

Advanced Information :• Stores (FIFO) 4 lines of incoming RAW Bayer data• For each pixel in the pixel datastream, compares the

neighboring pixels of the same color

• After this comparison the core 'knows' whether the actual pixel is a hot/warm or a cold pixel

• In this case the pixel values is replaced by the median value of the same color neighbors.


IP BLOCK DESCRIPTION DE-BAYERING (DE-MOSAIC)

Basic Description:Since each pixel has a filter with one of three colors (R/G/B),two-thirds of the color data is missing from each. To obtain afull-color image, various demosaicing algorithms can be used tointerpolate a set of complete red, green, and blue values foreach pixel.

Advanced Information:• De-Bayering 3x3 Billinear

• Color Filter Array (CFA) Demosaic with 3x3 Core• Logic-Efficient Billinear Approach• Stores (FIFO) 2 lines of incoming RAW Bayer data• The colors (red, green and blue) are interpolated in a bilinear way

• i.e. averaging of neighbor pixels

• De-Bayering 5x5 High Quality• Color Filter Array (CFA) Demosaic with 5x5 Core• High Image Quality with Fewer Image Artifacts • Stores (FIFO) 4 lines of incoming RAW Bayer data• Depending on the position of the actual pixel (red, green1, green2,

blue) and the wished for resulting color (red, green, blue), a different weighting scheme is used.

Debay3x3 Bi

Debay5x5 HQ


IP BLOCK DESCRIPTION CCM: COLOR CORRECTION MATRIX

CCMCCM

Many current image sensors provide incorrect color rendition due to ‘crosscolor’ effects. This effects leads to wrong color images (e.g. green with too much blue).

CCM modifies each single color channel by a 3x3 correction matrix:red-out = rr*red + rg*green + rb*blue

green-out= gr*red + gg*green + gb*blue

blue-out = br*red + bg*green + bb*blue

Builds the color statistics with selectable thresholds (min/max illumination, min/max color ratio). Adjusts the internal digital fine gain amplifiers for red gain and blue gain inside the IP core based on the positions of the individual Bayer filter elements.


IP BLOCK DESCRIPTION AWB: AUTO WHITE BALANCE

Basic Description:Cameras are not good at recognizing colors. AWB adjusts other colors in an image with reference to a known white color in the image. AWB determines white by examining the frequency (therefore wavelength) of incoming light

Advanced Information:• Builds the color statistics of a selectable image region (ROI) with selectable thresholds (min/max illumination, min/max color ratio)

• Adjusts the internal digital fine gain amplifiers for red gain and blue gain inside the IP core. Target is a so called grey world approach.

AWB


IP BLOCK DESCRIPTION HDR: HIGH DYNAMIC RANGE

Dynamic Range = Ratio of expressed in dB

Higher the dynamic range the better we can see detail in both light and dark areas. HDR combines multiple exposures of different duration and blends to form a clear image for both light and dark areas

Advanced Information:• High Dynamic Range Processing With Tone Mapping• Builds internal image statistics (histogram, average and

median values) to calculate for each frame two adaptive global tone map transfer curves and uses local tone mapping for contrast enhancements.

HDR

Intensity of brightest areaIntensity of darkest area


120DB INTRA-SCENE / ~170DB INTER-SCENE DYNAMIC RANGE

Intra-scene dynamic range:120dB intra scene dynamic range, based on 20 bits/pixel

Measured number from Aptina: 1 bit � 6dB. 20*6dB = 120dB

In linear mode the sensor has fewer than 12 bit intra scene dynamic range (<72 dB). A-1000 uses 3 integration times / readout. Each successive integration time is 16x shorter, shifting the dynamic range 4 bits (+24dB) at a time. Leads to an intra-scene dynamic range of 72+24+24 = 120dB

Inter-scene dynamic range:

120dB might be OK for a specific scene, but it is necessary to shift and adapt the intra-scene dynamic range when the scene illumination is changing (e.g. from night to day). e.g.: Auto manufactures BMW and Audi have a minimum system dynamic range requirement of 150dB.

The intra scene (in one image, but taken with 3 internal multi exposures) dynamic range is 120dB. The AE IP block shifts this intra scene dynamic range by setting the first integration time (the other 2 times are internally determined by the sensor).

Example:

By using 480 lines for integration, the minimum detectable illumination is E_min0, the maximum illumination is then:

E_max0 ~ E_min0 * 2^20, because of 120 dB intra scene dynamic range

By using instead 1 line for integration (selected by the AE module), the minimum detectable illumination is then:

E_min1 = Emin0 / 480 and the maximum illumination is to E_max1 = E_max0 / 480.

The definition of dynamic range is the ratio between the maximum signal to the minimum signal.

This leads to a system dynamic range of:

DR_system = 20*log( E_max0 / E_min1 ) dB = 20*log ( 2^20 * 480) dB = 120dB + 53.6dB = 173.6dB

120dB

x y

SCENE 1

x1

y1

x2

y2

x3

y3

170dB

120dB

120dB

120dB

SCENE 1

SCENE 2

SCENE 3


IP BLOCK DESCRIPTION GAMMA CORRECTION

GammaCorr

Corrects for non-linear response to image luminance between sensor and display, by applying a pre-distortion correction which becomes a linear response after again being distorted at the display end.

Uses a lookup table with interpolated values to apply a gamma correction for each color channel. The gamma value is selectable.


IP BLOCK DESCRIPTION OVERLAY

Overlay

Basic Description:• Overlays text & graphics on the video

image for display on screen.

Advanced Information:Overlay Engine

• Overlay of Digital Image Data • Free Selectable Size and Position • Up to 256 Levels of Transparency• Free Selectable Genlock Color


HDR DEMO CONFIGURATIONWITH APTINA MT9M033 (A-1000/MI-1000) SENSOR

- HDR alpha release 60fps 1200x720 readout with companded transfer curve. One sensor pixel is one TFT pixel.

- Streaming Image Data Path:

- No External Frame Buffer needed.

- Quality at a lower total system cost.

- System latency is extremely low (0.099ms – 0.510ms depending on IP blocks used)

- Full HD 1080p/60fps capable (A-1000 sensor is 720p)

- Integration time ratio is 16x by 16x

- 120dB Image Dynamic Range

- Very large range allows optimal light and dark scene processing

- 20 bits per color channel

- Independent of the incoming light

- For brightness beyond a certain threshold, the F# / Iris can be changed without any visible change on image brightness

- Fast Auto Exposure: Executed in 3 frames; no visible display bloom or blackout when going from dark to light and light to dark respectively

- Overlay functions are used to describe the stream processing functions currently active.


VIDEO ANALYTICSROADMAP

Lattice


VIDEO ANALYTICS IP PLANNED

Baseline VA Functions Planned

� Intelligent Video Motion Detector

- Analyze in real-time

- Filter out noise such as changing light, moving leaves on trees and animal movements during detection

- Provide real-time alerts; Start/Stop commands to DVR/NVR

� Intrusion Detector

- Automatic Perimeter Monitoring

- Monitor user-selectable zones for people or vehicle intrusion

- Filter out noise

- Provide real-time alerts

� Video counter

- Process video-streams in real-time

- Continuous multi-directional counting

- Count people, objects, vehicles

- Generate reports on people / vehicular traffic patterns

� Camera Tamper/Sabotage Detector

- Automatically Detect camera failure or Sabotage

- Detect partial blocking, complete blocking, drastic camera angle change

� Enhanced VA Functions Planned

- Object Left behind

- Object Tracking with intelligent zoom

- Intelligent Search

� Scan archived video for specific parameters: DVR/NVR application


LIST OF AVAILABLE IP BLOCKS BY FUNCTION

LatticeHelion


HELION IP – COLOR PIPELINE


HELION IP – IMAGE ENHANCEMENT


HELION IP – INTERFACES


HELION IP – PERIPHERALS & OVERLAY

THANK YOU

Documents

LATTICE CAMERA SOLUTIONS - T3LAB | Technology … Pixel Correction with-2D Noise Reduction 5x5 HQ DeBayer Right Hand Screen Sensor Port with -Adaptive Black Level Correction Linearization