LATTICE CAMERA SOLUTIONSVito Ruocco, Regional Sales Manager
Marco Boschini, Field Application Engineer [email protected]
Bologna - Consorzio T3LAB, 28 giugno 2012
Page: 2 Lattice Semiconductor Confidential
AGENDA
• Introduction to Cameras
• HDR-60 Video Camera Development KIT
• HiSPi Sensor Interfacing
• MACHXO2 Dual Sensor Interface Board (DSIB)
• @XO2 Speed Campaign
Page: 3 Lattice Semiconductor Confidential
AGENDA
• Introduction to Cameras
• HDR-60 Video Camera Development KIT
• HiSPi Sensor Interfacing
• MACHXO2 Dual Sensor Interface Board (DSIB)
• @XO2 Speed Campaign
Page: 4 Lattice Semiconductor Confidential
MARKET & TECHNOLOGY OVERVIEWSURVEILLANCE MARKET REVENUE
Overall Video Surveillance Revenue,World Market, Forecast: 2008 to 2015
Video Surveillance Camera Shipments by Type
World Market, Forecast: 2010 to 2015
Type Shipments 2010 2011 2012 2013 2014 2015
CAGR
(08-15)
CCTV (Millions) 29 27 25 24 22 21 -4%
IP (Millions) 3 6 13 24 38 52 63%
Total (Millions) 32 33 38 48 60 73 14%
Source: ABI Research
Video Surveillance Camera Revenue by Region
World Market, Forecast: 2010 to 2015
Region Revenue 2010 2011 2012 2013 2014 2015
CAGR
(08-15)
United States ($ Millions) 1,922 2,100 2,314 2,845 3,430 3,928 12%
Canada ($ Millions) 306 333 373 469 581 682 10%
Europe ($ Millions) 2,636 2,635 2,566 2,875 3,146 3,559 4%
Asia-Pacific ($ Millions) 1,409 1,581 1,785 2,316 2,988 3,600 16%
Rest of World ($ Millions) 1,079 1,148 1,300 1,592 2,032 2,357 15%
Total ($ Millions) 7,351 7,797 8,337 10,098 12,177 14,126 10%
Source: ABI Research
Video Surveillance Storage Revenue by Type, World Market, Forecast: 2008 to 2015
nearly doubling in 3 years!
Very strong in all regions!
Page: 5 Lattice Semiconductor Confidential
MARKET & TECHNOLOGY OVERVIEWTRENDS
TRENDS/DRIVERS
• Wireless Video: reduces infra structure cost, sometimes the only option, best suited for city surveillance
• Cloud-Based Video Surveillance (Video Surveillance as a Service (VSaaS))
• Video Analytics: Video Content Analysis (VCA) beyond Security (automotive, defense, medical, consumer, digital signage)
• From Analog to IP: Tipping Point for Network Video: Global in 2015, 2013 in Americas, 2012 in EMEA, Middle-East and Russia already tipped.
• From HD to 3D: 3D offers depth perception
• “Affordable” Thermal Imaging/Surveillance: New phenomenon. Commercial IR market is heating up.
• HDcctv gaining momentum: An open standard for transmission of HD video using coax-cable. Built upon HD-SDI standard. Not expected to impact the adoption of network cameras in the short term.
• From SD IP to HD IP (IMS Research: 30% HD/MP in 2011, 60% HD/MP by 2015)
• HD the resolution sweat spot: HD offers increased resolution but not to the extreme level where the cost of storage and bandwidth become unmanageable.
• ISP: From single sensor to multi-sensor processing (Dual and Quad)
Page: 6 Lattice Semiconductor Confidential
ISP
Flash SDRAM
SDI Tx
SDI Camera
SD/HD/3GSDI SDI Camera (HDcctv)
• Highest Quality• Uncompressed• Coax cable based
SensorI/FLens
ISP
Flash SDRAM
DACLens
Analog Camera
CVBSVGA/RGB
Analog Camera (cctv)• Highest volume• Lowest cost• Coax cable based
SensorI/F
ISP
Flash SDRAM
NetworkProcessor
IP Camera
RJ-4510/100MBps
IP Camera (SD/MP)• Video, Control, (& Power)
over one cable (CAT5)• Higher maintenance• Requires network / skillset
SensorI/FLens
MARKET & TECHNOLOGY OVERVIEWCAMERA ARCHITECTURES
Lattice
Page: 7 Lattice Semiconductor Confidential
AGENDA
• Introduction to Cameras
• HDR-60 Video Camera Development KIT
• HiSPi Sensor Interfacing
• MACHXO2 Dual Sensor Interface Board (DSIB)
• @XO2 Speed Campaign
Page: 8 Lattice Semiconductor Confidential
Price $399
Industry Leading Auto
Exposure,High Dynamic
Range &Auto White
Balance
Full HD 1080p60
Capable HDR ImageSignal
Processing (ISP) Pipeline
HDR-60 VIDEO CAMERA DEVELOPMENT KITOVERVIEW
Other Applications
: Traffic, Automotive,
Video Conferencin
g
Major Application: Security &
Surveillance Cameras
Fully Production-Ready HD
Video Camera
with HDR
Page: 9 Lattice Semiconductor Confidential
HDR-60 VIDEO CAMERA DEVELOPMENT KITTRI-PARTITE PARTNERSHIP
Lattice Semiconductor
Corp.
• Lattice ECP3-70 & HDMI PHY
Helion GmbH
• ‘IONOS’ Image Signal Processing IP
Aptina LLC
• Aptina A-1000 HDR Sensor
Page: 10 Lattice Semiconductor Confidential
HELION AT A GLANCE
More than
110
Image Processing IP cores available
• Founded 2003 as a spin-off company from the Fraunhofer Institute
• More than 15 years experience in HDR/WDR and image pre-processing
• Engineering Services in customized digital cameras and display systems
• Complete image processing solution from sensor to display, called IONOS™
• Consulting in image sensor configuration and FPGA programming
• Solutions in the field of Automotive, Medical, Security, Industrial
• Inventor of the WDR/HDR-BLENDFEST™ technology and the IONOS™ IP suite
• Exclusive Lattice IP core partner for image pre-processing
• Supported image sensors, like: Aptina, Panasonic, Sony, OmniVision, Cypress/OnSemi, and more...
Page: 11 Lattice Semiconductor Confidential
HDR-60 VIDEO CAMERA DEVELOPMENT KITKIT CONTENTS
Lattice HDR-60 Video Camera Development Kit
• Lattice HDR-60 Video Camera Main Board
• Lattice Nanovesta Sensor Board
• 1 x HDMI Cable
• 1x HDMI – DVI Adapter
• 2 x Standard USB Programming Cables
• Universal Power Supply
• Quick Start Guide
Page: 12 Lattice Semiconductor Confidential
HDR-60 VIDEO CAMERA DEVELOPMENT KITDEMO SETUP PLUG-AND-PLAY DEMO
HDMI / DVI cable
HDMI / DVI Monitor
Page: 13 Lattice Semiconductor Confidential
COLOR IMAGE SIGNAL PROCESSING (ISP)PIPELINE FROM HELION
Page: 14 Lattice Semiconductor Confidential
HDR-60 VIDEO CAMERA DEVELOPMENT KITHIGH DYNAMIC RANGE (HDR)
CAR?Person?
Under Exposed Over ExposedHDR
The Power of High Dynamic Range (HDR) High Dynamic Range (HDR) Examples:
HDR (also called WDR) enables the capture and display of both bright and dark areas in the same frame, in a way that there
are details in both areas, i.e. bright areas are not saturated, and dark areas are not too dark.
Page: 15 Lattice Semiconductor Confidential
HDR-60 VIDEO CAMERA DEVELOPMENT KITDEMO SPLIT SCREEN DIFFERENCES
Left Hand Screen
�Sensor Port with- Adaptive Black Level Correction
�Linearization 12 > 20
�Defective Pixel Correction with- 2D Noise Reduction
�5x5 HQ DeBayer
Right Hand Screen
�Sensor Port with- Adaptive Black Level Correction
�Linearization 12 > 20
�Defective Pixel Correction with- 2D Noise Reduction
�5x5 HQ DeBayer
�Auto Exposure
�AWB
�CCM
�Gamma
�HDR Tone Mapping
�Overlay
Page: 16 Lattice Semiconductor Confidential
HDR-60 VIDEO CAMERA DEVELOPMENT KIT32-BIT PIPELINE LOWERS LATENCY, ELIMINATES NOISE, REDUCES COST
Competition 18 bit PipeLattice-Helion 32-bit Pipe
De Bayer
CCM
AWB
FRAME BUFFER (adds cost)
18-bit ISP Pipeline MUST do 3D NR; else too much noise. But still can’t dare to throw away the last couple of noisy bits.Significant noise still present in output
3D Noise Reduction
LEGEND:
= Latency= Clean bit= Noisy bit
Internal PipelineWidth is 18 bit.Lost 2 bits already!.Low light performance now impacted.
(Each operation introduces noise)
32-bit ISP Pipeline nevereven maps the noisy bits to final RGB. Negligibleor no noise present in output.
(Each operation introduces noise)
ISPISP
Inside the
SENSOR
Internal PipelineWidth is 32 bit.
Pipeline is now very noisy, so,
can’t use it. Must do 3D NR.
Back and forth between buffer memory adds latency
Page: 17 Lattice Semiconductor Confidential
HDR-60 VIDEO CAMERA DEVELOPMENT KITFPGA RESOURCE USAGE
IP Block LUT EBR 18Kbit 18x18 Multiplier
Sensor Port with Auto Black Level Correction 1915
Linearization 12>20 100
Defective Pixel Correction & 2D Noise Reduction (2DNR) 1539 12
De-Bayer 5 x 5 HQ 1959 12
Color Correction Matrix (CCM) 1712 18
Statistics Engine 2000
Fast Auto Exposure* (AE) - - -
Auto White Balance & Saturation Control (AWB) 2167 2 8
High Dynamic Range (HDR) 1612 12 16
Gamma Correction 650 1 2
Overlay Helion Logo 336 5 3
Lattice Mico 32** 1800 22 2
I2C Master 132
Lattice HDMI PHY 276
TOTAL 16198 66 49
Percentage ECP3-35*** Resources Used 49.08% 91.67% 76.56%
* Fast AE is done by LM32 with data from Statistics Engine
** EBR for LM32 depends on code size. 22 used in demo*** Device on board is ECP3-70. Percentage usage of ECP3-35 shown since design fits an ECP3-35
Page: 18 Lattice Semiconductor Confidential
HELION’S IONOS IP SUITE
18
Helion’s IONOS-IP-CORE OVERVIEW
Defect Pixel CorrectionDPC
Color Filter Interpolation
Color Correction Matrix
Gamma Correction
Adaptive Black Level Correction
High Dynamic Range Imaging
Statistics
Overlay
Color Space Conversion
Automatic White Balance
BLC
CFI
CCM
Υ
CSC
AWB
HDRI
STAT
OSD
TG
AE
FRC
LIN
NR
AC
Test Pattern
Auto Exposure
Frame Rate Converter
Linearisation
Noise Reduction
Aperture Correction
LSC Lens Shading Correction
OI Output Interface
MI Memory Interface
SI Sensor Interface
COMPREHENSIVE IP LIBRARY:
Helion is providing a comprehensive intellectual property (IP) library. Containing more than 90 individual pieces of IP, the library is available for licensing either entirely or in parts. Developed by Helion, the library provides complete sensor to display ISP support.
All licensing is handled through Lattice!
Two Step licensing process:1. IP Evaluation License2. Full IP License
Page: 19 Lattice Semiconductor Confidential
HELION IP LICENSING MODEL
Customer wants to evaluate (Helion IP Evaluation Licensing Agreement)
Customer signs IP evaluation license agreement with Lattice:⇒ Customer will be given access to Helion FTP siteFTP site includes:• Complete ISP pipeline example projects• IP cores (time-bombed NGO format)• Documentation
Customer can evaluate and start designing a project!
No commitment required! No cost involved!
Customer wants to license IP (Helion IP Licensing Agreement)
Customer signs IP license agreement with Lattice:⇒Customer will be given license file for defusing IP time-bomb⇒Customer commits to ONLY use special part number devices with Helion ISP!
• special part numbers allow Lattice to track number of devices used with Helion IP and compensate Helion for IP usage (per device royalties)
Customer does not pay any Royalties, or NRE’s! No MOQ’s required!
⇒No Risk!
• Available part numbers for Helion IP usage: LFE3-35EA-8FTN256IAEJ ; LFE3-35EA-8FN484IAEJ ; LFE3-70EA-8FN484IAEJ ; LFE3-95EA-8FN484IAEJ ; LFE3-150EA-8FN672IAEJ
Page: 20 Lattice Semiconductor Confidential
Features Benefits
Fully Production Ready. Free Demo Bitstream, Schematics & Layout.Fits Commercially Available Camera Housings.
Accelerated Time to Market
Lowest Cost BOM compared to both competition FPGA and DSP, for 1080p60 HDR Camera. Needs No External Frame Buffer.
Full HD* Functionality and HDR at Lowest System Cost
Industry Leading Sensor Roadmap Support: up to 16MP Future-proof Kit Investment
Industry Leading:- Auto Exposure- Auto White Balance- HDR Range
Differentiation at Low System Cost
Available Comprehensive Wishbone Compliant
ISP LibraryVersatile Product Offering with
Single Source, fully-compatible IP
Broadcom BroadReach™ PHY and BNC ConnectorLegacy Cable Infrastructure Support.
Coax-ready for up to 700m Signal Travel.
ECP3-70 On Board.FTDI Chip On Board.RJ45 Ethernet Connector On Board.
Room to add H.264 Codec, Video Analytics IP and Customer’s own IP.Easy Programming via regular USB Cable
On Board Twin-Sensor Support Connectors Video Stitching Applications Support
HDR-60 VIDEO CAMERA DEVELOPMENT KITFEATURES & CUSTOMER BENEFITS
* Kit is 1080p60 capable. Current sensor on board is 720p.
Page: 21 Lattice Semiconductor Confidential
HDR-60 VIDEO CAMERA DEVELOPMENT KITSUMMARY
Fully production ready design
HDR 1080p60 capable FPGA stream processing
Industry-leading features:
- Fast auto exposure
- 120dB+ High Dynamic Range
- Support for up to 16MP sensors
Comprehensive Image Signal Processing library
Low cost BOM
Page: 22 Lattice Semiconductor Confidential
AGENDA
• Introduction to Cameras
• HDR-60 Video Camera Development KIT
• HiSPi Sensor Interfacing
• MACHXO2 Dual Sensor Interface Board (DSIB)
• @XO2 Speed Campaign
Page: 23 Lattice Semiconductor Confidential
SENSOR INTERFACINGHIGH SPEED SERIAL PIXEL (HISPI)
Background:
• Traditional parallel sensor interface can no longer handle increased sensor resolutions and frame rates. New sensor interfaces have been introduced by sensor manufacturers.
• A majority of ISP devices support only parallel sensor interface and require a bridge device converting high-speed serial data to a parallel format
• Lattice XO2 and XP2 FPGA’s provide an efficient and cost-effective solution for converting HiSPi-to-parallel and subLVDS-to-parallel bridging.
Available Reference Designs:
• Aptina HiSPi sensor I/F to Parallel bridge:
• MT9024 , AR0331
• Panasonic subLVDS sensor I/F to Parallel bridge:
• MN34081, MN34041
• Sony subLVDS sensor I/F to Parallel bridge:
• IMX036, IMX136
• Dual-Sensor Bridges:
• 2x Aptina MT9024 HiSPi to Parallel
http://www.latticesemi.com/products/intellectualproperty/referencedesigns/aptinahispitoparallelsens/index.cfm
Coming soon: MIPI Interface/Bridges
Page: 24 Lattice Semiconductor Confidential
APTINA HISPI TO TI DSP BRIDGEREFERENCE DESIGN FEATURES
Complete reference design
Designed to emulate parallel sensor output: output bus widths 8, 10, 12 or 14 bits
Supports HiSPi formats Packetized and Streaming SP: from 1-4 lanes running up to 700Mbps each
Bridge device offered in a small 132csBGA (8x8mm). TQFP packages also available
Requires no external PROM
Tested with TI TMS320DM3X5 and Aptina A1000 sensor
Parallel interface can be configured for 1.8v, 2.5v or 3.3v LVCMOS levels
Local Tone Mapping available as optional enhancement
Page: 25 Lattice Semiconductor Confidential
AGENDA
• Introduction to Cameras
• HDR-60 Video Camera Development KIT
• HiSPi Sensor Interfacing
• MACHXO2 Dual Sensor Interface Board (DSIB)
• @XO2 Speed Campaign
Page: 26 Lattice Semiconductor Confidential
• MachXO2-4000 PLD for sensor interfacing and driving a parallel ISP bus
• LP SDRAM: 16-bit data over a 32Mb address space
• Built-in USB 2.0 download to MachXO2
• Can be configured for a flywire ispDOWNLOAD® cable connection
• HiSPi and other serial sensor interfaces can be supported
• 36-pin flat ribbon cable connector for TI IPNC camera connections
• 3.3V, 2.5V and 1.8V voltages are possible for various MachXO2 I/O banks
MACHXO2 DSIBDUAL SENSOR INTERFACE BOARD
Page: 27 Lattice Semiconductor Confidential
Some common applications and uses for the MachXO2 DSIB
• 3D stereoscopic video cameras
• Aftermarket automotive black box DVR
• 3D camera for analytics in security/surveillance applications
• Traffic cameras where one sensor records video and the other sensor takes the photo
• Interfacing to the Texas Instruments (TI) IPNC DM812x, DM385 or DM385 camera base boards
• Evaluation of MachXO2 device interfacing to sub-LVDS signals
• Sensor bridges using a MachXO2 device to convert from serial sensors to parallel buses
MACHXO2 DSIBCOMMON APPLICATIONS
Page: 28 Lattice Semiconductor Confidential
MACHXO2 DSIBDEMO
• Targeting dual-sensor & stereoscopic video applications
• Demo supports the merging and synchronizing of two image sensors and outputting one bus for an ISP to process.
• DSIB plugs into HDR-60 mainboard
• DSIB can be used with TI DSP board or Helion ISP
Picture: HDR-60 mainboard w/ Dual-Sensor Headboard
Lattice’ XO2 based Dual Sensor Board (DSIB):TI DSP Interface
NanoVestaSensor #1(Right Eye)
NanoVestaSensor #1(Right Eye)
MachXO2 Dual-Sensor Interface Board (DSIB)
HDR-60Baseboard
Page: 29 Lattice Semiconductor Confidential
AGENDA
• Introduction to Cameras
• HDR-60 Video Camera Development KIT
• HiSPi Sensor Interfacing
• MACHXO2 Dual Sensor Interface Board (DSIB)
• @XO2 Speed Campaign
Page: 30 Lattice Semiconductor Confidential
@MACHXO2 SPEED CAMPAIGN
1. www.latticesemi.com
2. Home>Products>CPLD>MachXO2
3. Click on following picture
Page: 31 Lattice Semiconductor Confidential
@MACHXO2 SPEED CAMPAIGN
FunctionRequired
LUTs (MachXO)
Required LUTs
(MachXO2)
SPI Master 115 0
I2C Master 234 0
Timer/Counter 50-80 0
I2C Controllers
• Two controllers• Configurable master or slave• Programmable address• Up to 400KHz• Clock stretching supported
SPI Controller
• Configurable master or slave• Four wire interface• Up to 50Mbps
Timer/Counter
• 16-bit counter• Clock scalar (8, 64, 256, 1024)• Normal, clear on match,
Fast PWM, phase correct PWM, watchdog mode
Page: 32 Lattice Semiconductor Confidential
@MACHXO2 SPEED CAMPAIGNEMBEDDED FUNCTION BLOCK
Page: 33 Lattice Semiconductor Confidential
WHAT’S NEW WITH MACHXO2REFERENCE & DEMO DESIGNS, APP NOTES
Topic Function Type Doc # Board(s) Code Title
XO2 I2C Master Design Demo Design UG54 Pico CMachXO2 Master I2C Demo Using 'C' &
LM8
Reference Design RD1124 - RTLI2C Slave Peripheral Implementation
Using Lattice EFB
Demo Design UG55 Pico (x2) RTLMachXO2 Hardened I2C Master/Slave
Demo
XO2 SPI Master Design Demo Design UG54 Pico CMachXO2 Master SPI Demo Using 'C' &
LM8
Reference Design RD1125 - RTLSPI Slave Peripheral Implementation
Using Lattice EFB
Demo Design UG56 Pico (x2) RTLMachXO2 Hardened SPI Master/Slave
Demo
UFM Access via WB Reference Design RD1126 - RTL RAM-Type Interface for Embedded UFM
CFG Access via WB Demo Design UG57 - RTL MachXO2 Programming Via Wishbone
Embedded I2C
UFM & CFG AccessReference Design RD1129 Versa, Pico C
MachXO2 I2C Embedded Programming
Access Firmware
Accessing Power
Controller Through I2C,
SPI & User Logic
Demo Design UG58 Pico RTL MachXO2 Low Power Control Demo
- Usage Guide TN1204 - -Mach XO2 Programming
and Configuration
- Usage Guide TN1205 - -User Flash Memory & Hardened
Control Functions in Mach XO2
-Reference
GuideTN1246 - -
User Flash Memory & Hardened
Control Functions in Mach XO2
Application
Notes
I2C
XO2 I2C Slave Design
XO2 SPI Slave Design
SPI
Wishbone
Other
Page: 34 Lattice Semiconductor Confidential
LOW-COST HARDWARE KITSINCLUDING $29 PICO DEVELOPMENT KIT
MachXO2 Family
• Low Cost
• Low Power
• High System Integration
Pico Development Kit
• Discounted from $49 to $29
Embedded Function Block
• I2C, SPI, user Flash memory
• Pre-designed, pre-verified hard IP
• Saves design time & up to 429 LUTs
• New reference designs
• New re-usable demonstration designs
EFB
Download reference designs and Diamond design software today at www.latticesemi.com
THANK YOU
Page: 36 Lattice Semiconductor Confidential
HELION ISP IP BLOCK DESCRIPTIONS
Lattice
Page: 37 Lattice Semiconductor Confidential
IP BLOCK DESCRIPTIONSENSOR PORT
SensorPort
Basic Description:• Configures Image Sensor for Detection• Captures Image Data
• Detects the image sensor by scanning the I2C-bus. • Reads out the version and subversion register of the
image sensor. • Resets the sensor, loads the individual setup settings table
into the sensor
Advanced Information:• Maps sensor physical registers to IP block’s internal logical
registers • (e.g. integration time to the physical register
number, e.g. R0x3012)• Configures after each readout of a frame the following
registers: • Integration Times• Gains• Readout Coordinates
• Stores (FIFO) one line of incoming pixel data with pixel clock speed and reads outs this FIFO with a lower internal speed.
Integration Time
Gain
Coordinates
I2C
SENSORPHYSICAL
REGISTERS
LOGICALREGISTER
S
SENSOR IP BLOCK
Page: 38 Lattice Semiconductor Confidential
IP BLOCK DESCRIPTION LINEARIZATION
Linear10 > 12
Basic Description:• Decompresses Compressed Sensor Data
Advanced Information:• Linearization 10 – 12 bit• For MT9V022/V023/V024/V032/V033/V034 use only
• In case of the use of the 12Bit mode:• Decompands the companded image data in the
datastream • Applies an offset correction• At the output of this IP the datastream behavior is linear.
• Linearization 12 - 20 bit • For A-1000 HDR-Mode only• Selects the necessary decompanding scheme
(similar to Linear 10->12), • Depending on the selected HDR companding modes
Linear12 > 20
Page: 39 Lattice Semiconductor Confidential
IP BLOCK DESCRIPTION DEFECTIVE PIXEL CORRECTION
Def-2D
Basic Description:• Determines defective pixels• Corrects defective pixels by examining neighboring
pixels and modifying defective pixel
Advanced Information :• Stores (FIFO) 4 lines of incoming RAW Bayer data• For each pixel in the pixel datastream, compares the
neighboring pixels of the same color
• After this comparison the core 'knows' whether the actual pixel is a hot/warm or a cold pixel
• In this case the pixel values is replaced by the median value of the same color neighbors.
Page: 40 Lattice Semiconductor Confidential
IP BLOCK DESCRIPTION DE-BAYERING (DE-MOSAIC)
Basic Description:Since each pixel has a filter with one of three colors (R/G/B),two-thirds of the color data is missing from each. To obtain afull-color image, various demosaicing algorithms can be used tointerpolate a set of complete red, green, and blue values foreach pixel.
Advanced Information:• De-Bayering 3x3 Billinear
• Color Filter Array (CFA) Demosaic with 3x3 Core• Logic-Efficient Billinear Approach• Stores (FIFO) 2 lines of incoming RAW Bayer data• The colors (red, green and blue) are interpolated in a bilinear way
• i.e. averaging of neighbor pixels
• De-Bayering 5x5 High Quality• Color Filter Array (CFA) Demosaic with 5x5 Core• High Image Quality with Fewer Image Artifacts • Stores (FIFO) 4 lines of incoming RAW Bayer data• Depending on the position of the actual pixel (red, green1, green2,
blue) and the wished for resulting color (red, green, blue), a different weighting scheme is used.
Debay3x3 Bi
Debay5x5 HQ
Page: 41 Lattice Semiconductor Confidential
IP BLOCK DESCRIPTION CCM: COLOR CORRECTION MATRIX
CCMCCM
Many current image sensors provide incorrect color rendition due to ‘crosscolor’ effects. This effects leads to wrong color images (e.g. green with too much blue).
CCM modifies each single color channel by a 3x3 correction matrix:red-out = rr*red + rg*green + rb*blue
green-out= gr*red + gg*green + gb*blue
blue-out = br*red + bg*green + bb*blue
Builds the color statistics with selectable thresholds (min/max illumination, min/max color ratio). Adjusts the internal digital fine gain amplifiers for red gain and blue gain inside the IP core based on the positions of the individual Bayer filter elements.
Page: 42 Lattice Semiconductor Confidential
IP BLOCK DESCRIPTION AWB: AUTO WHITE BALANCE
Basic Description:Cameras are not good at recognizing colors. AWB adjusts other colors in an image with reference to a known white color in the image. AWB determines white by examining the frequency (therefore wavelength) of incoming light
Advanced Information:• Builds the color statistics of a selectable image region (ROI) with selectable thresholds (min/max illumination, min/max color ratio)
• Adjusts the internal digital fine gain amplifiers for red gain and blue gain inside the IP core. Target is a so called grey world approach.
AWB
Page: 43 Lattice Semiconductor Confidential
IP BLOCK DESCRIPTION HDR: HIGH DYNAMIC RANGE
Dynamic Range = Ratio of expressed in dB
Higher the dynamic range the better we can see detail in both light and dark areas. HDR combines multiple exposures of different duration and blends to form a clear image for both light and dark areas
Advanced Information:• High Dynamic Range Processing With Tone Mapping• Builds internal image statistics (histogram, average and
median values) to calculate for each frame two adaptive global tone map transfer curves and uses local tone mapping for contrast enhancements.
HDR
Intensity of brightest areaIntensity of darkest area
Page: 44 Lattice Semiconductor Confidential
120DB INTRA-SCENE / ~170DB INTER-SCENE DYNAMIC RANGE
Intra-scene dynamic range:120dB intra scene dynamic range, based on 20 bits/pixel
Measured number from Aptina: 1 bit � 6dB. 20*6dB = 120dB
In linear mode the sensor has fewer than 12 bit intra scene dynamic range (<72 dB). A-1000 uses 3 integration times / readout. Each successive integration time is 16x shorter, shifting the dynamic range 4 bits (+24dB) at a time. Leads to an intra-scene dynamic range of 72+24+24 = 120dB
Inter-scene dynamic range:
120dB might be OK for a specific scene, but it is necessary to shift and adapt the intra-scene dynamic range when the scene illumination is changing (e.g. from night to day). e.g.: Auto manufactures BMW and Audi have a minimum system dynamic range requirement of 150dB.
The intra scene (in one image, but taken with 3 internal multi exposures) dynamic range is 120dB. The AE IP block shifts this intra scene dynamic range by setting the first integration time (the other 2 times are internally determined by the sensor).
Example:
By using 480 lines for integration, the minimum detectable illumination is E_min0, the maximum illumination is then:
E_max0 ~ E_min0 * 2^20, because of 120 dB intra scene dynamic range
By using instead 1 line for integration (selected by the AE module), the minimum detectable illumination is then:
E_min1 = Emin0 / 480 and the maximum illumination is to E_max1 = E_max0 / 480.
The definition of dynamic range is the ratio between the maximum signal to the minimum signal.
This leads to a system dynamic range of:
DR_system = 20*log( E_max0 / E_min1 ) dB = 20*log ( 2^20 * 480) dB = 120dB + 53.6dB = 173.6dB
120dB
x y
SCENE 1
x1
y1
x2
y2
x3
y3
170dB
120dB
120dB
120dB
SCENE 1
SCENE 2
SCENE 3
Page: 45 Lattice Semiconductor Confidential
IP BLOCK DESCRIPTION GAMMA CORRECTION
GammaCorr
Corrects for non-linear response to image luminance between sensor and display, by applying a pre-distortion correction which becomes a linear response after again being distorted at the display end.
Uses a lookup table with interpolated values to apply a gamma correction for each color channel. The gamma value is selectable.
Page: 46 Lattice Semiconductor Confidential
IP BLOCK DESCRIPTION OVERLAY
Overlay
Basic Description:• Overlays text & graphics on the video
image for display on screen.
Advanced Information:Overlay Engine
• Overlay of Digital Image Data • Free Selectable Size and Position • Up to 256 Levels of Transparency• Free Selectable Genlock Color
Page: 47 Lattice Semiconductor Confidential
HDR DEMO CONFIGURATIONWITH APTINA MT9M033 (A-1000/MI-1000) SENSOR
- HDR alpha release 60fps 1200x720 readout with companded transfer curve. One sensor pixel is one TFT pixel.
- Streaming Image Data Path:
- No External Frame Buffer needed.
- Quality at a lower total system cost.
- System latency is extremely low (0.099ms – 0.510ms depending on IP blocks used)
- Full HD 1080p/60fps capable (A-1000 sensor is 720p)
- Integration time ratio is 16x by 16x
- 120dB Image Dynamic Range
- Very large range allows optimal light and dark scene processing
- 20 bits per color channel
- Independent of the incoming light
- For brightness beyond a certain threshold, the F# / Iris can be changed without any visible change on image brightness
- Fast Auto Exposure: Executed in 3 frames; no visible display bloom or blackout when going from dark to light and light to dark respectively
- Overlay functions are used to describe the stream processing functions currently active.
Page: 48 Lattice Semiconductor Confidential
VIDEO ANALYTICSROADMAP
Lattice
Page: 49 Lattice Semiconductor Confidential
VIDEO ANALYTICS IP PLANNED
Baseline VA Functions Planned
� Intelligent Video Motion Detector
- Analyze in real-time
- Filter out noise such as changing light, moving leaves on trees and animal movements during detection
- Provide real-time alerts; Start/Stop commands to DVR/NVR
� Intrusion Detector
- Automatic Perimeter Monitoring
- Monitor user-selectable zones for people or vehicle intrusion
- Filter out noise
- Provide real-time alerts
� Video counter
- Process video-streams in real-time
- Continuous multi-directional counting
- Count people, objects, vehicles
- Generate reports on people / vehicular traffic patterns
� Camera Tamper/Sabotage Detector
- Automatically Detect camera failure or Sabotage
- Detect partial blocking, complete blocking, drastic camera angle change
� Enhanced VA Functions Planned
- Object Left behind
- Object Tracking with intelligent zoom
- Intelligent Search
� Scan archived video for specific parameters: DVR/NVR application
Page: 50 Lattice Semiconductor Confidential
LIST OF AVAILABLE IP BLOCKS BY FUNCTION
LatticeHelion
Page: 51 Lattice Semiconductor Confidential
HELION IP – COLOR PIPELINE
Page: 52 Lattice Semiconductor Confidential
HELION IP – IMAGE ENHANCEMENT
Page: 53 Lattice Semiconductor Confidential
HELION IP – INTERFACES
Page: 54 Lattice Semiconductor Confidential
HELION IP – PERIPHERALS & OVERLAY
THANK YOU