Upload
doannhi
View
235
Download
0
Embed Size (px)
Citation preview
January 28, 2014 Sam Siewert
The Computational Photometer
Hybrid FPGA Digital Video Transformation for 3D – Joint UAA and
CU-Boulder Project
Outline Conceptual Design – 3D Scene Parsing Theory Current Rev-A Board Design 3D Configuration Options – Passive Binocular 3D (long range – UAV, security) – Active Structured Light 3D (range of 1 to 5 meters - medical)
The Team (Colorado and Alaska) and Sponsors (Intel, Mentor Graphics, Altera) The UAA CPAL Lab and Courseware The CU-Boulder Development Lab Status and Schedule Going Forward
Sam Siewert 2
3D Scene Parsing Human Depth Cues (Physiology, Psychology, Physics) – Between 9 and 15 Recognized Cues
Machine Vision Methods 1. Structure from Motion (Suitable for UAV, Photogrammetry
Elevation Estimations) - Passive 2. Binocular Stereopsis (Two Channel Registration and Triangulation
with Camera Extrinsic and Intrinsic Calibration) - Passive 3. Structured Light Projection (PrimeSense) - Active 4. Time of Flight (LIDAR) - Active
Active Methods vs. Passive IEEE RAS Paper on CV Improvement with 3D – “Change Their Perception”, December 2013 IEEE RAS Sam Siewert 3
Testing and Research Plans Enable UAA and CU-Boulder and University and Developer Programs (Open Hardware) UAA – Ice Formation (Multi-Spectral NIR + Visible Channels) CU-Boulder – Plant Growth Chamber (NIR + Visible) Wound Care – (RGB-D Active, TI DLP Module with NIR + Visible) Binocular Vision (Two Channel Visible)
Sam Siewert 4
OTS Alternatives Occipital – Boulder Colorado Kickstarter (Mobile Phone RGB-D) ASUS Xtion (Prime Sense) – We are testing with OpenNI Microsoft Kinect PrimeSense Developer Cameras Intel Creative Camera and Perceptual SDK We are testing all of them and comparing – None are Open! Many run OpenNI Middleware, but Hardware is Proprietary Sam Siewert 5
Research & Education Goals Education – Open Hardware, Firmware and Software – Analog layer, Digital,
Firmware, and Software Reference Design – Probing and Tracing at All Layers – Starting Point for Capstone Design and Student Research
Research – Compare Passive Binocular Vision with Computational
Photometry Parallelism to Active RGB-Depth Binocular = 2 Visible Coordinated Channels (UAV) RGB-D = Active Structure IR Projection, IR Channel, Visible Channel (Medical – e.g. Wound care)
– Low-Cost Infrared + Visible Computational Photometer for Remote Sensing and Safety/Security Applications
– Addition of IMU-on-Chip for Proprioception (Coordinated 3D Vision and Robotic Actuation)
Sam Siewert 6
2D & 3D Passive Computational Photometry
Sam Siewert 7
Analog Camera #1
LEFT (NIR, Visible) Altera FPGA
CVPU (Computer Vision Processing Unit)
Mobile Sensor Network
Processor (TI OMAP, Atom)
Networked Video
Analytics
HD Digital Camera Port (Snapshot)
USB 2.0, PCIe Host Channels
Flash SD
Card
Analog Camera #2
RIGHT (NIR, Visible)
2D Hough Transform
2D Skeletal Transform 3D Disparity & Depth Map
Rev-A Computational Photometer Board
Sam Siewert 8
Dual TI NTSC Decoders and FTDI Uplink – Plugs Into DE0, DE2i and DE4 Terasic Boards GPIO Header(s) – Can Uplink to DE2i or any Embedded Linux Board Via USB 2.0 – Reference Startix State Machine, FIFO, and Transforms
Rev-A Board Fabricated Some Issues to Fix – Connector Alignment (Ribbon Cable work around) – Debug Encoder port pin assignments for TVP module
Sam Siewert 9
DE0 (Can use DE0 Nano too)
2 Composite or RGB Component Inputs
FPGA State Machine YCrCb FIFO for Decoder Pixel Output Basic Frame Operations on FIFO Data – Time Decimation (every Nth frame) – Resolution down conversion (at 4:3 AR for SD video) – Pixel Re-encoding, for example to RGB from YCrCb
FTDI USB 2.0 Uplink from Each Channel Advanced Transformation – Sobel Edge Detection (Canny Option) – Hough Transforms – Research Transforms (E.g. Dr. Cenek’s Group H-MAX project)
Sam Siewert 10
3D Active Computational Photometry Concept (Rev-A + TI Kit)
Sam Siewert 11
Analog Camera #1
RGB (Visible)
Altera FPGA CVPU
(Computer Vision Processing Unit)
Mobile Sensor Network
Processor (TI OMAP, Atom)
Networked Video
Analytics
HD Digital Camera Port (Snapshot)
USB 2.0, PCIe Host Channels
Flash SD Card
Analog Camera #2
(Near Infrared)
TI DLP Light-crafter Kit http://www.ti.com/tool/dlplightcrafter
Depth Map
IR Pattern Projection
https://www.cs.purdue.edu/homes/aliaga/cs635-10/lec-structured-light.pdf
Photo credits and reference: Dr. Daniel Aliaga, Purdue University https://www.cs.purdue.edu/homes/aliaga/
2D Computer Vision Transforms Enable Intelligent Systems with Human-like Vision, but Wider Spectrum (Visible & Infrared) Real-Time 2D Scene Parsing & Understanding (OpenCV)
Sam Siewert 12
Canny Transform Skeletal Transform
Hough Linear Transform
3D is a Simple Concept, but Difficult in Practice
Triangulation is Basic Concept Camera Characteristics Complicate (Intrinsic) Binocular Apparatus Errors (Extrinsic) Common Reference Point (Left/Right) – Registration Far From Matching Human Capability RGB-D Scanning Becoming More Accurate, but Active
Sam Siewert 13
Simplified Planar Triangulation With Perfect Lenses and Detectors And Simple Point Registration
3D Computer Vision Transforms Long Range ( > 5 meters) Using Passive Binocular Methods – Impractical to Project from a UAV or Long Range Observer – Requires Image Registration – Accurate Camera Intrinsic (Camera Characteristics) & Extrinsic (e.g. Baseline)
Short Range ( < 5 meters), Structured IR Light Projection for RGB-D – Compare to ASUS Xtion and PrimeSense – Off-the-Shelf – Robust Depth Maps with Less Noise – Showing Significant Promise to Improve CV Scene Segmentation and Object
Recognition Compared to 2D – “Change Their Perception”, By Xiaofeng Ren, Dieter Fox, and Kurt Konolige,
IEEE RAS, December 2013.
Sam Siewert 14
Noise in Passive Depth Maps
Robust Active Depth Map “Change Their Perception”, By Xiaofeng Ren, Dieter Fox, and Kurt Konolige, IEEE RAS, December 2013.
The CP Team Sam Siewert – Asst. Professor UAA, Adj. CU-Boulder, Program PI Vitaly Ivanov – BS Student, UAA, Verification, Lab Content Jay Khandhar – MS Student, CU-Boulder, Linux Driver Development, lead Randall Meyers – TME, Mentor Graphics, PCB layout lead Jeries Shihadeh – Ph.D. Student, CU-Boulder, Hardware and Firmware development lead Sagar Sidhpura – MS Student, CU-Boulder, Linux Real-Time Kernel and Streaming integration and test, lead Sponsors: – Intel Embedded Education and Research Program
(DE2i) – Mentor Graphics, Longmont Colorado (DxDesigner,
Randall Meyer’s Engineering Time for PCB Work) – Altera University Program (DE0, DE4)
Sam Siewert 15
The UAA Computer Engineering Prototype and Assembly Lab
Supports Operating Systems (with Hardware) Computer Vision Lab – DE2i, DE0, TI-OMAP Alaska Space Grant Fellowship Lab (Autonomous Submersible) General Computer Engineering and Capstone
Sam Siewert 16
Computational Photometry – CU/UAA-Cam FPGA Interfaces for Multi-Channel, Multi-Spectral Computational Photography (Photometry) – Rev-A: Embedded Small: TI decoder -> DE0 (Stratix-III) -> FTDI ->
TI-OMAP – Rev-B: Embedded Medium: Intel DE2i Atom + Cyclone-IV <-
CameraLink (Grant) – Rev-C: Scalable Big: CameraLink -> DE4 -> PCIe -> Intel Haswell +
MICA (Xeon Phi)
Bread-board Funded by Intel Small Grant Completion of Board Design in November, FPGA Basic State-Machine, and Single-Channel Open Version Future Steps are Multi-Channel and Multi-Spectral – Ideally NIR to Visible, Perhaps to Thermal Imaging Range – 3D Stereo Vision for Point Cloud Models (PCL + OpenCV)
Sam Siewert 17
Breadboard Prototype Tested Summer 2013 NTSC -> TI Decoder -> Cyclone-III FPGA -> FTDI -> Beagle xM / Beagle Bone Black (Linux) Model-Sim Verification, Data-Flow Verification FTDI Interface and Linux USB Custom Driver (Stubbed) Spec to Mentor Graphics for PCB Layout, 4-Layer Fab
Sam Siewert 18
Verification by Q4-2013, Q1 Open Moving Fast Through Verification – PCB Layout Done Specification Provided to Mentor Graphics October Fab first boards Ideally in Nov/Dec for Run of 8 Demo at February Intel Embedded Education and Research Symposium, Q1-2014 ModelSim == Logic Trace Driver Stubbed in Linux for USB and UVC Working on FPGA Transforms and Applications!
Sam Siewert 19
Schedule 2013 – Rev-A Boards Fabricated in November 2013 – Rev-A Board Verification and Unit Test DE0 FPGA Firmware – December
2013 (Decoder Output Direct to VGA)
2014 – Prototype CU-Cam / CP Linux Driver Based on UVC Ready for Testing – Dataflow Tests from CU-Cam FTDI to DE2i Linux Driver (Beagle xM Driver) – Rev-1.0 Computer Vision Lab Materials Complete for Internal Review and
Testing with Students in CS&E A485 – Spring 2014 – Demonstration of Rev-A with DE0 at Intel Embedded Symposium (Poster) – FIFO Decimation, Resolution Down-Conversion, Pixel Format
Transformation Tests – Rev-2.0 Computer Vision Lab Material Updates – Stereo Vision Application Testing with OpenCV – OpenCV and Point Cloud Library Processing on DE2i Linux – Advanced Pixel, Macroblock, Frame Transformations on DE2i – Battery Power and Field Testing During Summer (Anchorage) – Summer 2014, Post Reference Designs on Intel and Mentor University
Program Sites
Sam Siewert 20
Summary Open Reference Design for Distribution by Mentor Graphics and Intel Embedded Education and Research Program
Exposes Students to High Quality 3D Vision Configurable Research Platform for 3D Passive, Active and 2D Multi-spectral Low Cost Research Platform that Can Be Battery Powered
Sam Siewert 21