October 12, 2012 Sam Siewert

A490 – Machine Vision and Computer Graphics

Image and Moving Picture Fundamentals

Lecture Week 7 (Lab #3 Overview and Exam #1

Review)

Using Digital Video for Machine Vision

From Cameras (Photometers) to Optical Navigation and Process Control

Sam Siewert

2

Machine Vision Outline for Reset of Semester Image Capture and Encode

– Pixel Encoding (YCrCb, sRGB) – Resolution (Down conversion) – Frame Rate (Decimation) – A/D Calibration (Flat-field Correction, Saturating with Known Stimulator) to Remove Artifacts

of Detector Caused by Variations Pixel to Pixel (“Christmas Lights” and Dark Currents)

Post Capture ImageTransformations – Enhancement with Pixel Convolution (e.g. Sharpen, Gamma Correction, Filtering) – Background Elimination (Focus on Changing Pixels Only) – Adjustment for Lighting Conditions and Color Encoding

Image Parsing - Edge and Feature Detection, Segmentation

– Basic Raster with Threshold Detection – Detection of Closed Areas/Regions (Segmentation)

Image Understanding - Object Recognition

– Based on Invariants (Shape, Color, Aspect Ratio, …, Principal Components) – Based on Models of Objects (Behavior and Signautre) – e.g. Eye Saccades

Image Perception - Object Tracking

– Computation of Centroid (Center of Pixel Mass of the Object) – Stereo Ranging with Multiple Cameras or Laser+Camera – Depth Perception – Proprioception – Robotic or Human Knowledge of Effector Part Locations in Space

Relative to Objects

Sam Siewert 3

Image Filters and Thresholding

CMV Chapters 3,4 Lab #3 – Tracking a Laser Spot

Sam Siewert

4

Review of CMV Conventions {P} – Originally Sampled Frame or Sub-frame of Pixels {Q} – First Transformation of {P} – E.g. Negative Image is {Q}=Saturation – {P}, Where Saturation

for 8-bit Gray-level is 255 – E.g. a Difference Image is {R} = {Q} – {P} is the time of sample of

{Q} is greater than {P} – A threshold

P0 is the Pixel of Interest in a Neighborhood – So, for all pixels in {P}, if Q0=P1, then {Q} is the same image

shifted one pixel to the left – If {Q}={P}+beta, modifies brightness – If {Q}={P} x gamma, modifies contrast – If {Q}={P} x gamma + beta, modifies both contrast and

brightness – This Can Define a PSF for an Image Convolution – e.g. Sharpen

Sam Siewert 5

P4 P3 P2P5 P0 P1P6 P7 P8

Common PSFs See www.dspguide.com/CH24.PDF – For all pixels, do {Q0 =

P0*h0+P1*h1+P2*h2+P3*h3+P4*h4+P5*h5+P6*h6+P7*h7+P8*h8; }

Convolution of causes no change to image

Sam Siewert 6

0 0 00 1 00 0 0

http://www.dspguide.com/

Median Filter Algorithm The Median Filter Uses Binning in range of 0 to Saturation of PGM (0 to 255) – Obviates need to Sort the Value of Neighbors in Kernel 3x3 – Q0 is the 5th largest neighbor in the Kernel – Must Zero Histogram of size 256 for Each Pixel – Loop through all neighbors and bin – Each P[m] neighbor’s value is the index into the histogram of

counts for pixels at that value

This filter provides noise suppression, but softens edges Truncated Media Filter Removes Noise and Sharpens

Sam Siewert 7

Lab #3 – Threshold Experiments See Laser Spot Videos – Dark Room – Dark Room with Clutter – Light Room with Clutter

Design a Threshold for the Laser Spot in Each Instance Based on Intensity Distribution Analysis of Edges Use Background Elimination by Taking Difference Images Raster Image to Find Maximum X distance edge-to-edge Track Y threshold crossings for each Column and Find Maximum Y distance edge-to-edge Compute X-bar, Y-bar Center of Spot Based on Threshold

Sam Siewert 8

Exam #1 Reivew

Chapters 1-4 in CMV, and Chapters 1,3,20,21,22 in FCG

& Notes from Class on MPEG

Sam Siewert

9

Exam Format 1 Question from CMV Ch. 1 1+ CMV Ch. 2 2+ CMV Ch. 3 & 4 2+ FCG 1, 3, 20, 21, 22 2+ from Notes 8 Questions Total – Some True/False (1 or 2) – Some Multiple Choice (1 or 2) – Some Short Answer, Pseudo-Code (C or MIPS ASM), Diagram,

Logic Table (no more than 4)

Full Class Period to Complete You Can Bring One Page of Notes if You Wish Closed Book

Sam Siewert 10

From Notes in Class – Thread Grids Flynn’s Taxonomy of Architectures for Processing Linux POSIX Threading – Mapping Threads to Image/Frame Grids – POSIX thread creation, parameters, join – Thread safety

Stack variables only Thread indexed global data Mutual Exclusion Semphore protection of shared global data

SIMD Vector Instructions

Sam Siewert 11

From Notes in Class – Speedup Speed-up Limited by Amdahl’s Law – P = Parallel Portion – (1-P) = Sequential Portion

Sam Siewert 12

Notes in Class – Analog vs. Digital Video

Analog vs Digital Encoding for Transmission – NTSC Frequency Modulation on Channels – Digital Broadband QPSK, QAM, 8VSB OTA – Digital Baseband Packet Switched Networks (Optical, Ethernet)

Digital is Routed (Diversely), Buffered, Compressed, Multiplexed (Shares Transmission Carrier) – Higher Latency, but More Flexible (Can be Processed) – Transported by Large (IP 1500 byte MTU) or Small (188 byte

MPEG) packets? Out-of-order arrival? – Bandwidth Delay Product – Bits in Network (e.g. Gbits/sec x

seconds of transport latency) – QoS – Does Packet Switched Approach Ideal of Constant Bit Rate?

Analog is Continuous Transmission with Instant Tuning – Known Deterministic Latency – Can’t be Processed Easily – Typically Lower Resolution and Not as Bandwidth Efficient

Sam Siewert 13

Notes in Class – MPEG Order of Operations and Lossy Steps

Sam Siewert 14

Notes in Class - Encoding Pixel (Point) Encoding – sRGB (ITU 709 R, G, B Primaries and CRT gamma), CIE RGB – RGB-α (Alpha is the opacity, a multiplier for each band for

foreground pixels, 1-Alpha for background pixels) – YCrCb

Frame Encoding – I-frame (compression limited to data contained in frame) – Uncompressed formats – PPM, PGM, PBM (TIFF) – Compressed formats – JPG, PNG, MPEG I-frame – Common Resolutions and Aspect Ratios (2K, 1080p, 720p, SVGA,

SD) Group of Pictures – I-frame starts, B (bidirectional) and P (predicted) frames reference I-

frame – Ffmpeg basics – Lossy vs. lossless steps (e.g. quantization and DCT for macroblock) – Ok for Machine Vision?

Sam Siewert 15

Color Graphics and MV use Additive Models – RGB cube – HSV cylinder

Human Physiology Limits Perception – Tri-stimulus – XYZ based on Human Eye Sensitivity to Short (420-

440nm), Medium (530-540nm), Long (560-580nm) Wavelength Light in Visible range of Spectrum

Gamut – complete subset of colors in visible spectrum based on CIE model for wavelength – Note that in gray levels, levels are brightness only at the same

chromaticity (e.g. white with more or less brightness) Gamma – non-linear correction factor to account for perception of adjacent gray levels that is not a linear progression of intensity Radiometry vs Photometry

Sam Siewert 16

CMV Chapter 1 Basic Concepts – Machine Vision as Inverse of Graphics – Rendering a Scene from List of Objects and Characteristics – Parsing an Imaged Scene into a List of Objects

The Challenge of Pattern Comparison with Training Sets – Does an Image Match a Known Pattern – Does this Generalize? – What if Object is Rotated? – Scaled? – Blurred? – Seen from and Angle? – In Bad Lighting?

Value of Color in Scene Segmentation

Sam Siewert 17

CMV Chapter 2 {P}, {Q}, {R} notation, Transformations, and Convolution Convolution Application of PSF – Must be 1/9 * to avoid change in intensity of original

CMV Convention for a PBM – Or bit-map is: {A}, {B}, {C} – All Pixels in each Image are 0 or 1 only – Simple threshold conversion from PGM, Graymap, is: for all

pixels do { if(P0 > thresh) A0=0; else A0=1} – For display purposes, use: for all pixels do {R0 = 255((1-A0)} for

image between 0 and saturation – Note Binary transformations like shrinking and edge finding

Sam Siewert 18

1 1 11 1 11 1 1

CMV Chapter 3 Noise Suppression Concepts and Filters Median Filters Mode Filters Truncated Median Filters Read P. 74, 3.13 – Note Cautions for Use of Filters in CMV Filters for Color – Apply Filter to Each Color Band Separately – Causes Color

Bleeding – Compute Distance (delta in grayscale) as square root of the

summation of the squares of the deltas – This Approach is Known as a Vector Median Filter – The Vector Distance is between 2 Locations in RGB Cube

Sam Siewert 19

CMV Chapter 4 Application of a Histogram to Count Occurrence of Intensity Levels in an Image or Kernel by Bin – Simple Single Peaked Histogram Has Well-defined Threshold – Bi-modal or Multi-modal Histogram has No Obvious Threshold – Background Elimination for a Stable Bakground Can Help

Alternative is Adaptive Thresholding – Threshold Varies According to Location in Image – Threshold Varies over Time

Analysis and Experimentation with Threhsolds are Focus of Lab #3

Sam Siewert 20

FCG Chapters 1, 3 IEEE Floating Point – Single and Double Precision – Normally 7 Digits of Precision on 32-bit Architecture – 15 Digits of Precision on 32-bit Architecture – Provides Handling of Infinity and NaN (See P. 6) – Must Clamp on Conversion from Floating Point to {P} at 0 and

Saturation – Use Test Code to Evaluate Precision -

http://www.ibm.com/developerworks/library/pa-bigiron1/ Compiler Optimization – Use for Graphics (e.g. Loop Un-rolling) and Vector Operations if Supported (SIMD) Raster Images, Devices (e.g. CRT, LCD) Basic Pixel Geometry and Values Gamma and Intensity Alpha Compositing and Opaque Layers – C = α x Cforeground + (1-α) x Cbackground JPG, TIFF, PPM, PNG

Sam Siewert 21

FCG Chapters 20,21,22

Sam Siewert 22

Ch 20 – Radiometry Definitions – Spectral Energy – Power – Irradiance – Radiance – Photometric Quantity – Useful to Human Observer based on

Luminous Efficiency Function of Observer Ch 21 - Color and Colorimetry – Tristimulus – Chromaticity Coordinates – Color Spaces and Models

Ch 22 – Visual Perception – Brightness and Contrast – L-cones, M-cones, S-cones – Acuity and Motion – Object Recognition – P. 585