June 1, 2021 Sam Siewert

Computer and Machine Vision

Lecture Week 2Part-1

Outline

Processing Images and Moving Pictures –High Level View and Computer Architecture for it

Linux Platforms for Computer/Machine Vision

I/O, Memory and Processing Challenges

Sam Siewert 2

Old School Moving Picture Media and Cameras

NTSC OTA (1941, 1953 color, 2009 dead)– Analog, Interlaced, Continuous Broadcast

Transmission or CCTV (Closed Circuit TV)– Coax Cable or Tuner with Immediate CRT Display– No Buffers, No Routing, No De-mux– No Compression

Analog CableAM/FM OTAFilm Projectors

Sam Siewert 3

Modern Digital CamerasCamera Link– High Frame Rates– High Data Rates and Resolutions– Industry Standard for Machine Vision Automation– E.g., Inspection Systems– E.g., Sony, IDT, National Instruments

SD-SDI and HD-SDI– Standard and High-Definition Synchronous Digital Interface– Standard for Studios, Broadcast

Digital Cinema– Red Camera– 1080p, 2K, 4K Resolutions and Much Higher– Automated Digital Delivery and Projection

Webcams and Mobile Phone Cameras– Very Low Cost– Proprietary– Performance Varies Dramatically

Sam Siewert 4

http://pro.sony.com/bbsc/ssr/cat-industrialcameras/cat-cameralink/product-XCLC30/http://www.idtvision.com/http://www.ni.com/vision/

DifferencesAnalog vs. Digital Encoding for Transmission

– Digital Allows for Image Processing– Adds Latency– Requires Compression for Packet Switched Networks and Storage

Routed (Diversely), BufferedCompressed (MPEG, JPEG) to Lower Bit-ratesMultiplexed (Shares Transmission Carrier for Audio, Video, Channels)Transported by IP (Large Packets)Continuous Transmission – Analog or Constant Bit-Rate / Frame-Rate

Sam Siewert 5

E.g., UAV Latency and JitterVerification of Video Frame Latency Telemetry for UAV Systems Using A Secondary Optical Method, Sam Siewert, Muhammad Ahmad, Kevin Yao

Sam Siewert 6

https://www.researchgate.net/publication/259344376_Verification_of_Video_Frame_Latency_Telemetry_for_UAV_Systems_Using_A_Secondary_Optical_Method

NTSC (Analog TV)

Sam Siewert 7

AM Video to CRTFM AudioChroma Added LaterOdd/Even Lines (Interlaced)29.97 FPS (30 before color)Vertical Blanking (CRT Retrace Time, Closed Captioning)525 Lines, 262.5 per Field, 60 Fields per Second

http://en.wikipedia.org/wiki/File:Ntsc_channel.svg

Linux in Computer VisionEmbedded Solutions– Texas Instruments OMAP (Beagle xM, Bone)– Numerous ARM SoCs (NVIDIA, Qualcomm,

Broadcomm, …)

Scalable Solutions– Multi-Core (Xeon Phi)– Vector Processing – CUDA, OpenCL GPU and GP-

GPU

Computer and Machine Vision is I/O, Memory and Processing Intensive

Sam Siewert 8

Camera InterfacesCCD (Charge Coupled Device) or CMOS (Common Metal Oxide Substrate) Detector– Integration Time for Photo-sensitive Elements in Array (to

Build up Charge)– Read-out Time to Sample Elements in Array

Luminance and Chroma Analog to Digital Conversion

Double Buffer for Read-out + Processing

Frame Capture– http://ecee.colorado.edu/~ecen5623/ecen/rtpapers/archive

/CHIPSET_DOCS/Bt878_Video_Decoder_Spec.pdf– Host Interface over PCI Bus or USB

Sam Siewert 9

http://ecee.colorado.edu/%7Eecen5623/ecen/rtpapers/archive/CHIPSET_DOCS/Bt878_Video_Decoder_Spec.pdf

Digital Video Transport QoSLatency– To Tune in a Program, Turn-on– To Deliver a Video Frame or Audio PCM Sample– To Start, FF, REW, Start-Over, Pause

Bandwidth– Resolution, Lossy/Lossless Compression, High Motion– Pixel Encoding for Color– Frame Rate– Constant Bit-rate Transport?– Variable Bit-rate Transport and Encoding?

Jitter– Decode and Presentation Rates– Elasticity in Decode to Presentation Buffering Necessary

Sam Siewert 10

June 1, 2021 Sam Siewert

Linux System Options

(Linux for Image Processing, Camera Interfacing and Computer Vision)

Sam Siewert 12

Processing OutlineMany-Core Linux Host(s)– Intel Atom– ARM– Xeon

GP-GPU Vector Processing PCI-E Co-ProcessorsNVIDIA Tesla/FermiAMD ATI

NPTL – Native POSIX Threads Library

NPTL Example Code Walkthrough

Sam Siewert 13

Conceptual View of RT ResourcesThree-Space View of Utilization Requirements– CPU Margin?– IO Latency (and

Bandwidth) Margin?– Memory Capacity (and

Latency) Margin?Upper Right Front Corner – Low-MarginOrigin – High-MarginMobile – Must Consider Battery Life Too (Power)

CPU-Utility

IO-Utility

Memory-Utility

Processing – Initial FocusProcessing and Scaling Frame Transformation, Encode, Decode is Critical

Memory for Buffering (Frame Transformations, CPU Integrated or GPU Offloaded – e.g. Linux VDPAU)

I/O for Networking (Transport)

I/O for Storage (On-Demand, Post, Non-Linear Editing)

Sam Siewert 14

Flynn’s Computer Architecture TaxonomySingle Instruction Multiple Instruction

Single Data SISD (Traditional Uni-processor)

MISD (Voting schemes and active-active controllers)

Multiple Data SIMD (e.g., SSE 4.2, GP-GPU, Vector Processing)

MIMD (Distributed systems (MPMD), Clusters with MPI/PVM (SPMD), AMP/SMP)

Sam Siewert 15

GPC has gone MIMD with SIMD Instruction Sets and SIMD Offload (GP-GPU)

NUMA vs. UMA (Trend away from UMA to NUMA or MCH vs. IOH)SMP with One OS (Shared Memory, CPU-balanced Interrupt Handling, Process Load Balancing, Mutli-User, Multi-Application, CPU Affinity Possible)

MIMD - Single Program Multi-Data vs. Multi-Program Multi-Data

Computer and Machine Vision

Treated as a Real-time and/or Interactive System– Requires Predictable Response (By Deadline)– Rate Monotonic– Earliest Deadline First– Least Laxity First

Covered in Depth in OS and Digital Media (Just be Aware for Computer Vision) Sam Siewert 16

Sam Siewert 17

CPU Scheduling TaxonomyExecution Scheduling

Global-MP Local-Uniprocessor

DistributedAsymmetric(AMP )

Symmetric(SMP OS)

Preemptive Non-Preemptive

Fixed-Priority

Hybrid

Dynamic-PriorityCooperative

Batch

FCFS SJN

Co-Routine ContinuationFunction

Heuristic EDF/LLF RR Timeslice(desktop)

Multi-FrequencyExecutives

StaticDynamic

RateMonotonic

DeadlineMonotonic

Dataflow

(Preemptive, Non-Preemptive SubtreeUnder Each Global-MP Leaf)

SMT(Micro-Paralell)

Sam Siewert 18

Response LatencyCi WCETInput/Output LatencyInterference Time

EventSensed Interrupt Dispatch Preemption Dispatch

Interference

Completion(IO Queued)

Actuation(IO Completion)

Input-LatencyDispatch-Latency

Execution ExecutionOutput-Latency

Time

Response Time = TimeActuation – TimeSensed(From Release to Response)

Resource ScalingProcessing – Co-Processors– GPU – CUDA, OpenCL– Many-Core – E.g., Intel Xeon Phi MICA– FPGA – E.g., Altera Stratix– Ideally Camera Interface

I/O – High-Rate Transport– HD-SDI, Camera Link, GigE/10GE

Memory – SSD, PCIe Nand, NVM– Samsung, Micron, Intel– Memristor (Future)

Sam Siewert 19

Computer and Machine VisionOutlineOld School Moving Picture Media and CamerasModern Digital CamerasDifferencesE.g., UAV Latency and JitterNTSC (Analog TV)Linux in Computer VisionCamera InterfacesDigital Video Transport QoSLinux System OptionsProcessing OutlineConceptual View of RT ResourcesProcessing – Initial FocusFlynn’s Computer Architecture TaxonomyComputer and Machine VisionCPU Scheduling TaxonomyResponse LatencyResource Scaling