"Processors for Embedded Vision: Technology and Market Trends," A Presentation from the Linley Group

EVA - September 10, 2014 © The Linley Group 2014 1

Processors for Embedded Vision Technology and Market Trends

Linley Gwennap, Principal Analyst, The Linley Group

Embedded Vision Alliance, Sept 2014


About Linley Gwennap

• Founder, principal analyst, The Linley Group

• Leading vendor of technical reports on mobile and

communications semiconductors

• Editor-in-chief of Microprocessor Report

• Author of recent articles on ARM, Broadcom,

Cavium, Intel, Marvell, Nvidia, Qualcomm, et al

• Author of “A Guide to Mobile Processors” and

“Mobile Semi Market Share Forecast”

• Former CPU designer at Hewlett-Packard


Agenda

• Mobile devices

• Automotive

• Cloud servers

• Internet of Things


Mobile Devices Need Vision Processing

• Billions of people already use smartphones and tablets to

take pictures and get information on their surroundings

• Face recognition can provide security access

to unlock phone (Android)

• Object recognition for shopping services

(Amazon Fire phone)

• Augmented reality to provide information on

surroundings such as in museum, shopping,

or tourism (e.g. AcrossAir, SkyMap)

• Computational photography for photo editing,

panoramic views, etc

“Firefly” on Amazon Fire Phone


Mobile Device Market—It’s the Big One!

0

500

1000

1500

2000

2500

2013 2014 2015 2016 2017 2018

Mill

ion

s o

f U

nit

s

Tablet

Smartphone

• Mobile devices require small size, low power and cost

• e.g. less than $3 to add a new feature

252M tablets in 2013 rising to 404M in 2018

1.04B phones in 2013 rising to 1.8B in 2018

(Source: The Linley Group)


A Few Companies Control the Platform

• Qualcomm, MediaTek,

Spreadtrum provide

complete chip sets

• Processor, baseband, RF,

Wi-Fi combo, NFC, power

management, etc.

• These vendors control

75% of all smartphones

• For vision processing to

succeed in smartphones,

it must come from these

vendors


Smartphone

Processor Share,

2014

Qualcomm

MediaTek

Apple

Spreadtrum

Samsung* Other

*Exynos processors only


Nvidia Tegra Image Processing Architecture

• Tegra K1 uses

two image

processors (ISP)

• Each ISP can

process up to

600Mpixels/s

• ISP handles basic

functions such as

YUV conversion,

AWB, gamma,

enhance edges,

noise reduction

ISP-A

Memory

Back

Camera

HW Accelerators HW Accelerators

CPU CPU

CPU CPU

AP

Memory Interface

Mu

x

GigaThread Engine

Raster Engine

SMX

ROP

L2 Cache

Memory Interface

ROP ROP ROP

Front

Camera

CSI x4

Kepler GPU Main CPUs

ISP-B

HW Accelerators HW Accelerators AP

Memory Interface

CSI x4

CSI x1


Heterogeneous Image Processing on Tegra

• Two crossbar buses enable sharing data among CPUs,

GPU, and ISPs

• Each unit provides different degrees of flexibility, power

efficiency, and performance

HW-ISP Kernels

GPU

K0

State Bus

K1

S0 …

S1 Sn

…

Kn

VI-

Mu

x

CS

I

Camera

Sensor

Kernels

CPU

K0

K1

…

Kn

LS BP AWB

DM EE CCM

ᵞ

Kernels

CPU

K0

K1

…

Kn

Kernels

GPU

K0

K1

…

Kn

Frame / Image Bus F0 … F1 Fn

Images

State

YUV


Tegra K1—Built for Vision

• Programmable engines can implement vision algorithms

while ISPs efficiently offload common imaging tasks

• GPU can handle multiple kernels (threads) with high FP

performance

• CPU cluster can handle four threads with general-purpose

programming

• Nvidia has demonstrated object tracking on Tegra

• Objects with as many as 4,096 focus points arranged in

matrices of 64x64 points

• Variable location and spacing


Tegra K1—Fast and Cheap

• Four Cortex-A15 CPUs at 2.3GHz

or two “Denver” CPUs at 2.5GHz

• Kepler GPU with 192 shaders (365 peak GFLOPS)

• Dual ISPs at 1.2 Gpixels/s total

• 2MB integrated cache memory

• DRAM controller with 17GB/s of peak bandwidth

• Complete SoC connects to camera, display, USB, HDMI,

serial ports, etc

• 15mm x 15mm package with PoP memory

• Operating power of about 2W to 4W

• Sells for ~$20 in high volume


Qualcomm Targets Smartphones

• Four Krait CPUs at 2.45GHz or

four Cortex-A53 CPUs at ~2.2GHz

• High-performance Adreno GPU

• Dual ISPs at >1.0 Gpixels/s

• Dual Hexagon DSPs at >600MHz

• DRAM controller with 25.6GB/s of peak bandwidth

• Complete SoC connects to camera, display, USB, HDMI,

serial ports, etc

• Integrated LTE baseband

• 14mm x 14mm package with PoP memory

• Sells for $25–$35 in high volume


Coprocessor Chips Available

• Irish startup Movidius offers Myriad 2

• Chosen as part of Google’s Project Tango

• Myriad 2 combines custom programmable cores with

hardware accelerators

• Two SPARC cores to run RTOS, scheduling

• 12 SHAVE cores at 600MHz total 86 peak GFLOPS

• Rated at 600 Mpixels/s

• Tiny 5mm x 5mm package

• Sells for $5-$10 in high volume

• Less performance than Tegra or Snapdragon


Coprocessor IP

• Cognivue offers vision accelerator with software

• APEX offers 32+ computational units

• Videantis targets vision with its IP

• Startup Adapteva offers FP accelerator

• Epiphany engine cranks out 71 GFLOPS in just 2mm2 (28nm LP)

• Some GPUs approach this level of FLOPS/mm2

• Ceva licenses industry-leading DSP cores

• Clock speeds greater than 1GHz in 28nm LP

• XC4410 performs 32 MACs/clock plus hardware accelerators

• IP will be successful only if major suppliers license it


Embedded Vision in Mobile—Summary

• Smartphone makers are desperately seeking innovation

• Samsung Galaxy S5 offers few new features

• Big innovation in iPhone 6 is copying Samsung’s bigger screens

• Vision processor enables cool new capabilities

• High-end mobile processors already include powerful

hardware that can be used for vision processing

• This level of performance will become ubiquitous in 3-5 years

• Need to demonstrate powerful use cases

• Need to integrate into OS for ease of use

• Major platform makers (Apple, Samsung, Qualcomm)

must lead innovation


Embedding Vision in Automobiles

• Many current vehicles offer advanced

driver assistance (ADAS) features

such as:

• Lane departure warning/keeping

• Automatic parking

• Collision avoidance

• Drowsy driver detection

• 2015 Mercedes C-class offers

autonomous “stop-and-go” driving

• Fully autonomous commercial

vehicles expected by 2020

Lane detection

Google self-driving car


Automobiles Use More Processors

0

20

40

60

80

100

120

140

2013 2014 2015 2016 2017 2018

Mill

ion

s o

f U

nit

s

Light Vehicles

With Nav Sys

• Cars require small size, high reliability for processors

• ADAS features at high end, moving into mainstream

• May have several processors in vehicle for different functions

About 100M cars and small trucks ship every year

~25% have nav system, rising to 45% in 2018



Auto Makers Turn to Mobile Processors

• Nvidia has shipped millions of Tegra chips into vehicles

(e.g. Audi, BMW, VW)

• Other mobile vendors starting to compete

• Traditional auto supplier Freescale has repositioned

mobile i.MX processor for automakers

• TI is bringing its DSPs (high FLOPS) to automakers

• Difficult to beat FLOPS per dollar of mobile processors


Embedding Vision in the Cloud

• Mobile devices have limits in performance, storage

• Some vision tasks can be offloaded to cloud data center

for additional processing

• Amazon Firefly service uses cloud server to recognize

object from library of possibilities

• This partitioning is also used for voice services such as Apple Siri

• Many augmented reality apps work the same way

• Impossible to store all possible scenes on the phone

• High-speed LTE network improves response time


Servers Have Small Units, Big Dollars

0

2

4

6

8

10

12

14

16

18

20

2013 2014 2015 2016 2017 2018

Mill

ion

s o

f U

nit

s

All Servers

Cloud Servers

• Servers average 1.8 processor chips at $375 ASP each

• Coprocessors can replace server processors at similar price

• But only a small portion of servers implement cloud vision

Server units rise from 10M to 20M in seven years

Cloud rises from 20% to 35% of all servers by 2018



What’s in Your Cloud?

• Most servers use standard Intel Xeon processors

• Data-center operators prefer this approach because all servers are

interchangeable to meet rapidly evolving market needs

• Easy to program and deploy new services

• But coprocessors can improve operating costs

• Doubling throughput of algorithm cuts number of servers in half

• Microsoft testing coprocessors (for Bing)

• Little deployment as yet

• Too few vision-based services to make it worth developing custom

hardware


Sorting Through the Options

• Many options for accelerating vision processing

• Discrete GPU chips/cards (e.g. GeForce, Radeon)

• High-performance DSP chips (e.g. TI C62xx)

• Design custom FPGA to accelerate algorithms

• Various levels of difficulty in building, programming

• GPUs can be programmed through APIs (e.g. OpenCL, CUDA)

• DSPs are typically programmed by hand for best performance

• FPGAs typically use a hardware design language (e.g. VHDL)

• Tools emerging to program FPGAs using C-like language

• Unlike true hardware (ASIC), FPGAs can be reprogrammed

• FPGAs are more flexible for a variety of (non-vision) applications

• Not clear what will emerge as winning approach


What Do “Things” Need to See?

• Internet of Things spans a broad set of applications

• Some of which are not yet defined or even known

• Smart electric meters, parking lots, vending machines, washer/dryer,

thermostat (Nest), security cameras, lighting, door locks…

• Most IoT devices don’t need vision

• Smart parking needs to sense if parking space is occupied

• Other industrial-control applications

• Security cameras could use vision processing

• Automatically detect change in scene

and notify owner via Internet

• Automatically adjust lighting, temperature

if no activity detected

Nest thermostat


Internet of Things Forecast Remains Vague

0

200

400

600

800

1000

1200

1400

1600

1800

2000

2013 2014 2015 2016 2017 2018 2019 2020

Mill

ion

s o

f U

nit

s

"Things"

Consumer

Commercial

• Commercial systems such as smart parking can cost hundreds

of dollars per unit (space) and still be profitable to install

• Consumer systems (e.g., cameras) must be low cost

Lights, doors, windows, etc

Washer, dryer, other appliances

Smart meters, parking, vending, industrial



Conclusions

• Mobile is the largest volume market for vision processing

• Vision hardware already integrated into leading hardware platforms

• Near-term opportunity to provide IP or coprocessors to other mobile-

platform suppliers (e.g. Apple, MediaTek, Samsung, Spreadtrum)

• Long-term opportunity to provide software for these platforms

• Some mobile vision processing will be in the cloud

• Most processing will be on standard Xeon platforms

• As volume grows, opportunity to provide hardware acceleration

• Low volumes but high ASP

• Automotive vision processing is growing rapidly

• ADAS requires several powerful processors per car

• Internet of Things is hot, but few opportunity for vision


You have questions?

We have answers.