• Home

  • Technology

  • "Caffe to Zynq: State-of-the-Art Machine Learning Inference Performance in Less Than 5...
18
Nick Ni and Vinod Kathail May 2017 Caffe to Zynq: State - of - the - Art Machine Learning Inference Performance in Less Than 5 Watts

"Caffe to Zynq: State-of-the-Art Machine Learning Inference Performance in Less Than 5 Watts," a Presentation from Xilinx

Embed Size (px)

Citation preview

Page 1: "Caffe to Zynq: State-of-the-Art Machine Learning Inference Performance in Less Than 5 Watts," a Presentation from Xilinx

Copyright © 2017 Xilinx, Inc. 1

Nick Ni and Vinod Kathail

May 2017

Caffe to Zynq:

State-of-the-Art Machine Learning Inference Performance in Less Than 5 Watts

Page 2: "Caffe to Zynq: State-of-the-Art Machine Learning Inference Performance in Less Than 5 Watts," a Presentation from Xilinx

Copyright © 2017 Xilinx, Inc. 2

• Why Zynq SoCs for Deep Learning Inference

• Caffe to Zynq SoC in Seconds

• A Full System Example

Agenda

Page 3: "Caffe to Zynq: State-of-the-Art Machine Learning Inference Performance in Less Than 5 Watts," a Presentation from Xilinx

Copyright © 2017 Xilinx, Inc. 3

Diverse Applications with Diverse Design Targets

ADAS

High accuracy

Low latency

Medical

Diagnosis:

Small networks

Robotics:

Real-time

Hearing Aids:

Small network

Low latency

Translate &

AlphaGo:

Huge networks

Page 4: "Caffe to Zynq: State-of-the-Art Machine Learning Inference Performance in Less Than 5 Watts," a Presentation from Xilinx

Copyright © 2017 Xilinx, Inc. 4

Zynq Offers the Most Efficient Deep Learning

Inference

Page 5: "Caffe to Zynq: State-of-the-Art Machine Learning Inference Performance in Less Than 5 Watts," a Presentation from Xilinx

Copyright © 2017 Xilinx, Inc. 5

Zynq SoCs Offer Superior Throughput, Latency

6xImages/sec/watt

1/5Latency (ms)

Machine

Learning

Inference

Xilinx

Benchmark

Xilinx

Benchmark• eGPU = nVidia TX1: nVidia GoogLeNet performance: https://devblogs.nvidia.com/parallelforall/jetpack-doubles-jetson-tx1-deep-learning-inference/

GoogLeNet

@ batch = 1

Xilinx ZU9 Xilinx ZU5 eGPU*

Images/s 370.0 155.0 70

Power (W) 7.0 4.5 7.9

Images/s/watt 53.0 34.5 8.9

Real Time

Applications

LatencyGoogLeNet

@ batch = 1

Xilinx ZU9 Xilinx ZU5 eGPU*

Images/s 370.0 155.0 70

Latency (ms) 2.7 6.4 14.2

GoogLeNet

@ batch = 8

Xilinx ZU9 Xilinx ZU5 eGPU*

Images/s 370.0 155.0 163

Latency (ms) 2.7 6.4 49.0

For large batch,

CPU/GPU/DSPs latency

increases significantly

https://devblogs.nvidia.com/parallelforall/jetpack-doubles-jetson-tx1-deep-learning-inference/
Page 6: "Caffe to Zynq: State-of-the-Art Machine Learning Inference Performance in Less Than 5 Watts," a Presentation from Xilinx

Copyright © 2017 Xilinx, Inc. 6

Training: Process for machine to

“learn” and optimize model from data

Inference: Using trained models to

predict/estimate outcomes from new

observations in efficient deployments

INFERENCE

Fewer

“dog” ✓

“dog”

Input

”cat”

TRAINING

= ?

Many

labels

Error

Input

FP-32FIXED-16

(INT16)

FIXED-8

(INT8)

Difference

vs FP32

VGG-16 86.6% 86.6% 86.4% (0.2%)

GoogLeNet 88.6% 88.5% 85.7% (2.9%)

SqueezeNet 81.4% 81.4% 80.3% (1.1%)

Inference now 8 bit and below

for maximum efficiency

The Divergence of Training and Inference

Top-5

Accuracy

https://arxiv.org/pdf/1510.00149v5.pdf

https://arxiv.org/pdf/1510.00149v5.pdf
Page 7: "Caffe to Zynq: State-of-the-Art Machine Learning Inference Performance in Less Than 5 Watts," a Presentation from Xilinx

Copyright © 2017 Xilinx, Inc. 7

Inference Precisions Moving to Lower and Variable

Precision

Citation: https://arxiv.org/pdf/1510.00149.pdf

# of weight bits # of weight bits

Convolution Layers Fully Connected Layers

Page 8: "Caffe to Zynq: State-of-the-Art Machine Learning Inference Performance in Less Than 5 Watts," a Presentation from Xilinx

Copyright © 2017 Xilinx, Inc. 8

Future Proof Architecture for Any Precisions

Limited to 32 bit operations

SIMD operations at INT8/16

New devices required to

support change in precision efficiently

Reconfigurable to scale and

optimize for different precisions

Page 9: "Caffe to Zynq: State-of-the-Art Machine Learning Inference Performance in Less Than 5 Watts," a Presentation from Xilinx

Copyright © 2017 Xilinx, Inc. 9

TOPS

8b• Reducing precision from 8b to 1b shrinks LUT cost by 40x

• Potential to scale CNN performance to above 23TOPS (ZU9)

BNN: Unparalleled Performance

8b

4b

1b1b1b1b1b1b1b1b

Q3TX2 ZU9

1.50.3 2.7

1b1b1b1b1b1b1b1b

1b1b1b1b1b1b1b1b

1b1b1b1b1b1b1b1b

1b1b1b1b1b1b1b1b

1b1b1b1b1b1b1b1b

1b1b1b1b1b1b1b1b

1b1b1b1b1b1b1b1b

1b1b1b1b1b1b1b1b

TOPS

(1b)

23

ZU9Q3TX2

1.50.3

Page 10: "Caffe to Zynq: State-of-the-Art Machine Learning Inference Performance in Less Than 5 Watts," a Presentation from Xilinx

Copyright © 2017 Xilinx, Inc. 10

Low Latency Inference by Layer to Layer Dataflow

On Chip

GPU/CPU: Dataflow using

Off chip memory

Xilinx: Maximum local data reuse

“layer merging” between layers

On-chip MemoryNvidia Tegra X1 (GPU Regfile + L2) Xilinx ZU7 (BRAM + URAM)

6 Mb 38 Mb

Up to 6x More On-chip Memory than SoCs and eGPUs

Nvidia TX1 spec: http://wccftech.com/nvidia-tegra-x1-super-chip-announced-ces-2015-features-maxwell-core-architecture-256-cuda-cores/

Page 11: "Caffe to Zynq: State-of-the-Art Machine Learning Inference Performance in Less Than 5 Watts," a Presentation from Xilinx

Copyright © 2017 Xilinx, Inc. 11

Frameworks

Libraries and Tools

Development Kits

DNN

CNNGoogLeNet

SSD

FCN …

OpenVX

integration

Page 12: "Caffe to Zynq: State-of-the-Art Machine Learning Inference Performance in Less Than 5 Watts," a Presentation from Xilinx

Copyright © 2017 Xilinx, Inc. 12

xFdnn: Direct Deep Learning Inference from Caffe

Compiles only ARM software code in minutes. No hardware compilation.

1 2 3Import .prototxt and

trained weights

Call prototxt

runtime API in

your application

Cross-compile for

Cortex-A53 and run

on a board

Page 13: "Caffe to Zynq: State-of-the-Art Machine Learning Inference Performance in Less Than 5 Watts," a Presentation from Xilinx

Copyright © 2017 Xilinx, Inc. 13

Deep Learning Design Examples

Page 14: "Caffe to Zynq: State-of-the-Art Machine Learning Inference Performance in Less Than 5 Watts," a Presentation from Xilinx

Copyright © 2017 Xilinx, Inc. 14

Building a Full Embedded Vision System

System Optimizing

Compiler Machine Learning

.prototxt

& Trained

Weights

DNN

CNNGoogLeNet

SSD

FCN …

Scheduling of Pre-Optimized

Neural Network Layers

Optimized Accelerators

& Data Motion Network

Page 15: "Caffe to Zynq: State-of-the-Art Machine Learning Inference Performance in Less Than 5 Watts," a Presentation from Xilinx

Copyright © 2017 Xilinx, Inc. 15

Building a Full Embedded Vision System

C/C++/OpenCL

Creation

Profiling to Identify

Bottlenecks

System Optimizing

Compiler

Computer Vision

Machine Learning

Scheduling of Pre-Optimized

Neural Network Layers

Optimized Accelerators

& Data Motion Network

.prototxt

& Trained

Weights

DNN

CNNGoogLeNet

SSD

FCN …

OpenVX

integration

Page 16: "Caffe to Zynq: State-of-the-Art Machine Learning Inference Performance in Less Than 5 Watts," a Presentation from Xilinx

Copyright © 2017 Xilinx, Inc. 16

Putting It All Together: CV and CNN with Multiple

Sensors

USB-3Stereo Vision

MIPI CSI ISP

SD Card File Read

VPSSScaler

Video Mixer

Frame Buffer

Optical Flow

CNN

500x500x10RGB

1280x720p60YUYV 4:2:2

2x720p60YUYV 4:2:2

Frame Buffer

Frame Buffer

Frame Buffer

HDMI TX

3840x2160p60RGB

Page 17: "Caffe to Zynq: State-of-the-Art Machine Learning Inference Performance in Less Than 5 Watts," a Presentation from Xilinx

Copyright © 2017 Xilinx, Inc. 17

• Zynq SoCs offer superior performance and lower latency compared to

other SoC offerings

• reVISION stack provides seamless inference of custom deep learning

networks from Caffe to Zynq SoCs

• Available publically in end of May 2017

• Visit Xilinx.com/reVISION for more information

Summary

Come Visit the Xilinx Booth in the

Technology Showcase to see the

reVISION stack in action!!

Page 18: "Caffe to Zynq: State-of-the-Art Machine Learning Inference Performance in Less Than 5 Watts," a Presentation from Xilinx

Copyright © 2017 Xilinx, Inc. 18

• INT8 Whitepaper

• Machine Learning Whitepaper

• reVISION Backgrounder

• Additional Papers & Tutorials

• Xilinx Embedded Vision Videos

• Forums

Resources

https://www.xilinx.com/support/documentation/white_papers/wp490-embedded-vision-int8.pdf
https://www.xilinx.com/support/documentation/backgrounders/Machine-Learning-Application-Landscape.pdf
https://www.xilinx.com/support/documentation/backgrounders/revision_backgrounder.pdf
https://www.xilinx.com/products/design-tools/embedded-vision-zone/revision-knowledge-center.html
https://www.xilinx.com/products/design-tools/embedded-vision-zone/revision-knowledge-center.html#video
https://forums.xilinx.com/t5/Embedded-Systems/ct-p/EMBED

Programming ARM TrustZone Architecture on the Xilinx Zynq

Programming ARM TrustZone Architecture on the Xilinx Zynq

Documents
Xilinx 7 Series FPGA and Zynq-7000 All Programmable SoC

Xilinx 7 Series FPGA and Zynq-7000 All Programmable SoC

Documents
Xilinx® Zynq®7000 series 5W Small, Efficient, Low … · Xilinx® Zynq®7000 series 5W Small, Efficient, Low-Noise Power Solution ). 1)) ) . IMPORTANT NOTICE FOR TI REFERENCE DESIGNS

Xilinx® Zynq®7000 series 5W Small, Efficient, Low … · Xilinx® Zynq®7000 series 5W Small, Efficient, Low-Noise Power Solution ). 1)) ) . IMPORTANT NOTICE FOR TI REFERENCE DESIGNS

Documents
Xilinx XAPP1092 Implementing SMPTE SDI Interfaces with Zynq …xilinx.eetrend.com/files-eetrend-xilinx/download/201401/... · 2019. 1. 17. · Using Zynq-7000 SoC GTX Transceivers

Xilinx XAPP1092 Implementing SMPTE SDI Interfaces with Zynq …xilinx.eetrend.com/files-eetrend-xilinx/download/201401/... · 2019. 1. 17. · Using Zynq-7000 SoC GTX Transceivers

Documents
Xilinx 7 Technology - Xilinx Serie 7 Zynq Technology · Xilinx 7 Technology Xilinx Serie 7 Zynq Technology BTE5380 ... Notes Notes Notes. ... Xilinx 7 Technology - Xilinx Serie 7

Xilinx 7 Technology - Xilinx Serie 7 Zynq Technology · Xilinx 7 Technology Xilinx Serie 7 Zynq Technology BTE5380 ... Notes Notes Notes. ... Xilinx 7 Technology - Xilinx Serie 7

Documents
XILINX CONFIDENTIAL. Design AXI Master Page 1. XILINX CONFIDENTIAL. Understanding Zynq AXI Master IP axi_user_npi Page 2 Agenda © Copyright 2012 Xilinx

XILINX CONFIDENTIAL. Design AXI Master Page 1. XILINX CONFIDENTIAL. Understanding Zynq AXI Master IP axi_user_npi Page 2 Agenda © Copyright 2012 Xilinx

Documents
Powering the Xilinx® Zynq® 7015 FPGA with … · TIDA-00551 - Test ReportPowering the Xilinx® Zynq® 7015 FPGA with TPS659110 Power Management IC 1 ... Xilinx® Zynq® 7010/15

Powering the Xilinx® Zynq® 7015 FPGA with … · TIDA-00551 - Test ReportPowering the Xilinx® Zynq® 7015 FPGA with TPS659110 Power Management IC 1 ... Xilinx® Zynq® 7010/15

Documents
Ubuntu on Zynq Tutorial 03 - Xilinx 2 ‐ Create the Second Stage Boot Loader (U‐boot ... Ubuntu On Zynq Tutorial Xilinx SDK 2013.4

Ubuntu on Zynq Tutorial 03 - Xilinx 2 ‐ Create the Second Stage Boot Loader (U‐boot ... Ubuntu On Zynq Tutorial Xilinx SDK 2013.4

Documents
Xilinx 7 Series FPGA and Zynq-7000 All Programmable SoC ...€¦ · Xilinx 7 Series FPGA and Zynq-7000 All Pr ogrammab le SoC Libraries Guide for HDL Designs UG768 (v14.7) October

Xilinx 7 Series FPGA and Zynq-7000 All Programmable SoC ...€¦ · Xilinx 7 Series FPGA and Zynq-7000 All Pr ogrammab le SoC Libraries Guide for HDL Designs UG768 (v14.7) October

Documents
Integrated Power Supply Reference Design for Xilinx® Zynq

Integrated Power Supply Reference Design for Xilinx® Zynq

Documents
Debian Linux on Zynq (Xilinx ARM-SoC FPGA) Setup Flow (Vivado 2015.4)

Debian Linux on Zynq (Xilinx ARM-SoC FPGA) Setup Flow (Vivado 2015.4)

Technology
PAN 3UVPX ZYNQ+ 3U VPX Xilinx Zynq UltraScale+ ….pdfPAN-3UVPX-ZYNQ+ is a 3U OpenVPX module based on the Zynq UltraScale+ MPSoC from Xilinx. The Zynq UltraScale+ integrates a …

PAN 3UVPX ZYNQ+ 3U VPX Xilinx Zynq UltraScale+ ….pdfPAN-3UVPX-ZYNQ+ is a 3U OpenVPX module based on the Zynq UltraScale+ MPSoC from Xilinx. The Zynq UltraScale+ integrates a …

Documents
2014/09/01 - XILINX - The Zynq book (tutorials)

2014/09/01 - XILINX - The Zynq book (tutorials)

Engineering
Xilinx Quick Emulator: User Guide QEMU€¦ · Xilinx provides a Quick Emulator (QEMU) for software developers targeting Zynq ®-7000 SoC, Zynq ® UltraScale+™ MPSoC, and MicroBlaze™

Xilinx Quick Emulator: User Guide QEMU€¦ · Xilinx provides a Quick Emulator (QEMU) for software developers targeting Zynq ®-7000 SoC, Zynq ® UltraScale+™ MPSoC, and MicroBlaze™

Documents
Zynq-7000 EPP ZC702 Evaluation Kit - Xilinx · The Xilinx® ZYNQ™-7000 EPP ZC702 Evaluation Kit gives developers a complete development platform including hardware, development

Zynq-7000 EPP ZC702 Evaluation Kit - Xilinx · The Xilinx® ZYNQ™-7000 EPP ZC702 Evaluation Kit gives developers a complete development platform including hardware, development

Documents
Debugging Embedded Cores in Xilinx FPGAs [Zynq] · Debugging Embedded Cores in Xilinx FPGAs [Zynq] 2 ©1989-2018 Lauterbach GmbH Debugging Embedded Cores in Xilinx FPGAs [Zynq] Version

Debugging Embedded Cores in Xilinx FPGAs [Zynq] · Debugging Embedded Cores in Xilinx FPGAs [Zynq] 2 ©1989-2018 Lauterbach GmbH Debugging Embedded Cores in Xilinx FPGAs [Zynq] Version

Documents
Caffe to Zynq: State-of-the-Art Machine Learning … 24... · Caffe to Zynq: State-of-the-Art Machine Learning Inference Performance in Less Than 5 Watts Vinod Kathail, Distinguished

Caffe to Zynq: State-of-the-Art Machine Learning … 24... · Caffe to Zynq: State-of-the-Art Machine Learning Inference Performance in Less Than 5 Watts Vinod Kathail, Distinguished

Documents
Xilinx UG480 7 Series FPGAs and Zynq-7000 All Programmable

Xilinx UG480 7 Series FPGAs and Zynq-7000 All Programmable

Documents
A Hardware Designer's Informal Guide to Xilinx® Zynq

A Hardware Designer's Informal Guide to Xilinx® Zynq

Documents
Xilinx UG925, Zynq-7000 EPP ZC702 Base Targeted Reference Design

Xilinx UG925, Zynq-7000 EPP ZC702 Base Targeted Reference Design

Documents
Xilinx Zynq-7000 All Programmable SoC and Express … · Xilinx Zynq-7000 ZC702 Evaluation Board, with Dual ARM Cortex-A9 processors, 10/100/1000Mbps Ethernet MAC, USB 2.0 OTG, CAN

Xilinx Zynq-7000 All Programmable SoC and Express … · Xilinx Zynq-7000 ZC702 Evaluation Board, with Dual ARM Cortex-A9 processors, 10/100/1000Mbps Ethernet MAC, USB 2.0 OTG, CAN

Documents
 · DS152 (v3.4) January 12, 2012 Product Specification 1 © 2009–2012 Xilinx, Inc. XILINX, the Xilinx logo, Virtex, Zynq, Artix, Kintex, Spartan, ISE

 · DS152 (v3.4) January 12, 2012 Product Specification 1 © 2009–2012 Xilinx, Inc. XILINX, the Xilinx logo, Virtex, Zynq, Artix, Kintex, Spartan, ISE

Documents
Programming ARM TrustZone Architecture on the Xilinx … · Programming ARM TrustZone Architecture on the Xilinx Zynq ... Programming ARM TrustZone Architecture on the Xilinx

Programming ARM TrustZone Architecture on the Xilinx … · Programming ARM TrustZone Architecture on the Xilinx Zynq ... Programming ARM TrustZone Architecture on the Xilinx

Documents
Xilinx Zynq®-7000 Platform Software Development Using the ARM

Xilinx Zynq®-7000 Platform Software Development Using the ARM

Documents
Python productivity for Zynq (Pynq) Documentation · PDF filePython productivity for Zynq (Pynq) Documentation, Release 2.0 Xilinx® makes Zynq® devices, a class of All Programmable

Python productivity for Zynq (Pynq) Documentation · PDF filePython productivity for Zynq (Pynq) Documentation, Release 2.0 Xilinx® makes Zynq® devices, a class of All Programmable

Documents
XA Zynq-7000 All Programmable SoC First Generation - Xilinx

XA Zynq-7000 All Programmable SoC First Generation - Xilinx

Documents
Xilinx XAPP1170 Zynq-7000 All Programmable SoC Accelerator for

Xilinx XAPP1170 Zynq-7000 All Programmable SoC Accelerator for

Documents
Xilinx Zynq-7000 All Progammable SoC Software …class.ece.iastate.edu/cpre488/resources/ug821-zynq-7000-swdev.pdf · SoC Software Developers Guide UG821 (v5.0) ... Zynq-7000 AP Soc

Xilinx Zynq-7000 All Progammable SoC Software …class.ece.iastate.edu/cpre488/resources/ug821-zynq-7000-swdev.pdf · SoC Software Developers Guide UG821 (v5.0) ... Zynq-7000 AP Soc

Documents
Zynq TRM - Xilinx - All Programmable TRM - Xilinx - All Programmable

Zynq TRM - Xilinx - All Programmable TRM - Xilinx - All Programmable

Documents
OpenAMP Framework for Zynq Devices - Xilinx€¦ · OpenAMP Framework for Zynq Devices Getting Started Guide UG1186 (v2016.2) July 5 , 2016

OpenAMP Framework for Zynq Devices - Xilinx€¦ · OpenAMP Framework for Zynq Devices Getting Started Guide UG1186 (v2016.2) July 5 , 2016

Documents