Online Webinar June 16, 2016 - Intel · Online Webinar June 16, 2016 Intel® Media SDK Intel® SDK for ... Embedded Systems & IoT (CPU) Intel® IoT Intel® System Developer Kit Studio

Online Webinar

June 16, 2016

Intel® Media SDKIntel® SDK for

opencl™ applications

Copyright © 2016, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.

Optimization NoticeOpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission by Khronos.

2

Welcome!Meet our Experts

Robert IoffeOpenCL™ Software Technical Consulting Engineer

Developer Products [email protected]

Jeff McAllisterMedia Software Senior Technical Consulting Engineer

Developer Products [email protected]

Intel Software & Services Group

2

mailto:[email protected]

mailto:[email protected]


Optimization Notice

3

What We’ll Cover Today

Introduction

Intel® Processor

Graphics -the Other Side of

the Chip

Programming Model

Examples

Intel OpenCL™ & Media

SDKs

Better Together

Q&A

3OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission by Khronos.

Take advantage of advanced media accelerators, graphics processing & programmable graphics


Optimization Notice

Intel CPUs + Intel® Processor Graphics (GPUs):A Winning Combination

Intel® HD Graphics6th Gen Intel Core Processor

If you’re only targeting CPUs, you’re not fully utilizing the chip

See Technical Specifications for hardware requirements & other details.Not shown Intel® Xeon® Processor Graphics 5

OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission by Khronos.

https://software.intel.com/en-us/intel-media-server-studio/details#technical


Optimization Notice

6

Hardware Overview

Graphics Technology Highlights

Execution Units (EUs) = general purpose cores

EUs, samplers, caches, etc. in “slices”

Fixed function is in “unslice”

eDRAM adds cache, increases bandwidth

Other Names Summary

Intel® HD Graphics

24 EUs

1x FF

GT2“4+2”

Good

Intel® Iris™ Graphics

48 EUs

2x FF

eDRAM

GT3“2+3e”

Better

Intel® Iris™ Pro Graphics

72 EUs

2x FF

eDRAM

GT3e,GT4e“4+4e”

Best

Naming Convention

Look for Intel Processor Graphics at ark.intel.com

GT2

GT3

GT4

Fixed Function (VDBox, VEBox)


ark.intel.com


Optimization Notice

7

Intel® SDK for OpenCL™ Applications &Intel® Media SDK: Better together for video apps

• AVC/H264, HEVC/H265, MPEG2, MJPEG …

Codecs

• Resize/crop, color conversion, deinterlace, composition/blend, FRC, telecine/interlace reversal…

Filters

Increasing GPGPU + fixed function = high performance custom pipelines



Optimization NoticeOpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission by Khronos.

8

Takeaways

Intel® Media SDK & Intel® SDK for OpenCL™ Applications used together allows programmers to exploit the full computational capabilities of Intel® processors by utilizing CPU, GPU & fixed function hardware.

Intel offers drivers, runtimes/compilers & a suite of tools allowing video application developers to create, build, debug, analyze & optimize.

Intel tools & hardware help you create exciting new applications & usages for image/video processing, virtual & augmented reality, artificial intelligence, robotics, & machine learning (including deep learning).

8

Get the most out of Intel® platforms with Intel® Software Development Tools


Optimization Notice

Intel® Software Development Tools

10

Technical Computing

Embedded Systems & IoT(CPU)

Intel® IoTDeveloper KitIntel® System

Studio

Heterogeneous Tools (GPU)

Intel® Media SDKIntel® SDK for

OpenCL™ Applications

(future)

New Interface

Intel® RealSense SDK


https://software.intel.com/intel-opencl




Optimization Notice

11

Developer Hardware Access

Media Fixed Function

VDbox/MFX VEbox/VQE EUs

LLC (eDRAM)

Core Core

LLC

Core Core

LLC

Ring

Media Accelerator Model (GPU)Intel® Media Server Studio• Intel® SDK for OpenCL™ Applications• Intel® Media SDK

GPU Components

CPU Components

Traditional CPU ModelIntel® Parallel StudioIntel® System Studio• CPU Compilers, performance

libraries, etc.System Agent


https://software.intel.com/en-us/intel-media-server-studio


https://software.intel.com/en-us/media-sdk


Optimization Notice

Media Accelerator vs. Traditional Model Comparison

12

Decode

MSDK Decode

DXVA/ VAAPI

Driver

VDBOX

Frame Processing

MSDK VPP

OpenCL GPU

Driver

EUs, VEBOX

Encode

MSDK Encode

VAAPI, OpenCL (+ VME extension)

Driver

EUs, VDBOX

GPU Driver

CPU

Library

App

Media Application

GPU (Accelerator Model, includes CPU)

CPU (Traditional Model)

CPU

Appor



Optimization Notice

OpenCL & Media SDK Programming ModelsM

ain

lo

op

Init

iali

za

tio

n

Decode

init

Parameters (from header)

stream frame VPP frame Encode stream

init

Parameters (in & out)

init

Parameters

__kernel void dot (…)

{

int tid = get_global_id(0);

c[tid] = dot(a[tid], b[tid]);

}

Application(written C, C++, …)

Khronos ICDOpenCL API

RuntimeCompiler

Front End (Clang)

Back End (LLVM)

GPU Device

CPU Device

Coprocessor Device

SPIR

Intel® SDK for OpenCL™ Applications

Kernels to Accelerate(OpenCL C)

Intel® Media SDK



Optimization Notice

Intel® Media SDK Value Propositions

14

Be

ne

fits

Hardware Acceleration: Utilize full capabilities of

Intel processors.

Easy optimizations: Simple high level

asynchronous API – no need for specialized

developers to code to the metal.

Multi-platform/OS support: Utilize Intel® Xeon®

and Core™ in Windows* and Linux*

Future proofed: Align with Intel hardware strategy

for the longterm. Optimize for new hardware

without code changes Imp

act

s Time-to-Market (TTM): Automatically utilize latest

hardware capabilities on Linux & Windows without

multiple implementations. Write in Windows,

deploy on Linux – or vice versa.

Fully utilize Intel hardware capabilities for more

powerful, cost efficient devices/platforms.

• Client: Better experience, longer battery life

• Server: Higher density = more streams per server

& lower space/power costs

Disruptive technology: more streams per server

with lower space/power costs changes the rules of

the game and makes new things possible.

One unified API for codecs & frame processing across hardware generations



Optimization Notice

15

Codecs + Frame Processing use Fixed Function + EUs

EU EU

EU EU

EU EU

EU EU

Sampler

EU EU

EU EU

EU EU

EU EU

3D

FFMedia Fixed Function

VDBOX VEBOX

EU EU

EU EU

EU EU

EU EU

VPP

Video Decoding BSD=VDBox decode

Caches

Video EncodingENC= EU+VDBox VME (MB type, motion vectors, bit budget/BRC)

PAK = VDBox (residue packing & entropy coding)

VDENC = low power encode (6th Generation Core® & forward)

VPHalVideo Processing Hardware

Acceleration Layer

VEBox

• Deinterlacing

• Denoise (Luma/Chroma)

• Frame Rate Conversion

• Color space conversions

• Composition/alpha blending

• ScalingSampler Sampler



Optimization Notice

16

Basic Structure of a Media SDK-optimized Application

Ap

pli

cati

on

Initialize Session, set parameters

Query + Allocate

Main loop

Find free surface

Q stages: decode, VPP Encode

Sync

Retrieve output

Drain loop Same as above

Clean up, exit



Optimization Notice

17

SDKs More Efficient Together

CPU

GPU

Entire pipeline on GPU for greatest efficiency

Decode

Process

Encode

X X

CPU

GPUDecode Process Encode

Problem Solution

Expensive GPU<->CPU copy/sync

Key

Intel® Media SDK Component

Intel® SDK for OpenCL™ Applications Component



Optimization Notice

18

Hardware View (OpenCL perspective)

Extensions: Built-in fixed function access• Load function in app init

• clCreateProgramWithBuiltInKernels

• set parameters (src/ref images, outputs, …)

• enqueueNDRangeKernel as usual

GPGPU for EUs• Low level C subset/SPIR provides instructions

• 2x 128-bit SIMD FPUs (8 32 bit float or int operations per cycle)

• Images (I/O for MSDK) read through sampler

• Buffers read through data port

• On many processors EU capabilities higher than CPUs



Optimization Notice

Motion Estimation Extensions

19

Two extensions for hardware video motion estimation

Host APIs & built-in kernels & associated constant definitions

cl_intel_motion_estimation (1.0)

• Defines an accelerator for cl_intel_accelerator extension

• Provides a built-in kernel for Intel Graphics Hardware motion estimation

cl_intel_advanced_motion_estimation (2.0)

• New built-in kernel for shape decisions

• Intra/Inter, bi/uni-directional prediction, partitioning, partition size

Available Today: Khronos Spec & Sample

www.khronos.org/registry/cl/extensions/intel/cl_intel_advanced_motion_estimation.txt

software.intel.com/en-us/articles/intro-to-advanced-motion-estimation-extension-for-opencl


http://www.khronos.org/registry/cl/extensions/intel/cl_intel_advanced_motion_estimation.txt

https://software.intel.com/en-us/articles/intro-to-advanced-motion-estimation-extension-for-opencl


Optimization Notice

20

Media SDK+OpenCL Application

Ap

pli

cati

on

Initialize Session, set parameters

Compile/load OpenCL kernels

Query + Allocate (add clCreateFromDX9MediaSu

rfaceKHR)

Main loop

Find free surface

Q stages: decode, VPP, …

Sync

Launch kernel

clEnqueueAcquireDX9MediaSurfacesKHR

set params, clEnqueueNDRangeKernel

clEnqueueReleaseDX9MediaSurfacesKHR

…



Optimization Notice

22

How-To

1. A minimal example (code walk through)2. Use case: Color conversion filter3. Use case: BRC

Advantages:• A few simple steps to keep your pipeline unified• Constant workload boosts GPU to peak frequency• Free your CPUs for other tasks• Create new capabilities not possible otherwise in small space/power/cost constraints

Fixed Function

Performance

Your innovation via GPGPU

New possibilities!



Optimization Notice

23

A Simple Kernel

__kernel void Flip(

__read_only image2d_t src,

__write_only image2d_t dst,

int h)

{

int2 coord_src = (int2)(get_global_id(0), get_global_id(1));

int2 coord_dst;

coord_dst.x = coord_src.x;

coord_dst.y = h - coord_src.y - 1;

uint4 pixel = read_imageui(src, CLK_FILTER_NEAREST, coord_src);

write_imageui(dst, coord_dst, pixel.xyzw);

}

Init Decode Process

Inner loop, only 4 pixels



Optimization Notice

24

Kernel Components (parameters)

__kernel void Flip(



int h)

{


int2 coord_dst;





}

kernel name + parameters



Optimization Notice

25

Kernel Components (coordinate calculations)

__kernel void Flip(



int h)

{


int2 coord_dst;





}

where to get data, where

to put it



Optimization Notice

26

Kernel Components (read data in)

__kernel void Flip(



int h)

{


int2 coord_dst;





}

fn to read from sampler, not

automatic like traditional CPU

model



Optimization Notice

Kernel Components (operation + output)

__kernel void Flip(



int h)

{


int2 coord_dst;




//do something more interesting here


}

Combine steps

fn to write back



Optimization Notice

28

Decode Steps

for (;;)

{

sts = mfxDEC.DecodeFrameAsync(&mfxBSin, &pmfxSurfaces[nIndex],

&pmfxFrameSurfOut, &syncp);

if (MFX_WRN_DEVICE_BUSY == sts) {MSDK_SLEEP(1); continue;}

if (MFX_ERR_MORE_DATA == sts) {sts = ReadBitStreamData(&mfxBSin, fSource);

continue;}

if (MFX_ERR_MORE_SURFACE == sts) {nIndex = getFreeSurfaceIndex(pmfxSurfaces,

numSurfaces); continue;}

if (MFX_WRN_VIDEO_PARAM_CHANGED == sts) continue; //todo: handle resolution

changes, etc.

if (MFX_ERR_NONE == sts) break; //new frame is ready after sync

}

sts = session.SyncOperation(syncp, 60000); // Wait until decoded frame is

ready

Media SDK decode may loop through several states before completing

Init Decode Process



Optimization Notice

Expected Return Codes for DecodeFrameAsync

29

Simplified Decode Flow

DecodeFrameAsync

(bitstream in)

MFX_ERR_MORE_SURFACE

MFX_ERR_MORE_DATA

DecodeFrameAsync

(null in)


Initialize

Finish(MFX_ERR_MORE_DATA indicates all surfaces drained)

Main loop Drain loop

Mo

re in

pu

t

Inp

ut

fin

ish

ed


•A new surface is required to proceed this is where decode will write its output

MFX_ERR_MORE_DATA

•More input bitstream data is required to proceed

MFX_WRN_DEVICE_BUSY

•Hardware device is unable to respond. This is an expected output for normal operation & should clear shortly. If this state persists more than a few milliseconds, this may indicate a problem.

MFX_WRN_VIDEO_PARAM_CHANGED

•The SDK decoder parsed a new sequence header. Decoding can continue with existing frame buffers. The app can optionally retrieve new video parameters by calling MFXVideoDECODE_GetVideoParam.

Other

•Other error codes may be bugs. Contact Intel Support for more info.



Optimization Notice

30

Launching the OpenCL Kernel

clEnqueueAcquireDX9MediaSurfacesKHR(m_clQueue, 4, m_oclSurfaces, 0, NULL, NULL);

// Y plane

clSetKernelArg(m_clKernel, 0, sizeof(cl_mem), &m_pOCLBuffers[OCL_IN][midIdxIn].OCL_Y); // In

clSetKernelArg(m_clKernel, 1, sizeof(cl_mem), &m_pOCLBuffers[OCL_OUT][midIdxOut].OCL_Y); // Out

clSetKernelArg(m_clKernel, 2, sizeof(cl_int), &m_frameSizeY[1]); // Frame height

clEnqueueNDRangeKernel(m_clQueue, m_clKernel, 2, NULL,

m_GlobalWorkSizeY, m_LocalWorkSizeY, 0, NULL, NULL);

// UV plane

clSetKernelArg(m_clKernel, 0, sizeof(cl_mem), &m_pOCLBuffers[OCL_IN][midIdxIn].OCL_UV); // In

clSetKernelArg(m_clKernel, 1, sizeof(cl_mem), &m_pOCLBuffers[OCL_OUT][midIdxOut].OCL_UV); // Out

clSetKernelArg(m_clKernel, 2, sizeof(cl_int), &m_frameSizeUV[1]); // Frame height

clSts = clEnqueueNDRangeKernel(m_clQueue, m_clKernel, 2, NULL,

m_GlobalWorkSizeUV, m_LocalWorkSizeUV, 0, NULL, NULL);

clSts = clEnqueueReleaseDX9MediaSurfacesKHR(m_clQueue, 4, m_oclSurfaces, 0, NULL, NULL);

Init Decode Process



Optimization Notice

31

Use Case: Color Format Conversion Filter

Y0 Y1 Y2 Y3

U0 V0 U1 V1

R G B R G B R

RGB3 (packed format)

NV12 (planar format)

__kernel void RGB3_to_NV12(


__write_only image2d_t Ydst,

__write_only image2d_t UVdst,

…) {

coord calculations

read_imageui(src)

(calculate Y,U,V from src RGB)

write_imageui(Ydst)

write_imageui(UVdst)

}

For additional efficiency chain more

operations here

Encodeother

filters?



Optimization Notice

Use Case: Bitrate Control (BRC)

32

Two paths to improve codec quality1. Core algorithms2. Bitrate control

EncodeVME

Decode

bitstream

Custom quality

metric filter

pre-check frames

data for QP decisions

Motion estimation kernel output and custom quality metrics can be combined in novel ways to build custom BRC with your own secret sauce.



Optimization Notice

Visualize FF+GPGPU+CPUs working together with Intel® VTune™ Amplifier



Optimization Notice

Why OpenCL + Intel® Media SDK?

Decode EncodeProcess

What Media SDK Covers (high level)

OpenCL CoversFuller range/lower level

Media SDK provides optimized implementations for• Codecs• Frame Processing Operations

For video processing tasks not in Media SDK’s scope, extend with OpenCL• Make use of growing GPU capabilities• Keep pipelines on GPU

Example uses: color conversions, custom bitrate control

Fixed Function

Performance

Add your innovation via

GPGPU

Build something awesome!



Optimization Notice

36

Unleash the Full Range of Gen Graphics Capabilities for Video Processing Applications

CALL TO ACTION

Develop fast, efficient, competitive media apps - access media accelerators in Intel GPUs

• Servers & Embedded: Intel® Media Server Studio – free Community Edition

• For clients: Intel® Media SDK & Intel® SDK for OpenCL™ Applications free standalone tools

Get Started - use Media Samples & Tutorials




https://software.intel.com/en-us/intel-opencl

https://software.intel.com/en-us/intel-media-server-studio-support/code-samples

https://software.intel.com/en-us/intel-media-server-studio-support/training#tutorial

For more information, visit:

Intel® Media Server Studio & how-to articles

Intel® Media SDK & Intel® SDK for OpenCL™ Applications

The webinar materials & recording will be emailed to you soon.

Watch for more webinars in Q3 & Q4.


https://software.intel.com/en-us/intel-media-server-studio/documentation


https://software.intel.com/en-us/intel-opencl


Optimization Notice

Legal Notices, Disclaimers & Optimization Notice

Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. No computer system can be absolutely secure. Check with your system manufacturer or retailer or learn more at intel.com.

Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit http://www.intel.com/performance.

All information provided here is subject to change without notice. Contact your Intel representative, sales office or distributor to obtain the latest Intel product specifications and roadmaps.

Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products.

The cost reduction scenarios described in this document are intended to enable you to get a better understanding of how the purchase of a given Intel product, combined with a number of situation-specific variables, might affect your future cost and savings. Nothing in this document should be interpreted as either a promise of or contract for a given level of costs.

INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL® PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.

Intel, the Intel logo, Xeon, Core, Iris Pro, and VTune are trademarks of Intel Corporation in the U.S. and other countries.OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission by Khronos.

38

Optimization Notice

Intel’s compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Notice revision #20110804



Optimization Notice

How to Get the Intel® Media SDK

40

Intel® Media SDK - FREE

Platform / Device Targets• Intel® Core™ or Core™ M processors

• Select SKUs of Intel® Celeron™, Pentium™ & Atom™ processors with Intel® HD Graphics supporting Intel® Quick Sync Video

• Client devices – Desktop/mobile applications

• OS - Windows only*

See Technical Specifications for System Requirements

Format Support - HEVC, AVC, MPEG-2, MPEG-Audio

See Technical Specifications for System Requirements

Intel® Media Server Studio – 3 Editions (includes Free Community)

Platform / Device Targets• Select SKUs of Intel® Xeon® & Core™ processor-based platforms• Applications for media, communications infrastructure, video

processing/conferencing, digital surveillance, video cloud & data center

software.intel.com/media-sdksoftware.intel.com/intel-media-server-studio







Optimization Notice

41

Intel® Media Server Studio Editions–At a Glance

Feature/Component Community Edition Essentials Edition Professional Edition

Media SDK

Graphics Drivers

Code Samples

OpenCL™ Code Builder and Runtime

Metrics Monitor (Linux* only)

Intel® Premier Support

HEVC Decoder & Encoder, GPU Assist APIs

Audio Decoder & Encoder

Video Quality Caliper

Intel® VTune™ Amplifier

Premium Telecine Interlace Reverser

Premium Components

https://software.intel.com/en-us/opencl-code-builder

Documents

Online Webinar June 16, 2016 - Intel · Online Webinar June 16, 2016 Intel® Media SDK Intel® SDK for ... Embedded Systems & IoT (CPU) Intel® IoT Intel® System Developer Kit Studio