Intel® Media Accelerator Reference Software for Linux* · 2019-05-30 · Intel® Media Accelerator Reference Software for Linux* White Paper May 2019 6 Document Number: 608831-001

Document Number: 608831-001

Intel® Media Accelerator Reference

Software for Linux*

White Paper

Document Number: 608831-001

Document Revision: 1.0

05 2019

Intel® Media Accelerator Reference Software for Linux*

White Paper May 2019

2 Document Number: 608831-001

INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR

OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS

OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING

TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE,

MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.

A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death. SHOULD

YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS

SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS

COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT

LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS

SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS.

Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of

any features or instructions marked "reserved" or "undefined". Intel reserves these for future definition and shall have no responsibility whatsoever for

conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with

this information.

You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel® products

described herein. You agree to grant Intel® a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter

disclosed herein.

No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.

All information provided here is subject to change without notice. Contact your Intel® representative to obtain the latest Intel® product

specifications and roadmaps.

The products described may contain design defects or errors known as errata which may cause the product to deviate from published

specifications. Current characterized errata are available on request.

Copies of documents which have an order number and are referenced in this document may be obtained by calling 1-800-548-4725 or by visiting:

http://www.intel.com/design/literature.htm

Intel® technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn

more at http://www.intel.com/ or from the OEM or retailer.

No computer system can be absolutely secure.

Intel® and the Intel® logo are trademarks of Intel Corporation in the U.S. and/or other countries.

*Other names and brands may be claimed as the property of others.

Copyright © 2019, Intel® Corporation. All rights reserved.

http://www.intel.com/design/literature.htm

http://www.intel.com/

Intel® Media Accelerator Reference Software for Linux* May 2019 White Paper

Document Number: 608831-001 3

Contents

1.0 Introduction ............................................................................................................................. 6

1.1 What is Intel® Media Accelerator Reference Software? .................................................... 6

1.1.1 Definition .......................................................................................................................................... 6

1.2 Scope of this Document ............................................................................................................................ 7

1.3 Terminology ...................................................................................................................................................... 7

1.4 Reference Documents ................................................................................................................................ 8

2.0 Intel® Media Accelerator Reference Software – Linux* ................................ 9

2.1 Overview of Intel® Media Accelerator Reference Software ............................................ 9

2.2 Technologies Used ........................................................................................................................................ 9

2.2.1 Advanced Linux* Sound Architecture (ALSA*) ...................................................... 9 2.2.2 FFmpeg* ........................................................................................................................................ 10 2.2.3 Intel® Media SDK ................................................................................................................... 10 2.2.4 Intel® Automotive Solution Compositor for Wayland* Protocol

Weston ............................................................................................................................................ 10

2.3 Intel® Media Accelerator Reference Software High-Level Architecture .............. 10

2.3.1 Intel® Media Accelerator Reference Software Video Manager ................ 12

3.0 A High Level Look at Popular Linux* Media APIs and Frameworks ......14

3.1 Advantages and Disadvantages of Using Higher Level Programming APIs such

as FFmpeg* vs Intel® Media SDK .................................................................................................. 14

3.2 Advantages and Disadvantages of Using FFmpeg*-QSV vs FFmpeg*-VAAPI .. 15

3.3 Advantages and Disadvantages of Migrating to iHD .......................................................... 16

3.4 Advantages and Disadvantages of Using FFmpeg vs. GStreamer ........................... 17

4.0 A High Level Look at Weston vs. X11 Rendering ..............................................19

4.1 Advantages and disadvantages of using Weston vs X11 ............................................... 19

4.2 Advantages and Disadvantages of Using Intel® Automotive Solution

Compositor for Wayland* Protocol Weston vs. Other Wayland Reference

Implementations ......................................................................................................................................... 19

5.0 Conclusion ...............................................................................................................................22

6.0 A Look Ahead at Future Enhancements .................................................................23

7.0 How to Acquire the Intel® Media Acceleator Reference Software ........24




Figures

Figure 1. Intel ® Media Accelerator Reference Software for Linux* High Level

Architecture .................................................................................................................................................... 11 Figure 2 Video Manager .............................................................................................................................................. 13

Tables

Table 1. Terminology ...................................................................................................................................................... 7 Table 2. Reference Documents ................................................................................................................................ 8 Table 3. Advantages and disadvantages: FFmpeg vs Intel® Media SDK ............................... 14 Table 4. Advantages and Disadvantages: QSV vs. VAAPI ................................................................. 15 Table 5. i965 vs. Intel® HD Graphics Drivercomparison ................................................................... 16 Table 6. Advantages and disadvantages: FFmpeg vs. GStreamer .............................................. 17 Table 7. Advantages and disadvantages: Wayland vs. X11 ............................................................. 19 Table 8. Advantages and disadvantages: Intel® Automotive Solution Compositor for

Wayland* Protocol Weston vs. others ......................................................................................... 20



Revision History

Date Revision Description

May 2019 1.0 Initial release of this document.




1.0 Introduction

The Intel® Media Accelerator Reference Software is a reference media player

application for digital signage use cases. It harnesses fixed-function hardware

acceleration for superior media streaming performance, reduced Central Processing

Unit (CPU) loading, and customizable Graphics Processing Unit (GPU) pipeline

solutions.

The architecture of the software stack includes the FFmpeg* VAAPI, FFmpeg* QSV, and

Intel® Automotive Solution Compositor for Wayland* Protocol Weston which improves

developer experience by promoting portability and compatibility with a range of

processors.

This white paper describes the architecture of Intel® Media Accelerator Reference

Software, a software stack applicable to many different market use cases, including

digital signage, kiosks, and more.

1.1 What is Intel® Media Accelerator Reference Software?

1.1.1 Definition

The main goal of the Intel® Media Accelerator Reference Software is to demonstrate

video content playback using VAAPI and QSV via FFmpeg and Weston Wayland for

rendering for retail, banking, hospitality, and education use cases. Intel® Media

Accelerator Reference Software also provides a configurable multi-zone composited UI

that enables the user to customize the number and types of playout zones .

Intel® Media Accelerator Reference Software is licensed reference code

showcasing best practices for video decode, playback, compositing, blending,

and rendering on Intel® platforms. Cross-platform support for access to Intel®

Graphics hardware acceleration and video processing pipeline.

Offloads graphics functions from CPU freeing up valuable processing for

analytics and other media player functions.

Code once and use on multiple current and future Intel® platforms.

Support for the latest formats and containers out of the box: JPG, GIF, BMP,

H.265/HEVC, H.264/AVC, VC-1, MJPEG, VP9, MKV, MP4, and MPEG2 TS.



1.2 Scope of this Document

This document provides a brief overview of Intel® Media Accelerator Reference

Software for Linux* along with the details about the advantages and disadvantages of

the Linux tools and frameworks selected for Intel® Media Accelerator Reference

Software.

1.3 Terminology

Table 1. Terminology

Term Description

ALSA* Advanced Linux Sound Architecture

DSG Digital Signage

FFmpeg* Fast Forward mpeg: A free software project that is a leader with their multimedia framework used to decode, encode, transcode, mux, demux, stream, filter and play.

HDR High Dynamic Range

GStreamer* An open source project that provides a library for constructing graphs of media-handling components. GStreamer playback, audio/video streaming as well as audio (mixing) and video (non-linear editing) processing.

GUI Graphical User Interface

IAS Reference implementation of a Wayland compositor called Intel® Automotive Solution Compositor for Wayland* Protocol

ICL IceLake

IWB Interactive White Board

Linux GNU Linux Operating System

OS Operating System

Qt* A cross-platform application framework and widget toolkit

SKL Intel® microarchitecture code name Skylake

TSD Transportation Solutions Division

VAAPI* Video Acceleration API

VPP Video Post/Pre Processing

VR Visual Retail

Wayland* Display Server Protocol

Weston* Reference implementation of a Wayland compositor




1.4 Reference Documents

Table 2. Reference Documents

Document Document No./Location

Intel® Media Accelerator Reference Software for Windows* 608837-001



2.0 Intel® Media Accelerator Reference Software –

Linux*

2.1 Overview of Intel® Media Accelerator Reference Software

The Intel® Media Accelerator Reference Software is an application that delivers digital

content across media signage devices such as digital posters, interactive kiosks, menu

boards, dynamic shelf labels, intelligent vending machines, point-of-sale devices, and

more. Intel® Media Accelerator Reference Software works to display the contents on the

screen, built by an Extensible Markup Language (XML) configuration file.

At a high-level, Intel® Media Accelerator Reference Software supports the features such

as multi-monitors, multiple zones, and a configuration Graphical User Interface (GUI).

Other features at a high-level include the following:

Video Acceleration API (VAAPI) and QSV via FFmpeg for decode

GPU vs CPU operations for decode, color conversion, scaling, and compositing.

Qt* based zone editor for display configuration

Hybrid multi-monitors.

Identical, Tiled, Different, Portrait, and Landscape display topologies.

Multiple zones: Each display could be configured with various media contents

via zones.

Media content: Video, audio, image, text ticker, RSS, benchmark zone.

Zone carousel, Zone border

Zone overlay and alpha blend

2.2 Technologies Used

The technologies used with Intel® Media Accelerator Reference Software include Intel®

Media SDK/Intel® Quick Sync Video and VAAPI via FFmpeg, Qt, Weston and Advanced

Linux* Sound Architecture (ALSA*).

2.2.1 Advanced Linux* Sound Architecture (ALSA*)

ALSA is a kernel driver providing audio software components access to the Linux kernel

for decoding audio frames.




2.2.2 FFmpeg*

FFmpeg is a community project that is a leader with their multimedia framework used

to decode, encode, transcode, mux, demux, stream, filter and play. FFmpeg is portable

and compiles and runs on multiple platforms and hardware architectures.

2.2.3 Intel® Media SDK

Intel® Media SDK accelerates rich media performance and speeds up video playback.

Use Intel® Media SDK to develop professional-grade media and video applications for

digital surveillance, retail, industrial, smart home, video conferencing, with the libraries,

tools, and samples included in this SDK. Its single API enables hardware acceleration

for fast video transcoding, image processing, and media workflows while helping you

use the capabilities in Intel® Quick Sync Video.

Accelerate rich media performance. Speed up video playback, encoding,

processing, and media formatting conversion.

Cut product time to market. Prototype, optimize, and productize your media

pipelines with a comprehensive, convenient API.

Speed up transition to new codecs. Use efficient codecs with hardware

acceleration to increase speed, compression, and quality across AVC, HEVC, and

MPEG-2.

Debug and customize your products quickly. Find and solve application bugs,

port applications to new operating systems, and add new features using source

code for open source Linux*.

2.2.4 Intel® Automotive Solution Compositor for Wayland* Protocol Weston

Weston is the reference implementation of a Wayland compositor, and a useful

compositor in its own right. Weston has various backends that lets it run on Linux

kernel modesetting and evdev input as well as under X11. According to the Weston

main page, “Weston is the reference implementation of a Wayland server. Wayland

server is a display server, a window manager and a compositor all in one.” Visit

https://github.com/intel/ias for more information about this project.

2.3 Intel® Media Accelerator Reference Software High-Level Architecture

The high-level architecture diagram in Figure 1 below depicts the major software

modules in the Intel® Media Accelerator Reference Software application.

https://www.intel.com/content/www/us/en/architecture-and-technology/quick-sync-video/quick-sync-video-general.html

https://github.com/Intel-Media-SDK/MediaSDK

https://github.com/intel/ias



Figure 1. Intel® Media Accelerator Reference Software for Linux* High Level Architecture

Intel® Media Accelerator Reference Software Linux Install

Config ManagerXML Config File

Display Manager

Display

SysInit Manager

Wayland

Video Manager

Image Manager

RSS ManagerAudio Manager

FFMPEG API

Input

Info

DecodeInitialize Codecs

FFMPEG SW Library VA API via libVA

Software Decode Hardware Decode

IAS Shell, Compositor

Initialize/Configure Modules

Ticker Manager

Composite & Render

Video / Audio / Image

RSS / Text

URL

Qt Module

QSV via libVA

Hardware Decode

FFmpeg API

FFmpeg SW Library

SysInit Manager is the software module that manages the initialization of the

application by first reading the configuration file and creating the configured zones via

manager instances. For example, if the configuration file has a video zone with five

videos for playback and a ticker zone to scroll text horiizonatally across the video it will

start and initialize a video manager and a ticker manager. As already mentioned the

SysInit Manger is responsible for instantiating appropriate modules based on the

configuration by the user. Each zone instance will execute as a separate thread in the

system. Certain zones like Video may have more than one thread (e.g. to read file data,

decode audio data, decode video data, rendering thread etc.).

Note: The C++11 standard is used for threads, mutexes and semaphores and managed via a

common software block called the KAL (Kernel Abstraction Layer) as illustrated in

Figure 2.

Once all the zone managers, per the user defined configuration, are initialized the

SysInit Manger will start the media playback. The playback can continue for a finite or




infinite duration based on the user configuration. In case of finite duration, the media

playback will end after the specified time.

For image, video ,and audio media types, Intel® Media Accelerator Reference Software

uses FFmpeg media APIs to demux, decode, filter, etc. the data. An abstraction layer is

created to abstract FFmpeg API calls from the application.

Finally, the SysInit Manager is responsible for starting and initializaing the Intel®

Automotive Solution Compositor for Wayland* Protocol Weston compositor for

rendering. All the media content will be passed to a compositor module which is

responsible for creating a composited scene and rendering on a display.

Note that Qt* is only used for the Configuration UI as part of Intel® Media Accelerator

Reference Software, and all manager modules are instantiated as follows:

Different configuration – One instance of each manager module per display.

Identical and Tiled configuration – One instance of each manager across

displays.

2.3.1 Intel® Media Accelerator Reference Software Video Manager

The below diagram is the high level architecture for the Video Manager as introduced in

Figure 1 above. As mentioned above, the SysInit manager starts the Video Manager

which in turn starts three concurrent threads for demuxing and parsing media files,

decoding audio and video media data via FFmpeg, and after, uses Intel® Automotive

Solution Compositor for Wayland* Protocol Weston for rendering the media data to the

display(s).



Figure 2 Video Manager

FFmpeg Decode Engine

Audio DecodeVideo Decode

Player Engine

System Manger

Media Manger Interface

File Read Thread

Video Decode Thread

AudioDecodeThread

Video Manager Interface(Main Task)

(KAL)

Kernel

Abstraction

Layer

(Threads, Semaphores,

Message queues

etc)

Video Client Thread EventLoop

FFmpeg APIs

S/W Decode VAAPI

S/W DecodeHardware

Decode (GPU)

Display Manager

ConfigurationManager

Renderer

Third Party Modules

Intel® Media Accelerator Reference Software Developed

Legends

Intel® Automotive Solution Compositor

for Wayland* Protocol




3.0 A High Level Look at Popular Linux* Media

APIs and Frameworks

This section presents a high level comparison of key advantages of:

Using higher level programming like FFmpeg vs Intel® Media SDK

Using Intel® Quick Sync Video (QSV) technology vs VAAPI

Moving to iHD

Using FFmpeg vs GStreamer

3.1 Advantages and Disadvantages of Using Higher Level Programming APIs such as FFmpeg* vs Intel® Media SDK

FFmpeg* is a very popular library which has been integrated into various applications

(such as MPV, VLC, Google Chromium/Chrome, Handbrake, and more). It can be

leveraged without much development to enable FFmpeg-qsv vs. FFmpeg-VAAPI.

Intel® Media SDK is focused on Intel® video decoding, encoding, processing which is not

enough for transcoding. With FFmpeg, you can have streaming/mux/demux, and also

various software audio and video codecs such as AAC, x264, 265, libvpx, and libaom.

Table 3. Advantages and disadvantages: FFmpeg vs Intel® Media SDK

Advantages Disadvantages

FFmpeg Intel® Media SDK FFmpeg Intel® Media SDK

More higher level

features such as

streaming, mux,

demux, RTP/RTSP, etc

while still having

access to MediaSDK

features via FFmpeg-

qsv.

Calls to Intel® Media

SDK directly gives

more control to use

the Intel® Media SDK

API without

FFmpeg/others

Unfamiliarity

or

complexity

with

licensing

Video processing of

elementary streams

only

Various audio and

video codecs

Not all features of Intel® Media SDK have

been exposed by

FFmpeg-qsv

More

restrictive

licensing

terms

Popular with

community support



3.2 Advantages and Disadvantages of Using FFmpeg*-QSV vs FFmpeg*-VAAPI

Intel® Quick Sync Video technology on Intel® Iris Pro Graphics and Intel® HD graphics

provides transcode acceleration on Linux* systems with FFmpeg* v2.8 and later.

FFmpeg* v2.8+ includes Intel® Quick Sync Video accelerated h264_qsv, mpeg2_qsv

and hevc_qsv codecs. These provide a fast time-to-market path to server transcode

solutions based on Intel® Xeon® processors with processor graphics.

VAAPI (Video Acceleration API) is an open-source library and API specification, which

provides access to graphics hardware acceleration capabilities for video processing. It

consists of a main library and driver-specific acceleration backends for each supported

hardware vendor.

Advantages of ffmpeg-qsv v ffmpeg-vaapi are listed in Table 4.

Table 4. Advantages and Disadvantages: QSV vs. VAAPI


QSV VAAPI QSV VAAPI

Can support Windows. Common API for Linux and Windows.

Linux* only - Can support different vaapi-drivers, including i965 driver, iHD driver. And also AMD/NVidia mesa driver.

Less popular in the FFmpeg community and Linux* players.

The libva driver needs to be attached to a DRM device to work

Better encoding quality with specified feature (such as LookAhead)

Easier secondary development due to focusing on Linux and less dependency on third-party such as Intel® Media SDK.

Better encoding performance with specified feature (such as MFE for 1:N/N:N transcoding on high performance GPU).

Wider codec support such as VP9 decoding and encoding. (Just current status. FFmpeg-qsv will support vp9 soon).

Some features have less dependency on FFmpeg framework and focusing on Intel® platform. Thus making these features easier to upstream.

Co-work closer with FFmpeg software codecs.

Automatically switches to FFmpeg software decoder if HW decoding failed.





QSV VAAPI QSV VAAPI

More popular in FFmpeg community, and more popular by various Linux player (Players don’t care encoder very much).

Automatically switches to FFmpeg software decoder if HW decoding failed.

3.3 Advantages and Disadvantages of Migrating to iHD

The i965-va-driver is the Intel® G45 and HD Graphics family driver for VAAPI. The

dependencies include at least 2 DRM packages, libdrm-intel1 and libdrm. Since this

driver has a long history being open source, the i965 driver can support legacy Intel®

GPUs, a legacy upstream Linux kernel, and various media frameworks such as ffmpeg

and GStreamer. I965 is convenient as you can just apt-get install on Ubuntu* or yum

install it on CentOS*/Fedora*.

The iHD driver was distributed with MediaServerStudio. Currently, it is open source

also, but just supports Generation 6 and above (Skylake+) platforms. The Intel® HD

Graphics Driver has some limitations with the upstream Linux kernel.

The iHD driver can work with Intel® Media SDK quite well, and supports more video

processing features such as tone-mapping/rotation.

Table 5. i965 vs. Intel® HD Graphics Drivercomparison


I965 Intel® HD Graphics Driver


Open Source Open Source Deprecated soon. i965 driver won’t support 10th or 11th generation and/or Ice Lake+ (ICL+) platforms. This means the Intel® HD Graphics Driver will be the only driver for 10th generation and above platforms

Limited platform support . See supported platforms at https://github.com/intel/media-driver

https://github.com/intel/media-driver

https://github.com/intel/media-driver






Supports legacy GPUs

Works with Intel® Media SDK

Limitations with upstream kernel prior to 4.16

Supports legacy kernel versions

Supports more video processing features

Supports various media frameworks – ffmpeg and gstreamer

Soon to be the only video driver for Intel®

3.4 Advantages and Disadvantages of Using FFmpeg vs. GStreamer

FFmpeg is a useful transcoding library

(streaming/decoding/encoding/mux/demux/processing) with a very simple and easy to

use API. FFmpeg supports a large codec library which is why FFmpeg can seem more

attractive than using GStreamer.

GStreamer is closer to a media framework and is very flexible and distributes various plugins. Besides the pre-packaged plugins GStreamer can be extended to support new codecs/filters by creating new plugins using basic GSteamer programming concepts.

Table 6. Advantages and disadvantages: FFmpeg vs. GStreamer


FFmpeg GStreamer FFmpeg GStreamer

Transcoding easily performed through command line or various popular GUI implementations readily available

GStreamer FFmpeg plugin available with a set of elements using the FFmpeg library

An API vs. media framework.

Easiser to create a new application which desires more of a media framework architecture

Media framework vs. APIs.

Easier to integrate APIs vs. adopting a new media framework into an existing software application





FFmpeg GStreamer FFmpeg GStreamer

Various audio and video codecs

Various plugins for codecs, filters, sinks and sources. GStreamer provides a framework to control different elements that perform different tasks and synchronized data flow.

Large redistributable framework size

Supported in a various popular applications (VLC, YouTube, MPV, Handbreak, Google Chrome/Chromium,

and more)



4.0 A High Level Look at Weston vs. X11 Rendering

4.1 Advantages and disadvantages of using Weston vs X11

Weston is a compositing window manager for Wayland. Weston is lightweight, fast and

can be used for embedded or mobile devices. Wayland is a protocol between a

compositor and clients.

The table below describes the advantages and disadvantages of using Weston vs. X11

architecture.

Table 7. Advantages and disadvantages: Wayland vs. X11


Wayland X11 Wayland X11

Small/Minimal redistributable size

Protocol for communication between the display server and the clients

X was developed over two decades ago and code base is very large (much of it is unused)

Designed with the premise that every frame is perfect. Essentially only completed frames from applications are rendered and timed with vsync updates.

No guarantee to prevent tearing (rendering of frames timed with vsync)

HDR support in progress

4.2 Advantages and Disadvantages of Using Intel® Automotive Solution Compositor for Wayland* Protocol Weston vs. Other Wayland Reference Implementations

The Intel® Automotive Solution Compositor for Wayland* Protocol Weston compositor

was developed to suit the Intel Transportation Solutions Division (TSD) use cases, but

also support features which are far broader. For example, Intel® Automotive Solution

Compositor for Wayland* Protocol Weston already supports where applications will be

positioned, how they will be z-layered, and how big windows be or whether they will

even be shown or not. For features not already supported easy-to-write plugins can be

developed and integrated.

To simplify the control of the the look and feel of a system the Intel® Automotive

Solution Compositor for Wayland* Protocol shell exposes an interface to the HMI




controller (a common central controller of the system Intel® Automotive Solution

Compositor for Wayland* Protocol in the automotive world) to modify the properties

for all other applications and Intel® Automotive Solution Compositor for Wayland*

Protocol would just follow this guidance. The goal of Intel® Automotive Solution

Compositor for Wayland* Protocol is to provide key hardware functionality to our

customers as fast as possible.

The goal of Weston is to provide a reference compositor to showcase Wayland

protocol and also something that works equally well on all hardware (.

For customers to use open source Weston (upstream non- Intel® Automotive Solution

Compositor for Wayland* Protocol Weston) they would need to essentially modify the

policies within the compositor and therefore own the entire compositor.

Table 8. Advantages and disadvantages: Intel® Automotive Solution

Compositor for Wayland* Protocol Weston vs. others


IAS Weston Intel® Automotive Solution

Compositor for Wayland* Protocol

Weston

HMI operations management

No plan to upstream Intel® Automotive Solution Compositor for Wayland* Protocol features into open source Weston such as rendering to a region of intrest (window position)

Written specifically for TSD/automotive use cases

No focus in the desktop world so definitely no support for Ubuntu/Fedora distros

Time to edit policies within compositor – thus owning the compositor

Support for multiple hw planes

IA (Intel Architecture) display controller functionality like display controller based rotation and scaling

No provision for window

Positioning

Z-layering

Size

Visibility

Supports remote sharing

Major gaps for upstream Weston (e.g. remote sharing, lack of desktop



As noted above, the Intel® Automotive Solution Compositor for Wayland* Protocol

Weston Wayland compositor implementation spans beyond the Intel Transportation

Solutions Division (TSD) use cases. Thus, the Intel® Media Accelerator Reference

Software takes advantage of Intel® Automotive Solution Compositor for Wayland*

Protocol Weston to efficiently composite videos, images, and/or tickers at specific

positions and z-layers on one more displays.

support for some major desktops like Ubuntu* 16LTS, no ability to draw to a region of interest on a display, no multi-display support for Tiled/Different/Clone modes)

Boots faster (optimized for Intel Architecture)

No support for multiple hw planes

Support for rendering to multiple diplays and rich graphics (3D as well)

No IA display controller functionality like display controller based rotation and scaling

Supports hypervisor based systems and can share buffers created from one domain with another domain

No support for rendering to multiple displays




5.0 Conclusion

The Intel® Media Accelerator Reference Software demonstrates using FFmpeg*-VAAPI

or FFmpeg*-QSV and Intel® Automotive Solution Compositor for Wayland* Protocol

Weston. It was designed to accelerate time to market for implementing key features in

each of the above technologies such as FFmpeg-QSV for decode and VPP operations

(using hardware or software codecs depending on the configuration) such as

deinterlace and Intel® Automotive Solution Compositor for Wayland* Protocol Weston

for compositing multiple videos/images/scrolling text(s) across multiple displays at

specific locations. Given FFmpeg’s popularity (by both the community and application

implementations), API architecture, relatively minimal footprint, and vast readily

available features and rich capabilities ( such as RTSP decode for IP cameras, MJPEG

decode for USB cameras, and more), it was decided as a key component in the Intel®

Media Accelerator Reference Software. Similarily, Intel® Automotive Solution

Compositor for Wayland* Protocol Weston as opposed to X11 and upstream Weston,

was chosen due to its rich capabilities that span beyond automotive use cases such as

rendering to specific locations on a one more displays and multi-display support such

as clone and extended modes. Intel® Automotive Solution Compositor for Wayland*

Protocol Weston (along with the upstream Weston) will also support High Dynamic

Range (HDR) in the near the future.



6.0 A Look Ahead at Future Enhancements

In addition to the performance enhancements created by GPU-based hardware

acceleration and reduced CPU loading, future versions of Intel® Media Accelerator

Reference Software will contain more advance features such as multi-zone compositing

for encode/broadcast, real-time transcode and broadcasting, and the integration of

deep learning analytics via the Intel® distribution of OpenVINO™ toolkit.

The integration of the Intel® distribution of the OpenVINO™ toolkit will allow the

creation of customized use cases such as how to monitor audience demographics in

front of a digital signage solution (e.g. the number of customers and their approximate

age and gender). All of the existing models in Intel Model Zoo packaged with

OpenVINO™ will be supported along with any future models which are supported by

OpenVINO™ Model Optimizer.

Additional use cases spanning retail, hospitality, banking, and education which could be

relialzed via easy-to-perform configuration changes in the Intel® Media Accelerator

Reference Software are:

Emotion detection, gaze tracking, and facial reidentification. With these

behavioral data references solution providers can gain insight into user

responses and craft improved messaging, personalization, prediction, and

more.

Student and teacher analytics in a classroom such as raising hands, sitting,

standing, attendance tacking, etc). Over time, the collected data can be used to

design and calibrate course content and interactions to be used by the factulty,

administration, and parents.

Autonomous/frictionless stores could use object detection models for tracking

inventory in a store, vending machine, or cooler as well.




7.0 How to Acquire the Intel® Media Acceleator

Reference Software

In order to acquire the Intel® Media Accelerator Reference Software, you’ll need to

contact your Intel representative to see if you qualify for access.