System Architecture and API Documentation - MPATmpat.eu/.../2016/08/D4.3_System_Architecture_and_API_Documentat… · D4.3 System Architecture and API documentation 4 Introduction

System Architecture and

API Documentation

D4.3

MPAT File ID: D4.3_v1.docx

Version: 1.0

Deliverable number:

D4.3

Author: Jean-Claude Dufourd (ParisTech)

Contributors: Fabian Schiller (IRT), Benedikt Vogel (IRT), Miggi Zwicklbauer (FOKUS) , Stefano Miccoli (FINCONS), Louay Bassbouss (FOKUS)

Internal reviewers:

Matthew Broadbent (ULANC), Christian Fuhrhop (FOKUS)

Ma

Work Package: WP4

Task: T4.2

Nature: R – Report

Dissemination: PU – Public

Status: Living

Delivery date: 31.07.2016

D4.3 System Architecture and API documentation

1

Table of Contents

Introduction ............................................................................................................4

Terminology ............................................................................................................5 Application Model ............................................................................................................................................... 5 Application............................................................................................................................................................. 5 Component ............................................................................................................................................................ 5 Component Model .............................................................................................................................................. 5 Page .......................................................................................................................................................................... 6 Page Model............................................................................................................................................................. 6 Layout ...................................................................................................................................................................... 6 Style .......................................................................................................................................................................... 6 Dynamic Server.................................................................................................................................................... 6 Front End ................................................................................................................................................................ 6 Back End ................................................................................................................................................................. 6 Transparent container ...................................................................................................................................... 6

Project Architecture ................................................................................................8 Editing Phase ........................................................................................................................................................ 9 Runtime Phase ................................................................................................................................................... 12

Front End Architecture .......................................................................................... 13 Introduction ........................................................................................................................................................ 13 MPAT Application ............................................................................................................................................. 13 About Time .......................................................................................................................................................... 13 Application Structure ...................................................................................................................................... 14 HbbTV Library ................................................................................................................................................... 14 Second Screen API ............................................................................................................................................ 14 Key Manager ....................................................................................................................................................... 15

Actions ........................................................................................................................................................................ 15 Key Management Stack ...................................................................................................................................... 15 Initial API .................................................................................................................................................................. 16

Stream Event Manager ................................................................................................................................... 16 Communication Manager .............................................................................................................................. 16 Media Time Manager ....................................................................................................................................... 17

Back End Architecture............................................................................................ 18 User Interface ..................................................................................................................................................... 18

Navigation Schemes ............................................................................................................................................ 18 Pageflow .................................................................................................................................................................... 19 Time Line .................................................................................................................................................................. 20 Static Compiler ....................................................................................................................................................... 21 Playout API .............................................................................................................................................................. 21 Event API ................................................................................................................................................................... 22 Second Screen API ................................................................................................................................................ 22 Import API ................................................................................................................................................................ 22 Asset API .................................................................................................................................................................... 23

Database Serialization .................................................................................................................................... 23 User roles and permissions .......................................................................................................................... 24

Dynamic Server Architecture ................................................................................. 27 Second Screen Support ................................................................................................................................... 27 Synchronization Support ............................................................................................................................... 27


2

Identification of device/user ........................................................................................................................ 27 Persistent storage of device/user information/activity ................................................................... 28 360° Video support .......................................................................................................................................... 28 Ad Insertion/Replacement ........................................................................................................................... 29

Ad insertion.............................................................................................................................................................. 29 Ad replacement ...................................................................................................................................................... 30

Bookmark Support ........................................................................................................................................... 30 Map Support ........................................................................................................................................................ 30 Asset Converter ................................................................................................................................................. 31

Edge Server ........................................................................................................... 33 Deployment/Playout/Testing ..................................................................................................................... 33


3

Table of Illustrations

Figure 1 Example of a transparent container with variable size grid ............................. 7

Figure 2 Project Architecture ....................................................................................... 8

Figure 3 Editing Phase............................................................................................... 10

Figure 4 Runtime Phase ............................................................................................ 12

Figure 5 Back End Architecture ................................................................................. 18

Figure 6 Time Line Editor UI ...................................................................................... 20

Figure 7 How to watch 360° Video content on TV via HbbTV .................................... 28

Figure 8 Ad Insertion ................................................................................................. 29

Figure 9 Ad Replacement .......................................................................................... 30

Figure 10 Asset Converter ......................................................................................... 31


4

Introduction

This document describes the architecture of MPAT. It starts by outlining a set of terminology, then describes the content architecture (a.k.a. Front End). Following this, an overall architecture is described, followed by the WordPress architecture (a.k.a. Back End). Finally, the Dynamic Server architecture and the Edge Server architecture are similarly described. Together, these elements form the MPAT architecture.


5

Terminology

The generic difference between a model and an instance is the same as between a class and an object in object oriented programming.

Application Model

Created by App Creator.

An MPAT Application Model is a template for an Application, in a broader sense than just a generic application with placeholders. An Application Model contains a set of allowed Page Models. An Application Model has a Style which is the default Style of its Page Models.

An Application Model can be of one of three types:

Website: Application Models of that type are used to create sets of Pages that use hyperlink-based navigation. There is no timing and no management of stream events in these application models.

Pageflow: Application Models of that type are used to create sequential sets of Pages that are usually navigated interactively, as extended slide shows.

Timeline: Application Models of that type are used to create sets of Pages that are triggered by a combination of stream events (broadcast time), interactivity, wall clock time and media time. This is the most general and complex type of application models.

The type of an Application Model is the first thing that is chosen and it cannot be changed. Available features in the editing flow are determined by that choice.

Application

Created by the Content Creator from an Application Model.

An MPAT Application is a coherent set of Pages implementing the interactive part of a TV show. An Application is specific to a TV show, complete with relevant text and media, and ready to be deployed. The Application may have multiple Styles and multiple Layouts.

Note: the interactive part of a TV show may be constructed with multiple Applications, and the viewers may not see the difference if these Applications are visually coherent.

Component

Manipulated by Content Creator.

Potential Components include a menu, a dialog, a vote, a popup, etc. These are the building blocks; the elements that are manipulated by the Content Creator. A component is an instance of a Component Model.

Component Model

Manipulated by App Creator. Present in the menus of the Content Editor.

A Component Model dragged into a Page creates a Component. When a Page is created from a Page Model, all the Component Models from the Page Model are instantiated into Components.


6

Page

Created by Content Creator from a Page Model.

A Page represents a single (sub)-page of an (MPAT) Application.

Page Model

Manipulated by the App Creator.

A Page Model is a type of screen organisation allowed in the Applications constructed from an Application Model. A Page Model has a Style, one Component Model or a set of Component Models and a Layout. A Page Model may contain one Page or multiple related Pages.

Layout

A Layout is a policy for spatial organization of components in a Page or Page Model. Layout types include:

Absolute: every element is placed at adjustable coordinates.

Grid: every element is placed in a cell of a grid. The grid may be regular or content-adjustable, etc.

Zone: elements are placed within selectable, pre-defined areas such as columns, sidebars, headers and footer, etc.

Style

A Style is a set of graphics and text attributes and choice of icons (visual appearance).

Dynamic Server

Node.js server that will answer the dynamic requests from interactive TV content served using MPAT, handle interaction with the companion screen framework and offer synchronization functionality.

Front End

The Front End is the part of the WordPress-related MPAT code that is dedicated to the production of the actual MPAT Application, HTML, JS, CSS, images and other resources.

Back End

The Back End is the part of the WordPress-related MPAT code that is dedicated to the management of the authoring UI, link with the WordPress database, etc.

Transparent container

A Transparent Container is a container that adds a feature but keeps the size and layout of its child unchanged. It usually has no “visual” meaning. The rendering of the child is identical too.


7

Figure 1 Example of a transparent container with variable size grid

Basic widgets can be filled with modules with variable sizing, within limits.

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.


8

Project Architecture

MPAT will reuse WordPress functionality for authoring, storage and production of interactive TV applications, including those built for the HbbTV platform. MPAT extends WordPress both from the inside (extensions and plugins) and from the outside, by providing complementary new tools and interfacing with existing tools.

As shown in the Figure 2, the project architecture has four internal components:

Figure 2 Project Architecture

The first component is the set of plugins and extensions of WordPress to provide:

Production of interactive TV applications, including HbbTV

Authoring user interface

Serialization of MPAT data into the WordPress database

The second component is a dynamic server. This server provides:

Support for legacy devices, e.g. alternative support for second screens or synchronization for devices that do not support it by default (such as early HbbTV devices)

Support for additional MPAT features, such as user management or integration with social networks

Support for application specific extensions, such as dynamic content

The third component is external to the project: the content server. Broadcasters have existing content servers, hosted either internally or externally. These can also be augmented with external CDN support. It is usually enough to have a URL pointing to the content to interface with this third component. When this content server is organized around “assets”, there is an additional layer of interfacing necessary. Assets are composite objects including various related media together with associated metadata. One example of asset metadata is TV Anytime. MPAT is going to provide interfaces from TV Anytime asset management to the authoring process.


9

The fourth component is a deployment assistant. The deployment assistant is responsible for making sure that all transformations from the authoring context to the deployment context are executed correctly:

Media assets may be edited in the authoring environment, and sometimes, lower quality versions of the media are used on the limited authoring machines

MPAT applications are produced and tested in the authoring environment and contain URLs to media and other applications in the authoring environment

The dynamic server used for debug and testing is normally in the authoring environment

The debug playout environment is on the author’s desk and is quite different from the final broadcast head end. For example, stream events descriptions may be in a different format

A fifth component, not shown in the figure, is the Edge Server. It is not sure yet whether specific MPAT developments will be required on generic Edge Servers or how much of those developments can be done within the deployer tool.

As a result of the above list, actions of the deployment assistant include:

Replace all media authoring URLs with final media URLs, with appropriate quality

Replace all authoring applications URLs with final applications URLs, and copy the files to the appropriate directory on the production HTTP server

Replace all dynamic request authoring URLs with final dynamic request URLs pointing to the production dynamic server

Replace debug playout instructions with final production broadcast headend instructions, and copy files, set up servers as needed

A first version of the MPAT deployment assistant will be purely text-based. The deployment assistant may also be implemented as a module within the dynamic server for convenience reasons.

Editing Phase

There are two distinct phases when creating an MPAT Application. The first of these is the Editing Phase. This is where the application is created, and involves the participation of a number of users.


10

Figure 3 Editing Phase

In Figure 3, within the WordPress box, “MPAT JS Components” (dark pink) represents the Front End, i.e. the part of the WordPress code that will produce the HTML, CSS and JS of the MPAT applications.

MPAT PHP Components (green) includes all the modules responsible for everything but the Front End:

Authoring User Interface: specified in WP2 and WP3,

Interface with the WordPress DB: to serialize all the MPAT specific information

Static Compiler: to produce the (constant) HTML, CSS and JS files that can be served to viewers’ TVs from optimized HTTP servers (not WordPress servers)

Playout API: to help the author with setting up and using a debugging environment for her applications

Event API: to help the author with setting up the stream events part of the debugging

Second Screen API: to manage the authoring part of relationship between the TV application and the second screen application

Import API: to help the author import TV Anytime Assets from the broadcasters’ asset managers, for use during authoring and debugging

Asset API: to help the author with managing assets that are not in sophisticated asset managers

In the Dynamic Server, nine modules are identified:

Front End API (used in runtime) o Multi Device Synchronization o User management, including Persistent storage and Bookmark support

Second Screen Solution (used in runtime)

Asset Manager


11

o Subtitle manager, to transcode from existing subtitle formats into the supported ones.

o Video transcoding

360 video support (does not appear in figure)

Ad Insertion (does not appear in figure)

Maps support (does not appear in figure)

Communication, as support for other features (does not appear in figure)

The remaining three elements are:

The Deployer Store: this storage is dedicated to all content, scripts, resources and instructions that are produced by the authoring phase and will be used in the run time phase; this includes all the MPAT applications, in their undeployed state

The Test Playout: the test playout is a system that allows the realistic (but not final) playout of interactive TV content, such as HbbTV; one example of such a system is a combination of multiplexer software with a DVB modulator.

The Test TV: it is an interactive television device, compatible with HbbTV for example. This may be either a TV or a set top box with a screen.


12

Runtime Phase

The second phase is the Runtime Phase, which occurs when the MPAT Application is deployed in production.

Figure 4 Runtime Phase

n Figure 4, only modules active in broadcast mode are shown:

Front End Related: o Synchronization: when the TV application detects that the TV does not have

multi-device communication, it connects to the dynamic server which supplies alternate synchronization mechanisms with other screens, based on existing supported web technologies

o User Management: provide a range of tracking features through communication with the dynamic server, storage of user preferences, etc.

Second Screen: when the TV application detects that the TV does not have second screen detection and communication, it connects to the dynamic server which supplies alternate discovery and connection mechanisms with local second screens, based on existing supported web technologies

MPAT authored content is extracted from the deployer store and installed in the document root of an HTTP server. The deployer tool is responsible for updating the URLs to this new location, as well as to the production servers for all types of resources. StreamEvent specifications and other broadcasting instructions are sent from the deployer store to the broadcast playout/head end, possibly converted. The TV of the final viewer receives A/V in DVB-T/C/S, gets MPAT content from the Internet and connects to the dynamic server using socket.io for various purposes (dynamic content, second screen legacy, synchronization legacy, etc.).


13

Front End Architecture

Introduction

The Front End is the part of the WordPress-related MPAT code that is dedicated to the production of the actual MPAT Application, HTML, JS, CSS, images and other resources.

There is a distinction between what the viewer perceives and what the designers and developers do. So the initial perception of one application for one TV show is relevant for the viewer, but not for the designers and developers in MPAT. The viewers may see a visually coherent set of pages, but MPAT may choose to implement this with multiple “applications”. MPAT Applications have an internal coherence determined by the capabilities of the system, and multiple MPAT applications may be associated to create the feeling of one “application” for the viewers of a TV show.

The typical structure from a viewer perspective is:

Hint towards the presence of an application: red button visual, appears and disappears periodically

Click on red button starts a launcher with various possibilities, including something for the current show (one or more), which is (are) an MPAT application

The MPAT application for the current show

MPAT Application

An (MPAT) Application is a set of Pages to be distributed with a TV show. We assume on this object:

A single style

A single entry point (the URL specified in the AIT)

A single origin (to avoid CORS issues) or a set of predefined origins (e.g. to include the dynamic server) with suitable CORS headers

Created from a single Application Model

An Application may have different states. If an Application contains only final, relevant content, it is usable and ready for deployment. If it contains placeholder content, then it is not yet complete and thus not ready for deployment. The extreme case is an Application Model which contains mostly placeholders. Placeholders could be linked with user roles: a user unrelated to a placeholder would not have to edit it, while a user with a particular role may have to replace the placeholder when editing the content.

About Time

The consideration of time is necessary in a subset of applications. When time related editing is present, there may be constraints on the adopted structure for an Application. If elements need to appear and disappear according to the current media time, then one single HTML per Application would be the only possible structure. Components can be in sub-pages (HTML files loaded by iframe, or HTML fragments loaded by AJAX), but one main HTML is loaded and stays “on top” until the end of the use of the Application.

Having one single HTML that is loaded over the whole life-cycle of the Application means that we can keep track of what the user has done, without requiring the storage of such information (using cookies or another alternative solution).


14

The notion of scene time does not exist in HTML. For example, there is no simple way to trigger the action of some HTML element at a specific time within a show (without StreamEvent support). It is however possible to trigger a HTML element at a certain UTC time, which may be different across devices. The only synchronization which is compatible across a wide range of devices uses the HbbTV standard’s StreamEvents. Existing timing attributes in the HbbTV standard include:

The time of tune-in or of the beginning of the show, i.e. the first time a script is run on the TV and the date/time logged; this will be different for each TV

A UTC time; the actual precision is unknown in this context

The reception of an HbbTV StreamEvent; synchronization between sets can be determined and, if compatible, somewhat guaranteed

The time of a user interaction

Any of the above, but including a specific offset

Each new element is a sub-page. This will be served by a web server if constant, or by the backend server if not. It will be included in a target div element, whose content is loaded by an AJAX request. These elements are added and removed by a script reacting to a stream event or key event.

Application Structure

An Application consists of at least one HTML page, one or more CSS pages, and one or more JS files. There is just one global MPAT JS object, called MPAT. The MPAT object holds one property per underlying module, such as a Key Manager or Second Screen.

The HTML page contains:

1. Hyperlink to CSS

2. Scripts to be preloaded

3. Initialisation of HbbTV library

4. Initialisation of Second Screen library

5. Key Manager setup

6. Stream Event Manager setup

7. Media Time Manager setup

8. Communication Manager setup

9. Depending on the application, a structure containing insertion points with IDs, where new elements can be added by interaction (Key Manager), server-sent event trigger (Stream Event Manager), or media time trigger (Media Time Manager)

HbbTV Library

The (very small) HbbTV Library contains a number of useful components for interacting specifically with HbbTV-compatible devices, including:

● Remote debugger ● Video player handling ● Event handling

○ Key events ○ Stream events

Second Screen API

The Second Screen API allows a companion application on a smartphone or tablet to connect to a HbbTV Application, exchange data and synchronize events. HbbTV 2.0 Terminals support the Second Screen API nativel. As a result, there is no need to include any JavaScript Library


15

in the HbbTV App. The following Companion Screen features are supported in the HbbTV 2.0 specification:

Launching a HbbTV Application from a companion screen: this uses DIAL (Discovery and Launch Protocol) to discover HbbTV 2.0 Terminals in a local network, and then launch the HbbTV App

Launch a Companion Screen App from a HbbTV App: the HbbTV App uses a JavaScript API (part of the HbbTV 2.0 Spec) to discover Companion devices and launch second screen App on them

App2App Communication: The HbbTV 2.0 Terminal provides a WebSocket Server for App2App communication. Each of the HbbTV and Companion Screen Applications establishes a WebSocket Connection to the WS Server on the same channel. The Server forwards the messages between both applications

As all terminals currently available on the market support HbbTV 1.x, we provide a Second Screen Framework called Famium on top of HbbTV 1.x that supports the same features mentioned above with some limitations:

Both HbbTV and Companion Screen Apps need to embed a JavaScript Lib in order to support the Second Screen API

The App2App communication Server is hosted in the Cloud. Local App2App communication is not possible

Local Discovery and Launch is not possible. Therefore a pairing step is required. During pairing, the HbbTV and Companion Screen Apps exchange a token that is stored locally in both Apps. There are different ways to exchange the token e.g. using QR-code, PIN-code or by entering a short URL in the Browser of the Companion Screen

The Famium API is compatible with the W3C Presentation API

Key Manager

The Key Manager is a script that manages key events sent by the viewer with the remote control. The Key Manager has the following features:

Actions

Software modules define Actions. For example, the Pageflow module requires two keys to navigate through pages: the Pageflow then defines two Actions, named Pageflow_next and Pageflow_previous, which are bound to two functions in the Pageflow module. Tfe Key Manager manages the association of the Actions with actual keys. The Key Manager exposes a function registerAction takes as parameters the name of the action and the function processing the key event. The Pageflow module is responsible for registering the actions it implements. Only Actions can be associated to keys.

Key Management Stack

The Key Manager has a stack of association lists. Each association list associates a key with a callback, i.e. a script function to execute if the key is pressed. The Key Event Listener uses the association list currently on top of the stack: it scans the association list for the key that was pressed and if there is a match, triggers the corresponding callback.

One association list is an interaction context. The interaction context may change for each page. Contexts can be pushed and popped onto the stack: the necessity of the stack becomes evident when you have a hierarchy of “forms”, returning when you finish a form.


16

In future versions of Key Management, there could be multiple stacks. HbbTV manages less than 20 keys, but it could be useful to group those keys into two or more groups, and have a stack for each group of keys.

Pageflow is one instance where this is obviously useful: when entering Pageflow, you want up-down to navigate the Pageflow. For all other keys, it depends on the content of each page of the flow. So from one page to the next, you want to change the active key context but not change the up-down association.

Initial API

MPAT.KeyManager {

KeyAssociation createKeyAssociation(); // returns a key association list

KeyAssociation getActiveAssociation();

// returns the name of the active key association, or null

bool registerAction(actionName, actionImplementation);

// returns true for OK

bool activate(keyAssociation);

bool push(keyAssociation);

bool pop();

bool registerKeyEventListener();

}

MPAT.KeyManager.KeyAssociation {

bool bind(key, actionName); // returns true for OK

bool unbind(key); // returns true for OK

}

Stream Event Manager

The Stream Event Manager listens for Stream Events and calls the appropriate callback for each event received.

The Stream Event Manager has an associative list of stream events to callback. When an

event of matching name is received, the callback is called with the data of the stream event.

Initial API

// requires the existence of MPAT.videoBroadcast

MPAT.StreamEventManager {

bool registerStreamEventListener(url, name, callback);

bool removeStreamEventListener(url, name, callback);

}

Communication Manager

The Communication Manager manages communication between the Application and a server. Communication will use socket.io, as it covers WebSockets and variants of Ajax…

Communication with a server is used to synchronize the Application with external events in case Stream Events are not available or not practical.

Initial API

MPAT.CommunicationManager {

bool initiateServerConnection();

bool postMessage(string);

bool onMessage(callback);


17

// callback has one string parameter for the received message

}

Media Time Manager

The Media Time Manager is only useful when there is media playing in the application, and that media is not the broadcast video. The broadcast video does not have a meaningful “currentTime”. Other playing media, such as downloaded/streamed video/audio streams, have a meaningful currentTime which can be used to trigger various scene actions: showing a component, removing a component, etc.

Although there are existing methods for describing and using subtitles, they can be nonetheless implemented in this way. In a sense, the Media Time Manager allows the creation of interactive videos. Possible actions triggered by the media current time are:

Show a component at time T

Hide a component at time T

Move a component

Change the style/highlight of a component

Change a text

Change the highlight of a text (karaoke)

Choose between multiple options in the video (choose a path in a game)

Loop a section of a video

Skip a section of a video

Conditionally loop or skip a section of a video

The Media Time Manager is similar to the Key Manager in a sense: it may have a simple association list of time-action pairs, or a stack of association lists, the top of the stack being the active association list.

A simple example of interactive video is: a newscast with multiple levels of details. For each news segment, there is possibly two or three levels of details, i.e. for each news there is a headline section, a short section and a longer section. Depending on an option set by the user, the viewing could have:

- Only the headlines, skipping to the next headline at the end of each; - Only the headlines and the short section, with at the end of each short section, a

button to select whether or not to see the long section; - All sections; - Any combination involving more politics or more sports or other choices.

All of this can be done with the use of a single video.


18

Back End Architecture

This section describes the “pure WordPress” part of the MPAT system, including the code managing the user interface of the authoring and the database connection, i.e. how the MPAT specific data is serialized into the WordPress database.

Figure 5 Back End Architecture

User Interface

The back end user interface is based on React.js. In order to accelerate the production og a working tool for the initial pilot phase of the project, input WP2 and WP3 will be partially considered. This will then be rectified in later revisions of the implementation. In the meantime, we are going to produce a temporary user interface, which incorporates some, but not all, of the desired features.

Different navigation schemes will use different user interfaces. These are described in the following section.

Navigation Schemes

A navigation scheme is one of the first choices made by an author. The choice is made in the general MPAT settings for the whole WordPress “site”, given that a WordPress site corresponds to an MPAT application.

There are three currently defined navigation schemes, but there is a mechanism for defining a new navigation scheme in a WordPress plugin and registering it in the MPAT authoring tool.

In the WordPress settings, a MPAT settings page is defined. The MPAT settings page has one option, which is the navigation scheme. A function can be called to register a navigation scheme, defined below. After selection of one navigation scheme, the specific options of that navigation scheme are added to the settings page.


19

Navigation Scheme API

This is the PHP API:

registerNavigationScheme(navSchemeName, navSchemeCallback);

navSchemeCallback has the task of adding to the MPAT settings the specific settings relevant to the navigation scheme implemented by the plugin. For example, for the TimeLine navigation scheme, a duration (real or indicative) is needed, so the callback add a text label “Duration” and a text field and appropriate code for its management.

Web Site

The Web Site navigation scheme is based on authoring individual single-screen WordPress pages and explicitly providing links to navigate between those pages. The author of an application designs individual pages. In most applications, these perform a single function (such as information display, video gallery), but it is also possible to combine multiple components (for example an image and an additional social media message display) within a page. In the predecessor to MPAT (the HAT editor), the possible distribution of components on the screen was limited. Layouts were full-screen, split in two regions (left and right) or three regions (left, middle and right), with one component in each region.

In MPAT, the layout handling implementation is based on React.js to allow an enhanced underlying capability. This provides more flexibility in the placement and sizing of the component, enabling the free positioning of arbitrarily sized rectangular regions on the screen. While there still are a number of constraints (such as the fact that regions are not allowed to overlap), it adds considerable freedom to the placing of components.

Navigation flow between individual pages is arbitrary and needs to be provided by the application editor. This is usually achieved either by making direct key shortcuts to the other pages or by providing selection of other pages from a pull-down menu. The former solution (key shortcuts) is preferred if pages have significant differences, while menu selection is preferable if the pages are conceptually on the same level (for example a list of actor biographies).

Pageflow

Pageflow navigation is based on the concept of spatial navigation between individual screens. Creation of individual screens (WordPress pages) and their content is identical to the Web Site navigation scheme.

The Pageflow navigation scheme differs from the Web Site navigation scheme in that pages are assumed to be either ‘above’ or ‘besides’ each other, so that an implicit navigation concepts, based on cursor keys, exists to navigate to the page ‘above’, ‘below’, ‘left’ or ‘right’ of the current one.

The Pageflow navigation scheme is best suited for presentations that have an implicit ‘narrative’ about a topic. Users will in almost every case, unless shortcuts are intentionally provided, need to follow the existing page order, making Pageflow an ideal choice for applications where information is best presented in a specific order, but less suitable where users want to access specific items quickly.

While the architecture doesn’t enforce any preferences in order, it is encouraged to have a narrative flow that goes ‘down’, with pages containing additional details to the right.


20

The interface automatically presents on-screen arrows to indicate the availability of additional pages that the user can navigate to.

To make the spatial arrangement of the pages clearer to the user and to enforce the visual metaphor, it is desirable to scroll (flow) between individual pages when moving between them, as opposed to immediately switching between them. Technically, smooth scrolling is difficult to achieve on earlier or low powered HbbTV devices. For such devices other indicators will be used to visualize the spatial relation and flow between the pages.

Time Line

The Time Line navigation scheme covers applications that are time oriented, depending either on media time in the case of applications such as VOD and catch-up, or on stream events in the case of live broadcasts where the precise synchronization of HbbTV elements is provided through stream events.

The principle of these applications is that they have a duration: immediately after choosing the Time Line navigation scheme, the author should indicate the total duration of the show. This duration decides the time span in which elements can be added and removed, and thus it is difficult to reduce it after elements have started being added.

Another option is the background image to use in the preview area, something that should be representative of the actual show. MPAT elements are designed (as WordPress pages) and edited in the same UI as the Web Site. MPAT elements cannot be edited directly in the Time Line Editor below.

The Time Line Editor will look like this:

Figure 6 Time Line Editor UI

Preview Area

The preview area displays:

The background image

The MPAT element that should be present at the time shown by the marker in the progress bar, if any

There is no interactivity, nor animation.


21

Progress Bar

The progress bar shows the duration of the application, and a marker. The marker can be moved by drag and drop along the progress bar. Near the marker is a test indication of the time value. As soon as the marker is dropped, the preview area is updated.

Property Sheet

The property sheet shows the characteristics of the selected time range and the associated MPAT element:

Starting time or stream event

Ending time or stream event

MPAT element

Possible interactions are:

Change one of the times with another value

Change one of the stream events attributes

Switch from time to stream event or back

Editable Time Line

The editable time line shows the time ranges (in green) when MPAT elements are shown.

Possible interactions are:

- Click on a time range to select it: the time range is highlighted, its attributes are shown in the property sheet

- Click on the selected time range or in an empty zone: deselects the time range, empties the property sheet

- Drag the body of a time range: changes the starting time - Drag the last 10% of a time range: changes the duration of the time range

Buttons

In the buttons zone (light brown), there are four elements:

- A MPAT element selector: this selector holds the list of all MPAT elements already defined, the author chooses which element to add in the selector, then presses the New button below.

- A “New MPAT Element” button: creates a new time range in the time line, starting in the first empty time and with maximum possible duration, i.e. no overlap with the next time range.

- A “Remove MPAT Element” button: can be used to remove the selected time range, inhibited if no time range is selected.

- A Publish button: pressing this button generates the page and playout information, including stream events description if relevant.

Static Compiler

It is envisaged that this will utilise existing functionality found in 3rd

party plugins

Playout API

The following capabilities are part of the TV-playout API incorporating the option to connect various playout systems/modulators:

● App signaling for the TV client ● URL-Input via AIT

● Static URL insertion ● Live URL insertion for testing/automatization of AIT ● Multiple applications per service (e.g. Launcher, Dashboard,, Teletext

Plus, etc.) ● Stream-events for dynamic events and synchronising


22

● Transmission via signaling in DSM-CC or XML-file on a webserver ● Event schedule (automatisation of events via a list of relative timestamps/-

range)

Live insertion

Manual trigger(s) from editor

Start/stop/skip/shift/delete events through an external app Sequencing of live stream events (mixture between scheduled and live

insertion) Other metadata

Content type (e.g. ad, news, etc.)

Context ID (e.g. media for Fiat 500 ad) ● Output for Testing and Production

● Switch for testing- and production-environment ● Multiplexed MPEG-TS ● Single program & multiple programs (TS-bouquet) ● Video/TS loop mode for testing ● Option for broadcast (DSM-CC) or broadband client download ● Playout center application form

Name of the application

Description of the application

Start date of availability (DD.MM.YYYY,HH:MM)

End date of availability (DD.MM.YYYY,HH:MM)

Icon

Title of application (launcher bar)

Subtitle (additional info launcher bar)

Teaser text for red button

Teaser start date (DD.MM.YYYY,HH:MM)

Teaser end date(DD.MM.YYYY,HH:MM)

Link to the actual video (if not live broadcast)

Event API

The event API is the part of the Time Line Editor, action of the Publish button.

Second Screen API

This module generates the MPAT internal description to feed the Second Screen module of the dynamic server with relevant information.

Import API

This module defines interfaces to integrate external asset management systems in MPAT. All standard CRUD operations, namely create, read/retrieve, update/modify and delete/destroy, are supported, enabling editors to manage media assets through MPAT administration UI.

Import functionality is implemented only logically: files continue to reside on external CMS which is in charge of the delivery to final users while metadata are stored in MPAT reflecting TV anytime specifications as instances of video class defined in Asset API module. The module provides dedicated UI to allow content editors to overwrite those information as needed. Also, the Import API module handles synchronization and conflict management between local and remote revisions of each asset.

This module imports TV Anytime-formatted information about a media asset.

● Items: on-demand video, broadcast and even live-broadcast ● Sources:


23

● Link URL or pre-produced content from CMS of broadcaster ● Dummy for live-broadcast applications (e.g. in a soccer game the overlay’s

position of the goal count is predictable/ the live-broadcast is not, the editor might want to trigger certain actions which are already pre-configured, i.e. player’s stats are shown after a goal was shot)

Asset API

The purpose of this module is to define asset structure and definition in MPAT cms, since WordPress implementation of media content is too simplistic, and to interact with asset converter to manipulate such contents. It is implemented as a WordPress plugin and its main features are:

● Extend WordPress attachment post type to define fields required by MPAT assets, like video metadata and reference to files. Metadata fields definition derives from TV-Anytime specifications. It relies on WordPress API for fields definition, their persistence and representation in administration UI.

● Feed asset converter with assets uploaded from MPAT administration UI to ensure highest coverage of devices with advanced features like adaptive streaming protocols.

● Provide admin UI to control asset converter tasks such as management of scheduled jobs, reprocessing of existing video files and generation of preview images.

Database Serialization

Since MPAT pages are managed as WordPress pages, data persistence is handled out-of-the-box by WordPress system. Also custom fields are handled this way, considering them as custom post meta.

Fields may vary a lot in term of complexity, from string and numbers, to more elaborated ones like multiple reference to other contents or grid structure, but in any case, WordPress stores information in table postmeta, with one record for each field. When the field has a complex structure, namely objects and array of values, it is automatically serialized

1 with PHP

serialize2 function before the insert or update query at database. Deserialization

3 is

automatically done as well when retrieving information from database using the php function unserialize

4. This serialization approach also applies to site and network options.

MPAT specific information is serialized to the WordPress database in three possible ways:

1. Anything concerning a page is stored in the wp_[X_]postmeta table (where X is the index of the site, wp_postmeta for the first site in a multisite), as a serialized string for structured data, as string otherwise

2. Anything concerning a particular site (MPAT application) is stored in the wp_[X_]options table (where X is the index of the site), as a serialized string for structured data, as string otherwise

3. Anything concerning the sites network (MPAT instance) is stored in the wp_sitemeta table as a serialized string for structured data, as string otherwise

Please note that “wp_” in the above examples is the default prefix used by WordPress to create tables, it may change based on WordPress configuration.

Database access from JavaScript code is going to be prevalent in the Authoring User Interface since it is going to be based on React.js. Database access will use the WordPress REST API.

1 https://codex.wordpress.org/Function_Reference/maybe_serialize

2 http://php.net/manual/en/function.serialize.php

3 https://codex.wordpress.org/it:Riferimento_funzioni/maybe_unserialize

4 http://php.net/manual/en/function.unserialize.php


24

User roles and permissions

Assumptions

- Each user can have multiple roles - Standard user roles, with related permissions, are provided by MPAT, while custom

roles can be defined with permissions from different standard roles - MPAT provides backend UI to assign roles to users and assign and revoke

permissions to roles

User roles defined in D2.1

- Concept Developer: Concept development & application design, external to MPAT; no user role needed

- API Developer: Feature implementation based on concept; no admin UI operations, no user role needed

- Layout Designer: Definition of the application’s visual look & feel (e.g., styles, templates); user role needed

- Application Creator: Application creation using MPAT authoring tools; user role required

- Content Editor: Integration of journalistic content into the application; user role required - Approval Editor: Approval of application for release/publishing; user role required - Technical Publisher: Roll-out & publishing of the application; user role required - Technical Administrator/Superuser: Manage roles grants and MPAT configuration,

have access to all areas; user role required

Implementation

- WordPress user API will be used (read more) - Ready made plugins to provide administration roles interface will be evaluated (more

likely members plugin) - Custom post statuses required by MPAT app and content creation workflows:

- Pending technical approval: pages created by App Creator which need validation

- Layout ready: pages designed by App Creator and which can be edited by Content Editor

- Pending content approval: pages completed by Content Editor which need validation by Approval Editor

Layout designer

Application Creator

Content Editor

Approval Editor

Technical Publisher

Technical Administrator

View styles X X

Create styles X X

Edit styles X X

Delete styles X X

View templates

X X

Create templates

X X

Edit templates X X

Delete templates

X X

https://codex.wordpress.org/Roles_and_Capabilities

https://wordpress.org/plugins/members/


25

View applications

X X[1] X

Create application

X X

Edit application

X X

Delete own applications

X X

Delete others applications

X

View pages X X[2] X

Create pages X X

Clone pages X X

Access to pages edit form

X X X

Select style X X

Edit style[3] X X

Select template

X X

Add modules X X

Remove modules

X X

Edit module settings

X X

Edit module content

X X

Submit page for “technical

approval” [4]

X X

Approve page structure[5]

X

Submit page for “content

approval” [6]

X X

Approve page content[7]

X X

Publish

page[8] X[9] X X

Execute static HTML generator

X X

Access to Asset

X X


26

converter dashboard

Trigger deploy

scripts[10] X X

[1] Content editor should see only applications for which he/she was previously authorized

[2] Content editor should see only pages being part of applications for which he/she was

previously authorized, and only with “layout ready”, “draft”, “pending content approval” and “publish” status, NOT “pending technical approval” status

[3] Only few style properties can be changed on page basis

[4] From “draft” to “pending technical approval” status

[5] From “pending technical approval” to “layout ready” status

[6] From “draft” or “layout ready” to “pending content approval” status

[7] From “pending content approval”, to “publish” status

[8] From any to “publish” status (sometimes “approve page content” and “publish” actions

overlap)

[9] Content editor can publish page only when editing previously created and approved, alias

“published” pages.

[10] TBD


27

Dynamic Server Architecture

This is a Node.js server. It includes the following modules:

- Communication - Second Screen - Synchronization (multi-device) - Identification of users / user management, together with related modules:

- Persistent Storage of user information (preferences) - Bookmarks support

- 360° video support - Ad insertion - Maps support - Asset converter

Whereas WordPress modules are strictly concerned with production of HTML/CSS/JS for the presentation part, to be used in the browser (TV or CS), the dynamic server modules do all the hidden, server-side work, e.g. sending fragments for dynamic presentations, sending sync information, managing the point of view, managing maps and POIs...

Second Screen Support

The Second Screen backend components are required for HbbTV 1.x Terminals since a local discovery, launch and communication is only supported in HbbTV 2.0. As fallback for HbbTV 1.x devices, the Second Screen Backend is needed. It consists of the following components:

Device and Connection Manager: It manages all devices (HbbTV Terminals and Companion Screens) connected to the Second Screen Backend. Devices need first to join a room in order to be able to communicate with each other.

Device Pairing Manager: to connect a Companion Screen to a HbbTV Application, a pairing of both devices is required. During pairing, a room token will be exchanges between both devices and stored in local storage or cookies in both apps. This allows to not repeat the pairing after each launch of the application. The HbbTV or Companion Screen App can remove the token at any time later. After this, the pairing step needs to be repeated in order to connect the devices again.

App2App Communication Proxy: This component allows establishing proxied bidirectional communication channel between companion screen and HbbTV applications.

Synchronization Support

Synchronization between multiple devices is not going to be available in the first pilot; hence this section is deferred to D4.4.

Identification of device/user

We need a way to identify a device, e.g. to identify that it is the same device as some time before.

The two things that a server will know about a device that connects are:

- The visible IP - The user agent string

Two TVs of the same brand and model in the same house/LAN with a single visible IP will not be distinguishable, but that may be rare enough to be an acceptable limitation.


28

Using cookies might be a viable alternative. The device gets a unique cookie upon its first connection. A second device will get another cookie. Note: If cookies are reset, the device will be (mistakenly) identified as a new device.

For companion screens, the problem is similar, with a bigger chance of ambiguity. However, in these cases, it is easier to ask for actual user identification.

Persistent storage of device/user information/activity

There are two possibilities for persistent storage: cookies and server-side storage. The two possibilities are useful in different situations. We also need to record the user activity, in particular for user profiling and tracking.

Persistent storage is not going to be available in the first pilot; hence this section is deferred to D4.4.

360° Video support

The 360° Video support in MPAT based on the Fraunhofer FOKUS Cloud-based 360° Video Playout for HbbTV. This Playout performs the rendering of the individual view on server side so that only the selected video content is streamed to the end device. This reduces the bandwidth needed or, put the other way around, allows for a higher quality video delivered on a given bandwidth. It also diminishes the requirements for the end device to simply play back a usual video stream in order to provide the full 360° video experience.

Figure 7 How to watch 360° Video content on TV via HbbTV

In MPAT we implement a 360° Video Plugin to integrate the Cloud-based solution into the MPAT system. The Fraunhofer FOKUS Cloud-based 360° Video Playout for HbbTV consists as depicted in the figure above of the following components:

360° Video Cloud Streaming Server: The 360° Video Cloud Streaming Server renders any kind of 360° content and streams the individual view to playback devices such as TVs and mobile devices. The 360° Video Cloud Streaming Server offers a REST API to control the individual view and playback state for each client.

360° Video TV Player: The 360° Video Cloud Streaming solution enables Smart and Hybrid TVs to provide 360° Video experience through usual video playback of the output from the cloud streaming server. Users can control their individual view and playback state via TV remote control.


29

Figure 8 How to predict future camera angle with

FOKUS Cloud-based 360° Video Playout for HbbTV

This player is replaced with the MPAT 360° Video Player, that will be also customizable for the editor.

Ad Insertion/Replacement

The following section describes the broadcast and IP-Video handling of the ad services at the TV client side. The gathering of user information and its analysis is outside of the scope of MPAT. As there are already accepted ad-systems, which are specialized in user tracking, the necessary data is handled by external ad serveice (e.g. Google analytics). The ad related functionality implemented in MPAT supports two use cases:

Ad insertion

This only relates to IP-videos in the HbbTV domain. VoD is to be paused during playback without user-interaction and inserted with targeted IP-video-ad(s). After the insertion, the video resumes to play from the previous paused position. This feature also includes a timer function to set an interval of how often the ad clips are interrupted in respect to the video playback.

Figure 8 Ad Insertion


30

Required functionality:

● External communication with ad server ● Interrupting on-demand-videos and inserting ad-videos ● Time-synched pre-buffering of ad-videos ● Insert options

● Insert at start of VoD ● Setting a time interval for insertion

Ad replacement

The broadcast video clips from an ad-break are overlaid (replaced) with IP-Videos in order to present personalized ads to the viewer. Therefore the ads of broadcast and IP have to be pre-buffered and synchronized in a frame accurate manner. This can be done one by one clip or on the whole ad-break.

Figure 9 Ad Replacement

Required functionality:

● External communication with ad server ● Frame accurate synchronization of IP-videos with broadcast using stream events ● Time-synched pre-buffering of ad-videos

Bookmark Support

This covers bookmarks, favorites, reading lists, save for later reading... referred to generically as bookmarks.

When a user interacts to this effect, info is passed to the backend and a piece of information (current URL usually) is added to some stored set of bookmarks.

This uses the general persistent storage module above. As such, it is not going to be available in the first pilot; hence this section is deferred to D4.4.

Map Support

Map support may include localizing the device to be able to send a map centered around the position. It also includes management for a list of PoIs and its translation into a presentation format that can be layered on to the map. This section is deferred to D4.4.


31

Asset Converter

Figure 10 Asset Converter

MPAT allows the content editors to insert videos in applications from several sources, which leads to the necessity of a tool that ensure compatibility with formats and codecs supported by HbbTV and mobile devices, namely the Asset Converter (AC). At the highest level, the asset converter digests video files and converts them with specified video and audio settings to assure compatibility with different interactive TV and mobile devices, considering also latest protocols for adaptive streaming. It relies on several third-party libraries and tools for its operation:

● Composer: Dependency manager for PHP projects ● Monolog: A PHP library which implements PSR3 interface for logging in PHP

applications and allows the asset converter to log messages on multiple streams at the same time.

● Components from Symfony framework: used for tasks like filesystem manipulation, application configuration and generation of a CLI application

● FFMPEG: open source library for video and audio manipulation, used in Asset Converter mostly for video conversion, image extraction and HLS segmentation to achieve adaptive streaming on mobile devices

● MP4Box: A segmenter tool to generate MPEG-dash compliant files to achieve adaptive streaming on HbbTV devices

In the first release, users can interact with Asset converter in two ways, executing commands through the CLI application provided or calling APIs after including the asset converter as a dependency. The integration plugin will use APIs provided by the AC classes to create CMS representations of the converted assets.

Assets can be added to the AC individually, providing their path or URL, or from a given folder. In the latter case, all the compatible files, are imported in the AC. Either way, the AC creates a copy of the original asset in its storage folder to ensure consistency throughout the time and prevent cases where further conversions fail because of missing source file. AC behavior on duplicated assets and folder cleaning can be configured at the application level. Scenarios like automatic digest of assets from third-parties like collaborators and agencies, can be easily achieved scheduling a periodic import from folders previously exposed by the FTP server.

The asset converter implements a simple but effective queue system with a lock mechanism while processing the assets. This prevent collisions between encoding jobs and performance degradation caused by uncontrolled proliferation of running processes. The simplest, as well as the most conservative, AC configuration considers one encoding job for one instance of the AC. Due to encoding time and number of requested assets, it might happen that the queue grows faster than AC capability to serve encoded files. In this case, some measures can be adopted. At first, FFMPEG threads number can be tuned accordingly to machine specifications to enable AC to use multiple processors. It is also possible to schedule multiple jobs at shifted intervals, reducing the time between one encoding task and the next, at the cost of more


32

resources used. Another approach could be to setup multiple instances of AC on several machines, sharing database and filesystem across them allows to have a single queue management while multiplying encoding capabilities.

While the Asset Converter need its own database and filesystem, there is nothing to prevent the reuse of those used elsewhere within MPAT. This configuration has some benefits, such as reduced configuration times and also import times, since assets which are already in AC folder are not copied but just added to the AC database. As a downside, mostly if the MPAT CMS is used as a production backend, assets conversion may lead to overloads resulting in a degradation of performance of MPAT applications. In that scenario, it is suggested to setup one or more separated AC instances.


33

Edge Server

An edge server, in a system administration context, is any server that resides on the “edge” between two networks, typically a private network and the Internet. An edge server will be used in the MPAT-Production environment for serving the final HbbTV app in form of HTML/JS/CSS and will act as a Proxy-Server for the communication with the Node.js backend aka dynamic backend.

The Edge Server is a requirement from Broadcasters for security reasons because it will decouple the MPAT-Authoring and the final HbbTV Application. As a side effect the Edge Server will be used to translate requests to the MPAT-Dynamic Server. Typically Services on the Dynamic Server are available on different ports.

Deployment/Playout/Testing

Due to the hybrid approach of the MPAT system, special attention has to be paid to the deployment of the applications. The multi-platform needs to provide data via broadband and broadcast, therefore two separate ways to deliver the data has to be chosen. On the one hand a static compiler located in the WordPress part of the system deploys the relevant web data (HTML/CSS/JS) to a conventional static HTTP webserver which is supplying the front end application.

The broadcast playout on the other hand needs to specify where the app is located in the web. This is done with the help of the so-called AIT (application information table) included in the MPEG transport stream. There, the start-URL of the designated application is signaled to the TV-client. According to the HbbTV specification the URL is bound to the TV channel and is auto-started when the user switches to that related TV channel. This is normally a fixed URL which doesn't change over the run-time of the application.

For dynamic events there is the possibility to use stream events which are part of the DSM-CC system (Digital storage media command and control) available in HbbTV consisting of small data packets that can be transmitted synchronously with the programme material. The purpose of the stream events is to send triggers to the client application. This can be used to having a timed link between the broadcast and the broadband world, e.g. for synchronising devices or just to trigger events at a certain time. As discussed previously, MPAT will provide number of methods to create the stream events. For example, an event schedule can be set up, where the author lists the triggers relevant to the application with the help of relative time-stamps. Furthermore, there is the potential to create stream events during the run-time of the applications via a manual live-trigger, e.g. by an editor. Both options have to be processed by the stream event inserter, a component in the external backend of MPAT.

While the static webserver feeds the frontend client with the broadband information, the TV-Playout API handles the broadcast side of the playout system. The MPAT consortium decided to define a generic interface, including an external AIT configuration component and the stream event inserter. For this purpose MPAT enables the connection of various TV modulators. Those modulators are needed to generate the transport stream, where the relevant application information and app triggers are inserted into the broadcast signal.

Ultimately MPAT supports two different ways of broadcast deployment. Firstly, finalised apps need to be tested in an appropriate environment, but this can't be done in the scope of a production playout of the broadcaster. Therefore a possibility to test the authored apps with various test systems has to be facilitated, which will be supported by a generic Playout API. As a first prototypical implementation, it has been decided to use IRT's BRAHMS solution for this


34

matter. Secondly the apps have to be ingested in the production system of the broadcaster for final playout via an appropriate interface.

Documents

System Architecture and API Documentation - MPATmpat.eu/.../2016/08/D4.3_System_Architecture_and_API_Documentat… · D4.3 System Architecture and API documentation 4 Introduction