A look at the new multimedia landscapepaginas.fe.up.pt/~mandrade/presentations/mapTele09.pdf · MAP-Tele Doctoral Course 2009-2010 FEUP Porto, 11 Dez’09 Context-awareness in multimedia

A look at the new multimedia landscape

Maria Teresa AndradeFEUP / INESC Porto

[email protected] ; [email protected]://www.fe.up.pt/~mandrade/ ; http://www.inescporto.pt

Introduction:

current situation and challenges in multimedia

communications

the online paradigm - anytime, anywhere

Typical scientific and technological

topics involved

multimedia coding and analysis

data bases and information systems

metadata

communication protocols and networks

mobile platforms

Presentation walkthrough

MAP-Tele Doctoral Course 2009-2010FEUP Porto, 11 Dez’09

Context-awareness in multimedia communications

New approaches to multimedia

communications

the user as the center of the service

the importance of context and challenges involved

standards support

Context-aware platforms

technologies, architectures, value delivered

trends



Introduction

Current situation and challenges in multimedia communications



Facts pertinent to mm comm

• huge amount of content available online

• multiplicity of formats

• different encoding schemes, different resolutions

• everyone can be a producer

• poorly annotated content

• different description schemes in use

• user generated tags

• free text annotations

• continuous progress of technology

• new multimedia-enabled devices appear each day offering different presentation capability

• convergence of networks

• content providers expand their offerings to multiple platforms

• more types of user devices supporting multimedia and IP connections



Current scenario

broadcast media

live media

stored media

media databases, repositories

♫♪

♪♫♪♪♫

♪♫

great diversity of client devices

great diversity of content sources, both professional

and domestic

different access and core networks

DVB-T/S/C/HUMTS, GPRS, cable, ADSL, dial-up, ...



Existing services• Entertainment / home

• media clips download and/or streaming (YouTube, ...)

• mp3 file sharing (iTunes, BearShare, LimeWire, ...)

• images/photos sharing (Flickr, ...)

• Internet radio, Internet TV

• on-line newspapers (OhmyNews, Google News, ...)

• blogs, RSS feeds

• IPTV, VoD

• Businesses, education

• file sharing

• live seminars, videoconferencing (Mbone, ...)

• streaming of educational content (classes, additional video material)

• VoIP

• collaborative work



Aspects to consider

• real-time

• real-time coding versus stored, pre-coded content

• real-time transmission and presentation

• interactive applications versus non-interactive

• static channels versus dynamic

• compression schemes

• constant bit rate (CBR) or Variable bit rate (VBR)

• features/capabilities of compression schemes

• download versus streaming

• required level of quality of service (QoS)

• content/applications accessible (on-line) and easily found



Aspects to consider: real-time• interactive applications (videoconference, VoIP, etc) and live

events (sports, seminar, keynote speech, etc) require real-time encoding and transmission

• if data arrives too late, it no longer can be used

• necessary to impose a limit to the end-to-end latency (maximum delay of coding+transmission+decoding+presentation) - typically 150 ms

• Video on Demand or video-clips streaming can be pre-encoded and stored but require real-time transmission

• pre-coding can be performed in non-real time, enabling the use of less expensive coders

• it also allows multiple-step encoding, resulting in better compromise between quality and bit rate (current practice in DVD)

• delay can be absorbed using a receiving buffer



Aspects to consider: channels

• static versus dynamic channels

• static channels offer a fixed bandwidth and usually fixed delays (small jitter) and small loss rate

• examples include ISDN (px64 kbit/s), cable or DVD

• dynamic channels present a higher degree of variability and usually higher bit and loss rate

• pose more challenges to the implementation of high-quality multimedia services

• examples include radio channels of wireless networks (such as UMTS) but also wired best-effort networks (such as Ethernet, shared media)



User’s expectations

• mobility

• personalization

• transparency

• more quality

• more choice

• easy person-to-person communication

• better/accurate search results

• any content, any time, anywhere



User’s expectations

• mobility

• personalization

• transparency

• more quality

• more choice

• easy person-to-person communication

• better/accurate search results

• any content, any time, anywhere



Current difficulties

• limited and variable network bandwidth

• non-reliable connections

• more bandwidth needed in many cases

• lack of a coherent platform for metadata

• lack of interoperability between applications and services

• non-scalable solutions

• protocols in constant evolution

• still rather poor user experience



Transport alternatives

• unicast, broadcast, multicast

• peer-to-peer

• unicast or point-to-point

• one-to-one communication, ex., videotelephony, media streaming on the Internet, VoD

• requires a copy of the content for each client

• can be customized to the client needs, preferences or context restrictions

• usually requires the existence of a return channel between the client and the source




• communication one-to-many (goal “one-to-all”)

• most well-known example, regular TV broadcasting

• efficient way to deliver popular content to a large audience simultaneously

• a broadcast system must be designed for each customer in the same way regardless of their requirements, terminal devices or connection conditions

• the system must be dimensioned for the worst case if all customers are to be served

• the system has limited possibilities to adapt to changing usage conditions

• given the great number of terminal devices, usually it is not practical to establish feedback channels for every consumer

• broadcast




• communication one-to-many (goal “one-to-all”)

• most well-known example, regular TV broadcasting

• efficient way to deliver popular content to a large audience simultaneously

• a broadcast system must be designed for each customer in the same way regardless of their requirements, terminal devices or connection conditions

• the system must be dimensioned for the worst case if all customers are to be served

• the system has limited possibilities to adapt to changing usage conditions

• given the great number of terminal devices, usually it is not practical to establish feedback channels for every consumer

• broadcast




• one-to-many communication but not one-to-all as in broadcast

• receiving terminals must take the action to connect to a multicast IP address

• in broadcast the signal arrives in fact to every terminal, regardless on whether the terminal is connected or not

• it is more efficient than unicast to reach simultaneously several terminals

• rather than having to establish one end-to-end connection between each consumer and the source, replicating the number of streams as much as the number of connected users

• it offers the same advantages as broadcast

• however, current IP networks have limited support for IP multicast

• other aprpoaches are being developed, working at the application level - overlay multicast

• multicast




• one-to-many communication but not one-to-all as in broadcast

• receiving terminals must take the action to connect to a multicast IP address

• in broadcast the signal arrives in fact to every terminal, regardless on whether the terminal is connected or not

• it is more efficient than unicast to reach simultaneously several terminals

• rather than having to establish one end-to-end connection between each consumer and the source, replicating the number of streams as much as the number of connected users

• it offers the same advantages as broadcast

• however, current IP networks have limited support for IP multicast

• other aprpoaches are being developed, working at the application level - overlay multicast

• multicast



unicast versus multicast

unicast

multicast




• Peer-to-peer

• unstructured architectures

• the network is flooded with query messages to locate streams and files (all nodes receive the query)

• structured architectures

• based on Distributed Hash Tables (DHT) that maintain an index to the available resources, facilitating location

• query messages are forwarded to only some nodes, which are more likely to have the location of the desired resource



Challenges in p2p for multimedia

• generally, users when searching for multimedia resources

• do not know the exact name of the file they are looking for

• often want to receive a useful set of results that are clearly related to the subject they are interested in

• while unstructured designs allow this kind of proximity behavior

• at the cost of larger search times and increased network traffic

• common structured, DHT-based, designs do not

• search is performed using a hash key and only exact matches are returned



Challenges in p2p - semantic queries

• Current research initiatives include

• Locality Sensitive Hashing (LSH) techniques (nearest-neighbor search problem in a Euclidean space)

• hierarchical multiple indexes, allowing query-to-query mapping and thus the implementation of a recursively query process

• the CUBIT project at Cornell University using an overlay where peers are organized in concentric rings

• though promising good results, require complex management and/or processing of intermediate results



Typical scientific and technological topics involved

multimedia coding and analysis

data bases and information systems

metadata

communication protocols and networks

mobile platforms



multimedia coding

• what is the desire of every consumer?

• to watch his/her favorite program at the best quality

• well, then, encode with as much bits as possible! ... or not?

• more complex encoders and decoders - more processing power needed, more memory required, larger storage capacity, professional and consumer equipment more expensive

• larger transmission pipes, allocation of huge amount of (non-existing!) resources, more money needed to invest on building adequate tx lines, more the users will have to pay



multimedia coding

• well then, it is necessary to compromise

• compromise between quality (distortion) and bit rate (money!)

• Rate-Distortion (R-D) optimization

• video is the most problematic because it has the highest bandwidth (6MHz analogue video)

• uncompressed video requires a too large bandwidth

• standard TV signals reach 260Mbit/s



multimedia coding

• compression algorithms can be lossy or lossless

• lossless means that the original signal can be reconstructed without any distortion (ex. zip)

• lossy means that some degradation will occur during reconstruction

• amount of loss of information should be according to the application requirements

• loss is a distortion from the original data that can be measured through the SNR (Signal to Noise Ratio) or MSE (Mean Square Error)

• it provides an indication of the distance between the original and the reconstructed signals



multimedia coding

• important characteristics to look at when selecting a compression algorithm

• efficiency

• relation between compression rate and quality

• compression can be measured by the relation of file sizes before and after compression

• complexity

• hardware or software only

• amount of processing power consumed, number of operations per unit of time

• real-time versus non rea-time

• delay

• how long is it necessary to wait before starting decoding a compressed signal? how much buffering is required?



multimedia coding• It is thus important to be able to measure the quality or the

efficiency (quality versus bit rate) of different compression schemes

• objective measures

• PSNR (peak signal to noise ratio), MSAD (mean sum of absolute differences), MSE (mean square error), ...

• can be automated and performed in real-time

• but sometimes are far away from the human perception of quality

• measured in dB

• a value of 50 dB indicates an almost perfect reconstruction

• subjective measures

• VQM (video quality metric)

• use objective metrics that try to model the human perception

• more difficult to perform

• may use an audience to visualize sequences and grade them



Evaluating video coding• quality evaluation



Evaluating video coding• quality evaluation

VQM - brighter blocks correspond to greater

differences, hence less quality



Principles of media coding

• how is it possible to compress and still obtain a good representation of the original content?

• taking advantage of redundancy that exists in the media signals

• redundant data either duplicates information or brings no new information

• taking advantage of the properties of the human perceptual system

• perceptual redundancy (psychoacoustic and psychovisual)



multimedia coding - redundancy

• Given that

• digital audio signal = succession of samples in time

• digital image = square matrix of pixels (spatial samples)

• video = sequence of images at a certain frequency in time

• neighbor samples in those signals (audio samples, pixels or images) are more or less correlated among them

• i.e., part of their information is the same or very similar, thus, redundant



multimedia coding - redundancy• several types of redundancy in audio and video signals

• spectral redundancy

• for example, duplication of information (redundancy!) between the RGB fundamental colors

• hence the benefits of using other color spaces :-)

• spatial redundancy

• for example, between pixels of an image that are spatially co-located

• temporal redundancy

• for example, between two consecutive images of a video sequence with moderate motion; pixels belonging to the background of two images in a video sequence where only people are moving

• statistical or coding redundancy

• code words with length larger than necessary

• code inefficiency



multimedia coding - major tools

• Predictive Coding

• eliminates temporal redundancy

• Transform coding

• eliminates spatial redundancy

• Quantization

• effectively reduces the amount of required bits taking profit of spatial redundancy reduction

• Entropy coding

• eliminates statistical redundancy



multimedia coding

• different compression schemes provide different results in terms of compromise quality-bit rate-complexity

scheme area of application bit rate

H.261 videoconference piecewise CBR, p*64 Kbit/s

MPEG1 storage and retrieval, streaming CBR, up to 1.15 Mbit/s

MPEG2 digital TV CBR and VBR, from 1.5 up to 20Mbit/s

H.263 videoconference CBR and VBR, from 33.6 Kbit/s

MPEG4-2 object-based coding, synthetic coding, ...

CBR and VBR

AVC/MPEG4-10/H.264 video streaming in mobile networks, HDTV, ...

CBR and VBR, from 10 Kbit/s up to several Mbit/s

MPEG4 - MVC (part of AVC)

3D TV content up to several Mbit/s



multimedia coding - open issues

• more efficient (R-D optimisation) algorithms in terms of

• subjective quality

• scalability

• encoder and decoder complexity

• ultra high definition and mobile HD

• higher compression ratios are needed

• more resolution (from 8 to 10 bits; full color resolution 4:4:4; larger spatial and temporal resolutions)

• 3D or stereoscopic video coding



multimedia coding - open issues

• more efficient (R-D optimisation) algorithms in terms of

• subjective quality

• scalability

• encoder and decoder complexity

• ultra high definition and mobile HD

• higher compression ratios are needed

• more resolution (from 8 to 10 bits; full color resolution 4:4:4; larger spatial and temporal resolutions)

• 3D or stereoscopic video coding



multimedia coding - new directions

• ISO/IEC JTC1 SC29 WG11 (MPEG)

• HVC (High-performance Video Coding)

• targeting substantially bit rate reduction over MPEG-4 AVC High Profile

• new 3D standard

• ITU-T SG16 Q6 (VCEG)

• H.NGC (Next Generation video Coding)

• Bit rate reduction of 50% at the same subjective quality

• Complexity ranging from 50% to 3 times H.264/AVC High Profile

• Goal is to have new standards in 2~3 years

• better visual experience for users and higher efficiency video coding

• Exploit perceptual characteristics, especially temporal features

• Exploit content features, especially local features



metadata

• data about data

• any kind of additional information concerning any type of content

• often includes items like file type, file name, creator name, date of creation, etc.

• main reason for metadata is to improve the management of and access to information



metadata

• structured information that describes, explains, locates a media resource

• a metadata record consists of a set of attributes, or elements, necessary to describe the resource in question

• metadata scheme

• Set of metadata elements, with associated content, semantics and syntax for describing a particular type of resources

• content

• values given to metadata elements

• semantics

• meaning/ definition of metadata elements

• syntax

• rules on how content must be represented, identified and formulated



metadata



• metadata scheme


• content


• semantics


• syntax




metadata



• metadata scheme


• content


• semantics


• syntax




metadata - content

• Descriptions about the content

• semantic or high-level

• low-level or intrinsic

• Usage

• content classification

• content search meeting user’s requests (useful content search)

• determine whether the content has the adequate format to be presented in a given user device

• to assist content transform or adaptation operations satisfying usage constraints



content metadata

<Elementary_Streams> <VideoParams> <VideoCoding>Mpeg2MP@ML</VideoCoding> <VideoResolution HorizontalSize="720" VerticalSize="576"/> <VideoBitRate>3000000</VideoBitRate> </VideoParams> <AudioParams> <AudioCoding>MPEG1 Layer 2</AudioCoding> <AudioMode>Stereo</AudioMode> <AudioBitrate>128000</AudioBitrate> </AudioParams></Elementary_Streams>



content metadata• textual descriptions can be subjective or incomplete



metadata - context

• Descriptions about the consumption context

• terminal capabilities

• characteristics and conditions of the network

• conditions or constraints of the usage environment

• user preferences and requirements

• Usage

• to determine whether the content has the adequate format to be presented under specific context consumption conditions

• to assist decision taking mechanisms when adaptation is required



metadata - context

• Descriptions about the consumption context

• terminal capabilities

• characteristics and conditions of the network

• conditions or constraints of the usage environment

• user preferences and requirements

• Usage

• to determine whether the content has the adequate format to be presented under specific context consumption conditions

• to assist decision taking mechanisms when adaptation is required



multimedia analysis

• analyse video and audio content extracting the values of intrinsic properties

• in video, color histograms, shape, movement, etc

• use knowledge-based techniques to infer higher-level concepts

• “countryside landscape”, “beach”, “car race”, soccer match”, “scene with church”, etc.

• it’s a way of getting metadata automatically!!



multimedia analysis

• analyse video and audio content extracting the values of intrinsic properties

• in video, color histograms, shape, movement, etc

• use knowledge-based techniques to infer higher-level concepts

• “countryside landscape”, “beach”, “car race”, soccer match”, “scene with church”, etc.

features extraction

features• color• shape• timbre

...

abstractionknowledge-based

processing

metadata• low-level

features• high-level concepts

summariesbrowsing retrieval

• it’s a way of getting metadata automatically!!



databases and information systems

• efficient database models to accommodate

• different types of descriptions

• repositories of multimedia information

• efficient search engines

• keyword-based

• semantic

• free-text based



communication protocols & networks

• transport of multimedia content imposes requirements on networks

• different network technologies require different aspects to be addressed differently to support the transport of multimedia content

• overall bandwidth, delays, variability in available bandwidth, loss, error rate

• QoS assurances, resources allocation




• IETF protocol stack

• signaling and transport protocols for multimedia networked applications

• transport

• TCP e UDP

• controlled transport

• RTP e RTCP

• session and interaction

• RTSP, SIP, SDP

• QoS assurances

• RSVP, DiffServ




• IETF protocol stack


• transport

• TCP e UDP

• controlled transport

• RTP e RTCP

• session and interaction

• RTSP, SIP, SDP

• QoS assurances

• RSVP, DiffServ

initial request and reply in HTTP

description of the offered service using SDP

messages between client and server using RTSP

Multimedia networked applications: standards, protocols and research trends 30/01/2009

MSc, University of Minho

IETF - protocol stack


session

transport

network

link

RSVPSIP

SDP

TCP UDP

RTP

IP

media coder

Ethernet ATM ...

application domain



mobile platforms

• research in multimedia communications is now focussing around mobile platforms given the growing acceptance of the online paradigm

• crucial to enable the “anywhere and anytime” paradigm

• require the development of client-server architecture

• development of client applications in different mobile platforms

• exchange of messages between client and server



mobile platforms

• research in multimedia communications is now focussing around mobile platforms given the growing acceptance of the online paradigm

• crucial to enable the “anywhere and anytime” paradigm

• require the development of client-server architecture

• development of client applications in different mobile platforms

• exchange of messages between client and server



mobile platforms

• usually, applications are of type client-server

• requires the development of software applications that run on the existing mobile platforms.

• for smartphones

• development platforms that have been used more commonly are BlackBerry, Palm webOS, Symbian and Windows Mobile

• more recently Android from Google has gained a hype and also iPhone from Apple

• Python, Java 2 Micro Edition (J2ME), Flash Lite and Symbian C++ are the programming languages

• the choice of development platform constraints the choice of tools and languages as well as the range of devices the applications can run on



mobile platforms

• Android open source software stack from Google for mobile devices

• based on Linux and using Java as the programming language

• includes an OS, middleware and key applications

• provides the SDK Android (System Developer Kit) with tools and APIs to develop applications that run on devices with the Android OS

• iPhone offers an OS and SDK with simulators

• applications are developed using the XCode IDE in Objective C (a superset of Ansi C)

• free simulators available

• Windows Mobile

• offers an SDK with simulators

• applications are developed using Visual Studio and usually written in C++ or C#



mobile platforms

• J2ME and Symbian C++ are very powerful languages

• have support in a number of devices

• however their learning curve is steep

• Phyton is a scripting language

• not as powerful but simples and good to automate simple tasks and perform simple logical processing

• limited support (Nokia series S60)

• Flash Lite is the mobile version of Flash from Adobe

• content is created using Adobe Flash Professional 8 or CS3

• supported in some Nokia smartphones



New approaches to multimedia communications


the importance of context and challenges involved

standards support




• delivering multimedia content to everyone requires that the user is considered the driving force of the service:

• disabilities

• visual and hearing disabilities

• physical and cognitive or neurological disabilities

• cultural aspects related with the user

• constraints regarding type of information (religious for example)

• non English speakers (language translation)

• level of expertise

• simple preferences of the user

• in type of content as well as format of the content

• the consumption environment where the user is

• scenarios that need or involve

• mobile solutions

• different operating systems

• noisy environments

• restricted access



The importance of context

• what is context in multimedia communications?

• it’s any kind of information that characterizes entities and situations involved in the consumption of multimedia content

• A. Dey, “"Context is any information that can be used to characterize the situation of an entity. An entity is a person, place, or object that is considered relevant to the interaction between a user and an application, including the user and application themselves."

• what is it good for?

• allows applications to automatically learn the usage consumption characteristics thus becoming context-aware



the importance of context

• context enables applications to be context-aware

• Context-aware applications are those having the ability to detect, interpret and react to aspects of user's preferences and environment characteristics, device capabilities or network conditions by dynamically changing or adapting their behavior based on those aspects that describe the context of the application and the user





client devicessources of content

different networks

♫♪

♪♫♪♪♫♪♫





client devicessources of content

different networks

♫♪

♪♫♪♪♫♪♫

context-aware

mediation

descriptions of content sources

capabilities, producers, adaptation

engines

descriptions about the user and his/her

terminal and connectivity conditions

descriptions of static and dynamic network

characteristics




• according to the entity it describes, context can be grouped into four main classes

• Resources, User, Physical, Time

• according to characteristics and nature of the contextual information

• the accuracy or level of confidence of the contextual information

• the period of validity

• contextual information may be static thus having an unlimited validity period or it may be dynamic and valid for only a given period of time..

• dependencies on other type of contextual information. Reasoning about one type of contextual information may depend on other types of contextual information

• according to the process by which context is obtained

• low-level, explicit or sensed contextual information when it is directly generated by some software or hardware appliance (sensors)

• high-level or inferred context

Time

User

Resource




• according to the entity it describes, context can be grouped into four main classes

• Resources, User, Physical, Time

• according to characteristics and nature of the contextual information

• the accuracy or level of confidence of the contextual information

• the period of validity

• contextual information may be static thus having an unlimited validity period or it may be dynamic and valid for only a given period of time..

• dependencies on other type of contextual information. Reasoning about one type of contextual information may depend on other types of contextual information

• according to the process by which context is obtained

• low-level, explicit or sensed contextual information when it is directly generated by some software or hardware appliance (sensors)

• high-level or inferred context




• resources context

• characteristics, capabilities and conditions of the resources that can or need to be used to provide the context-aware service - description of terminals, networks, servers or adaptation engines

• terminal: hardware platform, such as processor, screen size or network interface; software platform, such as operating system or software multimedia codecs installed.

• network: description of the network such as maximum capacity, instantaneously available bandwidth or losses;

• sources or multimedia servers: maximum number of simultaneous users or maximum throughput;

• adaptation engines: hardware and software platforms such as network interface or input/output formats allowed, bit rate range supported, etc.




• user context

• user general characteristics

• gender, nationality or age

• preferences of the user related with the consumption of content

• type of media or language preferred or related with the high-level semantics of the content (eg., local news versus international news or action movies versus comedy)

• user’s emotions and state

• anxious versus relaxed or happy versus sad

• online versus offline or sited versus walking

• physical context

• description of the natural environment surrounding the user such as lighting and sound conditions, temperature or location;

• time context

• indication of the time at which variations in the context have occurred or scheduling of future events



the importance of context and challenges

• using the context

• the meaning of context-awareness is to make use of sensed context to build understanding at higher levels of conceptual abstractions

• context-aware applications must initially acquire the explicit contextual information and then process it and reason about it to formulate concepts and take decisions if and how to react

• what type of context is relevant to my application?

• need to consider availability of context sources, periodicity of acquiring context, representing context, relating context, ...

• usually the knowledge and concepts that can be extracted greatly depend on the application at hand



the importance of context and challenges

• aspects to look at when using context dynamically

• accuracy or level of confidence of the contextual information

• period of validity – static versus dynamic context

• dependencies on other types of contextual information

• necessary to be aware and register context changes

• be able to react in useful time to those changes



the challenges in using context

• how to identify the required contextual information?

• how and when to sense and gather that contextual information?

• how to use the context to take decisions?

• how well do automatically generated context describe real world situations?

• how can automatically generated context be used to form rich concepts that translate well real world situations?

• how can systems know whether the user is satisfied?




• explicit contextual information is generated by sensors

• any hardware appliance, software component or entity capable of generating data describing some aspect of the context of usage can be designated as a sensor

• examples of low-level context

• a measurement of the available bandwidth performed by a network probe

• dimensions of a terminal display

• a temperature value obtained by a thermometer

• an indication of available Central Processing Unit (CPU) resources of a computer

• an alarm generated by a motion detection sensor

• gathering, representing and using different kinds of contextual information are essential steps of any system aiming to provide adapted services and content which satisfy usage constraints while meeting users’ expectations.



standards support

• representation of context is still an open research issue

• existing standards provide good frameworks to capture values of sensed context in a coherent/interoperable format

• three major initiatives for standard representation of context

• World Wide Web Consortium (W3C)

• Device Independence (DI) working group and Mobile Web Initiative (MWI)

• MPEG

• MPEG-7 and MPEG-21



standards support

• W3C, Device Independence

• provide "access to a unified Web from any device in any context by anyone"

• finalize the Composite Capability/Preference Profiles (CC/PP)

• liaise with the User Agent Profile (UAProf) of the Open Mobile Alliance (OMA)

• major work items

• content selection for device independence (DISelect)

• delivery Context: Client Interfaces (DCCI)

• Device Independent Authoring Language (DIAL)

• now looking to a knowledge-based framework

• Ubiquitous Web Applications (UWA) activity

• Delivery Context Ontology



standards support

• W3C DISelect

• normative processing model for general content selection or filtering

• attributes and elements defined enable conditional processing

• arbitrary expressions using a set of well‐defined XPath functions



standards support

• W3C DISelect

• normative processing model for general content selection or filtering

• attributes and elements defined enable conditional processing

• arbitrary expressions using a set of well‐defined XPath functions



standards support

• W3C CC/PP

• based on RDF to describe capabilities of client devices and user preferences

• uses the “user agent” field in HTTP requests

• user preferences sent in the HTTP header

• limited support though



standards support

• W3C CC/PP







standards support

• W3C CC/PP







standards support• W3C Resource Description Framework (RDF)

• generic specification to represent metadata

• three distinct forms of representation

• 1) “Triplet”, set of values ‘Subject, Predicate, Object’

• 2) XML syntax, used for data exchange between machines

• 3) graph representation



standards support

• MPEG-21

• digital object architecture to facilitate the management of and access to distributed heterogeneous multimedia resources

• open standard based on XML

• focuses on the development of an extensive set of specifications, descriptions and tools to facilitate the transactions of multimedia content in heterogeneous network environments

• divided in 18 parts

• more relevant for the implementation of context-aware content adaptation is MPEG-21 Part 7 – Digital Item Adaptation



standards support

• concepts of MPEG-21

• DI – Digital Item

• set of multimedia resources related to a certain topic (eg., are part of the same TV program or are related to a given song writer) and associated descriptions

• it includes an XML, the DID (Digital Item Declaration – part 2of the standard) that allows to declare and describe the constituent parts of teh DI

• user – any entity (system or person) who interacts with the DI



standards support• mpeg21 DI

• from the conceptual point of view it can be seen as a package of multimedia resources and metadata, include its XML declaration



standards support

• mpeg21 DIA, Digital Item Adaptation

• specifies a set of description tools to assist the adaptation of DIs

• contextual descriptions and structure of bitstreams

• relation between usage constraints and possible adaptation operations

• relation between adaptation and result in terms of quality

• tool Usage Environment Description – UED

• includes characteristics of users, terminals, network and natural environment

• Universal Constraints Description – UCD

• describes usage restrictions and the optimization of adaptation operations that satisfy those restrictions

• Terminal and Network QoS – AdaptationQoS, AQoS

• provides relationships between restrictions and adaptation operations satisfying those constraints as well expected quality



standards support














standards support














standards support• mpeg21 DIA









Context-aware platforms

technologies, architectures, value delivered

trends



technologies, architectures

• Including context-aware content adaptation in multimedia systems requires functionality to

• sense the context (and reason with it to infer new knowledge)

• take decisions if and how to react

• transforming an input multimedia signal into an output signal with different characteristics

• using manipulations at multiple levels: signal, structural or semantic

• to meet different resources’ constraints and user preferences

• maximizing/optimizing a given utility such as the quality of the multimedia content



technologies, architectures• different forms to adapt multimedia

• temporal scaling (in a video, modify the number of frames per second)

• spatial scaling (in a video, modify it aspect ratio)

• quality adaptation (modify the number of bits per second, augmenting for example the quantizer step size; eliminating DCT coefficients; etc.)

• modality conversion or transmoding (from video to a set of key frames; from audio/speech to text; etc.)

• transcoding between different encoding formats

• in different locations

• server-based (by selecting one among multiple existing variations of the content or by adapting it online, on request)

• proxy/router/gateway (adapting online, on demand)

• client-based (by subscribing or filtering out only adequate content; by adapting the content as it receives it)



technologies, architectures• example of adaptation

• extracted from C. Timmerer, “Multimedia Content Adaptation for Universal Access “, part 1, unit 2., May 2008



technologies, architectures



technologies, architectures• Region of interest adaptation

• detecting the focus of attention of the user



Fonte de conteúdo

Recursos de rede

adaptation decision

availability of content

availability of resources

<MediaFormat> <VisualCoding> <Format ... MPEG-2 Video Main Profile @ Main Level </Format> <Frame height="720" width="480" rate="30"/> <BitRate>5000000</BitRate> </VisualCoding></MediaFormat>

<DIA> ... <... "NetworkCapabilityType" maxCapacity="640000" minGuaranteed="96000"/> <... "NetworkConditionType" AvailableBandwidth maximum="460000"/> ...</DIA>

<TerminalCapability xsi:type="CodecCapabilitiesType"> ... <Decoding ... MPEG-4 Visual Simple Profile @ Level 1 </Decoding> ...</TerminalCapability>

<TerminalCapability xsi:type="CodecCapabilitiesType"> <Decoding ... JPEG </Decoding> ...</TerminalCapability>

adaptation 2

Consumidor

request for content using the keyword “Gladiator”

adaptation 1



trends• first generation of context-aware systems used only low-

level context

• usually only addressed terminal capabilities in astatic way

• second generation introduced dynamics and also started to look at network characteristics and user preferences

• mostly using low-level or sensed context

• new generation is addressing dynamic user preferences and introducing knowledge-based techniques to infer high-level concepts

• characterising in a away closer to human thinking, the situation the user is in while consuming content

• use of ontologies and machine learning techniques

• also new locations for adaptation - in the client rather than in the server


Contextt-awareness in multimedia communications

Thank you very much for your attention!

Documents

A look at the new multimedia landscapepaginas.fe.up.pt/~mandrade/presentations/mapTele09.pdf · MAP-Tele Doctoral Course 2009-2010 FEUP Porto, 11 Dez’09 Context-awareness in multimedia