03.03.2004 Halldór Matthías SigurðssonPage 1 The MPEG Standards With special focus on MPEG-7/21 Halldór Matthías Sigurðsson Ph.D. student at CTI – [email protected]@cti.dtu.dk

03.03.2004 Halldór Matthías Sigurðsson Page 1

The MPEG StandardsWith special focus on MPEG-7/21

Halldór Matthías SigurðssonPh.D. student at CTI – [email protected] - 03.03.2004


General

MPEG– Moving Picture Experts Group

Experts from standardisation groups and from industry

ISO/IEC– International Standards Organisation– International Electro-technical

Commission

MPEG


Why standards

The essence of all standardisation efforts is INTEROPERABILITY

Interoperability for consumers means that they use content without being bothered by incompatible formats, codecs, metadata, and so forth

Interoperability for the industry means that different industries can exchange, share and profit from the same content

Paves the way for horizontal markets (in contrast to vertical)

MPEG


Standardization

The MPEG standards are “open” on two sides

Methods for generation and consumption are not defined by the standards

Generation MPEG Consumption

Standardization

MPEG


Standardization

Methods, algorithms and implementations are open for industry competition and future innovation

Intellectual property owners are committed to license their patents on terms that are fair, reasonable and non-discriminatory

Source: MPEG-4 – The Media StandardSource: Telenor R&D

Bit rate required for constant broadcast quality

MPEG


The MPEG standards

MPEG-1 MPEG-2

MPEG-4

MPEG-7

MPEG-21

Video coding standards

Content Management

Static

Objects

Description

Lifecycle

Consume

Generate

DR

M

Pay

MPEG


Review

MPEG-1– established in 1992, was designed to produce

reasonable quality images and sound at low bit rates (e.g., 352x288 images with VHS quality at 1.5 Mbits/sec).

MPEG-2– established in 1994, was designed to produce

higher quality images at higher bit rates (e.g., 720x576 D-1 at up to 15 Mbits/sec).

MPEG 1 & 2


Quality vs. bandwidth

Source: Optibase.com

Selecting the right format

MPEG 1 & 2


GeneralMPEG 4

A new object based representation of multimedia content

Integrates synthetic and natural multimedia objects

Interactivity with multimedia objects

Object based scalability

Source: Kazem Najafiwww.comm.toronto.edu/~najafi/


User InteractivityMPEG 4

Presentation level interactivity: – A user makes changes to scene by

controlling an individual object or group of objects. This also includes presentation creation

Session level interactivity:

– A user controls the playback process of the presentation, i.e. VCR like functionality for the whole session

Local level interactivity: – A user only makes changes that can

be taken care of locally, e.g. changing the position of an object on the screen, volume control or a audio object etc.


Object combinationMPEG 4

2D

SOUNDTRACK

SOUND EFFECTS

TEXT

ANIMATION

VIDEO

3D

INTERACTION

MUSIC

LAYOUT

DIALOG


Publish

CABLE

Deliver

SATELLITE

WIRELESS

DSL

CD/DVD

End UserPlayback

PC

SET-TOP BOX

DVD PLAYER

GAME CONSOLE

INTERNET DEVICE

PDA

CELLPHONE

Deploying Multimedia ContentMPEG-4

Process AuthorV

isu

al C

on

ten

tA

ud

io C

on

ten

tS

cen

e In

form

atio

n

2D

SOUNDTRACK

SOUND EFFECTS

TEXT

ANIMATION

VIDEO

3D

INTERACTION

MUSIC

LAYOUT

DIALOG

MPEG-4

Rights Holder


ConclusionMPEG 4

MPEG-4 offers a new representation of multimedia information

Full integration of various coding methods is not fully achieved

Powerful segmentation pre-processing tools are required


General

Text is self contained and can besearched and indexed

A picture can say more than thousand words

How do you describe multimedia?

How detailed can/should you get?

How can you search for multimedia?

Why would you want to?

Try describing the picture on the right– Do you thing everyone describes it in

the same way?

MPEG-7


Motivation

The multimedia context:– More information is in digital form and is on-line.– Multimedia content covers: still pictures, audio, speech, video,

graphics, 3D models, etc.– Multimedia content is available at all bitrates and on all

networks.– Increasing number of multimedia applications, services.

Necessity of describing content:– Increasing amount of information.– More needs to have “information about the content”.– Difficult to manage (find, select, filter, organize, etc) content.– User: human or computational systems.

MPEG-7

Source: Chiariglione.org/mpeg


Application Areas

Architecture, real estate, and interior design (e.g., searching for ideas). Broadcast media selection (e.g., radio channel, TV channel). Cultural services (history museums, art galleries, etc.). Digital libraries (e.g., image catalogue, musical dictionary, bio-medical imaging

catalogues, film, video and radio archives). E-Commerce (e.g., personalized advertising, on-line catalogues, directories of e-

shops). Education (e.g., repositories of multimedia courses, multimedia search for support

material). Home Entertainment (e.g., systems for the management of personal multimedia

collections, including manipulation of content, e.g. home video editing, searching a game, karaoke).

Investigation services (e.g., human characteristics recognition, forensics). Journalism (e.g. searching speeches of a certain politician using his name, his voice

or his face). Multimedia directory services (e.g. yellow pages, Tourist information,

Geographical information systems). Multimedia editing (e.g., personalized electronic news service, media authoring). Remote sensing (e.g., cartography, ecology, natural resources management). Shopping (e.g., searching for clothes that you like). Social (e.g. dating services). Surveillance (e.g., traffic control, surface transportation, non-destructive testing in

hostile environments).Source: Chiariglione.org/mpeg

MPEG-7


Scope

MPEG-7– “Multimedia Content Description Interface”– Representation of multimedia metadata in XML– Describes multimedia content at a number of levels– Provides interoperable metadata system– Allows fast and efficient indexing, searching and filtering

AcquisitionAuthoring

Editing

Browsing

Navigation

FilteringManagement

TransmissionRetrieval

Streaming

Coding

Compression

SearchingIndexing

MPEG-1,-2,-4

MPEG-7

MPEG-7


Scope

FeatureExtraction

MPEGDescription

SearchEngine

Extraction MPEG-7 Scope Use

Content analysis (D,DS)Feature extraction (D, DS)Annotation tools (DS)Authoring (DS)

Description Schemes (DSs)Descriptors (Ds)Language (DDL)Coding Schemes (CS)

Searching & filteringClassificationComplex queryingIndexing

StandardizedIndustrial competition Industrial competition

MPEG-7

Source: Dr. John SmithMultimedia Information Retrieval and Management, Springer 2003


Filter

Type of ApplicationsMPEG-7

MPEG-7enabled

Database

Search

PULL PUSH

Personalisation

Universal Multimedia Access

Universal Multimedia Access applications involve adapting multimedia content according to usage context, which includes user preferences, device capabilities, network conditions, user environment, and spatial, temporal and operational context.


Searching and Indexing

MPEG-7 Search Engine(XML Metadata)

Model Query

SemanticsQuery

SimilarySearch

MPEG-7Model

Descriptions

MPEG-7Descriptors

MPEG-7Sematics

Descriptions

MPEG-7Metadata Storage

MultimediaDatabase

Semantics-based– people, places, events,

objects, scenes

Content-based– Color, texture, motion,

melody, timbre

Metadata– Title, author, dates

Examples– Sketch up a logo and search

to search for the company

– Whistle a tune and search for the song

– Search for information about a person from a picture of him

MPEG-7


Main Elements

Description Definition Language:– The Description Definition Language (DDL) is the

language specified in MPEG-7 for defining the syntax of Description Schemes and Descriptors. The DDL is based on the XML Schema Language

Description Schemes (DS):– Description Schemes (DS) are description tools

defines using DDL that describe entities or relationships pertaining to multimedia content

Descriptors(D):– Descriptors are description tools defined using DDL

that describe features, attributes, or groups of attributes of multimedia content.

Feature– Features are defined as a distinctive characteristic of

multimedia content that signifies something to a human observer, such as the “color” or “texture” of an image

Data– Multimedia Data is defined as a representation of

multimedia in a formalized manner suitable for communication, interpretation, or processing by automatic means (i.e. for example image or video)

MPEG-7

MultimediaContent Item

Data

Feature

Observer

Descriptor

DescriptionDefinitionLanguage

Description Scheme

defines

signifies

describes

DS

DS

DDL

DD

DS

D

D D

MPEG-7 Standard Scope Application Domain

i.e. MedicalImaging,Remote Sensing,Surveillance video


Low level Descriptors

• Color • Camera motion• Motion activity• Mosaic

• Color • Motion trajectory• Parametric motion• Spatio-temporal

shape

• Color • Shape• Position• Texture

Video segments Still regions

Moving regions Audio segments

• Spoken content • Spectral

characterization• Music: timbre,

melody

Low level audio and video Descriptors

MPEG-7

Source: Ali TabatabaiSony US Research Labratories


From Low-Level to Semantic Level

Low-Level VisualFeatures

SemanticContent

Mapping

Context

Color Texture Shape Motion Shot Boundaries

People / Objects Location Events / Actions Time

Image Processing Image Understanding

Information about the content: recording date & conditions, title, author, copyright, coding format, classification, etc.

Information present in the content: Combination of low level and high level descriptors High level description

– Efficient and powerful– Lack of flexibility

Low level description– Generic and flexible– Intelligent / efficient search engine

MPEG-7


MPEG-7 Example

The following example gives an MPEG-7 description of the event of handshake between people:

<Mpeg7> <Description xsi:type="SemanticDescriptionType"> <Semantics> <Label> <Name> Shake hands </Name> </Label> <SemanticBase xsi:type="AgentObjectType" id="A"> <Label href="urn:example:acs"> <Name> Person A </Name> </Label> </SemanticBase> <SemanticBase xsi:type="AgentObjectType" id="B"> <Label href="urn:example:acs"> <Name> Person B </Name> </Label> </SemanticBase> <SemanticBase xsi:type="EventType"> <Label><Name> Handshake </Name></Label> <Definition> <FreeTextAnnotation> Clasping of right hands by two people </FreeTextAnnotation> </Definition> <Relation type="urn:mpeg:mpeg7:cs:SemanticRelationCS:2001:agent" target="#A"/> <Relation type="urn:mpeg:mpeg7:cs:SemanticRelationCS:2001:accompanier“ target="#B"/> </SemanticBase> </Semantics> </Description></Mpeg7>

MPEG-7

Source: Dr. John SmithMultimedia Information Retrieval and Management, Springer 2003


Application Examples

From Low-Level Features to SemanticsResearch results from A. Murat Takalp

University of Rochesterhttp://www.ece.rochester.edu/~tekalp

MPEG-7


Application ExamplesMPEG-7

Location-Based Browsing Using Established shotsResearch results from A. Murat Takalp




Location-Based Browsing Using Established shotsResearch results from A. Murat Takalp




Source: Source: Chiariglione.org/mpeg

Analysing a sucker game



Application Examples from research articles on Multimedia Information Retrieval and Management

Search of human faces from a face databaseDr. Kin-Man Lam

Data management for live plant identificationDr. Zheru Chi

Fast startup and interactive retrievals of broadcast videosDr. Jimmy To and Dr. C. K. Li

Biometrics feature retrieval using palmprint imagesDr. David Xhang and Dr. Jane You

Content-Based retrieval for medical dataDr. Tom Weidong Cai, Prof. David Dagan Feng, and Dr. Roger Fulton

Information Discovery on the World-Wide-WebDr. Ben Kao, and Dr. David Cheung


Content ManagementMPEG-7

Package solutions for Intranets / Company webs– Store, edit and publish files of any type – Maintain history of file changes – Recognize relationships between files – Enforce access controls to projects and files – Platform for management functions

Security and Workflow– Permit safe delegation of control to areas of the site– Restrict visibility of content to users with appropriate roles– Empower knowledge owners as content authors– Ensure content compliance with policies

Empowering Content Owners– Delegation of responsibility– Tailorable workflow provides safeguards

Organizing Content– Searching both metadata and content – Topics as alternative organization


ConclusionMPEG-7

MPEG-7:

– AV content description for interoperable application

Description Definition Language:

– XML Schema (flexibility) + Binary version (efficiency)

Great possibilities / problems

– Value of multimedia data increases when it can be

found and consumed in a better way

– It is extremely difficult to get a computer to

“understand” or analyze semantic content

– Many things can not currently be implemented in

praxis

– Automated feature extraction is not possible with

current technology


IntroductionMPEG-21

Source: MultimediaLAB, Ghent University

MPEG-21 the Multimedia Framework

Universal management of content, repurposing content based on user preferences and device capabilities, protection of rights, protection from unauthorized access / modification, protection of privacy of providers and consumers …


Digital Media Value ChainMPEG-21

Sell

Deliver

Distribute

Store

Maintain

Produce

OrganizeAcquire

Create

Plan MetadataMgmt.

Search

Collate

Package

Extract

Annotate

Adapt

Annotate

Search

Author

Tran

sact

Create

Man

age

Index

Source: John R. SmithIBM T. J. Watson Research Center


General

A future where every human on the Earth is potentially an element of a network involving– billions of content providers– billions of value adders– billions of packagers– billions of service providers– billions of consumers– billions of resellers

To make this future real we need an infrastructure enabling electronic commerce of digital content

MPEG-21



ObjectivesMPEG-21

– The past

– MPEG technologies have been used to create infrastructures on which business can flourish

– This happened in the simple transposition of the physical/analogue world to the digital world

The future

– MP3, DviX and peer-to-peer protocols etc. have shown the power of digital content in people’s hands

– These technologies have resulted in mass abuse of other people’s IPR because of the absence of an appropriate infrastructure



Network adaptationMPEG-21

Use different image formats within a single multimedia presentation, without having to transcode one format into another or vice versa.

Makes it possible to show a video fragment on a small display of a mobile phone, and on a professional digital TV; so-called "scalability" of multimedia data is very important in this context.

Allows applications to modify themselves, depending on the resources that are available, e.g., by switching to a lower image resolution when the available network bandwidth is decreasing.

Source: Robert Bleidt, EETimes


Digital Rights ManagementMPEG-21

Is MPEG trying to tame the hackers?

DRM is a necessity, not only for industry support, but for a global market

Explains the basic concepts of a machine-interpretable language for expressing the rights of users

Does not provide specifications for securty in trused systemsk, pose specific applications, or describe the details of the accounting systems required.

It does describe the language’s syntax and semantics


ConclusionMPEG-21

What– A Digital Item is a structured digital object with a standard

representation, identification and metadata within the MPEG-21 framework.

Who– A User is any entity that interacts in the MPEG-21 environment

or makes use of a Digital Item.

User A User BTransaction/Use/Relationship

Digital ItemAuthorization/Value Exchange



The Multimedia FrameworkMPEG-21

Identification and

Description

Content Manageme

nt and Usage

Terminals & Networks

IPMP

Content Represent-

ation

Digital Item Declaration

Event R

eportin

g

Metrics &

Inte

rface

sEvent

Report

ing

M

etr

ics

& Inte

rface

s

User A User BTransaction/Use/Relationship

Digital ItemAuthorization/Value Exchange



ConclusionMPEG-21

Open standard-based framework for multimedia delivery and consumption?

Enables the use of multimedia resources across a wide range of networks and devices

Consists of Users that interact with Digital Items

No technical distinction between providers and consumers

Glues all kinds of multimedia content together

Digital Rights Management infrastructure


DemoMPEG-21

http://multimedialab.elis.ugent.be/demo.asp


CABLE

Deliver

SATELLITE

WIRELESS

DSL

CD/DVD

End UserPlayback

PC

SET-TOP BOX

DVD PLAYER

GAME CONSOLE

INTERNET DEVICE

PDA

CELLPHONE

ConclusionMPEG

Process Author

Vis

ual

Co

nte

nt

Au

dio

Co

nte

nt

Sce

ne

Info

rmat

ion

2D

SOUNDTRACK

SOUND EFFECTS

TEXT

ANIMATION

VIDEO

3D

INTERACTION

MUSIC

LAYOUT

DIALOG

MPEG-4

Rights Holder Publish

SECURE

TARGETED INTEGRATED

CONTENT MANAGEMENT

DIGITAL RIGHTS MANAGEMENT

SUBSCRIBER MANAGEMENT

AD SERVING

ECOMMERCE

ACCOUNTING & BILLING

MPEG-21

SEMANTICS

CONTENTS

METADATA

PULL

PUSH

UNIVERSALMM ACCESS

MPEG-7

Documents

03.03.2004 Halldór Matthías SigurðssonPage 1 The MPEG Standards With special focus on MPEG-7/21 Halldór Matthías Sigurðsson Ph.D. student at CTI – [email protected]@cti.dtu.dk