E.G.M. Petrakis Introduction 1

Technical University of CreteDepartment of Electronic and Computer

Engineering

Multimedia Data Management

Euripides G.M. Petrakishttp://www.intelligence.tuc.gr/~petrakis

http://courses.ced.tuc.gr

Chania 2010

E.G.M. Petrakis Introduction 2

Definition

Multimedia: composite entities combining text, audio, images, video (bit-stream objects), graphics

Multimedia Information Systems: database systems that support all multimedia data types and handle very large volumes of information

E.G.M. Petrakis Introduction 3

TEXT

The most common type of informationThe least space intensive data typeThe form in which text is stored varies

(plain ascii, word files, spreadsheets, annotations, database fields etc.)

Text fonts are becoming complex allowing special effects (color, shade, fill etc.)

E.G.M. Petrakis Introduction 4

AUDIO Space intensive (one minute can take up

Mbytes), presented as analog, digital or MIDI

Analog waveform: electrical signal, amplitude specifies the loudness of the sound in microphones, tapes, records, amplifiers,

speakersDigital waveform audio: digital,

less sensitivity to noise and distortion involves larger processing and storage

capacitiesDigital Audio Tape (DAT), Compact discs (CD)WAV (Microsoft’s wave file format)

E.G.M. Petrakis Introduction 5

MIDI (Musical Instrument Digital Interface)

Commands that describe how the music should be played are stored (instead of sound)

A music synthesizer generates sound Provides high data compression, Widely accepted

Furht et.al.96

E.G.M. Petrakis Introduction 6

IMAGES

Digital images: sequences of pixelsPixels: numbers interpreted to display

intensity, color, contrast etc Binary (0-1 values), gray-scale (8

bits/pixel), colour (3x8 values for RGB)Space overhead depends on image

type, resolution, compression schemeImage formats: tiff, bmb, jpeg etc.

E.G.M. Petrakis Introduction 7

Image Concepts and Structures

Binary images: 1 bit/pixel black & white photos, facsimile images

Computer Graphics: 4 bits/pixelGrayscale images: 8 bits/sampleColor images: 16, 24 bits/pixel

E.G.M. Petrakis Introduction 8

RGB Representation

A color is produced by addingred, green and blue

The straight line R=G=B specifies gray values ranging from black to white

E.G.M. Petrakis Introduction 9

YUV Representation

YUV describes the luminance and chrominance components of an image1 luminance: gray scale version of an

imageY = 0.299R + 0.587G + 0.114B

2 chrominance components:U = 0.564(B - Y)V = 0.713(R - Y)

E.G.M. Petrakis Introduction 10

Conversions

Conversion between RGB and YUV requires multiplication operationsan approximation: Y = R/4 + G/2 +B/2, U=(B-Y)/2, V=(R-

Y)/2R = Y + 2V, G = Y – (U + V), B = Y + 2U

YCbCr is another color format for compressionCb = U/2 + 0.5, Cr = V/1.6 + 0.5

E.G.M. Petrakis Introduction 11

VIDEO

The most space intensive data typeA sequence of framesRealistic video playback, transmission,

compression/decompression require transfer rates about 30frames/sec

Microsoft’s AVI and Apple’s Quicktime file formats integrate video and audio in the same presentation

E.G.M. Petrakis Introduction 12

Sequence of FramesKhoshafian Baker 96

E.G.M. Petrakis Introduction 13

Audio-video Modes of Operation

Can be either stored or used / transmitted live in real-time

Can be used interactively or non-interactively

Furht et.al.96

E.G.M. Petrakis Introduction 14

GRAPHICS

Objects described through their basic elements (e.g., 2D, 3D shapes)these elements can have different sizes,

position, orientation, surface, fill etc. compact representationsgenerated and can be manipulated by

design tools (e.g., CAD tools)Their descriptions are stored in files

E.G.M. Petrakis Introduction 15

Khoshafian Baker 96

2D and 3D graphics objects

E.G.M. Petrakis Introduction 16

MULTIMEDIA objects

Text, audio, images, video, graphics are elements of complex multimedia objects

Various tools or applications integrate, process and combine multimedia

Applications: multimedia authoring applications that output documents and databases and end-user applications (e.g., video on demand)

Tools: for viewing, updating, querying (presentation viewers, browsers etc.)

E.G.M. Petrakis Introduction 17

Multimedia Databases (MDB)

Means stored information or database management systems (dbms)

Multimedia dbms (mdbms) integrate conventional database capabilities together with different technologies such as Hierarchical storage management (HSM) and Information retrieval (IR)

E.G.M. Petrakis Introduction 18

Multimedia Technologies

Technologies integrated within a mdbms HSM support IR support (exact and approximate)Spatial data types and queriesInteractive querying, relevance feedback,

refiningAutomatic feature extractionAutomatic content retrieval and indexing Query optimization

E.G.M. Petrakis Introduction 19

Database capabilities

Persistence: object persist through invocations Transactions: content is inserted, deleted, updated Concurrency control: transactions run concurrently Recovery: failed transactions are not propagated to the

db Querying: content can be retrieved Versioning: access previous states of objects Integrity: transactions guarantee consistency of content Security: constraints for accessing/updating objects Performance: optimal data structures and programs

E.G.M. Petrakis Introduction 20

Hierarchical Storage Management (HSM)

Support storage of multimedia objects On-line: on RAM, magnetic diskNear-on-line: on optical storageOff-line: on tapes, shelves

Each level has differentPerformance: decreases from top to bottomCapacity: increases from top to bottomCost: decreases from top to bottom

E.G.M. Petrakis Introduction 21

Information Retrieval (IR) Capabilities

Retrieval is the most common operationDeletions and updates are less commonExact match: search based on exact informationInexact: search based on inexact information

e.g., partial, neighborhood search, can be fuzzy or probabilistic

The results are ranked by order of relevance to the query

Query refinementIterate over query results Adjust weights of query terms or featuresAnd finally resubmit queries

E.G.M. Petrakis Introduction 22

Khoshafian Baker 96

MDBMS architecture

E.G.M. Petrakis Introduction 23

MDBMS Implementation

Relies on 3rd party vendors for each componentRelational dbms for typical records separate optical storage module for

text/audio/graphics/images/videoText retrieval system (e.g. Lucene)Audio/image/video retrieval systemFeature extraction systemMultimedia object interface system

E.G.M. Petrakis Introduction 24

Object-Oriented Multimedia Databases

Better design, better suited for multimedia applications Uniform handling of data and operationsData types are objects with internal structures

and operations that capture the behavior of objects (e.g., audio playback, video browsing)

OO dbms does not satisfy all MM requirementsProvides primitives for object handling Multimedia components need to be

implemented or integrated

E.G.M. Petrakis Introduction 25

Khoshafian Baker 96

Object orientation in MMDBMS’s

E.G.M. Petrakis Introduction 26

Client-Server Architectures

Multimedia databases operate in client-server architectures

A number of interconnected server nodes provide I/O capabilities for multimedia objectsBasic DBMS moduleIR moduleHSM module More services may include scanner, fax

services etc.

E.G.M. Petrakis Introduction 27

Khoshafian Baker 96

Client-Server in mdbms

E.G.M. Petrakis Introduction 28

Khoshafian Baker 96

Various modules and servers in an MMDBMs architecture

E.G.M. Petrakis Introduction 29

Multimedia Applications

Multimedia Systems suggest a variety of applicationsMultimedia conferencing Multimedia on demand (interactive TV,

news on demand)See next page for more …

E.G.M. Petrakis Introduction 30

Furht et.al.96

Multimedia Applications

E.G.M. Petrakis Introduction 31

Multimedia Conferencing

Multimedia conferencing enable a number of participants to exchange multimedia informationSkype, PoWWoWNow, WebexEach participant has a workstation linked

to other workstations over high-speed networks

Each participant can send or receive mm data and perform certain collaborative activities

E.G.M. Petrakis Introduction 32

Furht et.al.96

The biggest performance challenge occurs when the participants transmit voice and videoThese are mixed together to form a composite

stream consisting of video and voice streams

A video conference system

E.G.M. Petrakis Introduction 33

Furht et.al.96

Software Architecture

E.G.M. Petrakis Introduction 34

System Functions

Multipoint connection set-up: the system negotiates for network resources

Dynamic session control: add/delete participants

Conference directory service: supports registration, announcement, query etc.

Automatic scheduling and recording: planning of network resources.

Conference teardown: release reserved resources

E.G.M. Petrakis Introduction 35

Architectures

Fully distributed: direct connections between the participants Processing and mixing of media at every locationShortest delayThe connections increase rapidly

Centralized (star) network: a central is connected to every participantProcessing and mixing at central nodeThe central node waits until all media is received

before mixing and broadcasting

E.G.M. Petrakis Introduction 36

Architectures (cont.)

Double star network: a central node from one star network is connected to another central node of another star network

Hierarchical network: intermediate nodes, root and leaves (participants) connected hierarchicallyintermediate nodes perform mixing and

processingthe completely mixed data is sent to root who

broadcasts directly to the leavesreduces network traffic significantly

E.G.M. Petrakis Introduction 37

Furht et.al.96

Multimedia conferencing network architectures

E.G.M. Petrakis Introduction 38

Video on Demand

Fast networks coupled with powerful computers and compression techniques will be capable of delivering stream data in real-time

On-demand multimedia servicesinteractive entertainment video news distributionvideo rental services digital multimedia libraries

E.G.M. Petrakis Introduction 39

Interactive Television (ITV)

An ITV system must be capable of providingbasic TVsubscription TVpay per viewvideo on demandshoppingeducationelectronic newspaperfinancial transactionssingle-user and multi-user games

E.G.M. Petrakis Introduction 40

ITV network Architectures

Distributed architectures, componentsinformation (content) servers:

entertainment companies, TV stations connected to stb’s offering different types of services

network: wide area network (ATM) connects the head-ends which are connected with subscribers

set top boxes (stb) : terminal devices that allow users interact with a network providing personalized, on-demand, interactive services

E.G.M. Petrakis Introduction 41

Furht et.al.96

A General ITV Architecture

E.G.M. Petrakis Introduction 42

Furht et.al.96

Hierarchical ITV Architecture

E.G.M. Petrakis Introduction 43

News on-Demand

Delivers news on demand to various corporate and financial services

The Nynex and Dow Jones systems:The Dow Jones production center

produces news which are send through a wideband network to the Nynex media service center

Customer sites can request specific news from the Media Service Center which are delivered over the network and stored at customer sites

E.G.M. Petrakis Introduction 44

Furht et.al.96

Architecture of Nynex and Dow Jones system