REQUIREMENTS FORMULTIMEDIA PROCESSING

MULTIMEDIA SYSTEMS

IREK DEFEE

• WHAT ARE THE RQUIREMENTS FOR

HANDLING MULTIMEDIA DATA ?

1. PERCEPTUAL REQUIREMENTS

For the user we would like to provide

maximal perceptual stimulation

THIS REQUIRES:

- OPTIMAL STIMULATION OF EACH

SENSE

- OPTIMAL COORDINATION OF ALL

SENSES - INVOLVEMENT OF POSSIBLE MANY SENSES (ACTIVE) THIS MEANS INVOLVEMENT OF: - ACOUSTICAL (HEARING,VOICE) - VISUAL - HAPTIC (TOUCH, MOTORIC, VESTIBULAR (orientation in space) ) ON FULL SCALE (e.g. SPATIAL PROPERTIES)

THE PROBLEM IS HOW THIS CAN BE

DONE TECHNICALLY:

- ACOUSTICAL STIMULATION

THIS IS NOW RELATIVELY EASY

(SPATIAL (MULTICHANNEL SOUND)

- VISUAL STIMULATION

SPATIAL (3-D) IMAGES ARE

PROBLEM, HIGH RESOLUTION TOO

HAPTIC STIMULATION

THERE ARE MANY INPUT DEVICES

(JOYSTICKS), BUT FULL STIMULATION INCLUDING MOTOR

SENSE, SKIN SENSORS AND

VESTIBULAR SYSTEM IS DIFFICULT

IN FACT WE CAN TALK ABOUT

TWO TYPES OF SYSTEMS FOR

STIMULATION:

- CLOSELY COUPLED TO SENSORY SYSTEM (BY GLOVES, UNIFORMS, EYE DISPLAYS, HEADPHONES, MOVING SIMULATORS) THEY ALLOW ”FULL IMMERSION”, OR CREATION OF VIRTUAL REALITY MIGHT BE TIRING TO USE - LOOSELY COUPLED TO SENSORY

SYSTEM (NORMAL SCREENS, SPEAKERS, JOYSTICKS ETC.)

MIGHT BE MORE COMFORTABLE

IN THIS COURSE WE ARE TALKING

ABOUT ’LOOSELY’ COUPLED

MULTIMEDIA SYSTEMS WITH

STANDARD SENSORY INTERFACES

VIRTUAL REALITY IS A TOPIC OF

OTHER COURSES

WHAT ARE THE TECHNICAL

REQUIREMENTS FOR MULTIMEDIA

SYSTEMS ?

1. HANDLING OF LARGE DATA SETS

E.G. AUDIO

5 CHANNELS X 16-BIT X 44 000 SAMPLES/SEC = 3 520 000 BITS/SEC

~450 KB/sec = 27 MB/min=1,6GB/h

SECOND EXAMPLE, VIDEO

RESOLUTION OF STANDARD TV

PICTURE IS 720X576 POINTS, EACH

POINT IS 24-BIT (8-BIT R, G, B)

THERE ARE 25 PICTURES PER SECOND

SO THE DATA RATE IS

720X576X25X24~ 250 Mb/sec = 1.8GB/min

= 108 GB/h

FOR HIGHER RESOLUTIONS:

A FORMAT FOR HIGH-DEFINITION TV

IS PROPOSED

PICTURE SIZE IS 1920X1080 POINTS

IT IS ABOUT 5X TIMES LARGER THAN

STANDARD TV. DATA RATES ARE

THEN, 1.3 Gb/sec, 9 GB/min, 540 GB/h

DATA RATES FOR AUDIO AND VIDEO ARE NOT PRACTICAL,

COMPRESSION IS REQUIRED

- COORDINATION AND SYNCHRONIZATION

- HUMAN SENSORY SYSTEM IS VERY SENSITIVE TO COORDINATION OF EVENTS IN TIME AND SPACE

MULTIMEDIA DATA, ESPECIALLYAUDIO AND VIDEO DEPEND CRITICALLY ON TIME AND SPACE

• EXAMPLES:

- LIP SYNCHRONIZATION IN TV AND

CINEMA, VOICE AND LIP MOVEMENT

MUST BE SYNCHRONIZED, OTHERWISE IT IS VERY ANNOYING

SYNCHRONIZATION ERROR FOR BIG

PICTURES MUST BE IN THE LIMIT OF +20 ; –40 MILISECONDS

- SPATIAL SOUND

HEARING SYSTEM ESTABLISHES

LOCATION OF SOUND SOURCES

BASED ON DIFFERENCE OF TIME OF

ARRIVAL AT EARS

PRECISION REQUIRED IS ON THE ORDER OF SEVERAL 10’s OF MICROSECONDS.

• THIRD REQUIREMENT: FAST RESPONSE

WHENEVER WE RESPOND (E.G. BY

CLICKING), WE EXPECT IMMEDIATE

RESPONSE.

IMMEDIATE MEANS ”NO NOTICEABLE DELAY” FOR PEOPLE.

IT MEANS LESS THAN 300 ms=0.3 sec

• THUS, MULTIMEDIA DATA REQUIRE

HIGH-DATA RATE

& TIME/SPACE CRITICAL PROCESSING

WITH

FAST RESPONSE TO EXTERNAL

EVENTS

THESE ARE ATTRIBUTES OF

REAL-TIME PROCESSING

• DEMANDS FOR COMPUTATIONAL POWER ARE HIGH

• ONLY RECENTLY THIS BECAME POSSIBLE

Multimedia systems requirements

CONTENT DELIVERY

DISTRIBUTION

CONSUMER

Recall the multimedia system is for service offering:

What users expect from a service they use?- They expect satisfying Quality of Service – QoS QoS includes technical aspects which are most

important for engineers (e.g. good communication link)

QoS in broad sense includes all other aspects which make customer satisfied (e.g. support, price)

What is QoS ?

• QoS is degree to which customer expectations are satisifed

• QoS includes technical parameters for

quality: e.g. reliability of communication

• QoS may also include other parameters

for satisfaction (attraction, support, price, etc.)

• What service owners expect? They expect proper environment that they can run their service Especially that their buisness can run without problems Unfortunately for network buisnesses there are big problems with this What is absolutely necessary is SERVICE PROTECTION

Good buiness over the network needs as the most critical requirement

STRONG SERVICE PROTECTION

CONTENT DELIVERY

DISTRIBUTION

CONSUMER

Every part of the system has to be strongly protected againstthose who want to destroy the service integrity

Overall architecture of multimedia system

There are two ”layers”: Hardware layer: servers, networks and terminalsSoftware layer: content, service, provisioning, networking, middleware, applications

Multimedia system

• Composed of hardware/software components• Goal: Creation and delivery of media content from

production to consumer at proper QoS in an economic way

This means business is sustainable and consumers are satisfied

Examples of multimedia systems

Content delivery system examples:

• Television/(radio?) (old, basic system)

• Web/Internet type

• Mobile (radio networks)

Content distribution type examples:

• Music

• Movies (audio/visual content

• Games

• Virtual worlds (Check Second life on the Web)

• eBooks? eMagazines? eNewspapers?