What is Multimedia? Multimedia APIs

T-111.5350 Multimedia ProgrammingPablo Cesar

What is Multimedia?Multimedia APIs

Pablo [email protected]

http://www.tml.hut.fi/~pcesar



Outline• Definitions of Multimedia• Multimedia Elements:

– Multimedia Objects: Audio, video, graphics, text– Visual Style– Layout of those objects

• Temporal dimension (animation, synchronization)• Graphical layout

– Application Logic: State of the application (e.g., Games)– User Interaction: Passive to authoring (Visualization, Navigation, WIMP

concepts)• Taxonomy of Authoring Content Formats

– Expressive Power, Easiness of Use, Safety of Distribution, Interoperability

• Compiled Languages (C, C++)• Virtual Machine Languages (Java)• XML Based Languages (SMIL, XForms)


MultimediaHeller

• Media Type: text, sound, graphics, and motion

• Media Expression: describes the level of abstraction using the media– Elaboration: no edited

information– Representation: edited or

stylized– Abstraction: For example

icons (most abstract)• Context: includes properties

such as – Interactivity– Aesthetics– Audience

Multimedia

Motion

Graphics

Sound

Text

Ela

bora

tion

Rep

rese

ntat

ion

Abs

trac

tion

Med

ia T

ype

Media Expression(Increasing abstraction)

Con

text

AudienceDisciplineInteractivity

QualityUsefulnessAesthetics


MultimediaPurchase

• Modality: – Aural: audio – Visual: graphics

• Nature of the sign: – Concrete iconic (photorealistic

image)– Abstract iconic (map)– Symbolic (written word)

• Syntax / Arrangement: – Individual– Augmented– Temporal– Linear– …

Augmented

Individual

Network

Schematic

Linear

Temporal

Syn

tax

Con

cret

eic

onic

Sign

icon

icA

bstr

act

Sym

bolic

Aur

alV

isua

l

Modality


MultimediaBulterman and Hardman

• Media Assets: How to reference the multimedia objects of the presentation

• Synchronization Composition:– Hard timing relationships, Relative structural ordering– Constraints

• Spatial Layout– Implicit (video), explicit, and dynamic

• Asynchronous Events– Content-based (timing) and user interaction (navigation)

• Adjunct/Replacement Content– Alternative content / adaptation content

• Performance Analysis– performance optimization for various delivery scenarios


MultimediaVuorimaa

• Multiple media– Types: text, graphics, animation, image, audio, and video– Source: Natural (e.g., video) vs. artificial (e.g., 3D graphics)

• Interaction– Stand-alone vs. Networked applications– Level of interaction (user interface, application, and service)– Amount of interaction

• E-mail, video-on-demand, video conference, video• Game, and virtual reality

• Timing– External synchronization of different media (e.g., video and slides)– Internal timing within single medium (e.g. video)– Usually applications have time dimension (e.g., story line)


MultimediaSummary (1/2)

• Multimedia Objects– Audio, video, graphics, text

• Visual Style• Layout of those objects

– Temporal dimension (animation, synchronization)– Spatial layout

• User Interaction– Passive to authoring (Visualization, Navigation, WIMP concepts)

• Application Logic– State of the application (e.g., Games)


MultimediaSummary (2/2)

”Computer mediated applications that integrate and present different media objects, which are arrange spatially and temporally. Moreover, user interaction can control the behavior of the application.”

MultimediaObjects

VisualStyle

TemporalDimension

SpatialLayout

UserInteraction

Application Logic


Multimedia ElementsObjects and Visual Style

• Discrete Media– Icons: semantic images (e.g., stop symbol). Require

the user to have previous knowledge– Graphics: computer generated. Can be 2D or 3D

graphics depending on the goal– Images: natural source (e.g., photograph)

– Text: Size, ��, Color

• Continuous Media– Motion Pictures (audio + video)


Multimedia ElementsSpatial Layout (Pihkala 2003, Boll 2001)

• Absolute – coordinates relative to origin

• Directional relations: – define order in space

• Topological relations:– disjoint, touch, equals, inside

of, covered by, contains, cover, and overlap

• Text Flow: – one-dimensional flow showed

in two-dimensional area

Contains

East

North


Multimedia ElementsTemporal Dimension (Pihkala 2003, Boll 2001)

• Temporal Models:– Definite: for example 6 seconds– Indefinite: for example, when user clicks– Parallel and Sequential relations (e.g., start these two videos at

this moment or start this video after this other one)• Animation:

– Mixture of temporal dimension and spatial layout (i.e., position of an object changes in time)

Time


Multimedia ElementsUser Interaction

• Different Levels of Interaction (Aleem):– Passive: only visualization– Reactive: limited interaction (e.g., Scroll Pane functionality).– Proactive: choose a path or make selections (e.g., Button).– Reciprocal: corresponds to user authoring of information

• Interaction Models (Boll):– Navigational: choice to decide where to go next– Design: user can modify the visual style of the presentation

(e.g., colors)– Movie: user can control the global time (e.g., VCR capabilities)


Multimedia ElementsApplication Logic

• Traditionally multimedia presentations did not have that much logic:– Virtual visit to a museum, DVD menus...

• Real – time interactive systems:– Virtual Reality worlds, games

• Application Logic needs of a programming language (if, case, goto...)– Compiled Languages: C, C++– Virtual Machine: Java– World Wide Web, MPEG-4, Director: scripting


Taxonomy of Authoring Content FormatsRequirements

• Supported Media Types: audio, video, text, graphics, and animation• Arrangement of the signs: spatial and temporal• Interaction: passive, reactive, proactive, and reciprocal• Difficulty to use (threshold)• Expressional power (i.e., ceiling)• Safety of Distribution• Interoperability

+

++

+++

Ceiling

+++

++

+

Safety of Distribution

++++Compiled Languages

++++VM Languages

++++XML based Languages

InteroperabilityThreshold


Compiled Languages

Pro• Efficient approach• Expressive power (closer

to computer hardware)

Con• Interoperability (each

service has to be compiled to target device)

• Less safer to distribute (it can include harmful code)

Normally, used for system software (e.g., operating system) and resource demanding services: C, C++


Compiled LanguagesSystem Software

• ”User Interface Software Tools” (1995, Myers) defines a layered model

• Applications implemented using higher-level tools

• Toolkit: a library of widgets used by applications

• Windowing System: helps user to monitor and control different contexts (input and output functionality)


Compiled LanguagesWindowing System (1/3)

ProtocolX Network

Xlib

One perSession

FluxboxEnlight.

ProtocolX Network

Sawfish

Xlib

XServer

GLib

LayerBase

Window ManagerKwin

GIMP Toolkit (GTK)

GDK

Toolkit

Gnome Libraries

Window Manager

KDE Libraries

Qt ToolkitToolkit

KDE DesktopEnvironment

Gnome Desktop Environment


Compiled LanguagesWindowing System (2/3)

• X-Window– X.org: fonts management, graphics card support, composite functionality– Desktop environments: KDE, GNOME (Toolkits + Applications)– Window Managers: FluxBox, Sawfish…

• DirectFB– XDirectFB: X-Window Support on DirectFB– DirectFBGL

• Microsoft Windows– DirectX

• Mac– Video: QucikTime– 3D: OpenGL– 2D: QuickDraw


Screenshots – X.org


Screenshots – DirectFB


Compiled LanguagesWindowing System: DirectFB

DirectFB Application

DirectFB Chipset Driver

User Space

Kernel Space

Hardware

Framebufffer Driver

Framebufffer Timing and Mode Accelerator


Compiled LanguagesWindowing System: Direct-X

Win32 Application

Direct3D API

HALDevice

Device Driver Interface (DDI)

Graphics Hardware

Win32 Application

GDI


Compiled LanguagesToolkits

• Toolkits provide– Interaction: to handle user input– Canvas Operations: both rendering region, canvas, and

graphics primitives– Set of Widgets: predefined user interface elements (e.g.,

Button)– Graphical Layout: to control the location of the widgets

• Examples: QT, GTK• Virtual Toolkit

– Device independent Toolkit– Mapped to actual Toolkit in the device– Example: AWT


Compiled LanguagesMedia Providers

• Audio/Video: Xine, MPlayer• Television: linuxtv• Games: SDL• Other Languages: For example libflash• 3D graphics:

– OpenGL– OpenGL ES

• Home media platforms: LIMMBO, MythTV


VM Languages

Pro• Platform independence• Safer to distribute

(restricts potential security attacks)

• Expressive power (programming language)

• Well documented APIs

Con• Heavy applications

(because of VM concept)• Difficult of use

(programming language)• Less powerful than

compiled languages

A Virtual Machine is an abstraction of the computing environment. JVM + APIs


VM LanguagesJava Overview

• Nowadays, trying to target all kind of computer devices• Editions:

– Java 2 Enterprise Edition (J2EE): for servers and enterprise computers– Java 2 Standard Edition (J2SE): for servers and personal computers– Java 2 Micro Edition (J2ME): for embedded devices, PDAs, mobile

phones, and Digital television set-top boxes– Java Card: for smart cards

KVM

MIDP

CLDC

Profile

Configuration

• Profile– Requirements for a specific vertical

market of devices (set of APIs)• Configuration

– Minimum platform for a horizontal grouping of devices (VM + core APIs)


OptionalPackages

Java 2Enterprise

Edition(J2EE)

Java Virtual Machine KVM

Java 2 Micro Edition(J2ME)

Java 2StandardEdition(J2SE)

OptionalPackages

OptionalPackages

PersonalProfile

FoundationProfile

CDC

OptionalPackages

MIDP

CLDC Java Card

Card VM

Servers Personal Computers

TV STBsHigh End PDAs

Mobile PhonesLow end PDAs

Smart Cards


VM LanguagesMultimedia

• User interface development (AWT/Swing)– Layout: Grid, North-South-East-West, Flow– Set of Widgets: Button, TextArea– User Interaction: awt.ui.* (Mouse, Keyboard…)

• Video/Audio and Synchronization (JMF)– Manager, Player, Data Source, and Controller

• 3D Graphics– Java3D– Java wrappers for OpenGL

• Different Devices– Television: MHP/OCAP/ACAP/ARIB -> GEM– Handheld: MIDP


VM LanguagesUser Interface Development


VM LanguagesJMF (1/2)

Decodes and plays the media data

Retrieves the actual media data

Implements the state machine


VM LanguagesJMF (2/2)

• Unrealised: when it does not have all the information to acquire the needed resources

• Realised: when it has all the information to acquire the needed resources

• Prefetched: when it has all the needed resources, and has already prefetched enough media data to start playing immediately

• Started: when it is actually playing the media


VM Languages3D Graphics

• Java3D– Completely new API for stand-alone 3D graphics applications– Can use any underlying architecture (Direct-X, OpenGL...)– It might not be the most efficient approach– Developers have to learn a new API

• Java wrappers of OpenGL– Functionality from OpenGL– Developers knows the API already– Only wrappers: uses Java Native Interface (JNI)– Much intercommunication between layers (Java -> C)– API is not standardised yet (Java Specification Requests)

• JSR 231: OpenGL• JSR 239: OpenGL ES


VM LanguagesJ2ME

• Defines two Configurations:– CDC: High end consumer devices

• RAM Java Memory: around 2MB• ROM Java Memory: around 2.5MB

– CLDC: Low end consumer devices• Processor:16 bit/16 MHz or higher• Java total memory: 160-512 KB

• CDC (Connected Device)– Personal Profile

• Adds support for lightweight AWT– Foundation Profile

• Basic application APIs (no GUI)• CLDC (Connected Limited Device)

– Mobile Information Device Profile (MDIP)• Application APIs + GUI APIs KVM

OptionalPackages

PersonalProfile

FoundationProfile

CDC

OptionalPackages

MIDP

CLDC

TV STBsHigh End PDAs

Mobile PhonesLow end PDAs

JVM


VM LanguagesHandheld


VM LanguagesTelevision

DataInteroperableApplication

DataInteroperableApplication

InteroperableApplication

ApplicationManager

TransportProtocol(s)

Sun JavaAPIs

HAViAPIs APIs

DAVIC DVB SpecificAPIs

Java Virtual Machine

Operating System, drivers, firmware

System Software


VM LanguagesSummary

Temporal

Spatial

Arrangement of the signs

AWT

Java Threads

AWT EventsInteraction

GEMTelevision

MIDPHandheld

Different Devices

Supported Media Types

AWTText, Graphics

JMFVideo, Audio


XML Based Languages

Pro• Easiness of use (you can

even use a text editor)• Interoperability (only

needs a compatible browser)

• Safest to distribute

Con• Expressive power (quite

limited, not a programming language!)

• Use of scripting for application logic (or not?)

• Needs of a service under it (browser)

Declarative programming language (only what has to be done, not how). Major contributor is W3C


XML Based LanguagesOverview

XML BasedLanguage

Document

XML BasedLanguage

Document

XML BasedLanguage

Document

• HTML & XHTML• Multimedia

– SMIL, Timesheets

• User Interface– XForms, XIML

• Vector Graphics– SVG

• Voice– VoiceXML


XML Based LanguagesHTML & XHTML

HTML• HTML 4.01: (24 Dec. 1999) W3C

Recommendation• Lingua franca for publishing

hypertext on the WWW.• Non-proprietary• Can be created by a wide range of

tools:– Text editors– Authoring tools

• All kind of features (mixed together):

– UI components– Fonts– Lists

XHTML• XHTML 1.0 (26 Jan. 2000, revised 1

Aug. 2002) W3C Recommendation• XHTML 2.0: (22 July 2004) W3C

Working Draft• Reformulation of HTML 4 in XML• Intention

– To only describe the structure of the document (CSS formatting)

• XHTML 1.0– Well formed documents– Proper nesting– ...

• XHTML 2.0 – No backwards compatible– Reduces scripting– Includes XForms and XML Events


XML Based LanguagesXHTML Modularization and XHTML 1.1

ModulesPrivate

Applet

Presentation

Edit

Bi−directional Text

Other XHTML Modules

Forms

Tables

Basic Forms

Basic Tables

Server−Side Image Map

Client−side Image Map

Image

Object

Name Identification

IFrame

Target

Frames

Intrinsic Events

Legacy

Metainformation

Scripting

Stylesheet

Style Attribute

Link

Base

Structure

Modules

Text

Hypertext

List

Core

ModulesOther W3C

Applet

Presentation

Edit

Bi−directional Text

Other XHTML Modules

Forms

Tables

Basic Forms

Basic Tables

Server−Side Image Map

Client−side Image Map

Image

Object

Name Identification

IFrame

Target

Frames

Intrinsic Events

Legacy

Metainformation

Scripting

Stylesheet

Style Attribute

Link

Base

ModulesOther W3C

Ruby Annotation

ModulesPrivate

Structure

Modules

Text

Hypertext

List

Core


XML Based LanguagesMultimedia

SMIL • SMIL 2.0 (07 Aug. 2001) W3C

Recommendation• Easy to write, like HTML• Doesn’t define media formats,

only integrates them• <img>, <video>, <audio>

• Defines the spatial and temporal dimensions of the document

• Limited Interaction• <a>, <area>: for links

• Absolute Layout

Timesheets• Similar to CSS, but for

temporal dimension• Document composed of:

– Content: XHTML– Formatting: CSS– Timing: Timesheets

• Similar syntax than SMIL<time><par>

<item select=”#isect1” dur=10s><item select=”#isect2” dur=10s>

</par></time>


XML Based LanguagesUser Interface

XForms• XForms 1.0 (14 Oct. 2003) W3C

Recommendation• Next generation of web forms• Not intended as a self-standing

document type• Uses host language for the

document layout (e.g., XHTML, SMIL)

• Advances user interface features:• text input, select one, select

many, submit• User input can be validated in the

client-side• Calculations are done, as well, in

the client side

XUL• XML User Interface Language• Only supported in Mozilla and

Netscape 6 (or later) browsers• Only for window-based graphical

UI (mobile phones?)• Abstraction only at the platform

level (not at the UI level, voice?)• It separates:

• Client application definition and programmatic logic

• Presentation (using CSS)• Language-specific text labels

• Look & feel changed as wished• Interaction achieved by scripting• Interface elements: windows,

menubar, scrollbar


XML Based LanguagesVector Graphics, Voice

SVG• SVG 1.0 (04 Sept. 2001) W3C

Recommendation• SVG 1.1 (14 Jan. 2003) W3C

Recommendation• Describes vector-based graphics

for the Web (no pixel based)• Shapes (e.g., lines & curves)• Images• Text

• Drawings can be• Interactive (e.g., Mouse clicked )• Animated (e.g., Change location)

VoiceXML• VoiceXML 2.0 (16 Mar. 2004)

W3C Recommendation• Creation of audio dialogs (user

interfaces)• Input

• Speech Recognition and/or touch tone (keypad)

• Output• Pre-recorded audio and Text-to-

Speech Synthesis (TTS)• Describes:

– Spoken prompts: synthetic speech

– Recognition of spoken words and touch tone key presses (fields)

– Control of dialog flow (menu, form that can be submitted to server)

– Telephony control (call transfer)


XML Based LanguagesTerminals & Browsers (Desktop)

http://www.xsmiles.org

http://www.mozilla.org

http://home.netscape.com/

http://www.opera.com/

http://www.microsoft.com/windows/ie/


XML Based LanguagesTerminals & Browsers (Embedded)

Mobile Phone

Web TV

EspialBrowser


XML Based LanguagesSummary

Links

Yes

Yes

Yes

Yes

SMIL

FullLinksLinksInteraction

--NoNoTemporal

--AbsoluteFlow & Absolute

Spatial

Arrangement of the signs

--YesYesText, Images

--NoNoVideo

--YesNoAudio

Media Types

XFormsSVGXHTML


Multimedia Languages

Please,

select atopic byusing yourremotecontrol

OKOrderedGroup

(Scene)

Transform2D(Graphics)

Please,

select atopic byusing yourremotecontrol

Text and Rectangle(Background)

Button Button Button

(Image of Topics)ImageTexture

(Panel)TouchSensor

MovieTexture(Video)

OK


Multimedia LanguagesMPEG-4 Overview (1/2)

• Evolution:– MPEG traditionally targeted to audio/video codecs (MPEG-1, MPEG-2)– Complex toolkit capable of providing solutions for multimedia

applications• Scene:

– Composition of different multimedia objects (2D, 3D, video) including their spatial and temporal relationships

• Entry points:– BInary Format for Scene (BIFS):

• Hierarchical structure (scene graph)• Properties: color, size, position, and timing• Behavior: BIFS commands (conditional) and Animations

– MPEG-Java: set of Java APIs– eXtensible MPEG-4 Texttual (XMT): XML language that describes

scenes


Multimedia LanguagesMPEG-4 Overview (2/2)

• Some of the Scene Nodes:– Top: root of the graph (e.g., Layer3D and Layer2D)– Grouping: containers of multimedia objects– Sensor: nodes capable of detecting events (e.g., Time and Touch)– Shape: Graphical Primitives that include two fields: Geometry (e.g.,

rectangle and circle) and Appearance (e.g., texture and material)– Face: integration of synthetic 3D human-like objects

• Interaction: – Sensors detect events and Route distribute them– Predefined behaviors: resize, relocate – Complex behavior: script or Java

• Widgets: – Can be implemented (e.g., sensor + Shape)

• Layout: – Local coordinates of the objects (more complex automatic layout is not

permitted)


Multimedia LanguagesMHEG Overview

• Content Classes– Multimedia objects (e.g., video or audio clips)– Contained in MHEG object (small data) or reference (e.g., filename, web server

address)– Author can reference to smaller sections (e.g., track 5)

• Behavior Classes: – Synchronization of events and user interaction– User Interaction

• The action class:– Event triggers – Sequential and parallel

• The link class: establishes relationships between events and objects i.e. what actions to take on what objects in response to a particular event.

• Selection and modification classes:– E.g., Push button, checkbox, radio button, slider, text entry field and text lists – Selections, input information and trigger events.


Multimedia LanguagesMHEG Example

(scene:InfoScene1<other scene attributes here>group-items:

(bitmap: BgndInfocontent-hook: #bitmapHookoriginal-box-size: (320 240)original-position: (0 0)content-data: referenced-content: "InfoBngd"

)(text:

content-hook: #textHookoriginal-box-size: (280 20)original-position: (40 50)content-data: included-content: "1. Lubricate..."

) links:

(link: Link1event-source: InfoScene1event-type: #UserInputevent-data: #Leftlink-effect: action: transition-to: InfoScene2

))


Conclusion

• Multimedia– Multimedia objects, visual style– Spatial layout, temporal dimension– Application logic, user interaction

• Four alternatives (from taxonomy)– Compiled languages (C): most efficient, less safer to distribute– VM languages (Java): programming language, interoperable– XML based languages: most interoperable, less expressive power– Multimedia Languages: intended for multimedia

• Number of APIs– C: OpenGL/Direct-X, DirectFB, SDL, linuxTV– Java: AWT, Swing, JMF, Java3D, Java OpenGL– XML: XHTML, SMIL, Timesheets, XForms, SVG, VoiceXML


References1. T. A. Aleem. A Taxonomy of Multimedia Interactivity. Doctoral

dissertation, The Union Institute, USA, September 1998.2. S. Boll. ZYX, Towards Flexible Multimedia Document Models for Reuse

and Adaptation. Doctoral dissertation, University of Vienna, Austria, August 2001.

3. D. C. A. Bulterman and L. Hardman, Structured Multimedia Authoring, ACM Transactions on Multimedia Computing, Communications, and Applications, 1(1): 89-109, February 2005.

4. P. Cesar, Graphics Software Architecture for High End Interactive Television Terminals, Helsinki University of Technology, Finland, December 2005 (in print).

5. R. S. Heller, C. D. Martin, N. Haneef, and S. Gievska-Krliu. Using a theoretical multimedia taxonomy framework. ACM Journal of Educational Resources in Computing, 1(1): article number 6, 2001.

6. K. Pihkala. Extensions to the SMIL Language. Doctoral dissertation, Helsinki University of Technology, Finland, November 2003.

7. H. Purchase. Defining multimedia. IEEE Multimedia, 5(1):8-15, 1998.8. M. Williams. A Taxonomy of Media Usage in Multimedia. Doctoral

dissertation, Nova Southeastern University, USA, May 2003.9. P. Vuorimaa, Multimedia Technology Course

(http://www.tml.hut.fi/Opinnot/T-111.350)

Documents

What is Multimedia? Multimedia APIs