Introduction to Imaging and Multimedialibvolume1.xyz/animation/bsc/2ndyear/semester1/an2104multimedia… · storyboard pruning and concise video summarization. Using voice-recognition

Introduction to Imaging and Multimedia 9/5/13

A. Elgammal, Rutgers 1

Introduction to Imaging and Multimedia

Fall 2013 Ahmed Elgammal

Rutgers University

  Introduction to Imaging and Multimedia   This course has two integrated components:

  Digital Multimedia   Image Processing and Computer Vision

  You can think of it as an introduction to multimedia with a special focus on digital imaging and video



  What is Multimedia ?

What is Multimedia   When different people mention the term multimedia, they often

have quite different, or even opposing, viewpoints.   A PC vendor: a PC that has sound capability, a DVD-ROM

drive, and perhaps the superiority of multimedia-enabled microprocessors that understand additional multimedia instructions.

  A consumer entertainment vendor: interactive cable TV with hundreds of digital channels available, or a cable TV-like service delivered over a high-speed Internet connection.

  A Computer Science (CS) student: applications that use multiple modalities, including text, images, drawings (graphics), animation, video, sound including speech, and interactivity.

  Evolving definition…



Multimedia in Computer Science   Digital Multimedia - Computational Multimedia

Digital Multimedia is the field concerned with computer-controlled integration of text, graphics, images, videos, audio, and any other medium where every type of information can be represented, transmitted and processed digitally.

Digital Multimedia: Historical Perspective   The word multimedia was coined in the beginning of

the 1990s   Mainly after the success of digital audio recording on

CDs   The next anticipated step was to create digital content

involving image, text, video, along with the audio   Multimedia CD-ROMs: ex, Encyclopedia Britannica   Experience only limited to a single user interacting

with a PC   Things have changed dramatically since then.



Example MM Applications   Multimedia involves multiple modalities of text, audio, images,

drawings, animation, and video. Examples of how these modalities are put to use:   Video teleconferencing.   Distributed lectures for higher education.   Tele-medicine.   Co-operative work environments.   Searching in (very) large video and image databases for target visual

objects.   “Augmented” reality: placing real-appearing computer graphics and

video objects into scenes.   Including audio cues for where video-conference participants are

located. Taking into account gaze direction and attention of participants as well

Example MM Applications   Building searchable features into new video, and enabling

very high to very low-bit-rate use of new, scalable multimedia products.

  Making multimedia components editable. allow the user side to decide what components, video, graphics, etc., are actually viewed; allow the client to move components around or delete them. Making components distributed.

  Building “inverse-Hollywood” applications that can recreate the process by which a video was made. This then allows storyboard pruning and concise video summarization.

  Using voice-recognition to build an interactive environment, say a kitchen-wall web browser



Multimedia life cycle   Four main phases in multimedia life cycle

  Multimedia Generation, Authoring, Capturing…   Multimedia Representation, and processing   Multimedia Retrieval: answering user queries.   Multimedia Delivery

Capturing (digitization)

Authoring Editing

Indexing Archival

Client

Retrieval

Sens

ors /

inpu

t

MM Delivery

MM Generation MM Archival MM Retrieval

User ?

User ?

User ?

User ?



Inherent Quality of Multimedia Data   Digital: it is always bits and bytes   Voluminous: the storage and transmission bandwidth

limitations require that the data be compressed   Interactive   Real-time and synchronization: very small and

bounded delay while transmitting information. Intra-media and inter-media synchronization

Multimedia Research Topics and Projects To the computer science researcher, multimedia consists of a wide

variety of topics:   Multimedia processing and coding: multimedia content

analysis, content-based multimedia retrieval, multimedia security, audio/image/video processing, compression, etc.

  Multimedia system support and networking: network protocols, Internet, operating systems, servers and clients, quality of service (QoS), and databases.

  Multimedia tools, end-systems and applications: hypermedia systems, user interfaces, authoring systems. Multi-modal interaction and integration: “ubiquity” web-everywhere devices, multimedia education including Computer Supported Collaborative Learning and design, and applications of virtual environments.



  Multimedia and Computer Science: Multimedia is in the intersection among different areas   Graphics,   HCI,   Visualization,   Computer vision,   Data compression,   Graph theory,   Networking,   Database systems,   Data mining   Architecture and operating systems,   ….

Multimedia Systems   A Multimedia System is a system capable of

processing multimedia data and applications.   A Multimedia System is characterized by the

processing, storage, generation, manipulation and rendition of Multimedia information.





Components of a Multimedia System

  Capture devices   e.g. Video Camera, Microphone, Digitising/Sampling Hardware, etc.

  Storage Devices   e.g. Hard disks, CD-ROMs, DVD, Blu-ray, etc

  Communication Networks   Internet, wireless internet, etc.

  Computer Systems   e.g. Multimedia Desktop machines, Workstations, smart phones, iPads

  Rendering Devices   e.g. CD-quality speakers, HDTV, Hi-Res monitors, Color printers etc.

Challenges for Multimedia Systems   Distributed Networks   Temporal relationship between data

  Render different data at same time continuously.   Sequencing within the media (e.g. playing frames in correct

order/time frame in video)   Synchronization — inter-media scheduling

  E.g. Video and Audio — Lip synchronization is clearly important for humans to watch playback of video and audio and even animation and audio.



Hardware   All contemporary personal computers are quite

capable of displaying MM content   Even Almost all cell phones   For authoring content, a more powerful machine is

needed   Bandwidth is still and always will be an issue

Multimedia System Desirable Features

  Very High Processing Power   Special Hardware/Software needed (e.g. GPUs)   Efficient I/O   Large Storage and Memory   High Speed Network Support



Multimedia History   Before the digital age…



Multimedia: Text

Parchment Papyrus Stone

Paper   Han Dynasty of China (202 BC).Raw material such as tree bark was finely chopped, mixed with

water, spread onto screens, and dried. Well guarded secret   Introduced in Europe in 600AD through the Middle East   First paper mill in Europe was in Spain, in 1120. More mills appeared in Italy in

about the 13th century. They used hemp and linen rags as a source of fiber. Paper is recorded as being manufactured in both Italy and Germany by 1400.



Printing Press

  In 1451, Johannes Gutenberg and Johannes Fust went into partnership and produced a forty-two line Bible and a thirty-two line Latin Grammar

  By 1465, printing presses based on Gutenberg's moveable type could be found in Italy, by 1470 in Paris. London followed in 1480. By 1499, there were presses in Stockholm, Constantinople and Lisbon.

  By 1500, Europe contained in excess of nine million volumes, of thirty thousand titles all of which came off the presses of more than one thousand printers.

Multimedia Data: Text   Text

  Input: keyboard, touch pad   Stored and input character by character.   Storage of text is 1 or 2 bytes per character.   Other forms of data (e.g. Spreadsheet files, XML) may store format as

text (with formatting).   Format: Raw text or formatted text

e.g HTML, Rich Text Format (RTF), Word or a programming language source

  Not temporal — BUT may have natural implied sequence e.g. HTML format sequence, Sequence of Java program statements.

  Size Not significant compared with other multimedia data.



Images

Niepce 1825 (8 hours)

Daguerre 1838 (10 mn)

1888 Eastman's Kodak camera on the market with the slogan "You press the button, we do the rest"

Maxwell 1861 (color)

Now: digital CCD 1969 at AT&T

Camera obscura Mozi 400BC Aristotle 350BC Ibn al-Haitham, 1000AD

Painting on some media: From Prehistoric times till today

Multimedia Data: Images   Images

  Still pictures which (uncompressed) are represented as a bitmap (a grid of pixels).

  Input: Digital camera, scanner or generated by graphics editor programs (e.g. Paint)

  Analog sources will require digitizing.   Stored at 1 bit per pixel (Black and White), 8 Bits per pixel (Grey Scale,

Color Map) or 24 Bits per pixel (True Color)   Size: a 512x512 Grey scale image takes up 1/4 Mb, a 512x512 24 bit

image takes 3/4 Mb with no compression.   Storage increases with image size   Compression is commonly applied



Stored music

15th century

1598

1870 Player piano

Stored audio

1860:phonotaugraph



Modern period: music

Then digital

1877 1904

Read-Write !

Multimedia Data: Audio   Audio

  Audio signals are continuous analog signals.   Input: microphones and then digitized and stored   usually compressed.   CD Quality Audio requires 16-bit sampling at 44.1 KHz   1 Minute of Mono CD quality audio requires 5 Mb.



Multimedia Data: Video   Video

  Input: usually captured by a video camera.   There are a variety of video (analog and digital) formats   Raw video can be regarded as being a series of single images.   There are typically 25, 30 or 50 frames per second.   a 512x512 size monochrome video images take 25*0.25 = 6.25Mb for 1

second to store uncompressed.   Digital video clearly needs to be compressed.

Environment

user

Traditional Multimedia Interface vs. Ubiquitous Interface



Multimedia in a Ubiquitous Computing World

  Multimedia is becoming available all the time around us.   Ubiquitous computing focus on integrating computation into the

environment, rather than having computers as distinct objects.   Ubiquitous computing covers wide range of research topics,

such as distributed computing, mobile computing, and sensor networks.

  Mobile phones now are the most commonly used multimedia devices.

What this course is not about   This course is not about

  Web design   Multimedia tools   Computer graphics and animation   Internet technology



Resources   Multimedia Systems Ch 1   Some Slides by Prof G. Medioni @ USC

Documents

Introduction to Imaging and Multimedialibvolume1.xyz/animation/bsc/2ndyear/semester1/an2104multimedia… · storyboard pruning and concise video summarization. Using voice-recognition