1

EE 6850, F'02, Chang, Columbia U 1

Prof. Shih-Fu Chang

EE 6850, Fall 2002Department of Electrical Engineering

Columbia Universitysfchang@ee.columbia.edu, http://www.ee.columbia.edu/~sfchang

Course URL: http://www.ee.columbia.edu/~sfchang/course/vis/

Visual Information Systems

EE 6850, F'02, Chang, Columbia U. 2

Definition

� A computer hardware/software system used for � Acquiring and Storing� Indexing and Searching� Manipulating (editing and quality enhancement)� Distributing� Protectinglarge amount of visual information

� Images, video, animations, and associated multimedia� Examples:

� Web Media Search Engines� Home Media Server� Mobile Multimedia Portals

2

VIS Content Chain and Areas of Interest

content management & exchange

Internet

Broadcast users

(filter, navigation)

Internet users

(streaming, search)

Mobile users (browsing, messaging,transcoding)

broadcast

productionaggregation

EE 6850, F'02, Chang, Columbia U. 4

Why it’s important?

� Adoption of Digital Video� New Content Creation/Presentation Tools

� Editor, slide show, production software

� High-Speed Access Networks� New Services and Devices

� Multimedia Messaging� Multimedia Enabled IA� Media portals

� Standards become available� Digital TV, DVD, MPEG-4, MPEG-7, and IPMP

3

EE 6850, F'02, Chang, Columbia U. 5

Example: PDR (Tivo) Any-Time Paradigm

� Time-shift, local storage: 60 hours � Skipping mode� Instant record, live pause, simultaneous

record/playback� Highlight/trail viewing� Search/retrieval, bookmarking, multi-source

comparison/summarization � Personal profile, multi-user profile� Target services, ads, consumer usage data, � Pay per choice e-Commerce

EE 6850, F'02, Chang, Columbia U. 6

Other Example Systems

� Hermitage Museum Web Site (www.hermitagemuseum.org)

� Media Portal: Columbia WebSeek, Google Image Search

� Video Indexing Engine: � News Video: Virage and ABC� Presentation Video: Fuji Xerox� Search sports video by motion: VideoQ� Video mosaic

� Automatic Video Editing

4

EE 6850, F'02, Chang, Columbia U. 7

Hermitage Museum

� Mixed media search tools: keyword and visual

� Virtual tour (using panoramic and 3D view)� Zoom view gallery: multi-resolution� Watermark protection (invisible)� Balanced use of aesthetic, informational, and

technical components� Issues:

� Acquisition, representation, user interfaces, search tools, copyright protection

EE 6850, F'02, Chang, Columbia U. 8

Web Media Portal

� Google Image Search: claim to be the largest search engine for images on the Web� “jpg” 18M images, “gif” 11.6M images

� WebSEEk: (with J. Smith 1997)� Collect and Categorize Images/Videos on the Web

(600K images, 10K videos)

� Issues� Search keywords, display interfaces, more

advanced search tools

5

EE 6850, F'02, Chang, Columbia U. 9

Image/ compression/

attributeextraction

HSV

PPM

featurevectors

icon/

height,

Format

detection

Format

conversionJPG

VideoVideo

colorconversion

GIF

(re-animation)

width, etc.

motionicon

regions,features, attributes

colorspace

WebSEEk: Extracting Visual Features

EE 6850, F'02, Chang, Columbia U. 10

WebSEEk: Extracting Text Features

� Term Extraction — URL, hyperlink text, tags, file names, directory names, surrounding paragraph, etc.

� URL= http://www.mynet.net:80/animals/domestic-beasts/dog37.jpg

� URL= http://camille.gsfc.nasa.gov/rsd/movies2/Shuttle.gif

� URL=http://www.arch.columbia.edu/DDL/projects/amiens/slide6b.gif

� Demo: WebSEEk

6

EE 6850, F'02, Chang, Columbia U. 11

News Video Indexing Systems

Key frames

Linking transcripts to video

Keywordsearch

(Virage & ABC News)

image

image

image

EE 6850, F'02, Chang, Columbia U. 12

VideoQ:Video Search Engine

� automatic motion analysis, region segmentation and tracking

� spatio-temporal search of multiple video objects

� test on a video collection(3000 video clips)(30,000 video objects)

� Demo

7

EE 6850, F'02, Chang, Columbia U. 13

VideoQ System Overview

video scene cut detectionobject

segmentationindexfeatures

keywords

off-line

videoserver

video archiveviewer

user animated sketch

browsing

queryserver

features

on-line

EE 6850, F'02, Chang, Columbia U. 14

Game 1

Game 2

Commercial

Serve 1, base-line play, 2 strokes

Serve 2 Key Frame

Columbia’s Sports Video Navigator and Streaming

image

8

EE 6850, F'02, Chang, Columbia U. 15

Technical Issues

� Digital video compression� Metadata description and browsing� Video Indexing: shot and scene segmentation� Browsing and visualization interfaces� Content-based multimedia search� Fast Indexing and Relevance Feedback� Copyright protection and authentication� Mobile multimedia access� Evaluation: metrics and events

EE 6850, F'02, Chang, Columbia U. 16

Compression Standards

� MPEG-1� 1.5 Mbps, CD-ROM interactive applications

� MPEG-2� 2-10 Mbps, Digital video, e.g., DTV, DVD

� MPEG-4 � Wide range of bitrates� Low-bit-rate video, e.g., wireless � Natural-synthetic hybrid video� High-quality audio

� H-263(+), H-26L� Video conferencing� Internet video

9

Still region SR1: Creation inform a tion Text annotation

Still region SR2: Text annotation Color structure

Still region SR3: Text annotation Matching hint Color structure

Spatial segment decompos i tion: No overlap, gap

Directional spatial segment relation: left

Content S tructure

Agent object AO1: Label Person

Agent object AO2: Label Person

Event EV1: Label Semantic time Semantic place

Concept C1: Label Property Property

Comradeship

Shake hands

Alex Ana

Object-event relation: hasAccompanierOf

Concept-semantic base rel a tion: hasProperty Of

Content Semantics

Object-event relation: hasAgentOf

Metadata: MPEG-7 Description

Segment-semantic base relation: hasMediaPerceptionOf

Segment-semantic base relation: hasMediaSy mbolOf

Photographer: Seungyup Place: Columbia University Time: 19 September 1998

704x480 pixels True color RGB http://www.alex&ana.jpg

Columbia University, All rights reserved

Creation information: Creation Creator Creation corrdinates Creation location Creation date

Media information: Media profile Media format Media instance

Usage unformation: Rights

Content Management

EE 6850, F'02, Chang, Columbia U. 18

Shot boundary detection

� Shot: a continuous take of video content� Task: how to break long video into individual

shot units?� Shot boundary detection based on color,

motion, edge� Demo: sports, sitcom, consumer, news� issues:

� flash, lighting changes, motion, special effects� alignment of captions with shot boundaries

10

EE 6850, F'02, Chang, Columbia U. 19

Challenge: How to find scene boundaries?

transient2:20 4:045:22

Sceneboundary

image image image image image

EE 6850, F'02, Chang, Columbia U. 20

Watermarking and Content Protection

Tx Rx

Watermark Verify thewatermark

WM

� Stenography vs. watermarking� Watermark

� Visible vs. invisible� Spatial vs. frequency domain� Fragile vs. robust vs. semi-fragile

� Applications: copyright, user ID, content ID, authentication, data hiding

image

11

Multimedia Authentication: Producer Identification and Content Integrity Verification

Editor 1 Transcoder 2Transcoder 1

Editor X Transcoder YTx Rx

Tx Rx

verification verification

verificationverificationverification

verification

Editor 1 Transcoder 2Transcoder 1

Editor X Transcoder YTx Rx

Tx RxEditor 1 Transcoder 2Transcoder 1

Editor X Transcoder YTx Rx

Tx Rx

signature

verification

� Is the received content authentic?

� Who signs the specific content?

� Trustable digital government, business, hospitals, etc.

EE 6850, F'02, Chang, Columbia U. 22

Detecting Forgeries

Manipulated Images

AuthenticationOutput

12

EE 6850, F'02, Chang, Columbia U. 23

Copyright Protection & Copy Control

ScrambleTx Rx

Descramble

VCR

� Ownership Identification, Copy Control have to survive multi-stage transcoding, A/D, D/A conversion

� Prevent unauthorized access� Prevent illegal distribution, e.g., digital cinema

EE 6850, F'02, Chang, Columbia U. 24

Issues

� Content feature extraction� Watermark embedding algorithms� System architecture, combining other

security techniques� General rights management systems� Attack methods� Evaluation metrics

13

Mobile Multimedia Access

� Technical issues� Scalable coding, skimming, bandwidth-restricted

adaptive streaming

Broa dca stCont ent

EE 6850, F'02, Chang, Columbia U. 26

Course format

� Weekly lectures� 3 warm-up homeworks

� (video coding, video indexing, image search)

� 1 final project� team projects� mid-term proposal and brief presentation� final report and presentation� may be extended to “Advanced Projects”

� Test data and tools will be provided

14

EE 6850, F'02, Chang, Columbia U. 27

Example Course Projects (1)

� Experiment-type projects� Components for the video testbed

�Scene change�Video content summarization�Multimedia metadata visualization tools

� Speech recognition and text parsing tools� MPEG-7 authoring and parsing tools� Mobile streaming and transcoding� Automatic Content-based video editing

(e.g., muvee.com)

EE 6850, F'02, Chang, Columbia U. 28

Project topics (2)� video indexing

� Echo video visualization combined with 3D graphics� Segmentation of video objects and content based search� Mosaic-based video browsing

� Example: dartfish http://www.dartfish.com/en/solutions/in_showcase.jsp

� Evaluation and comparison� Watermarking: protection and attacks� Similarity metrics

� Survey � content management systems in medical, biological, graphic, music

domains� Music information retrieval� DVD copyright protection standards� MPEG-21