TERM PAPER PRESENTATION

QUERY PROCESSING IN MULTIMEDIA DATABASES

TURKER YILMAZ

1. Conceptual Modeling and Querying in Multimedia Databases.CHITTA BARAL, GRACIELA GONZALEZ, TRAN SON,2. An Approach to a Content-Based Retrieval of Multimedia Data.GUISEPPE AMATO, GIOVANNI MAINETTO, PASQUALE SAVINO, 3. Integrated Spatial and Feature Image QueryJOHN R.SMITH, SHIH-FU CHANG, 4. An Image Database System with Support for Traditional Alphanumeric Queries and Content-Based Queries by Example.DANIEL L.SWEETS, YOGESH PATHAK, JOHN J.WENG,

STUDIED ARTICLES

PRESENTATION PLAN

1. Introduction.

2. Presentation of articles in subject order, which support or completes each other.

3. Conclusion

INTRODUCTION•Multimedia Data is any unstructured piece of information stored in the Multimedia Database.

•A Multimedia Database differs from a conventionaldatabase.

•Large image databases are commonly employed inapplications like criminal records, customs, plan rootdatabases and, voters’ registration databases.

RESEARCHES

• Most researches in Multimedia Database design have focused on a particular kind of MM data.

• Another direction of research on Multimedia Databases is focused on data structures and algorithms for storing and processing Multimedia content.

FOCUS

• An object oriented approach to multimedia database design.

• Spatial approaches to image retrieval methods.

• Content based queries, including automated feature extraction methods supported with the alphanumeric query modules.

• Solutions to modeling problems and query display methods.

A GENERAL APPROACHMULTIMEDIA MODEL CLASSIFICATIONS:•Multimedia Description Model provides the linguisticmechanism for identifying the huge amount of conceptual entities stored in raw objects.•Multimedia Presentation Model(MPM) describes the temporal and spatial relationships among differently structured multimedia data.•Multimedia Interpretation Model The feature level, manages recognizable measurable aspects of description level objects.The concept level, describes the semantic content of the description level objects.

A GENERAL APPROACH (cont’d)

• A Canonical media object (CO) is a higher level view of a raw object and corresponds to the entire raw objects where a media object represents a relevant portion of a canonical media object.• Examples of MO’s are regions of images, sequencesof regions of video frames, video shots and, words or paragraphs in text documents.• Operations defined on MO’s are the usual editing primitives like creation, modification, access etc.

ANALYSIS AND RETRIEVAL

MULTIMEDIA DATABASE GENERATION

• DATABASE POPULATION

• ACCES STRUCTURE GENERATION

• QUERY FORMULATION AND EXECUTION

DATABASE POPULATION

• Data is stored completely.

• Features are extracted .

• Recognition of concepts are associated with, relevant objects.

ACCES STRUCTURE GENERATION

Using feature & concept values the system createsappropriate access structures that will speed up thesubsequent process.

QUERY FORMULATION AND EXECUTION

• The user formulates the query by interacting with the graphical interface provided by theQuery formulation tool or writes appropriatequery command into the shell.

Concepts can be recognized either; • During the database population • At retrieval time.

QUERYING THE MULTIMEDIA DATABASE

• Browsing: Users have foggy ideas of what they’relooking for.• Content Based Retrieval: Where a request is specified and retrieval of objects satisfying the queries is expected.Content Based Retrieval in Multimedia environments generally takes the form of similarity queries,which are needed when;• an exact comparison is not possible,• retrieved objects need to be ranked.

QUERY RESTRICTIONS•Feature and Concepts: The user may express restrictions on the values of the object’s features and on the values of concepts.•Object Structure: Allowing the user to make restrictions on the structure of the Multimedia objects to be retrieved.•Spatio Temporal Relationships: Formulatingrestrictions on the spatial and temporal relationships of the objects to be retrieved.•Uncertainty: Users may not be certain of the attribute of an object.

THE MULTIMEDIA QUERY LANGUAGE

• If the user specifies a certain concept in the query, the answer set may also contain objects that do not contain that concept but other related concepts, which defined through a relationship between concepts.• Weights are included in order to provide a ranked based retrieval.• Selectors are needed to cope with features, recognition degrees and structure.

THE MULTIMEDIA QUERY LANGUAGE (cont’d)

Example:

After this schema, we can identify four classes containing canonical objects: MPEG, MJPEG, Frames, JPEG and GIF.

THE MULTIMEDIA QUERY LANGUAGE (cont’d)

By looking at this conceptual level schema we can say that skyscrapers, churches and bell towers aresubclasses of BUILDING class.

THE MULTIMEDIA QUERY LANGUAGE (cont’d)

Example:Let us suppose that the user needs to “retrieve all images of all skyscrapers that are higher than 200m”This can be done with;SELECT I FROM I in imagesWHERE I match any

(SELECT SS FROM SS in SKYSCRAPERS WHERE SS.height>200)

SPATIAL APPROACHES TO FEATURE IMAGE QUERY

Image queries can be performed by regions andtheir spatial and feature attributes.

SaFe

In spatial image query (SaFe) the images arematched based upon the relative locations of symbols. For example a relative SQ may ask for images in which symbol A is to the left of symbol B.

HOW DOES IT WORK?• Regions and their feature and spatial attributes are extracted from the images.• The overall match score between images is computed.

IN SaFe SYSTEM• Each object is assigned a minimum bounding rectangle.• Distances between objects are computed.• The user assigns the relative weighting “x” to eachobject. • The overall single region query distance between region q and t is given by;

ftqf

mtqm

atqa

stqstq ddddd ,,,,,

STARTEGIES FOR SPATIAL IMAGE QUERIES

Two strategies for image queries:• Parallel attribute query strategy • Pipeline attribute query strategy

FEATURE QUERY• In order to provide color image retrieval, query-by-color method is used.

• In order to support query-by-color method, an automated color region extraction system is proposed which is named “single-color quadratic back projection system” (SCQBP).

• The system first generates a color histogram hfor each image. • For each image m such that h[m] r in a binary set c is generated .

• Then, each binary set is back projected onto the image using;

FEATURE QUERY (cont’d)

mc

B[x,y]= )(max,1...0 jcAkjMj

FEATURE QUERY (cont’d)

RELATED IMAGE RETRIEVAL TECHNIQUES

• Synthetic color region image retrieval• Color photographic image retrievalare proposed and implemented. This implementation can be found in the WEB at

URL: http://disney.ctr.columbia.edu/safe

CONTENT BASED QUERIES and ALPHANUMERIC

QUERY SUPPORT• The proposed system offers support for both alphanumeric query, based on alphanumeric data attached to the image fileand, content based query utilizing image examples which is accessible from within a user friendly GUI.

SHOSLIF-OThe proposed system implements image retrievalmethod using;

Self Organizing Hierarchical Optimal Subspace and LearnIng Framework for Object Recognition.

SHOSLIF-O (cont’d)The system incorporates 3 major modules:

SHOSLIF-O (cont’d)The SHOSLIF-O module analyzes all images in the database and builds a hierarchical structure for efficient search providing the query-by-image contentcapability of the system.Alphanumeric database fields can be defined by the user in the definition phase and a flat file imported bythe user can act as a database provided it matches thefield count given in the definition phase.In the query phase, the user can enter a text query and the alphanumeric database modules search the database and come out with the image files that satisfy the given conditions.

SHOSLIF-O (cont’d)

PROCESS

PROCESS (cont’d)•MEFs (Most Expressive Features)•MDFs (Most Discriminating Features)

PROCESS (cont’d)

When building the tree, the system can proceed in a supervised or unsupervised learning mode.

KEYWORD-BASED QUERY SUPPORT

• In large image databases, alphanumeric data associated with an image is entered in an alphanumeric database. • This system uses a relational database structurefor storage and retrieval of images and associated data.

GENERAL FEATURES • The matched items of appearance-based retrieval have pointers to the associated text.• One can also start searching with a key field and retrieve images.• One can use alphanumeric search to find all the matched persons and their face images. Then the usercan use those images to find people who look similarto those matched.

GENERAL FEATURES

PROPOSALS FOR EASING THE QUERY PROBLEMS AND RESULT DISPLAY

An example: A movie database can be created using the following attributes:

MOVIE(Title, Year, Producer, Director, Length, Movie_type, Prod_studio)

Cont’dHere another data type called “CORE” is proposed in order to refer to the digitized item directly without causing any confusion between special attribute names.

New definition is:

MOVIE (Title, Year, Producer, Director, Length, Movie_type, Prod_studio, CORE)

•No need to be familiar with the schema.

CER DIAGRAM FOR WWW

CER FOR WWW (cont’d)• After creating CORE ER Diagram (CER), the table definitions are:HTMLDoc(h_url, title, type, length, lastmodify,

CORE)Links (l_url, label)Include (h_url, l_url)• After defining the following methods:-contains (HTMDoc.Title, string)-reach_by (HTMLDoc.url, url_to, by_n, by_type)-mentions (HTMLDoc,string)-linktype (HTMLDoc,url)

CER FOR WWW (cont’d)“Starting from the Computer Science home page,find all documents that are linked through pathsof lengths two or less containing only local links.Keep only the documents containing the string ‘database’ in their title.”SELECT Links.l_urlFROM HTMLDoc,Links,IncludeWHERE substring(“database”, HTMLDoc.title)AND HTMLDoc.h_url= Include.h_urlAND Links.l_url=Include.l_urlAND reach_by(“http://cs.bilkent.edu.tr”,Links.l_url,2,local)

CER FOR WWW (cont’d)Adding two additional methods which are • displayDoc(HTMLDoc)• displayObj (WebObject, properties.position,

properties.size, properties.props)Additional queries can be performed such as;“List all documents that have video clip or picturelabeled ‘Atatürk’”SELECT HTMLDoc.h_urlFROM HTMLDoc, WebObject, IncludeWHERE HTMLDoc.h_url = Include.h_urlAND WebObject.w_url = Include.w_urlAND (WebObject.objectType= “IMAGE” OR WebObject.objectType = “VIDEO”)AND WebObject.label = “Atatürk”

CER FOR WWW (cont’d)

A RESULT DISPLAY PROPOSAL: SQL+D

•“DISPLAY” word is proposed to be reserved.Example: Consider a database for a video rentalstore containing movie titles and other generalinformation of the movies, plus a movie clip anda picture of the promotional poster. Also availableis a list of the actors in a movie, and otherinformation about the actors, including their picture.• The Schema looks as follows:MOVIE (Available, title, director, producer, date,

classification, rating, CORE, poster)MOVIE_ACTORS(title, name, role)ACTORS (name, dob, biography, picture)

A RESULT DISPLAY PROPOSAL: SQL+D (cont’d)

“List all actors in ‘Gone with the wind’ with theirpictures and biographies.”SELECT MOVIE_ACTORS.name,

ACTORS.biography,ACTORS.picture

FROM MOVIE_ACTORS,ACTORSWHERE MOVIE_ACTORS.title=”Gone with the wind” AND ACTORS.name=MOVI_ACTORS.nameDISPLAY PANEL main, PANEL info ON main (east),WITH MOVIE_ACTORS.name AS list ON main (west),

ACTORS.picture AS image ON info (north),ACTORS.biography AS text ON info (south)

A RESULT DISPLAY PROPOSAL: SQL+D (cont’d)

A RESULT DISPLAY PROPOSAL: SQL+D (cont’d)

A RESULT DISPLAY PROPOSAL: SQL+D (cont’d)

A RESULT DISPLAY PROPOSAL: SQL+D (cont’d)

CONCLUSION• There are many proposed systems to make query processing in multimedia databases easier. Although all of them are useful in themselves, some coordinationis needed in order to evaluate and combine the theoretical and practical issues hidden in them.

• Object oriented modeling is necessary for multimediadatabase design.

CONCLUSION (cont’d)• Query language is defined from traditional query language and extended to support;

• Partial match retrieval,• Expressions of conditions on the values of features,• Possibilities to take into account the imprecision ,of the interpretation of the content of the multimedia object.

• Usage of automated feature extraction methods improves image detection and query effectiveness.

• Extensions for the display of the query results, improve multimedia database query flexibility.

• By using spatial image querying mechanisms, we can improve effectiveness over non-spatial imagequery mechanisms.

• There is not any answer for image queries that searches for a picture taken in different lighting and weather conditions hence the problem of distortion continues to affect the effectiveness of multimedia databases.

CONCLUSION (cont’d)

END OF TERM PAPER

PRESENTATION

QUERY PROCESSING IN MULTIMEDIA DATABASES

TURKER YILMAZ