NAVS Spring 2008 Concertation Meeting - CERTH/ITI1 Information Society Technologies Dr. Petros Daras, Informatics & Telematics Institute

NAVS Spring 2008 Concertation Meeting - CERTH/ITI 1

Information Society Technologies

Dr. Petros Daras, Informatics & Telematics

Institute


VICTORY Overview

• VICTORY is a 30-month STREP• Start: Jan 2007• End: Jun 2009

• 9 partners• CERTH/Informatics & Telematics Institute Greece• University of Ljubljana Slovenia• Politecnico di Torino Italy• Alcatel – Lucent Deutschland AG Germany• TELETEL S.A. Greece• HYPERTECH S.A. Greece• EMPOLIS GmbH Germany• TWT GmbH Germany• LivingSolids GmbH Germany


VICTORY Objectives

To develop an innovative, distributed search engine that will introduce MultiPedia search and retrieval capabilities to a standard (PC-based) and a mobile P2P network. The 3D search engine will be based on:

• content, which will be extracted taking into account low-level geometric characteristics and

• context, which will be high-level features (semantic concepts) mapped to low-level features.

To bridge the gap between low and high-level information through automated knowledge discovery and extraction mechanisms.

• Annotation mechanisms• Relevance feedback


VICTORY Objectives

To achieve delivery of audio-visual content on low-power mobile terminals as to enable their integration in a PC network, where the missing resources will be provided collaboratively by other participants.

To develop a multimodal personalized user interface. Different MultiPedia search and retrieval modalities will be supported:

• Text (annotation)• 2D images (taken by the user's mobile device)• Sketches (made by the user)• 3D objects

To ensure that the shared MultiPedia content will not be distributed uncontrollably against the owner's will and to protect the intellectual property rights of the content owners.

• DRM (Digital Right Management)• 3D Object Watermarking


Achievements so far

System Architecture

P2P Network

3D search & retrieval algorithms

Relevance feedback algorithms

Annotation propagation framework


The overall VICTORY architecture


Super peer module (broker)

Superpeers are alwayson, always connected machines, with a static IP addresses associated.

Super-peers are trustworthy machines for the point of view of security.A super-peer manages a set of client peers that are “directly connected” to it.Each superpeer knows all the services and contents available on the

controlled clients. Each super peer maintains and updates a certain number of metadata

catalogues describing clients directly connected to it. Moreover, a table containing IP addresses of other super-peers directly connected is stored on each superpeer.

Super-peers play a basic role in processing user queries in order to: search for Multipedia Objects, negotiate resources (QoE), and manage VICTORY's services.

They also serve as relays thus enabling the clients that are ‘hidden’ behind NATs and firewalls to successfully traverse them and establish connections to other client peers.


Edge-peers

Each edge-peer may be occasionally off or disconnected from the network;

When it reconnects to the network, its IP address may not be the same as before.

When an edge-peer gets connected to the network, it should register (or create a “connection”) with one and only one superpeer,

It should provide it with all the information regarding the contents and services available, plus other general information.

The edge-peer may have a list of more than one superpeer to which connect, in order to tolerate superpeer failures.


P2P networking

VICTORY island groups users that share common interest (like automotive group or tourist group)

VICTORY island may provide some specific services (collaboration), while other services (search and download) are common to whole VICTORY community

Super peer

Client peer

Client peer

Client peer

Victory island

Super peer

Super peer

Victory islandVictory island


Search architecture

Low-level feature extraction

3D object Repository

Low-level feature extraction

GeometricDescriptors

Online

Submitting new 3D object

Browsing / Selecting existing 3D object

Feature Retrieval

Results

Offline

Spherical HarmonicsEllipsoidal Harmonics

Medial Surface… Feature

Matching


Low-level feature extraction Module

• Input: 3D Object• Output: Feature vector with geometrical

descriptors • Goals: Discriminative power

Robustness

Invariance to geometric transformations

• Formats: VRML, OFF, OBJ, 3DS, X3D


Pre-processing Sub-module

Input 3D object (VRML, OFF, 3DS,

OBJ, X3D)

Voxel Model

Uniform 3D Mesh Represetation

(points, triangles)

Format-specific Parser

Mesh-to-Volume Transformer


Mesh-to-Volume Transformer(Voxelization)

- Creation of the bounding cube, which is partitioned in equal cubed shape voxels

- Creation of the Binary volume function

3D model Voxel model Binary Volume Function

( )f x


Preprocessing

• Translation• Scaling• Rotation (PCA)


Descriptor Extraction Sub-module

Voxel Model

3D Radon – based Descriptors

Generalized 3D Radon Transforms

Spherical Trace – based Descriptors

3D Krawtchouk – based Descriptors

Geometric Descriptor Vectors

– 3D Radon Transform– RIT– EnRIT– SIT

Spherical Trace Transform– "Mutated" RIT– 1D Fourier Transform– Radon Transform– Polar-Fourier Transform– Hu Moments– Zernike Moments– Krawtchouk Moments– Spherical Harmonics

Weighted 3D Krawtchouk Moments

Low-level Feature Extraction Algorithms


Combining Topology & Geometry (1)

• The model is segmented into “meaningful parts” based on the segmentation of medial surface

• A meaningful part is a component that can be perceptually distinguished from the remaining object.


Combining Topology & Geometry (2)

• Every part is approximated with a Super Quadratic surface and described using the novel 3D Distance Field Descriptor

• 3D Distance Field Descriptor (3D DFD) is a measure of the difference between the surface of the quadratic and the surface of the part.

• An attributed graph is formed, where the attributes are the Super Quadric parameters and the 3D DFD

• The matching is based on state-of-art Approximate Attributed Graph Matching.

• The matching approach is capable of partial matching


Ellipsoidal Harmonics

• The 3D object is described using Ellipsoidal Harmonics.

• Ellipsoidal Harmonics are solutions of Laplace’s equations in ellipsoidal coordinates

• Ellipsoidal Harmonics offer:– Compact 3D object description– Better 3D object approximation


Impact Descriptor(1)

• The key idea of Impact Descriptor is the indirect description of the 3D object’s geometry, by computing a descriptor that describes the impact of the 3D object in the surrounding space.

• Every object is treated as a distributed mass and the gravitational impact is described– Assuming that the time-space is following the rules of Euclidean

geometry, generalizations of Newton’s Laws are utilized in order to compute the Potential and the Density of the surrounding gravitational field

– Assuming that the time-space is following the rules of Riemannian geometry, Einstein’s General Relativity Laws are adopted in order to estimate the curvature of the surrounding time-space (or, how the 3D object curves the surrounding time-space).


Impact Descriptor(2)

• Histograms of the Riemannian curvature and Newtonian fields are forming the descriptors


Watermarking

• A watermarking approach for copyright protection based on spheroidal harmonic analysis

• The object surface is segmented into patches.• Patches that are appropriate for watermarking

are selected• Every selected patch is mapped on a spheroid• Spheroidal harmonic coefficients are

appropriately modified (based on the watermark sequence)


Relevance Feedback

Objective:• To refine the retrieved results• To give a “human-centric” approach to the

system, since it will be based on the user’s individual actions and choices.

Methods:

• Semantic Force Relevance Feedback

Coulomb-like forces are applied between query and the objects

• Ranked-based Relevance Feedback

Rank lists are calculated for all objects

Relevance Feedback Steps

Step 1: The user defines a query object

Step 2: The system compares the query with all database objects

Step 3: The system returns the most similar objects to the query

Step 4: The user marks the degree of relevance of the retrieved results

Step 5: The user feedback is used to train the system


Relevance Feedback Results


Annotation and annotation propagation mechanisms

Objective:• To model and develop a mechanism for the manual

annotation of 3D objects.• To develop a method for the automatic annotation

propagation to non-annotated 3D objects.

Method:

• Geometry exploitation

• Fuzzy logic, neural networks, active learning, relevance feedback

• Every object i maintains 2 vectors:

– Geometric features vector Gi

– Probabilities vector Si = <pi1, … , piK>

pij : Probability for object Oi to have attribute Aj

K : Number of Attributes


Annotation Propagation

Step 1: Object that provides maximum knowledge gain is selected for manual annotation

Step 2: User annotates the object

Step 3: Probability vector of object is updated

Step 4: User marks similar and non-similar objects

Step 5: Feature vector of marked objects is updated

Step 6: The annotation is used to train the neural network

Step 7: Probability vectors of all non-annotated vectors are updated

Geometric Features Probabilities

Geometric characteristics

of i

Input

User annotation

(Probability {0,1})

Expected output

Probability for i to have specific

attributes

Output

Knowledge Gain Estimator

NeuroFuzzy Controller Annotator

Relevance Feedback


Annotation Propagation Results

Sp

heri

cal H

arm

onic

s

Automatic Annotation on ITI DB Automatic Annotation on SHREC DB

Sp

heri

cal Tra

ce


Access

Over the Project Website (http://www.victory-eu.org) / Results / Victory Search


Publications

[1] A.Mademlis, P.Daras, D.Tzovaras and M.G.Strintzis: "3D Volume Watermarking Using 3D Krawtchouk Moments", 2nd International Conference on Computer Vision Theory and Applications (VISAPP 2007), Barcelona, Spain, March 2007.

[2] G. Kordelas and P. Daras: "Recognizing 3D Objects Using Ray-Triangle Intersection Distances", IEEE International Conference on Image Processing (ICIP 2007), San Antonio, Texas, USA, September 2007.

[3] A.Mademlis, P.Daras, D.Tzovaras and M.G.Strintzis: "On 3D Partial Matching of Meaningful Parts", IEEE International Conference on Image Processing (ICIP 2007), San Antonio, Texas, USA, September 2007.

[4] D.Zarpalas, P.Daras, A.Axenopoulos, D.Tzovaras, and M.G.Strintzis:"3D Model Search and Retrieval Using the Spherical Trace Transform", EURASIP Journal on Advances in Signal Processing Volume 2007.

[5] A.Mademlis, P.Daras, A.Axenopoulos, D.Tzovaras, and M.G.Strintzis :"Combining Topological and Geometrical Features for Global and Partial 3D Shape Retrieval", IEEE Transactions on Multimedia, Accepted for Publication

[6] E.Onasoglou and P.Daras: "Semantic Force Relevance Feedback, Content-Free 3D Object Retrieval and Annotation Propagation: Bridging the Gap and Beyond", SPRINGER Multimedia Tools and Applications Journal (MTAP), Special Issue on Multimedia Semantics, Accepted for Publication

[7] A.Mademlis, P.Daras, D.Tzovaras and M.G.Strintzis: "3D Object Retrieval based on Resulting Fields", 29th International conference on EUROGRAPHICS 2008, workshop on 3D object retrieval, 15 April 2008, Crete, Greece.

[8] M.Lazaridis and P.Daras: "A Neurofuzzy Approach to Active Learning based Annotation Propagation for 3D Object Databases", 29th International conference on EUROGRAPHICS 2008, workshop on 3D object retrieval, 15 April 2008, Crete, Greece


Under review

[1] J. Constantinides, A.Mademlis, P.Daras, and M.G.Strintzis: “Blind Robust 3D Mesh Watermarking based on Oblate Spheroidal Harmonics", IEEE Transactions on Multimedia

[2] P. Daras, M. Lazaridis, Timotheos Kastrinogiannis,Vasileios Karyotisand Christos Malavazos, “A Novel Annotation Propagation Framework for 3D Object Databases Combining Geometric Features and Relevance Feedback Mechanisms”, IEEE Transactions on Multimedia

Special Issue on Integration of Context and Content for Multimedia Management

[3] A.Mademlis, P.Daras, D.Tzovaras and M.G.Strintzis: “3D Object Retrieval using a 3D Shape Impact Descriptor” IEEE Transactions on Multimedia

[4] A.Mademlis, P.Daras, D.Tzovaras and M.G.Strintzis: “Ellipsoidal Harmonics for 3D Shape Description” IEEE Transactions on Multimedia


Future Work

• 3D low-level feature extraction algorithms:– Refinement & Integration

• P2P Network – Intraconnectivity through different Victory Islands– Support to some Victory island specific services, like rendering

and collaboration– Support for mobile clients – Mobile Gateway

• Annotation and annotation propagation mechanisms– Integration

• Relevance feedback algorithms– Integration

• Ontology based retrieval, QoE, Visualization on mobile devices, integration

Documents

NAVS Spring 2008 Concertation Meeting - CERTH/ITI1 Information Society Technologies Dr. Petros Daras, Informatics & Telematics Institute