EE 576 - Object - Scene Representationisl.ee.boun.edu.tr/courses/ee576/lectures/sunum/objsceanalpres.pdf · Object–SceneAnalysis Introduction ObjectRecognition Model-BasedObjectRepresentation

OutlineObject – Scene Analysis

EE 576 - Object - Scene Representation

H.I. Bozma

Electric Electronic Engineering

Bogazici University

May 22, 2013

H.I. Bozma EE 576 - Object - Scene Representation


Object – Scene AnalysisIntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding



IntroductionObject RecognitionModel-Based Object RepresentationGraph RepresentationScene Understanding

Objects vs Scenes

◮ Objects – Single meaningful entities

◮ Scenes – Collection of objects

◮ Learning ⇔ Representation




Problem

◮ Representation

◮ Scale invariance, occlusion

◮ Learning: A priori knowledge is essential




Representation

◮ Levels of abstraction◮ Specific objects◮ Generic objects◮ Functional representations

◮ Properties of objects: Shape, color, texture, function

◮ Shape representation: Collection of points, boundaries,surfaces, volumes, collection of parts




Complexity

◮ Object type◮ Type – Monocular, stereo, range◮ Nature – Natural, artificial

◮ Similar objects◮ Complex shapes◮ Occlusion◮ Illumination




General Approach

◮ Choose a representation model

◮ Find instances of known models in the scene

◮ Estimate their pose




Basic Models

◮ 2D Features◮ Color◮ Shape - Area, moments, Fourier coefficients◮ Filter responses◮ Histograms◮ Corners, SIFT, SURF

◮ 3D Features◮ Geons (or geometric ions) - Simple volumes such as cubes,

spheres, cylinders, and wedges.

◮ Features and the relationships between them




Visual Filters

◮ Robot at x ∈ X , ∀f ∈ F ,◮ Visual filters V with |V| = Nv (Nv = 56 used)

◮ Two color surfaces (a,b from CIELab)H.I. Bozma EE 576 - Object - Scene Representation



Difficulties

Projection and occlusion – create difficulties.Objects 3-D – images 2-D.Projected shapes – Depend on the viewpoint.Objects occlude each other → Only parts of objects visible →Problems in segmentation.Perceptual grouping




Perceptual Grouping

Organization of image features into higher level feature

◮ Descriptive – Make geometrical structure explicit

◮ Hierarchical

◮ Invariant – if possible




Perceptual grouping




Complexity

◮ The number of models in the database

◮ Complexity of representation

◮ Complexity of matching




Representation and Recognition Paradigm

◮ Could the image be a projection of the model?

◮ Match part of a scene with a model → Search for best fit◮ Estimate object pose◮ Project model and verify

◮ What types of features to match and how?◮ Feature matching◮ Structured description matching




Matching Models

◮ Template matching◮ Correlate a template with the image - Invariant to translation,

sensitive to rotation, scaling and viewpoint◮ Using other metrics




Feature Matching

◮ Match image and model features → Search for best fit

◮ Check for consistency using constraints that depend onrepresentation




Feature Matching

Figure : Feature matchingH.I. Bozma EE 576 - Object - Scene Representation



Representation By Parts

◮ One approach to structural shape representation is graphrepresentation

◮ A graph consists of two entities: nodes P and edges E (P)between nodes.

◮ Properties of nodes and edges in the model database.

Some examples are:

◮ Line drawings consisting of vertices (nodes) and lines (edges)

◮ Surfaces (nodes) and their relationships (edges)

◮ Complex shape represented by parts (nodes) and therelationships (edges)




Recognition by Structure Matching

◮ Here, the model is based on a graph that represents theobjects’ parts and their relationships.

◮ Given an incoming image, a similar model is constructed andmatched with those in the model database.




Graph Matching

Given G = (P ,E (P),R(P)) and G ′ = (P ′,E ′(P ′),R ′(P ′)

◮ Two nodes n ∈ P and n′ ∈ P ′ form an assignment ifR(n) = R ′(n′)

◮ Two assignments (n1, n′1) and (n2, n

′2) are compatible if

E (n1, n2) = E (n′1, n′2)

◮ Isomorphism: Compatibility for all nodes

◮ Isomorphism: Compabilility for a subset of nodes




Search Tree: Breadth-First

Figure : Breath-first search of a match tree.H.I. Bozma EE 576 - Object - Scene Representation



Graph Isomorphism

Finding an isomorphism is NP-hard:




Graph Isomorphism – Ex 2




Aspect Graphs

Aspect graph – A series of two-dimensional views (nodes) of aknown object that are related to each other (edges) via 3Dtransformations.

◮ Generate the aspect graph

◮ Maintain them in some convenient representation in storage,

◮ Match one or more two-dimensional views of an unknownobject against the stored views of the known object,

Recognition problem → Transformed to a series of two-dimensionalmatching problems.




Varying viewpoints




Varying Viewpoints




Aspect Graph




Problem

Scene analyis (scene understanding) is the ability to make usefuldescriptions of the given scene.

◮ Identify objects and relationships between them

◮ Locate a certain object

◮ Make measurements for inspection




Scene Representation

◮ Identify objects and relationships between them

◮ Locate a certain object

◮ Make measurements for inspection




Scene Representation

◮ Bottom-up approaches: First find objects and then put themtogether

◮ Objects ⇒ A graph of objects

◮ Top-down – Holistic approaches◮ Image histogram◮ Bag-of-words◮ Spatial envelope◮ Hull census transform (HCT)◮ Bubble surfaces and descriptors




Problems in Scene Analysis

◮ Representation

◮ Occlusion

◮ Image complexity

◮ Scene complexity

◮ A priori knowledge is essential




Bag-of-Words

◮ Image representation - A collection of words, disregardinggrammar and even word order.

◮ Words – Obtain many samples and then cluster to get asmaller set

◮ Frequency of occurrence of each word - Used as a feature fortraining a classifier




Spatial Envelope

naturalnessopennessroughnessexpansionruggednessSpectral Spatial domain




Hull Census Transform (HCT)




Bubble Space

Robot base

x = [c α]T

Base space X = R2 × S1

Pan-tilt movement:

f = [f1f2]T

Camera state space F ⊂ S2

Bubble space B = X × FRobot’s base + its viewing directions




Bubble Surfaces

◮ Bubble surface Bi (x , t) : Im(h(x))× R≥0 → R≥0.

Bi (x , t) =

{[

f

ρi (b, t+)

]

| ∀f ∈ F and b = [x f ]T}

ρi : B × R≥0 → R≥0 - Encodes i th visual feature response

◮ Initialization: S2 sphere with radius ρ0 ∈ R≥0 – ρi (b, 0) = ρ0

◮




Problems in Scene Analysis

◮ Representation

◮ Occlusion

◮ Image complexity

◮ Scene complexity

◮ A priori knowledge is essential




Complexity

◮ Images are complex◮ Nature – Monocular, stereo, range◮ Quality – Indoors, outdoors,

◮ Scenes are complex◮ Illumination


Documents

EE 576 - Object - Scene Representationisl.ee.boun.edu.tr/courses/ee576/lectures/sunum/objsceanalpres.pdf · Object–SceneAnalysis Introduction ObjectRecognition Model-BasedObjectRepresentation