Object Recognition Using Attention

Tammy Avraham and Assoc.Prof Michael Lindenbaum

Computer Science Dept., TechnionHaifa, Israel

Projects for implementing a systemfor:

The system we are going to build

Fast! How ? By directing attention to the important parts of

the image. Faces is an example. We want to be able to recognize

also some other types of objects

Our system

Do you see faces ? How many & where ?

Yes, I see 10 faces in locations…

Computer Vision : Giving computers the ability to understand what a scene (=a matrix of pixels) represents.

A subtask – object recognition: an expert system that detects a certain type of objects. For instance: faces, cars, chairs…

A very difficult task – variations due to illumination conditions, viewing position, posture. .– Categorical (e.g. – many types of chairs)

Usually object recognition systems scan all the image for the desired objects.

The candidates are, for instance, all possible rectangles in the image in a few sizes

Object recognitionProject: build object recognizer

……

Segmentation

Instead of exhaustive scanning perform segmentation as a pre-process.

Segmentation/Grouping=dividing the image into regions, each hopefully representing an object.

Project: implement a segmentation algorithm

Adding Attention

Objective: minimize

those calls

Ordering\Attention algorithm

candidate

identity

Image CandidateSelection

High-levelObject Recognizer

Human Attention

Yarbus 67 – eye movement study

Human Attention – Psychology theory #1

Triesman & Gelade 80 - Feature integration Theory Pop-out vs. sequential scan

Pop-out: when the target is different in on feature from all othersSequential: when the target is distinct by a combination of features

Theory of the visual search process:– Pre-attentive stage: parallel feature extraction – Combining the feature maps to one saliency map – Attention spotlight (Posner)

A computer model based on Feature integration Theory

Itti,Koch&Niebur 1998:

– Bottom-up saliency – Top-down information

Project: build saliency maps

Saliency Map

Human Attention – Psychology theory #2- based on inner-scene similarity

Duncan & Humphreys 89 – Similarity Theory– Search Difficulty depends on:

Target vs. Non-target similarity Non-target vs. Non-target similarity

– Theory: Stage 1 – hierarchical segmentation to structural

units including similarity grouping/linkage.Stage 2 – attention, spreading suppression.

Using Inner Scene similarity for computer vision

Mutual Similarity of candidates as useful information

Dynamic search – use previous recognizer answers to change the priorities of the non-yet-attended candidates.

Example: if the middle green

creature turned out to be not

interesting, mark also the other

similar creatures as not

interesting, and try a micki-mouse.

Project: measuring similarity

Project: dynamic attention

Coming back to the system we want to build

A computer-vision system for object recognition using ideas from human attention to direct us faster to the important objects in the scene.

Candidate selection:– Simple selection of rectangles, or:– Segmentation

The attention will rely on:– Saliency– Prior knowledge on desired objects– Similarity between objects in the scene.

We should be able to test all combinations of above options.

The system’s scheme

candidates description

3.Similarity measures

4.Attention

order control

5.Object

recognition

locations of targets &

search orderdistance matrix

Image + user parameters

candidate index recognition

results

a few target and nontarget models

(optional)

2.Candidates selection

1.Compute saliency

saliency map

candidates priorities

features maps

Real-Time presentation

More details on:

Saliency map

1. Computing saliency

Params:

Calculate feature maps

Feature maps

imageCalculate saliency per feature

Saliency map per feature

Combine to one saliency map

model images

If model images are used, they can effect the importance of each Feature when combining them to one map of saliency.

More details on:

2. Candidate selection

Segmentation algorithmImage: Colored/gray

Saliency map

Trivial candidate selection

Params:

-segmentation or simple rectangular candidates

-which algorithm to use for segmentation

-Remove regions beyond a certain size ?

-Provide only most salient candidates. How many ?

-…

Compute priority of each candidate

Description of indexed candidates.+Priority of each

If possible – consider only the salient regions during segmentation to save time

3. Similarity measures : option 1

Itti

More details on:

Extract some partial description of each candidate: A feature vector or a histogram

Similarity matrix (distance matrix)Candidates

description

Params:

-which features to extract

-which metric to use as the distance function

- …

Feature extraction Distance function

Feature vectors/histograms

image

Use some distance between the features of the candidates to measure dissimilarity

Models images

If model images are used, treat them as candidates and get also a similarity measure between them and the other candidatesAnd between themselves.

3. Similarity measures : option 2 : use the feature maps calculated in the saliency stage instead of stating from the image pixels.

Itti

More details on:

From the feature maps that describes each pixel individually create a feature vector\histogram per candidate

Similarity matrix (distance matrix)

Candidatesdescription

Params:

-calc a feature vector or histogram

-which metric to use as the distance function

- …

Calculate feature vectors

Distance function

Feature vectors/histograms

Feature maps

Use some distance between the features of the candidates to measure dissimilarity

4. Attention order control

More details on:

Iteratively decide on next candidate to attend

Index of candidate to attend to

Recognitionresults

The target candidates

The scan order

Distance matrixPriorities vector

Params:

-consider priorities

-consider similarity

-when to stop

- …

5.Recognition

More details on:

Learning\training offline stage (?)

Image

Candidates description

Params

Recognition (classification)

Candidate index

training set (target and non-target examples)

All projects implementing object recognition should test their software to recognize a few categories of objects, and supply interface for testing more categories.

Yes/no

Amount of certainty

Implementation & Integration

The system will combine C and Matlab code. In some cases we will force one of them. Each part should be able to work alone with its own graphical user Interface, and as a part of the

system.How? For instance - Each project will implement a main function :

OutParams MainFunction( InParams )When this part is tested alone: The User Interface callback function will call this function.When it is used as a part of the system: The main program that will run all the system will call it.

Integrators: – Provide UI for all the system for feeding parameters needed for all its parts.– Design the way parameters are passed form one part to another.– Write the main programs for running the system.– Responsible that the system will work !

Meaning – meeting with the other groups all through the semester.– Responsible that the system will work in all possible paths –

Using saliency or skipping Use inner-scene similarity consideration for attention Use top-down knowledge for attention. If yes, use it when computing the saliency map

or when computing the distance matrix. The type of candidates selection. Use a high resolution image only for recognition and a lower resolution image for the

rest.– All above (and much more) are parameters the user can set using the UI.– Responsible for testing the system with a variety of images.– Responsible for presenting the results both in real time and the results statistics.

Important (also for your grade)

DOCUMENTATION – both in the code and in the report. Each class and each method/function should be explained in details

RUNNING TIME – important to save time when possible. Add time measures and report them.

OPEN TO CHANGES – your code will be submitted as well and it is possible that someone else will want to use it and change it in the future.

Time Table & Grading

StageDate/End DateGrade

Get a projectMarch 22

Theoretic stage : understand the algorithms

April 1515%

Software designMay 315%

Implementation and Testing

June 6

Integration (all along)June 15

Project PresentationJune 2115%

Final Report and code Submission

June 2855%

Remark: where ‘grade’ is filled, it means you have to submit something until that date

A little more on each stage

Theoretic stage: 1. Read the paper/papers you are given, understand it, summarize it. 2.Explain how it fits in the system. What the input and output parameters are. 3.How do you plan to test it.

Software design: Design of your code. Includes all functions and parameters you are going to implement. Description of your user Interface and your software interface for the system. (Already using the datatypes designed for the input and output parameters by the integrators).

Software implementation and testing: implement and perform many tests. Pass the code to the integrators, even before it is perfect. At the end date of this stage it should be perfect.

Integration stage: Putting the system together. The integrators are the most busy, but the other must be available to correct bugs if needed. At this time you should be working on your presentation.

Presentation: Prepare a short power-point presentation (about 20 minutes) describing the important theory involved and show some results.

Final Report: A wide description of the theory, the software documentation, and the results. Submit the code well documented (Electronically).

The integrators at each stage

The integrators must be always one step ahead. During the ‘theoretic stage’ they do ‘software design’: Design the

datatypes that will be used as input and output parameters for each stage. Plan the software they have to write (for instance – simple candidate selection). Describe the user interface and the way they are going to present the system’s results.

During the ‘software design’ stage the integrators should implement their software with dummy functions where the models should be.

During the ‘implementation stage’ they should find and film many images to be used for the full system’s testing and bring them to the other people. Gradually integrate parts to the systems when people are done (or almost done).

Integration stage: Here they are the most busy people. Presentation: describe all the system and show some results.

Remarks

If we correct mistakes in your theoretic report and the software design, you must correct them before going to the next stage.

This slides don’t include all the details of the systems. Each group, when assigned with the project will get much more details concerning their part.

Set a meeting with Tammy A.S.A.P. Some of the projects will be guided by Tammy, some

by Micha. Meeting with us will be held each one-two weeks. You can set a meeting each time you have questions. Start working ASAP ! For instance: read the paper and

come to ask questions before Passover