INAOE at ImageCLEF2007Towards Annotation based Image Retrieval

H. Jair Escalante, Carlos Hernández, Aurelio López, Heidi Marín, Manuel Montes, Eduardo Morales, Enrique Sucar, Luis Villaseñor

Language Technologies LaboratoryNational Institute of Astrophysics, Optics and Electronics

Tonantzintla, Mexico

mmontesg@inaoep.mxhttp://ccc.inaoep.mx/~mmontesg

Overview of the talk

• Our first participation at ImageCLEF; the goal

was to build the basic infrastructure

– Some textual and mixed strategies for image retrieval

• However we could do something more…

– A Web based query expansion method, and

– An annotation based image retrieval approach

Textual and mixed strategies

• VSM IR System for textual retrieval (baseline)

• Late fusion of independent retrievers (LF)

• Intermedia feedback (IMFB)

TBIR

CBIR

FusionRelevantImages

Query

Example images

Topic statement

Query CBIR

QueryExpansion

TBIRRelevantImages

Topic statement

Example images Terms for annotations

Some new things…

• Web-based query expansion:– Original statement + top-k snippets (NQE)

– Original statement + top-l more repeated words from the top-k snippets (WQE)

• Annotation based expansion (ABE)

– Use automatic image annotation methods for obtaining text from images, then…

– Expand documents and/or queries with automatic annotations, finally…

– Apply some strategy for textual image retrieval

Basis of our idea

skypalm

clouds

sea

sandsand

grass

palm, sky, sand, grass, sea, clouds

Flamingo Beach Original name in Portuguese: “Praia

do Flamengo”; Flamingo Beach is considered as one

of the most beautiful beaches of Brazil;Flamingo Beach Original

name in Portuguese: “Praia do Flamengo”; Flamingo

Beach is considered as one of the most beautiful

beaches of Brazil;

Flamingo Beach Original name in Portuguese: “Praia

do Flamengo”; Flamingo Beach is considered as one

of the most beautiful beaches of Brazil;

• Region-level annotations are generally complementary to manual (image-level) annotations

Automatic image annotation

• Assign labels (words) to regions within segmented images

x1 x2 …….. xn Automatic imageAnnotation

method

Grass 0.6Sky 0.2Tree 0.1Ground 0.1

Grass 0.5Tree 0.3Ground 0.1Jet 0.1

Rock 0.5Church 0.2Elephant 0.2Entrance 0.1

Elephant

Grass

Sky

x1 x2 …….. xn

x1 x2 …….. xn

. . .

Annotation improvement

Improving the automatic annotation

Grass 0.6Tree 0.2rock 0.1building 0.1

People 0.4Tree 0.3Mountain 0.2Jet 0.1

Church 0.3Grass 0.3Sky 0.2Elephant 0.2

Tree 0.5Grass 0.3Sky 0.1Jet 0.1

R1 R2 R3 R4

c1 grass people tree churchc2 grass tree tree churchc.. …. …. …. ……ci rock people tree churchc.. …. …. …. ……cj tree people tree churchc256 building jet jet elephant

Idea: select the best label’s configuration, taking into account:1. The prior probabilities of each label, and2. The semantic cohesion of the entire configuration

Grass, Tree, Rock, Building, People,

Mountain, Jet, Sky, Church, Elephant

Set of labels

Some problems with the labels

2000 training annotated-regions (2%) 98000 regions to annotate (98%) Imbalanced training set Limited vocabulary

Annotation based query expansion

sand boats sky tree people water sand sky tree buildings sky water tree

accommodation with swimming pool

accommodation with swimming pool +sand boats sky tree people water buildings +

three given images

The volcano Tungurahua

Baños, EcuadorMarch 2002

sand clouds sky mountain

The surroundings of the Valle Francés Torres del

Paine National Park, Chile March 2002

furniture grasspeople clouds

Annotation based document expansion

Experimental results

Top ranked runs for each configuration considered.

Visual-English run

• No textual query was used, but at the end the recovery was done based on textual data.

• It combines intermedia feedback and our annotation based expansion technique.

CBIR

TBIRRelevantImages

Exampleimages

Terms for manual annotations

AutomaticAnnotation

Automaticannotations

Textual vs. mixed strategies

Initial conclusions

• Intermedia feedback is an effective way for mixing visual and textual information

• Methods based on web-query expansion showed better performance

• Anotation based expansion is a promising way for expanding text using image’s visual content

• Annotations can be useful for image retrieval, though several issues should be addressed

Our current work

• Work on the improvement of automatic image annotation methods

• Investigate different (better) ways for measuring the semantic cohesion between labels and manual annotations

• Use such semantic cohesion estimates for improving image retrieval from annotated collections

Thanks for your attention

Language Technologies LaboratoryNational Institute of Astrophysics, Optics and Electronics

Tonantzintla, México

Manuel Montes y Gómezmmontesg@inaoep.mx

http://ccc.inaoep.mx/~mmontesg