Non-Local Characterization of Scenery Images: Statistics, 3D Reasoning, and a Generative Model Tamar Avraham and Michael Lindenbaum Technion

Non-Local Non-Local Characterization of Scenery ImagesCharacterization of Scenery Images: : Statistics, 3D Reasoning, and a Generative Statistics, 3D Reasoning, and a Generative

ModelModel

Tamar Avraham and Michael Lindenbaum

Technion

Characterization of Scenery Images: OverviewCharacterization of Scenery Images: Overview

● Statistical Characterization● Rough shape of regions ● Relative location of regions ● Shape of boundaries

● 3D Reasoning● Why are background contours horizontal?

● A Generative Model● Provides a prior on scenery image annotation● Generates image sketches,

capturing the gist of scenery images

manual segmentation and region annotation

Given the above segmentation (without texture), which region labeling is more likely ?

skymountain

sea

scenery images (LabelMe)

rocks

groundtrees

mountainsand

OR ?

Property 1 : HorizontalnessHorizontalness

Most background objects exceed the image width

Background objects are wide and of low height while foreground objects’ shape tend to be isotropic

background objects : sky, mountain, sea, trees, field, river, sand, ground, grass, land, rocks, plants, snow, plain, valley, bank, fog bank, desert, lake, beach, cliff, floor

foreground objects: all others

The relative top-bottom locations of types of background are often highly predictable

The probability for a background region of identity A to appear above a background region with identity B, summarized in a histogram for various background identity pairs

Topological ordering of background identities can be defined: this DAG is associated with the reachability relation R : {(A,B)| p(A above B) > 0.7}

sky

mountain desert

trees field valley

river lake

land

sand

sea

plants

groundgrass

rocks

Property 2: Order / Relative Order / Relative LocationLocation

Property 3: Boundary ShapeBoundary Shape

Chunks of upper boundaries as 1D signals: Curves associated with sea, grass or field resemble DC signals. Curves associated with trees and plants are high frequency signals. Curves associated with mountains resemble signals with low frequency and high amplitude

A sample of contour segments associated with background object classes mountain, sea, and trees

Flatland Flatland Flatland Flatland

3D Reasoning: Why are background regions horizontal?Why are background regions horizontal?Land regions whose contour tangents in aerial images are uniformly distributed

appear with strong horizontal bias in images taken by a photographer standing on the ground

Land regions whose contour tangents in aerial images are uniformly distributed appear with strong horizontal bias in images taken by a photographer standing on the ground

Θ - the set of tangent angles for contours in aerial images (relative to an arbitrary 2D axis on the surface) Θ’ - the set of angles that are the projections of the angles in Θ on the camera’s image plane

”Place a penny on the middle of one of your tables in Space ... look down upon it. It will appear a circle....gradually lower your eyes ... and you will find the penny becoming more and more oval to your view....” From Flatland, by Edwin A. Abbott, 1884

lake

sand

grass

flora

soil

weed

p

px

zh

X1X2

X3

A schematic illustration of an aerial image

An image is taken by a photographer standing on the ground

tantan 'tan

hz x

The distribution Θ’, assuming Θ=U[0,180°), h~2[m], z~U[0,1000[m], x~U[0,500[m]]

θ’ Θ’ θ Θ

3D Reasoning cont.

Ground elevation and slope Ground elevation and slope statisticsstatistics

Two landscape image contour types:

1) The contours between different types of regions on the terrain

2) The contours of mountains associated with occluding boundaries (e.g., skylines)


Two landscape image contour types:

1) The contours between different types of regions on the terrain

2) The contours of mountains associated with occluding boundaries (e.g., skylines)

3D Reasoning cont.


The contours between different types of regions on the terrain


The contours between different types of regions on the terrain

(cos sin sin cos cos ) sin sintan '(cos sin sin cos cos ) (cos cos sin cos sin )

H zx z

The distribution Θ’ assuming Θ =U[0,180°), ϕ~slope statistics, ω~[-90°,90°]. H’s distribution was estimated from sampling an elevation map in pair locations up to 9km apart

A point p lies on a boundary between land regions, located on an elevated surface with gradient angle ϕ. The plane is rotated at an

angle ω relative to X1 axis

( , , )P x H z

X1

X2

X3 O

ω

Estimated terrain slope distribution using the IIASA-LUC dataset

3D Reasoning cont.


The contours of mountains associated with occluding boundaries

Tangents in images bounded by the

max-slope-over-land-regions statistics


The contours of mountains associated with occluding boundaries

Tangents in images bounded by the

max-slope-over-land-regions statistics

Estimated distribution of the maximum slope over land regions, each covering approx. 9 square kilometers

* The paper also discussed the effect of land cover and points out other factors that should be considered in a more complete analysis.

modeled by PCA of “1D” signals

The Generative ModelThe Generative Model

1

1 1 1

{ ,...}

( ,..., ) (the annot

,..., , ,.

atio

..,

( |

)

)

n

?

n

n

n

n

L sky, ground, sea, trees, grass, plants, rocks

l l l L

S h h S S

P l S

1h

2h

4h3h

1S

2S

3S

1 2 3 1

1,..., 1,..., 1'

( | ) ( )( | ) ( ) ( | ) ( | )

( | ') ( ')n

i i i i

i n i nl L

P S l P lP l S P l P h l P S l

P S l P l

top-bottom ordertop-bottom order

region heightregion heightA normal

distribution for the height covered by each region type

upper boundaries upper boundaries

1

1 1 11

( ( ,..., )) (' ', ) ( , ) ( , ' ')n

n i i ni

P l l l M top l M l l M l bottom

top

bottom

sky trees ground sea

The generative nature of the model makes it possible to:

1) Generate image sketches, capturing the gist of scenery images

ECCV10

2)Obtain priors for region annotation

more recent work

The Generative Model: The Generative Model: advantagesadvantages

– Given a set of manually segmented and annotated scenery images:• Top-bottom order: estimate the transaction matrix M

counting number of occurrences of the different ‘moves’.

• Relative region coverage: estimate mean and variance for the relative average height of each type of region

• Upper boundary: for each background region type, collect 64-pixels length chunks. Find the first k principle components and Eigen values so that 95% of the variation in the training set is modeled. ( )

– Possible to train different models for different scenery categories.here we trained together of 3 categories: coast, mountain, open country.

The Generative Model: The Generative Model: TrainingTraining

top

bottom

sky treesground sea

, ,

– randomly selecting the top-bottom sequence by a random walk on the Markov network , starting at ‘top’, stopping at the sink ‘bottom’.

– randomly select the relative average height of each region– randomly generate the boundaries:

• For each generate 4 chunks

The Generative Model: The Generative Model: Generating SketchesGenerating Sketches

Sky

mountain

mountain

trees

Trees

sky

mountain

mountain

trees

trees

, , ~ (0, )i m j jS b b N

,1 ,4,...,i iS SiS

typical scenery images (LabelMe) manual segmentation and typical scenery images (LabelMe) manual segmentation and region annotationregion annotation

semantic sketches of scenery images semantic sketches of scenery images generated by our modelgenerated by our model

typical scenery images (LabelMe) manual segmentation and typical scenery images (LabelMe) manual segmentation and region annotationregion annotation

semantic sketches of scenery images semantic sketches of scenery images generated by our modelgenerated by our model

The Generative Model: The Generative Model: Generated ResultsGenerated Results

The Generative Model: The Generative Model: (More) Generated Results(More) Generated Results

Region ClassificationRegion Classification

Q: Can the new cues contribute to region classification/annotation?

A: Complimentary to textural & color cues

Goal: to show that region classification using global + local descriptors is better than only local descriptors

sky

mountainsea? ground?

rocks?plants?

only layout

sky? sea?

mountain? ground?sea

rocks

only color&texture

+ =sky

mountainsea

rocks

Region Classification - Region Classification - HMMHMM

Marginals by the sum-product message passing algorithm

Classification by max

1H

2H

4H

5H

1T

2T

3T

4T

5T

2S

3S

4S

5S

3H

1il

1iT 1iH 1iS

1l

1T 1H

il

iT iHiS

nl

nT nHnS

j

i

Lp

arg max( )j

i

j

L

iL

c p

Region Classification - Region Classification - DiscussionDiscussion

General object annotation and detection using context: G. Csurka and F. Perronnin. An efficient approach to semantic segmentation. IJCV, 2010. C. Desai, D. Ramanan, and C. Fowlkes. Discriminative models for multi-class object layout. ICCV, 2009. C. Galleguillos and S. Belongie. Context based object categorization:A critical survey. Comput. Vis. Image Understand, 2010. X. He, R. S. Zemel, and D. Ray. Learning and incorporating top-down cues in image segmentation. ECCV, 2006. S. Kumar and M. Hebert. A hierarchical field framework for unified context-based classification. ICCV, 2005. A. Rabinovich, A. Vedaldi, C. Galleguillos, E. Wiewiora,and S. Belongie. Objects in context. ICCV, 2007. J. Shotton, J. Winn, C. Rother, and A. Criminisi. Textonboost for image understanding: multi-class object recognition

and segmentation by jointly modeling appearance, shape and context. IJCV, 81(1):2–23, 2009.

Approximated inference needed (e.g., greedy iterative methods, loopy belief propagation)

Background region classification of scenery images: a 1D problem

Enables exact inference

Region Classification - Region Classification - DetailsDetails

Textural & color features: as in Vogel&Scheile IJCV 07:

HSV Color histograms

Edge direction histogramsGray-level co-occurrences (GCLM, Haralick et al. 73). 4 offsets. For each, contrast, energy, entropy, homogeneity, inverse difference moment, and correlation.

and are each modeled with a multiclass probabilistic SVM (LibSVM, Wu, in, Weng 04), RBF kernel.

5-fold cross validation at image level. Each training includes parameter selection by inter-training set cross validation.

Dataset of 1144 images (LabelMe: coast, open country, mountains). Regions:

( | )i j ip l L T ( | )i j ip l L S

sky mountain sea trees field river sand ground grass land rocks plants snow plateau valley bank lake beach cliff Total Regions1120 1489 401 622 366 150 182 94 36 41 201 143 50 28 20 20 9 3 4 4979

Region Classification – Region Classification – Results 1Results 1

ground truthInput image relative location boundary shape color&texture all cues

sky

mountainmountainsea

sky

mountainmountainmountainmountainsea

sky

mountainmountainfield

SKY

MOUNTAIN-TREESMOUNTAIN-TREESMOUNTAIN-TREESMOUNTAIN-TREESPLAIN-SAND

mountainmountainsea

MOUNTAIN-TREESMOUNTAIN-TREESMOUNTAIN-TREESMOUNTAIN-TREESWATER

sky

mountainmountain

sky

SKY

MOUNTAIN-PLANTSMOUNTAIN-ROCKSMOUNTAIN-ROCKSMOUNTAIN-ROCKSSKY

sky

mountainmountain

sea

SKY

MOUNTAIN-PLANTSMOUNTAIN-ROCKSMOUNTAIN-ROCKSMOUNTAIN-TREESWATER

sky

mountainsea

sky

mountainsea

sky

mountainmountain

SKY

MOUNTAIN-TREESMOUNTAIN-SAND

mountainsea

MOUNTAIN-TREESWATER

sea

mountainsea

WATER

MOUNTAIN-SANDWATER

sky

mountainsea

SKY

MOUNTAIN-SANDWATER

sky

treessea

sand

sky

treesseasand

sky

mountainmountain

field

SKY

MOUNTAIN-TREESMOUNTAIN-TREESPLAIN-SAND

mountainsea

mountain

MOUNTAIN-TREESWATERMOUNTAIN-TREES

sky

treessky

mountain

SKY

MOUNTAIN-TREESSKYMOUNTAIN-SNOW

sky

treessea

sand

SKY

MOUNTAIN-TREESWATERMOUNTAIN-SAND

sky

mountain

mountain

skymountainmountain

mountainmountainmountain

sky

mountain

mountainfield

SKYMOUNTAIN-TREESMOUNTAIN-TREES

MOUNTAIN-TREESMOUNTAIN-TREESPLAIN-ROCKS

mountain

mountain

MOUNTAIN-TREESMOUNTAIN-TREES

MOUNTAIN-TREESMOUNTAIN-TREESMOUNTAIN-TREES

sky

mountainmountain

SKYSKYSKY

MOUNTAIN-TREESMOUNTAIN-TREESMOUNTAIN-PLANTS

sky

mountain

mountain

SKYMOUNTAIN-TREESMOUNTAIN-TREES

MOUNTAIN-TREESMOUNTAIN-TREESMOUNTAIN-SNOW

sky

mountainfield

field

skymountain

fieldfield

sky

mountainmountain

field

SKYMOUNTAIN-TREES

MOUNTAIN-TREESPLAIN-SAND

mountainfield

field

MOUNTAIN-TREES

PLAIN-SANDPLAIN-SAND

sea

field

fieldfield

PLAIN-SANDPLAIN-TREES

PLAIN-TREESPLAIN-GROUND

sky

mountainfield

field

SKYMOUNTAIN-TREES

PLAIN-SANDPLAIN-SAND

Region Classification – Region Classification – Results 2Results 2ground truthInput image relative location boundary shape color&texture all cues

sky

sea

sand

sky

sea

sand

sky

mountain

mountain

SKY

MOUNTAIN-TREES

MOUNTAIN-SAND

sea

mountain

WATER

MOUNTAIN-TREES

sea

sea

sky

WATER

WATER

MOUNTAIN-SAND

sky

sea

sand

SKY

WATER

MOUNTAIN-SAND

sky

mountainsea

sand

sky

mountainsea

sand

sky

mountainmountain

field

SKY


PLAIN-SAND

mountainsea

mountain

MOUNTAIN-TREESWATER

MOUNTAIN-SAND

mountain

mountainsea

plants

MOUNTAIN-SAND

MOUNTAIN-SANDPLAIN-SAND

MOUNTAIN-PLANTS

sky

mountainsea

sand

SKY

MOUNTAIN-SANDWATER

MOUNTAIN-SAND

sky

mountain

mountain

sky

mountain

mountain

mountain

sky

MOUNTAIN-TREESSKY

MOUNTAIN-SAND

mountain

mountain

MOUNTAIN-TREES

MOUNTAIN-TREES

skymountain

SKYMOUNTAIN-ROCKS

SKY

skymountain

mountain

SKY

MOUNTAIN-SNOW

MOUNTAIN-ROCKS

skymountain

mountain

skymountainmountain

skymountain

mountain

SKYMOUNTAIN-TREESMOUNTAIN-SAND

sea

mountain

WATERMOUNTAIN-TREES

skymountainmountain

SKYMOUNTAIN-GROUNDMOUNTAIN-GROUND

sky sea

mountain

SKYWATERMOUNTAIN-SAND

sky

field

mountain

sky

field

mountain

sky

mountain

mountain

SKY

MOUNTAIN-SAND

MOUNTAIN-TREES

mountain

mountain

MOUNTAIN-TREES

MOUNTAIN-TREES

sky

field

field

SKY

PLAIN-GRASS

PLAIN-GRASS

sky

field

mountain

SKY

PLAIN-GROUND

MOUNTAIN-TREES

sky

mountainfield

sky

mountainfield

sky

mountainmountain

SKY

MOUNTAIN-TREESMOUNTAIN-SAND

mountainmountain


sea

mountainsea

WATER


sky

mountainfield

SKY


sky

trees treesbankriver

bank

sky

treestreestreestrees

bankriver

bank

sky

mountainmountain

trees

mountain

field

SKY

MOUNTAIN-TREESMOUNTAIN-TREESMOUNTAIN-TREESMOUNTAIN-TREES

PLAIN-TREESMOUNTAIN-TREES

PLAIN-ROCKS

mountainmountain

mountainmountain

mountain


MOUNTAIN-SANDMOUNTAIN-TREES

MOUNTAIN-TREES

sky

treesmountain

treesmountain

trees

SKY


MOUNTAIN-TREESMOUNTAIN-GROUND

MOUNTAIN-TREES

sky

treestrees

fieldriver

trees

SKY


PLAIN-GROUNDPLAIN-GROUND

PLAIN-TREES

sky

trees treesbankriver

bank

sky

treestreestreestrees

bankriver

bank

sky

mountainmountain

trees

mountain

field

SKY


PLAIN-TREESMOUNTAIN-TREES

PLAIN-ROCKS

mountainmountain

mountainmountain

mountain


MOUNTAIN-SANDMOUNTAIN-TREES

MOUNTAIN-TREES

sky

treesmountain

treesmountain

trees

SKY


MOUNTAIN-TREESMOUNTAIN-GROUND

MOUNTAIN-TREES

sky

treestrees

fieldriver

trees

SKY


PLAIN-GROUNDPLAIN-GROUND

PLAIN-TREES

Accuracy per class:

Color&texture: higher accuracy for trees, field, rocks, plants, snow

New cues: better for sky, mountain, sea, sand

Other classes performance: very low due to their number.

Discussion

We achieved the goal of showing that the new cues improve texture&color only based region classification.

Many classifications counted as errors are actually correct

Related to recent work on object categorization with huge amount of categories (Deng, Berg, Li, Fei-Fei ECCV10, Fergus, Weiss, Torralba ECCV10)

Work in progress.

Region Classification – Region Classification – Results 3Results 3Cue Accuracy

Color&Texture 0.615Relative Location 0.503Boundary Shape 0.452

Relative Loc. + Boundary Shape 0.573Color&Texture + Relative Loc. 0.676Color&Texture + Boundary Shape 0.641

All (ORC) 0.682

19 categories

SummarySummary

Focus of characterization of scenery images

Intuitive observations regarding the statistics of co-occurrence, relative location, and shape of background regions were explicitly quantified and modeled

Some 3D reasoning

Non-local properties can capture the gist of images

Contextual background region classification with exact inferences.

The new cues improve local-descriptors based region classification

Future & General Future & General DiscussionDiscussion

A better way to evaluate region classification: work in progress

Use the layout cues for better top-down segmentation (Felzenszwalb&Veksler, CVPR 10). Shape prior to address “shrinking bias” (Vicente, Kolmogorov, Rother, CVPR 08)

Use the layout cues to improve scene categorization

Augment foreground objects into the model. Extend model to other domains.

Use the cues to align pictures.

Generated sketches as a basis for rendering.

Scenery : too simple?

Lets first succeed in understanding those images, following the biological visual system evolution

Thank You For Your Time

Documents

Non-Local Characterization of Scenery Images: Statistics, 3D Reasoning, and a Generative Model Tamar Avraham and Michael Lindenbaum Technion