CS 1699: Intro to Computer Vision...

CS 1699: Intro to Computer Vision

Color

Prof. Adriana KovashkaUniversity of Pittsburgh

September 22, 2015

Today

• Review: SIFT features

• Physics and perception of color

• Color matching

• Color spaces

• Uses of color in computer vision

Announcement

• Homework 2 released 9/17

• Small changes made 9/18

• Homework 1 due tonight at 11:59pm

• Review late policy

• Reminder: Do not look for or use existing implementations

Harris Detector: Summary

• Compute image gradients Ix and Iy for all pixels

• For each pixel– Compute

by looping over neighbors x, y

– compute

• Find points with large corner response function R (R > threshold)

• Take the points of locally maximum R as the detected feature points (i.e., pixels where R is bigger than for all the 4 or 8 neighbors).

4D. Frolova, D. Simakov

(k :empirical constant, k = 0.04-0.06)

K. Grauman

Example of Harris application

Local Descriptors: SIFT Descriptor

[Lowe, ICCV 1999]

Histogram of oriented

gradients

• Captures important texture

information

• Robust to small translations /

affine deformationsK. Grauman, B. Leibe

Computing gradients

• tan(α)= 𝑜𝑝𝑝𝑜𝑠𝑖𝑡𝑒 𝑠𝑖𝑑𝑒

𝑎𝑑𝑗𝑎𝑐𝑒𝑛𝑡 𝑠𝑖𝑑𝑒

Gradients

m(x, y) = sqrt(1 + 0) = 1Θ(x, y) = atan(0/1) = 0

Gradients

m(x, y) = sqrt(0 + 1) = 1Θ(x, y) = atan(1/0) = 90

Gradients

m(x, y) = sqrt(1 + 1) = 1.41Θ(x, y) = atan(1/1) = 45

Basic idea:

• Take 16x16 square window around detected feature

• Compute gradient orientation for each pixel

• Create histogram over edge orientations weighted by magnitude

Scale Invariant Feature Transform

L. Zitnick, adapted from D. Lowe

0 2angle histogram

SIFT descriptor

Full version• Divide the 16x16 window into a 4x4 grid of cells (2x2 case shown below)

• Compute an orientation histogram for each cell

• 16 cells * 8 orientations = 128 dimensional descriptor

L. Zitnick, adapted from D. Lowe

Full version• Divide the 16x16 window into a 4x4 grid of cells (2x2 case shown below)

• Compute an orientation histogram for each cell

• 16 cells * 8 orientations = 128 dimensional descriptor

• Threshold normalize the descriptor:

SIFT descriptor

0.2

such that:

L. Zitnick, adapted from D. Lowe

CSE 576: Computer Vision

Making descriptor rotation invariant

Image from Matthew Brown

• Rotate patch according to its dominant gradient

orientation

• This puts the patches into a canonical orientation.

K. Grauman

Examples of Using SIFT

Examples of Using SIFT

Today

• Review: SIFT features

• Physics and perception of color

• Color matching

• Color spaces

• Uses of color in computer vision

Color and light

• Color of light arriving at camera depends on

– Spectral reflectance of the surface light is

leaving

– Spectral radiance of light falling on that patch

• Color perceived depends on

– Physics of light

– Visual system receptors

– Brain processing, environment

K. Grauman

The Eye

The human eye is a camera!• Lens - changes shape by using ciliary muscles (to focus on objects

at different distances)

• Pupil - the hole (aperture) whose size is controlled by the iris

• Iris - colored annulus with radial muscles

• Retina - photoreceptor cells

Slide by Steve Seitz

Retina up-close

Light

D. Hoiem

Color Sensing in Cameras: Bayer Grid

Estimate RGB at each cell

from neighboring values

http://en.wikipedia.org/wiki/Bayer_filter Slide by Steve Seitz

© Stephen E. Palmer, 2002

Cones

cone-shaped

less sensitive

operate in high light

color vision

Two types of light-sensitive receptors

Rods

rod-shaped

highly sensitive

operate at night

gray-scale vision

slower to respond

Slide Credit: Efros

Rod / Cone Sensitivity

Slide Credit: Efros

© Stephen E. Palmer, 2002

Distribution of Rods and Cones

Night Sky: why are there more stars off-center?

.

0

150,000

100,000

50,000

020 40 60 8020406080

Visual Angle (degrees from fovea)

Rods

Cones Cones

Rods

FoveaBlindSpot

# R

ece

pto

rs/m

m2

- Rods responsible for

intensity

- Cones responsible for

color

- Fovea: small region (1

or 2°) at the center of the

visual field containing

the highest density of

cones (and no rods).

– Less visual acuity in the

periphery

Adapted from A. Efros, K. Grauman, S. Seitz, P. Duygulu

Image credit: nasa.gov

Electromagnetic spectrum

Human Luminance Sensitivity Function

K. Grauman

The Physics of Light

Any patch of light can be completely described

physically by its spectrum: the number of photons

(per time unit) at each wavelength 400 - 700 nm.

400 500 600 700

Wavelength (nm.)

# Photons(per ms.)

© Stephen E. Palmer, 2002

The Physics of Light

.

# P

hoto

ns

D. Normal Daylight

Wavelength (nm.)

B. Gallium Phosphide Crystal

400 500 600 700

# P

hoto

ns

Wavelength (nm.)

A. Ruby Laser

400 500 600 700

400 500 600 700

# P

hoto

ns

C. Tungsten Lightbulb

400 500 600 700

# P

hoto

ns

Some examples of the spectra of light sources

© Stephen E. Palmer, 2002

The Physics of Light

Some examples of the reflectance spectra of surfaces

Wavelength (nm)

% P

hoto

ns R

eflecte

d

Red

400 700

Yellow

400 700

Blue

400 700

Purple

400 700

© Stephen E. Palmer, 2002

© Stephen E. Palmer, 2002

.

400 450 500 550 600 650

RE

LA

TIV

E A

BS

OR

BA

NC

E (

%)

WAVELENGTH (nm.)

100

50

440

S

530 560 nm.

M L

Three kinds of cones:

Physiology of Color Vision

• Why are M and L cones so close?

3 is better than 2…

• “M” and “L” on the X-chromosome– Why men are more likely to be color blind (see what it’s like:

http://www.vischeck.com/vischeck/vischeckURL.php)

• “L” has high variation, so some women are tetrachromatic

• Some animals have 1 (night animals), 2 (e.g., dogs), 4 (fish, birds), 5 (pigeons, some reptiles/amphibians), or even 12 (mantis shrimp)

http://en.wikipedia.org/wiki/Color_visionD. Hoiem

Human photoreceptors

Possible

evolutionary

pressure for

developing

receptors for

different

wavelengths in

primates

Osorio & Vorobyev, 1996

K. Grauman

Measuring spectra

Foundations of Vision, B. Wandell

Spectroradiometer: separate input light

into its different wavelengths, and measure

the energy at each.

K. Grauman

Spectral

reflectances for

some natural

objects: how much

of each wavelength

is reflected for that

surface

Forsyth & Ponce, measurements by E. Koivisto

Metamers

K. Grauman

We don’t perceive a spectrum (or even RGB)

• We perceive – Hue: mean wavelength, color

– Saturation: variance, vividness

– Intensity: total amount of light

• Same perceived color can be recreated with combinations of three primary colors (“trichromacy”)

D. Hoiem

Color mixing

Source: W. Freeman

Cartoon spectra for color names:

Additive color mixing

Colors combine by

adding color spectra

Light adds to black.

Source: W. Freeman

Examples of additive color systems

http://www.jegsworks.com

http://www.crtprojectors.co.uk/

CRT phosphors multiple projectors

K. Grauman

Subtractive color mixing

Colors combine by

multiplying color

spectra.

Pigments remove

color from incident

light (white).

Source: W. Freeman

Examples of subtractive color systems

• Printing on paper

• Crayons

• Most photographic film

K. Grauman

Fun with color!

Chromatic adaptation

http://www.planetperplex.com/en/color_illusions.htmlK. Grauman

Chromatic adaptation

http://www.planetperplex.com/en/color_illusions.htmlK. Grauman

Brightness perception

Edward Adelson

http://web.mit.edu/persci/people/adelson/illusions_demos.html

K. Grauman

Edward Adelson

http://web.mit.edu/persci/people/adelson/illusions_demos.html

K. Grauman

Edward Adelson

http://web.mit.edu/persci/people/adelson/illusions_demos.html

K. Grauman

Color constancy

• Interpret surface in terms of “true color”, rather than observed intensity– Humans are good at it

– Computers are not nearly as good

D. Hoiem

• Content © 2008 R.Beau Lotto

• http://www.lottolab.org/articles/illusionsoflight.asp

Look at blue

squares

Look at yellow

squares

K. Grauman

• Content © 2008 R.Beau Lotto

• http://www.lottolab.org/articles/illusionsoflight.asp

K. Grauman

• Content © 2008 R.Beau Lotto

• http://www.lottolab.org/articles/illusionsoflight.asp

K. Grauman

• Content © 2008 R.Beau Lotto

• http://www.lottolab.org/articles/illusionsoflight.asp

K. Grauman

• Content © 2008 R.Beau Lotto

• http://www.lottolab.org/articles/illusionsoflight.asp

K. Grauman

• Content © 2008 R.Beau Lotto

• http://www.lottolab.org/articles/illusionsoflight.asp

K. Grauman

Name that color

High level interactions affect perception and processing.

K. Grauman

http://www.math.unt.edu/~tam/selftests/stroopeffects.html

Reasons for illusions

• Chromatic adaptation: we adapt to a particular

illuminant

• Assimilation, contrast effects, chromatic

induction: nearby colors affect what is perceived;

receptor excitations interact across image and

time

• Afterimages: tired receptors produce negative

response

Color matching ~= color appearance

Physics of light ~= perception of light

K. Grauman

Chromatic adaptation

• The visual system changes its sensitivity depending

on the luminances prevailing in the visual field

• The exact mechanism is poorly understood

• Adapting to different brightness levels

• Changing the size of the iris opening (i.e., the aperture)

changes the amount of light that can enter the eye

• Think of walking into a building from full sunshine

• Adapting to different color temperature

• The receptive cells on the retina change their sensitivity

• For example: if there is an increased amount of red light, the

cells receptive to red decrease their sensitivity until the

scene looks white again

• We actually adapt better in brighter scenes: This is why

candlelit scenes still look yellow

http://www.schorsch.com/kbase/glossary/adaptation.htmlK. Grauman

Today

• Review: SIFT features

• Physics and perception of color

• Color matching

• Color spaces

• Uses of color in computer vision

Color matching experiments

• Goal: find out what spectral radiances

produce same response in human

observers

K. Grauman

Color matching experiments

Foundations of Vision, by Brian Wandell, Sinauer Assoc., 1995 After Judd & Wyszecki.

Observer adjusts weight (intensity) for primary

lights (fixed SPD’s) to match appearance of

test light.

Color matching experiments

• Goal: find out what spectral radiances

produce same response in human

observers

• Assumption: simple viewing conditions,

where we say test light alone affects

perception

– Ignoring additional factors for now like

adaptation, complex surrounding scenes, etc.

K. Grauman

Color matching experiment 1

Slide credit:

W. Freeman

Color matching experiment 1

p1 p2 p3Slide credit:

W. Freeman

Color matching experiment 1

p1 p2 p3Slide credit:

W. Freeman

Color matching experiment 1

p1 p2 p3

The primary color

amounts needed

for a match

Slide credit:

W. Freeman

Color matching experiment 2

Slide credit:

W. Freeman

Color matching experiment 2

p1 p2 p3Slide credit:

W. Freeman

Color matching experiment 2

p1 p2 p3Slide credit:

W. Freeman

Color matching experiment 2

p1 p2 p3p1 p2 p3

We say a

“negative”

amount of p2

was needed to

make the match,

because we

added it to the

test color’s side.

The primary color

amounts needed

for a match:

p1 p2 p3

Color matching

• What must we require of the primary lights

chosen?

• How are three numbers enough to represent

entire spectrum?

K. Grauman

Trichromacy

• In color matching experiments, most people

can match any given light with three primaries• Primaries must be independent

• For the same light and same primaries, most

people select the same weights• Exception: color blindness

• Trichromatic color theory• Three numbers seem to be sufficient for encoding color

• Dates back to 18th century (Thomas Young)

L. Lazebnik

Grassman’s laws

• If two test lights can be matched with the same

set of weights, then they match each other: – Suppose A = u1 P1 + u2 P2 + u3 P3 and B = u1 P1 + u2 P2 + u3 P3.

Then A = B.

• If we scale the test light, then the matches get

scaled by the same amount:– Suppose A = u1 P1 + u2 P2 + u3 P3.

Then kA = (ku1) P1 + (ku2) P2 + (ku3) P3.

• If we mix two test lights, then mixing the

matches will match the result (superposition):– Suppose A = u1 P1 + u2 P2 + u3 P3 and B = v1 P1 + v2 P2 + v3 P3.

Then A+B = (u1+v1) P1 + (u2+v2) P2 + (u3+v3) P3.

Here “=“ means “matches”.

K. Grauman

Computing color matches

• How do we compute the

weights that will yield a

perceptual match for any test

light using a given set of

primaries?

1. Select primaries

2. Estimate their color matching

functions: observer matches

series of monochromatic lights,

one at each wavelength

)()(

)()(

)()(

313

212

111

N

N

N

cc

cc

cc

C

…

…

…

K. Grauman

Foundations of Vision, by Brian Wandell, Sinauer Assoc., 1995 Slide credit: W. Freeman

Color matching functions for a

particular set of primaries

p1 = 645.2 nm

p2 = 525.3 nm

p3 = 444.4 nm

Rows of matrix C

Computing color matches

)(),(),( 321 iii ccc i matches

Now have matching functions for all monochromatic light sources,

so we know how to match a unit of each wavelength.

)(

)( 1

Nt

t

t

…

Arbitrary new spectral signal is a linear combination of the

monochromatic sources.

t

Computing color matches

K. Grauman

Fig from B. Wandell, 1996

Computing color matches

So, given any set of primaries and their associated

matching functions (C), we can compute weights (e)

needed on each primary to give a perceptual match to

any test light t (spectral signal).

K. Grauman

• Why is computing the color match for any color signal for a given set of primaries useful?

– Want to paint a carton of Kodak film with the Kodak yellow color.

– Want to match skin color of a person in a photograph printed on an ink jet printer to their true skin color.

– Want the colors in the world, on a monitor, and in a print format to all look the same.

Adapted from W. Freeman

Computing color matches

Image credit: pbs.org

Today

• Review: SIFT features

• Physics and perception of color

• Color matching

• Color spaces

• Uses of color in computer vision

Why specify color numerically?

• Accurate color reproduction is commercially valuable

– Many products are identified by color (“golden” arches)

• Few color names are widely recognized by English

speakers

– 11: black, blue, brown, grey, green, orange, pink, purple, red,

white, and yellow.

– Other languages have fewer/more.

– Common to disagree on appropriate color names.

• Color reproduction problems increased by prevalence of

digital imaging – e.g. digital libraries of art.

– How to ensure that everyone perceives the same color?

Forsyth & Ponce

Standard color spaces

• Use a common set of primaries/color

matching functions

• Linear color space examples

– RGB

– CIE XYZ

• Non-linear color space

– HSV

K. Grauman

Linear color spaces

• Defined by a choice of three primaries

• The coordinates of a color are given by the

weights of the primaries used to match it

mixing two lights produces

colors that lie along a straight

line in color space

mixing three lights produces

colors that lie within the triangle

they define in color space

L. Lazebnik

Color spaces: RGB

0,1,0

0,0,1

1,0,0

Image from: http://en.wikipedia.org/wiki/File:RGB_color_solid_cube.png

Some drawbacks• Strongly correlated channels

• Non-perceptual

Default color space

R(G=0,B=0)

G(R=0,B=0)

B(R=0,G=0)

D. Hoiem

Trichromacy and CIE-XYZ

Perceptual equivalents with RGB Perceptual equivalents with CIE-XYZ

D. Hoiem

Color Space: CIE-XYZ

RGB portion is in triangleD. Hoiem

Color Space: CIE-XYZ

Perceptual uniformity

D. Hoiem

Distances in color space

• Are distances between points in a color

space perceptually meaningful?

K. Grauman

Distances in color space

• Not necessarily: CIE XYZ is not a uniform color

space, so magnitude of differences in coordinates

are poor indicator of color “distance”.

McAdam ellipses:

Just noticeable differences in colorK. Grauman

• Attempt to correct this

limitation by remapping

color space so that just-

noticeable differences

are contained by circles

distances more

perceptually meaningful.

• Examples:

– CIE u’v’

– CIE Lab

Uniform color spaces

CIE XYZ

CIE u’v’

K. Grauman

Color spaces: CIE L*a*b*

“Perceptually uniform” color space

L(a=0,b=0)

a(L=65,b=0)

b(L=65,a=0)

Luminance = brightness

Chrominance = colorD. Hoiem

Color spaces: HSV

Intuitive color space

H(S=1,V=1)

S(H=1,V=1)

V(H=1,S=0)

D. Hoiem

HSV color space

• Hue, Saturation, Value

(Brightness)

• Nonlinear – reflects topology of

colors by coding hue as an

angle

• Matlab: hsv2rgb, rgb2hsv.

Image from mathworks.comK. Grauman

Today

• Review: SIFT features

• Physics and perception of color

• Color matching

• Color spaces

• Uses of color in computer vision

R G B

• Color histograms:

Use distribution of

colors to describe

image

• No spatial info –

invariant to

translation,

rotation, scale

Color intensity

Pix

el

co

un

ts

Color as a low-level cue for CBIR

K. Grauman

R G B

n

i

ii yxyxI1

,min),(

Given two histogram

vectors, sum the

minimum counts per bin:

= [2, 0, 3]

= [1, 3, 5]

[ 1, 0, 3 ]

K. Grauman

Color as a low-level cue for CBIR

Color-based image retrieval

• Given collection (database) of images:

– Extract and store one color histogram per image

• Given new query image:

– Extract its color histogram

– For each database image:

• Compute intersection between query histogram and

database histogram

– Sort intersection values (highest score = most similar)

– Rank database items relative to query based on this

sorted order

K. Grauman

Color-based image retrieval

Example database

K. Grauman

Example retrievals

Color-based image retrieval

K. Grauman

Example retrievals

Color-based image retrieval

K. Grauman

Color histograms for image matching

http://labs.ideeinc.com/multicolrSource: L. Lazebnik

Color-based image retrieval

K. Grauman

Color-based skin detection

M. Jones and J. Rehg, Statistical Color Models with Application to Skin

Detection, IJCV 2002.K. Grauman

Color-based segmentation

for robot soccer

Towards Eliminating Manual Color Calibration at RoboCup. Mohan

Sridharan and Peter Stone. RoboCup-2005: Robot Soccer World Cup IX,

Springer Verlag, 2006

http://www.cs.utexas.edu/users/AustinVilla/?p=research/auto_vis

K. Grauman

Summary

• Color perception differs from the physics

of color

• Various color spaces exist, with different

strengths and weaknesses

• Color has limited application in computer

vision

Next time

• Segmentation

• Clustering

[Figure by J. Shi]