Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
CS 1699: Intro to Computer Vision
Color
Prof. Adriana KovashkaUniversity of Pittsburgh
September 22, 2015
Today
• Review: SIFT features
• Physics and perception of color
• Color matching
• Color spaces
• Uses of color in computer vision
Announcement
• Homework 2 released 9/17
• Small changes made 9/18
• Homework 1 due tonight at 11:59pm
• Review late policy
• Reminder: Do not look for or use existing implementations
Harris Detector: Summary
• Compute image gradients Ix and Iy for all pixels
• For each pixel– Compute
by looping over neighbors x, y
– compute
• Find points with large corner response function R (R > threshold)
• Take the points of locally maximum R as the detected feature points (i.e., pixels where R is bigger than for all the 4 or 8 neighbors).
4D. Frolova, D. Simakov
(k :empirical constant, k = 0.04-0.06)
K. Grauman
Example of Harris application
Local Descriptors: SIFT Descriptor
[Lowe, ICCV 1999]
Histogram of oriented
gradients
• Captures important texture
information
• Robust to small translations /
affine deformationsK. Grauman, B. Leibe
Computing gradients
• tan(α)= 𝑜𝑝𝑝𝑜𝑠𝑖𝑡𝑒 𝑠𝑖𝑑𝑒
𝑎𝑑𝑗𝑎𝑐𝑒𝑛𝑡 𝑠𝑖𝑑𝑒
Gradients
m(x, y) = sqrt(1 + 0) = 1Θ(x, y) = atan(0/1) = 0
Gradients
m(x, y) = sqrt(0 + 1) = 1Θ(x, y) = atan(1/0) = 90
Gradients
m(x, y) = sqrt(1 + 1) = 1.41Θ(x, y) = atan(1/1) = 45
Basic idea:
• Take 16x16 square window around detected feature
• Compute gradient orientation for each pixel
• Create histogram over edge orientations weighted by magnitude
Scale Invariant Feature Transform
L. Zitnick, adapted from D. Lowe
0 2angle histogram
SIFT descriptor
Full version• Divide the 16x16 window into a 4x4 grid of cells (2x2 case shown below)
• Compute an orientation histogram for each cell
• 16 cells * 8 orientations = 128 dimensional descriptor
L. Zitnick, adapted from D. Lowe
Full version• Divide the 16x16 window into a 4x4 grid of cells (2x2 case shown below)
• Compute an orientation histogram for each cell
• 16 cells * 8 orientations = 128 dimensional descriptor
• Threshold normalize the descriptor:
SIFT descriptor
0.2
such that:
L. Zitnick, adapted from D. Lowe
CSE 576: Computer Vision
Making descriptor rotation invariant
Image from Matthew Brown
• Rotate patch according to its dominant gradient
orientation
• This puts the patches into a canonical orientation.
K. Grauman
Examples of Using SIFT
Examples of Using SIFT
Today
• Review: SIFT features
• Physics and perception of color
• Color matching
• Color spaces
• Uses of color in computer vision
Color and light
• Color of light arriving at camera depends on
– Spectral reflectance of the surface light is
leaving
– Spectral radiance of light falling on that patch
• Color perceived depends on
– Physics of light
– Visual system receptors
– Brain processing, environment
K. Grauman
The Eye
The human eye is a camera!• Lens - changes shape by using ciliary muscles (to focus on objects
at different distances)
• Pupil - the hole (aperture) whose size is controlled by the iris
• Iris - colored annulus with radial muscles
• Retina - photoreceptor cells
Slide by Steve Seitz
Retina up-close
Light
D. Hoiem
Color Sensing in Cameras: Bayer Grid
Estimate RGB at each cell
from neighboring values
http://en.wikipedia.org/wiki/Bayer_filter Slide by Steve Seitz
© Stephen E. Palmer, 2002
Cones
cone-shaped
less sensitive
operate in high light
color vision
Two types of light-sensitive receptors
Rods
rod-shaped
highly sensitive
operate at night
gray-scale vision
slower to respond
Slide Credit: Efros
Rod / Cone Sensitivity
Slide Credit: Efros
© Stephen E. Palmer, 2002
Distribution of Rods and Cones
Night Sky: why are there more stars off-center?
.
0
150,000
100,000
50,000
020 40 60 8020406080
Visual Angle (degrees from fovea)
Rods
Cones Cones
Rods
FoveaBlindSpot
# R
ece
pto
rs/m
m2
- Rods responsible for
intensity
- Cones responsible for
color
- Fovea: small region (1
or 2°) at the center of the
visual field containing
the highest density of
cones (and no rods).
– Less visual acuity in the
periphery
Adapted from A. Efros, K. Grauman, S. Seitz, P. Duygulu
Image credit: nasa.gov
Electromagnetic spectrum
Human Luminance Sensitivity Function
K. Grauman
The Physics of Light
Any patch of light can be completely described
physically by its spectrum: the number of photons
(per time unit) at each wavelength 400 - 700 nm.
400 500 600 700
Wavelength (nm.)
# Photons(per ms.)
© Stephen E. Palmer, 2002
The Physics of Light
.
# P
hoto
ns
D. Normal Daylight
Wavelength (nm.)
B. Gallium Phosphide Crystal
400 500 600 700
# P
hoto
ns
Wavelength (nm.)
A. Ruby Laser
400 500 600 700
400 500 600 700
# P
hoto
ns
C. Tungsten Lightbulb
400 500 600 700
# P
hoto
ns
Some examples of the spectra of light sources
© Stephen E. Palmer, 2002
The Physics of Light
Some examples of the reflectance spectra of surfaces
Wavelength (nm)
% P
hoto
ns R
eflecte
d
Red
400 700
Yellow
400 700
Blue
400 700
Purple
400 700
© Stephen E. Palmer, 2002
© Stephen E. Palmer, 2002
.
400 450 500 550 600 650
RE
LA
TIV
E A
BS
OR
BA
NC
E (
%)
WAVELENGTH (nm.)
100
50
440
S
530 560 nm.
M L
Three kinds of cones:
Physiology of Color Vision
• Why are M and L cones so close?
3 is better than 2…
• “M” and “L” on the X-chromosome– Why men are more likely to be color blind (see what it’s like:
http://www.vischeck.com/vischeck/vischeckURL.php)
• “L” has high variation, so some women are tetrachromatic
• Some animals have 1 (night animals), 2 (e.g., dogs), 4 (fish, birds), 5 (pigeons, some reptiles/amphibians), or even 12 (mantis shrimp)
http://en.wikipedia.org/wiki/Color_visionD. Hoiem
Human photoreceptors
Possible
evolutionary
pressure for
developing
receptors for
different
wavelengths in
primates
Osorio & Vorobyev, 1996
K. Grauman
Measuring spectra
Foundations of Vision, B. Wandell
Spectroradiometer: separate input light
into its different wavelengths, and measure
the energy at each.
K. Grauman
Spectral
reflectances for
some natural
objects: how much
of each wavelength
is reflected for that
surface
Forsyth & Ponce, measurements by E. Koivisto
Metamers
K. Grauman
We don’t perceive a spectrum (or even RGB)
• We perceive – Hue: mean wavelength, color
– Saturation: variance, vividness
– Intensity: total amount of light
• Same perceived color can be recreated with combinations of three primary colors (“trichromacy”)
D. Hoiem
Color mixing
Source: W. Freeman
Cartoon spectra for color names:
Additive color mixing
Colors combine by
adding color spectra
Light adds to black.
Source: W. Freeman
Examples of additive color systems
http://www.jegsworks.com
http://www.crtprojectors.co.uk/
CRT phosphors multiple projectors
K. Grauman
Subtractive color mixing
Colors combine by
multiplying color
spectra.
Pigments remove
color from incident
light (white).
Source: W. Freeman
Examples of subtractive color systems
• Printing on paper
• Crayons
• Most photographic film
K. Grauman
Fun with color!
Chromatic adaptation
http://www.planetperplex.com/en/color_illusions.htmlK. Grauman
Chromatic adaptation
http://www.planetperplex.com/en/color_illusions.htmlK. Grauman
Brightness perception
Edward Adelson
http://web.mit.edu/persci/people/adelson/illusions_demos.html
K. Grauman
Edward Adelson
http://web.mit.edu/persci/people/adelson/illusions_demos.html
K. Grauman
Edward Adelson
http://web.mit.edu/persci/people/adelson/illusions_demos.html
K. Grauman
Color constancy
• Interpret surface in terms of “true color”, rather than observed intensity– Humans are good at it
– Computers are not nearly as good
D. Hoiem
• Content © 2008 R.Beau Lotto
• http://www.lottolab.org/articles/illusionsoflight.asp
Look at blue
squares
Look at yellow
squares
K. Grauman
• Content © 2008 R.Beau Lotto
• http://www.lottolab.org/articles/illusionsoflight.asp
K. Grauman
• Content © 2008 R.Beau Lotto
• http://www.lottolab.org/articles/illusionsoflight.asp
K. Grauman
• Content © 2008 R.Beau Lotto
• http://www.lottolab.org/articles/illusionsoflight.asp
K. Grauman
• Content © 2008 R.Beau Lotto
• http://www.lottolab.org/articles/illusionsoflight.asp
K. Grauman
• Content © 2008 R.Beau Lotto
• http://www.lottolab.org/articles/illusionsoflight.asp
K. Grauman
Name that color
High level interactions affect perception and processing.
K. Grauman
http://www.math.unt.edu/~tam/selftests/stroopeffects.html
Reasons for illusions
• Chromatic adaptation: we adapt to a particular
illuminant
• Assimilation, contrast effects, chromatic
induction: nearby colors affect what is perceived;
receptor excitations interact across image and
time
• Afterimages: tired receptors produce negative
response
Color matching ~= color appearance
Physics of light ~= perception of light
K. Grauman
Chromatic adaptation
• The visual system changes its sensitivity depending
on the luminances prevailing in the visual field
• The exact mechanism is poorly understood
• Adapting to different brightness levels
• Changing the size of the iris opening (i.e., the aperture)
changes the amount of light that can enter the eye
• Think of walking into a building from full sunshine
• Adapting to different color temperature
• The receptive cells on the retina change their sensitivity
• For example: if there is an increased amount of red light, the
cells receptive to red decrease their sensitivity until the
scene looks white again
• We actually adapt better in brighter scenes: This is why
candlelit scenes still look yellow
http://www.schorsch.com/kbase/glossary/adaptation.htmlK. Grauman
Today
• Review: SIFT features
• Physics and perception of color
• Color matching
• Color spaces
• Uses of color in computer vision
Color matching experiments
• Goal: find out what spectral radiances
produce same response in human
observers
K. Grauman
Color matching experiments
Foundations of Vision, by Brian Wandell, Sinauer Assoc., 1995 After Judd & Wyszecki.
Observer adjusts weight (intensity) for primary
lights (fixed SPD’s) to match appearance of
test light.
Color matching experiments
• Goal: find out what spectral radiances
produce same response in human
observers
• Assumption: simple viewing conditions,
where we say test light alone affects
perception
– Ignoring additional factors for now like
adaptation, complex surrounding scenes, etc.
K. Grauman
Color matching experiment 1
Slide credit:
W. Freeman
Color matching experiment 1
p1 p2 p3Slide credit:
W. Freeman
Color matching experiment 1
p1 p2 p3Slide credit:
W. Freeman
Color matching experiment 1
p1 p2 p3
The primary color
amounts needed
for a match
Slide credit:
W. Freeman
Color matching experiment 2
Slide credit:
W. Freeman
Color matching experiment 2
p1 p2 p3Slide credit:
W. Freeman
Color matching experiment 2
p1 p2 p3Slide credit:
W. Freeman
Color matching experiment 2
p1 p2 p3p1 p2 p3
We say a
“negative”
amount of p2
was needed to
make the match,
because we
added it to the
test color’s side.
The primary color
amounts needed
for a match:
p1 p2 p3
Color matching
• What must we require of the primary lights
chosen?
• How are three numbers enough to represent
entire spectrum?
K. Grauman
Trichromacy
• In color matching experiments, most people
can match any given light with three primaries• Primaries must be independent
• For the same light and same primaries, most
people select the same weights• Exception: color blindness
• Trichromatic color theory• Three numbers seem to be sufficient for encoding color
• Dates back to 18th century (Thomas Young)
L. Lazebnik
Grassman’s laws
• If two test lights can be matched with the same
set of weights, then they match each other: – Suppose A = u1 P1 + u2 P2 + u3 P3 and B = u1 P1 + u2 P2 + u3 P3.
Then A = B.
• If we scale the test light, then the matches get
scaled by the same amount:– Suppose A = u1 P1 + u2 P2 + u3 P3.
Then kA = (ku1) P1 + (ku2) P2 + (ku3) P3.
• If we mix two test lights, then mixing the
matches will match the result (superposition):– Suppose A = u1 P1 + u2 P2 + u3 P3 and B = v1 P1 + v2 P2 + v3 P3.
Then A+B = (u1+v1) P1 + (u2+v2) P2 + (u3+v3) P3.
Here “=“ means “matches”.
K. Grauman
Computing color matches
• How do we compute the
weights that will yield a
perceptual match for any test
light using a given set of
primaries?
1. Select primaries
2. Estimate their color matching
functions: observer matches
series of monochromatic lights,
one at each wavelength
)()(
)()(
)()(
313
212
111
N
N
N
cc
cc
cc
C
…
…
…
K. Grauman
Foundations of Vision, by Brian Wandell, Sinauer Assoc., 1995 Slide credit: W. Freeman
Color matching functions for a
particular set of primaries
p1 = 645.2 nm
p2 = 525.3 nm
p3 = 444.4 nm
Rows of matrix C
Computing color matches
)(),(),( 321 iii ccc i matches
Now have matching functions for all monochromatic light sources,
so we know how to match a unit of each wavelength.
)(
)( 1
Nt
t
t
…
Arbitrary new spectral signal is a linear combination of the
monochromatic sources.
t
Computing color matches
K. Grauman
Fig from B. Wandell, 1996
Computing color matches
So, given any set of primaries and their associated
matching functions (C), we can compute weights (e)
needed on each primary to give a perceptual match to
any test light t (spectral signal).
K. Grauman
• Why is computing the color match for any color signal for a given set of primaries useful?
– Want to paint a carton of Kodak film with the Kodak yellow color.
– Want to match skin color of a person in a photograph printed on an ink jet printer to their true skin color.
– Want the colors in the world, on a monitor, and in a print format to all look the same.
Adapted from W. Freeman
Computing color matches
Image credit: pbs.org
Today
• Review: SIFT features
• Physics and perception of color
• Color matching
• Color spaces
• Uses of color in computer vision
Why specify color numerically?
• Accurate color reproduction is commercially valuable
– Many products are identified by color (“golden” arches)
• Few color names are widely recognized by English
speakers
– 11: black, blue, brown, grey, green, orange, pink, purple, red,
white, and yellow.
– Other languages have fewer/more.
– Common to disagree on appropriate color names.
• Color reproduction problems increased by prevalence of
digital imaging – e.g. digital libraries of art.
– How to ensure that everyone perceives the same color?
Forsyth & Ponce
Standard color spaces
• Use a common set of primaries/color
matching functions
• Linear color space examples
– RGB
– CIE XYZ
• Non-linear color space
– HSV
K. Grauman
Linear color spaces
• Defined by a choice of three primaries
• The coordinates of a color are given by the
weights of the primaries used to match it
mixing two lights produces
colors that lie along a straight
line in color space
mixing three lights produces
colors that lie within the triangle
they define in color space
L. Lazebnik
Color spaces: RGB
0,1,0
0,0,1
1,0,0
Image from: http://en.wikipedia.org/wiki/File:RGB_color_solid_cube.png
Some drawbacks• Strongly correlated channels
• Non-perceptual
Default color space
R(G=0,B=0)
G(R=0,B=0)
B(R=0,G=0)
D. Hoiem
Trichromacy and CIE-XYZ
Perceptual equivalents with RGB Perceptual equivalents with CIE-XYZ
D. Hoiem
Color Space: CIE-XYZ
RGB portion is in triangleD. Hoiem
Color Space: CIE-XYZ
Perceptual uniformity
D. Hoiem
Distances in color space
• Are distances between points in a color
space perceptually meaningful?
K. Grauman
Distances in color space
• Not necessarily: CIE XYZ is not a uniform color
space, so magnitude of differences in coordinates
are poor indicator of color “distance”.
McAdam ellipses:
Just noticeable differences in colorK. Grauman
• Attempt to correct this
limitation by remapping
color space so that just-
noticeable differences
are contained by circles
distances more
perceptually meaningful.
• Examples:
– CIE u’v’
– CIE Lab
Uniform color spaces
CIE XYZ
CIE u’v’
K. Grauman
Color spaces: CIE L*a*b*
“Perceptually uniform” color space
L(a=0,b=0)
a(L=65,b=0)
b(L=65,a=0)
Luminance = brightness
Chrominance = colorD. Hoiem
Color spaces: HSV
Intuitive color space
H(S=1,V=1)
S(H=1,V=1)
V(H=1,S=0)
D. Hoiem
HSV color space
• Hue, Saturation, Value
(Brightness)
• Nonlinear – reflects topology of
colors by coding hue as an
angle
• Matlab: hsv2rgb, rgb2hsv.
Image from mathworks.comK. Grauman
Today
• Review: SIFT features
• Physics and perception of color
• Color matching
• Color spaces
• Uses of color in computer vision
R G B
• Color histograms:
Use distribution of
colors to describe
image
• No spatial info –
invariant to
translation,
rotation, scale
Color intensity
Pix
el
co
un
ts
Color as a low-level cue for CBIR
K. Grauman
R G B
n
i
ii yxyxI1
,min),(
Given two histogram
vectors, sum the
minimum counts per bin:
= [2, 0, 3]
= [1, 3, 5]
[ 1, 0, 3 ]
K. Grauman
Color as a low-level cue for CBIR
Color-based image retrieval
• Given collection (database) of images:
– Extract and store one color histogram per image
• Given new query image:
– Extract its color histogram
– For each database image:
• Compute intersection between query histogram and
database histogram
– Sort intersection values (highest score = most similar)
– Rank database items relative to query based on this
sorted order
K. Grauman
Color-based image retrieval
Example database
K. Grauman
Example retrievals
Color-based image retrieval
K. Grauman
Example retrievals
Color-based image retrieval
K. Grauman
Color histograms for image matching
http://labs.ideeinc.com/multicolrSource: L. Lazebnik
Color-based image retrieval
K. Grauman
Color-based skin detection
M. Jones and J. Rehg, Statistical Color Models with Application to Skin
Detection, IJCV 2002.K. Grauman
Color-based segmentation
for robot soccer
Towards Eliminating Manual Color Calibration at RoboCup. Mohan
Sridharan and Peter Stone. RoboCup-2005: Robot Soccer World Cup IX,
Springer Verlag, 2006
http://www.cs.utexas.edu/users/AustinVilla/?p=research/auto_vis
K. Grauman
Summary
• Color perception differs from the physics
of color
• Various color spaces exist, with different
strengths and weaknesses
• Color has limited application in computer
vision
Next time
• Segmentation
• Clustering
[Figure by J. Shi]