Colors and sensors Slides from Bill Freeman, Fredo Durand, Rob Fergus, and David Forsyth, Alyosha...

Colors and sensors

Slides from Bill Freeman, Fredo Durand, Rob Fergus, and David Forsyth, Alyosha Efros

Agenda

• Project 1 delayed till Thursday October 9th

• Color• Sensors• Matlab intro

Project 1: Demosaicing

• Warning; it might take some time– Getting familiar with Matlab– Writeup solutions in html and submit to EEE– I put up a project template (with sample code &

writeup)

Image Formation

Digital Camera

The Eye

Film

Digital camera

• A digital camera replaces film with a sensor array– Each cell in the array is light-sensitive diode that converts photons to electrons– Two common types

• Charge Coupled Device (CCD) • CMOS

– http://electronics.howstuffworks.com/digital-camera.htm

Slide by Steve Seitz

Sensor Array

CMOS sensor

Sampling and quantizing brightness

The real world of colored light: why is color useful?

• Find things to eat

• Spot dangerousthings

What’s the physics behind color?

The Physics of Light

Any patch of light can be completely describedphysically by its spectrum: the number of photons (per time unit) at each wavelength 400 - 700 nm.

400 500 600 700

Wavelength (nm.)

# Photons(per ms.)

© Stephen E. Palmer, 2002

The Physics of Light

.

# P

ho

ton

s

D. Normal Daylight

Wavelength (nm.)

B. Gallium Phosphide Crystal

400 500 600 700

# P

ho

ton

s

Wavelength (nm.)

A. Ruby Laser

400 500 600 700

400 500 600 700

# P

ho

ton

s

C. Tungsten Lightbulb

400 500 600 700

# P

ho

ton

s

Some examples of the spectra of light sources

© Stephen E. Palmer, 2002

The Physics of Light

Some examples of the reflectance spectra of surfaces

Wavelength (nm)

% P

hoto

ns R

efle

cted

Red

400 700

Yellow

400 700

Blue

400 700

Purple

400 700

© Stephen E. Palmer, 2002

Electromagnetic Spectrum

http://www.yorku.ca/eye/photopik.htm

Human Luminance Sensitivity Function

Computer Vision - A Modern Approach

Set: Color Slides by D.A. Forsyth

Black body radiators• Construct a hot body with near-zero albedo (black body)

– Easiest way to do this is to build a hollow metal object with a tiny hole in it, and look at the hole.

• The spectral power distribution of light leaving this object is a simple function of temperature

• This leads to the notion of “color temperature” --- the temperature of a black body that would look the same

€

E λ( )∝1λ5 ⎛ ⎝

⎞ ⎠

1exp hc kλT( ) −1

⎛

⎝ ⎜

⎞

⎠ ⎟

Why do we see light of these wavelengths?

© Stephen E. Palmer, 2002

.

0 1000 2000 3000

En

erg

y

Wavelength (nm)

400 700

700 C

2000 C

5000 C

10000 C

VisibleRegion

…because that’s where theSun radiates EM energy

Visible Light

Plank’s law for Blackbody radiationSurface of the sun: ~5800K

The Psychophysical Correspondence

There is no simple functional description for the perceivedcolor of all lights under all viewing conditions, but …...

A helpful constraint: Consider only physical spectra with normal distributions

area

Wavelength (nm.)

# Photons

400 700500 600

mean

variance

© Stephen E. Palmer, 2002

The Psychophysical Correspondence

Mean Hue

yellowgreenblue

# P

hoto

ns

Wavelength

© Stephen E. Palmer, 2002

The Psychophysical Correspondence

Variance Saturation

Wavelength

high

medium

low

hi.

med.

low# P

hoto

ns

© Stephen E. Palmer, 2002

The Psychophysical Correspondence

Area Brightness#

Pho

tons

Wavelength

B. Area Lightness

bright

dark

© Stephen E. Palmer, 2002

Spectral Image Formation

I(λ)

R(λ)

Si(λ)

I(λ) R(λ)

From Foundation of Vision by Brian Wandell, Sinauer Associates, 1995

Spectral Image Formation

I(λ) – Illumination Spectrum

R(λ) - Surface reflectance/transmission

Si(λ) - Spectral sensitivity of photoreceptor i

Pixel value / Perceived color depends on all 3 terms! Problem of color constancy

Pi (λ) = I(λ)R(λ)Si (λ)

Color appearance depends on nearby colors

Top pink should look stronger

Color names for cartoon spectra

400 500 600 700 nm

400 500 600 700 nm

400 500 600 700 nm

red

gree

nbl

ue

400 500 600 700 nm

cyan

mag

enta

yell

ow

400 500 600 700 nm

400 500 600 700 nm

Slide credit:W. Freeman

Additive color mixing

400 500 600 700 nm

400 500 600 700 nm

red

gree

n

Red and green make…

400 500 600 700 nm

yell

ow

Yellow!

When colors combine by adding the color spectra. Example color displays that follow this mixing rule: CRT phosphors, multiple projectors aimed at a screen, Polachrome slide film.

Slide credit:W. Freeman

Additive color mixing of illuminants

Subtractive color mixing

When colors combine by multiplying the color spectra. Examples that follow this mixing rule: most photographic films, paint, cascaded optical filters, crayons.

400 500 600 700 nm

cyan

yell

ow

400 500 600 700 nm

Cyan and yellow (in crayons,called “blue” and yellow) make…

400 500 600 700 nmGreen!gr

een

Slide credit:W. Freeman

Subtractive color mixing of materials

• Light reflecting off colored object

• E.g. printing inks

Wikipedia

Color matching experiment

Foundations of Vision, by Brian Wandell, Sinauer Assoc., 1995

Color matching experiment 1

Color matching experiment 1

p1 p2 p3

Color matching experiment 1

p1 p2 p3

Color matching experiment 1

p1 p2 p3

The primary color amounts needed for a match

Color matching experiment 2

Color matching experiment 2

p1 p2 p3

Color matching experiment 2

p1 p2 p3

Color matching experiment 2

p1 p2 p3 p1 p2 p3

We say a “negative” amount of p2 was needed to make the match, because we added it to the test color’s side.

The primary color amounts needed for a match:

p1 p2 p3

Measure color by color-matching paradigm

• Pick a set of 3 primary color lights.• Find the amounts of each primary, e1, e2, e3, needed

to match some spectral signal, t.• Those amounts, e1, e2, e3, describe the color of t. If

you have some other spectral signal, s, and s matches t perceptually, then e1, e2, e3 will also match s, by Grassman’s laws.

• Why this is useful—it lets us:– Predict the color of a new spectral signal– Translate to representations using other primary lights.

Goal: compute the color match for any color signal for any set of primary colors

• Examples of why you’d want to do that:– Want to paint a carton of Kodak film with the Kodak

yellow color.

– Want to match skin color of a person in a photograph printed on an ink jet printer to their true skin color.

– Want the colors in the world, on a monitor, and in a print format to all look the same.

Color matching functions for a particular set of monochromatic primaries

p1 = 645.2 nmp2 = 525.3 nmp3 = 444.4 nm

Foundations of Vision, by Brian Wandell, Sinauer Assoc., 1995

Questions?

Some other color spaces…

NTSC color components: Y, I, Q

B

G

R

Q

I

Y

312.0523.0211.0

322.0274.0596.0

114.0587.0299.0

NTSC - RGB

HSV hexcone

Forsyth & Ponce

Hue Saturation Value• Value: from black to white

• Hue: dominant color (red, orange, etc)

• Saturation: from gray to vivid color

• HSV double cone

value

saturation

saturation

hue

CCD color sampling

The eye’s approach to color imaging

Cross-section of eye

Ganglion cell layer

Bipolar cell layer

Receptor layer

Pigmentedepithelium

Ganglion axons

Cross section of retina

© Stephen E. Palmer, 2002

Cones cone-shaped less sensitive operate in high light color vision

Two types of light-sensitive receptors

cone

rod

Rods rod-shaped highly sensitive operate at night gray-scale vision

Human eye photoreceptor spectral sensitivities

Foundations of Vision, by Brian Wandell, Sinauer Assoc., 1995

What colors would these look like?

Color Sensing in Camera (RGB)• 3-chip vs. 1-chip: quality vs. cost

• Why more green?

http://www.cooldihttp://www.cooldictionary.com/words/Bayer-filter.wikipediationary.com/words/Bayer-filter.wikipedia

Why 3 colors?

Slide by Steve Seitz

CCD color filter pattern

detector

The cause of color moire

detector

Fine black and white detail in imagemis-interpreted as color information.

Typical color moire patterns

Blow-up of electronic cameraimage. Notice spuriouscolors in the regionsof fine detail in the plants.

Color sampling artifacts

Human receptors vs CCD sensors

Distribution of incoming luminance into CCD sensors

Gamma correction

• Iout = Iin^(gamma), where gamma < 1