Visuelle Perzeption für Mensch- Maschine Schnittstellen · Visuelle Perzeption für Mensch-Maschine Schnittstellen Vorlesung, ... Forsyth and Jean Ponce ... (Georgia Institute of

Interactive Systems Laboratories, Universität Karlsr uhe (TH)Dr. Edgar Seemann, 23.10.09 1

Visuelle Perzeption für Mensch-Maschine Schnittstellen

Vorlesung, WS 2009

Prof. Dr. Rainer StiefelhagenDr. Edgar Seemann

Institut für AnthropomatikUniversität Karlsruhe (TH)

http://[email protected]

[email protected]

Dr. Edgar Seemann, 23.10.09 2

Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciTermine (1)

2

TBA21.12.2009

Stereo and Optical Flow18.12.2009

Termine Thema19.10.2009 Introduction, Applications

23.10.2009 Basics: Cameras, Transformations, Color

26.10.2009 Basics: Image Processing

30.10.2009 Basics: Pattern recognition

02.11.2009 Computer Vision: Tasks, Challenges, Learning, Performance measures

06.11.2009 Face Detection I: Color, Edges (Birchfield)

09.11.2009 Project 1: Intro + Programming tips13.11.2009 Face Detection II: ANNs, SVM, Viola & Jones

16.11.2009 Project 1: Questions

20.11.2009 Face Recognition I: Traditional Approaches, Eigenfaces, Fisherfaces, EBGM

23.12.2009 Face Recognition II

27.11.2009 Head Pose Estimation: Model-based, NN, Texture Mapping, Focus of Attention

30.11.2009 People Detection I

03.12.2009 People Detection II

07.12.2009 Project 1: Student Presentations, Project 2: Intro11.12.2009 People Detection III (Part-Based Models)

14.12.2009 Scene Context and Geometry


Book Recommendations

� „Computer Vision a Modern Approach“, David Forsyth and Jean Ponce

� „Pattern Classification“, Duda&Hart

� „Multiple View Geometry“, Hartley&Zisserman


Organisatorisches

� Programmier-Aufgaben� Gruppen a 3 Studenten

� Ergebnisse haben Auswirkung auf Gesamtnote

� C++, Okapi � Mehr Details und eine kleine Einführung am 9. November

� Bitte tragt euch in die Liste ein


Programmier-Umgebung

� Ihr könnt mit eurem Lieblings-Editor programmieren

� Empfehlung:� Eclipse (CDT)

� QtCreator

� vi, emacs ;-)

Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ci

Basics:

Cameras, Transformations and Color Spaces

WS 2009/10

Dr. Edgar Seemann

[email protected]


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciToday‘s Overview

� Cameras and� Pinhole Camera Model

� Image formation� perspective projection

� calibration

� Transformations� Euclidian, similarity, affine, projective transformation

� some invariants

� ColorCredits: Bernt Schiele (Darmstadt), Bastian Leibe (Aachen),Kirsten Grauman (University of Texas)David Forsyth (University of Illinois)James Rehg (Georgia Institute of Technology)


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciQuiz

� When was photography invented?

� By whom? � Exposure time?

http://www.digicamhistory.com


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciQuiz

� When was photography invented? 1825

� By whom? Niépce� Heliography

� Exposure time? 8 hours

� William Henry Fox Talbot inventsthe calotype in 1834 which prettymuch invents the negative


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciFirst production camera?


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciFirst production camera?

� 1839. Daguerrotype


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciBeginning of hobby photography?

� 1900 Kodak Brownie


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciQuiz

� Who did the first color photography?

� When?


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciQuiz

� Who did the first color photography? � James Cleark Maxwell (also known

from the Maxwell equations) When? 1861

� Oldest color photos still preserved: Prokudin-Gorskiihttp://www.loc.gov/exhibits/empire/


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciProkudin-Gorskii

� Digital restoration


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciProkudin-Gorskii


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciProkudin-Gorskii


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciInstant photography?


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciInstant photography?

� 1947, Edwin Land (Polaroid founder)


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciFirst microprocessor in a camera

� Canon AE-11976


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciFirst digital camera?


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciFirst digital camera?

� 1975, Steve Sasson, Kodak

� Uses ccd from Fairchild semiconductor, A/D from Motorola, .01 megapixels, 23 second exposure, recorded on digital cassette


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciFirst megapixel sensor

� Of reasonable size?


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciFirst megapixel sensor

� Of reasonable size?

� (Kodak) Videk 1987, 1.4MPixels


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciReferences

� http://www.digicamhistory.com/

� http://www.photo.net/history/timeline� http://inventors.about.com/library/inventors/blphotography.htm� http://www.loc.gov/exhibits/empire/� http://www.spartacus.schoolnet.co.uk/USAphotographers.htm� http://www.eyeconart.net/history/photography.htm� http://www.scphoto.com/html/history.html� http://www.g4tv.com/callforhelparchive/features/44534/Witness_to_History_

The_Digital_Camera.html� http://www.digicamhistory.com/� http://www-users.mat.uni.torun.pl/~olka/� http://inventors.about.com/od/pstartinventions/a/Photography.htm� http://www.ted.photographer.org.uk/camera_designs_3.htm� http://accad.osu.edu/~waynec/history/timeline.html

� http://en.wikipedia.org/wiki/History_of_the_single-lens_reflex_camera


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciCameras

� First photograph due to Niépce(1820s)

� First digital photographs (Bell, Texas Instruments, Kodak 1970s)

� Basic abstraction is the pinhole camera� lenses required to ensure image is not too dark

� various other abstractions can be applied


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciPinhole Camera

� (simple) standard and abstract model today� box with a small hole in it

� Pinhole cameras work in practice


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciDiffraction limit

� Optimal size for visible light: sqrt(f)/28 (in millimiters) where f is focal length


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciPinhole too big-many directions areaveraged, blurring theimage

Pinhole too small-diffraction effects blurthe image

Generally, pinhole cameras are dark, becausea very small set of raysfrom a particular pointhits the screen.


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciProperties

� Distant objects are smaller

� Moving the image plane simply scales the image

� Each set of parallel lines (=direction) meets at a different point� The vanishing point for this direction

� Sets of parallel lines on the same plane lead to collinear vanishing points. � The line is called thehorizon for that plane

� Good ways to spot faked images� scale and perspective don’t work

� vanishing points behave badly

� Shadows and ligthing are not consistent


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciPinhole Camera Model

� origin of the coordinate system is O

� the optical axis of the camera is parallel to the z-axis of the coordinate system

� f is the focal length

� the z-coordinate of the image plane is: -f

image planexy

z

f

O


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h


� the image of an object is the projection of the object onto the image plane

� intersection of all projection-lines is the focal point F of the camera� the origin is defined to be the focal point of the camera: F = O

f

xy

z

image plane

F=O


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h


� due to similar triangles and the intercept theorems:

′z = − f′x

− f=

x

z ⇒ ′x =

− fx

z

′y

− f=

y

z ⇒ ′y =

− fy

z

xy

z

f P

′P

P =x

y

z

′P =

′x

′y

′z


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciMathematical Simplification

� translation of the image plane in front of the focal point

′z = f

′x =fx

z

′y =fy

z

xy

z

f P

′P

image plane


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciLinear Transformation

Goal: formulation of the projection as a linear transformation

- i.e. find matrix A such that:

′P =′x

′y

′z

= A

x

y

z

= A P

• Transformation matrix depends on z

• It should therefore be different for each point p

-> NOT WORKING

′z = f

′x =fx

z

′y =fy

z


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciHomogenous Coordinates

� define equivalence class:

′x

′y

′z

→

′x

′y

′z

1

≅

s ′x

s ′y

s ′z

s

=

sfx

z

sfy

zsf

s

=

fx

fy

fz

z

x

y

z

→

x

y

z

1

≅

sx

sy

sz

s

≅


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciMathematical Simplification

′x

′y

′z

→

′x

′y

′z

1

≅

fx

fy

fz

z

=

f 0 0 0

0 f 0 0

0 0 f 0

0 0 1 0

x

y

z

1

fx

fy

fz

z

= A

x

y

z

1


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciCompletion of the Projection

� so far:� projection of a point P onto the image plane

� what’s missing:� pixel-coordinates (u,v) of the projected points

v = − k v ′y + v 0

u = k u ′x + u 0

′x

′y

u

v

u 0

v 0

image plane

With ku and kv scaling factorswhich denote the ratio between world and pixel coordinates


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciCompletion of the Projection

� in matrix notation:

� again in homogeneous coordinates:

u

v

=

k u 0

0 − k v

′x

′y

+

u 0

v 0

u

v

1

=k u 0 0

0 − k v 0

0 0 0

u 0

v 0

1

′x

′y

′z

1


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciComplete Projection

� multiplication of the two projection matrices:

u

v

1

=k u 0 0

0 − k v 0

0 0 0

u 0

v 0

1

f 0 0 0

0 f 0 0

0 0 f 0

0 0 1 0

x

y

z

1

u

v

1

=k u f 0 u 0

0 − k v f v 0

0 0 1

0

0

0

x

y

z

1


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciPerspective Projection

� 4 parameters:

� have to be known to perform the projection

� called ‘internal camera parameters’ since they depend on the camera only

� perform calibration to estimate those parameters

α u uk f=α v vk f= −u 0

v 0


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciCalibration

� calibration object� point-coordinates known in 3D

� left: model of the calibration object

� right: image taken with the camera


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciCalibration

� left: edge detection

� right: extraction of the corner-points of the squares


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciCalibration

� transform the (corner-)points pi of the calibration object into the coordinate system of the camera: � R: rotation (3 degrees of freedom), 3x3 matrix� t: translation (3 degrees of freedom), 3x1 vector

� in homogenous coordinates:

x i

y i

z i

= R p i + t

x i

y i

z i

1

=R t

0 1

p i

1

=

r1 1 r1 2 r13 t x

r2 1 r22 r2 3 t y

r3 1 r32 r3 3 t z

0 0 0 1

p i

1


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciCalibration

� the observation of the projected points are in image-coordinates

� i.e. we have to project the pi onto the image plane:

u i

v i

1

=α u 0 u 0

0 α v v 0

0 0 1

0

0

0

x i

y i

z i

1

x i

y i

z i

1

=R t

0 1

p i

1

=

r1

r 2

r 3

0

t x

t y

t z

1

p i

1


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciCalibration

� the resulting matrix is a 3x4 matrix

� theoretically this matrix has 12 parameters,

� in reality it has only 10 degrees of freedom

� therefore we need (minimally) 5 points to solve the system of linear equations (more points are better…)

su i

sv i

s

=α u 0 u 0

0 α v v 0

0 0 1

0

0

0

r1

r 2

r 3

0

t x

t y

t z

1

p i

1

su i

sv i

s

=α u r1 + u 0 r 3

α v r 2 + v 0 r 3

r 3

α u t x + u 0 t z

α v t y + v 0 t z

t z

p i

1


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciInternal and External Parameters

� internal parameters of the camera:

� external parameters of the cameras:� rotation R

� translation t

α u uk f=α v vk f= −u 0

v 0


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciThe reason for lenses


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciHow to trace rays

� Start by rays through the center


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciHow to trace rays


� Choose focal length, trace parallels

f


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciHow to trace rays


� Choose focal length, trace parallels

� You get the focus plane for a given scene plane� All rays coming from points on a plane parallel to the lens are focused on

another plane parallel to the lens

f


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciFocusing

� To focus closer than infinity� Move the sensor/film further than the focal length

f


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciThin lens formula

fDD’


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciThin lens formula

fDD’


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciThin lens formula

fDD’

Similar triangles everywhere!

y’

y

y’/y = D’/D


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciThin lens formula

fDD’

Similar triangles everywhere!

y’

y

y’/y = D’/D

y’/y = (D’-f)/f


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciThin lens formula

fDD’

1D’ D

1 1f

+ =


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciMinimum focusing distance

� By symmetry, an object at the focal length requires the film/sensor to be at infinity.

Film/sensor

Rays from infinity

Rays from object at f


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciField of view & focusing

� What happens to the field of view when one focuses closer?� It's reduced

film focused at infinity

film focused close


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciFocal length in practice

24mm

50mm

135mm


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciRecap

� Pinhole is the simplest model of image formation

� Lenses gather more light� But get only one plane focused

� Focus by moving sensor/film

� Cannot focus infinitely close

� Focal length determines field of view� From wide angle to telephoto

� Depends on sensor size


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciSpherical aberration


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciLens systems


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciVignetting


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciOther (possibly annoying) phenomena

� Chromatic aberration� Light at different wavelengths follows different paths; hence, some

wavelengths are defocussed� Machines: coat the lens� Humans: live with it

� Scattering at the lens surface� Some light entering the lens system is reflected off each surface it

encounters (Fresnel’s law gives details)� Machines: coat the lens, interior� Humans: live with it (various scattering phenomena are visible in

the human eye)

� Geometric phenomena (Barrel distortion, etc.)


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciA simpler abstraction model

� perspective projection can be approximated by a ‘scaled orthographic’ projection� if the object is far away from the camera (w.r.t. the focal length of

the camera

� then we can neglect the depth of the object (the object is regarded as small and flat)

� (analogous: the focal length of the camera is infinite)

� orthographic projection:� the projection is orthogonal to the image plane

xyP′P


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciScaled Orthographic Projection

� Reason:� if the object is ‘nearly’ planar, then all points on the object have

‘nearly’ the same distance to the camera� the scale-factor of the projection is therefore ‘nearly’ constant for

all points on the object

� this is called the ‘scaled orthographic’ projection� this is quite often a reasonable assumption

′ = =xfx

zs x

00′ =x

fx

z

′ =yfy

z′ = =y

fy

zs y

00


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ci

2D Alignment


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ci

� 2 objects in the database

� new test image - question: which object

Recognition: Example


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciRecognition: Example

� Alignment:� alignment: transform the image such that one object is

in complete correspondence with the other object

� Objects are assumed to be segmented (i.e. we know which pixels belong to the background and which to the object)

� estimate the parameters of the transformation


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciRecognition: Example

� translation:

� rotation:

� scaling and mirroring

� what remains is to compare the 2 (segmented) binary object-masks


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciDegrees of Freedom (DoF) for Objects

� Euclidean transformation� transformations in the image plane� rotation (1 DoF)� translation (2 DoF)

� invariants under these transformations� length - yes� angle - yes� length-ratio - yes� parallelism - yes

� object can be localized with 2 points� 2 points in the image plane have 4 coordinates - which can be used to

determine the 3 DoF


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h


� Similarity transformation:� rotation (1 DoF)� translation (2 DoF)� scaling of the object (1 DoF)

� invariants under these transformations� length - no� angle - yes� length-ratio - yes� parallelism - yes

� object can be localized with 2 points� 2 points in the image plane have 4 coordinates - which can be used to

determine the 4 DoF


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h


� Affine Transformation:� has 6 DoF

� 2x2 = 4 parameters + 2 parameters for translation

� rotation(1), translation (2), scaling (1)

� the remaining 2 DoF are � non-uniform scaling of the axes (1 DoF)

� shear of the axes (1 DoF)


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h


� invariants under affine transformation:� length- no

� angle - no (because of shear)

� length-ratio - no (because of non-uniform scaling)

� parallelism - yes

� object can be localized with 3 points� 3 points in the image plane have 6 coordinates -which

can be used to determine the 6 DoF


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h


� Perspective transformation� adds 2 DoF

� Scaling depending on x-, y-position

� invariants under projective transformation:� length- no

� angle - no

� length-ratio - no

� parallelism - no

� object can be localizedwith 4 points:� 4 points need to determine

the complete coordinate system…


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ci

Color


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciOutline

� Color and Radiometry

� Human Color Perception

� Color spaces

� Lightness and Color Constancy

� Physics-based Vision: Specularities


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciWhy does a visual system need color?

(an incomplete list…)� To tell what food is edible.

� To distinguish material changes from shading changes.

� To group parts of one object together in a scene.

� To find people’s skin.

� Check whether a person’s appearance looks normal/healthy.

� To compress images


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciColor and Radiometry

� What is color ?


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciWhat is Color?

� A perceptual attribute of objects and scenes constructed by the visual system

� A quantity related to the wavelength of light in the visible spectrum

� A challenge� “There are no second-rate brains in color vision”

– Edwin Land


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

cicolor


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciReflectance Model


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciIllumination Spectra

Blue sky Tungsten light bulb


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciReflectance Spectra

Spectral albedoes for several different leaves, with color names attached. Notice that different colourstypically have different spectral albedo, but that different spectral albedoes may result in the same perceived color (compare the two whites). Spectral albedoes are typically quite smooth functions. Measurements by E.Koivisto.


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciColor names for cartoon spectra

400 500 600 700 nm

400 500 600 700 nm

400 500 600 700 nm

red

gree

nbl

ue

400 500 600 700 nm

cyan

mag

enta

yello

w

400 500 600 700 nm

400 500 600 700 nm


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciAdditive color mixing

400 500 600 700 nm

400 500 600 700 nm

red

gree

n

Red and green make…

400 500 600 700 nm

yello

w

Yellow!

When colors combine by adding the color spectra. Example color displays that follow this mixing rule: CRT phosphors, multiple projectors aimed at a screen


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciSubtractive color mixing

When colors combine by multiplying the color spectra. Examples that follow this mixing rule: most photographic films, paint, cascaded optical filters, crayons.

400 500 600 700 nm

cyan

yello

w

400 500 600 700 nm

Cyan and yellow (in crayons,called “blue” and yellow) make…

400 500 600 700 nmGreen!gr

een


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciWhy specify color numerically?

� Accurate color reproduction is commercially valuable � Many products are identified by

color (“golden” arches);

� Few color names are widely recognized by English speakers -� About 10; other languages have

fewer/more, but not many more.

� It’s common to disagree on appropriate color names.

� Color reproduction problems increased by prevalence of digital imaging - eg. digital libraries of art. � How do we ensure that

everyone sees the same color?


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciAn assumption that sneaks in here

� For now we will assume that the spectrum of the light arriving at your eye completely determines the perceived color.

� But we know color appearance really depends on:� The illumination

� Your eye’s adaptation level

� The colors and scene interpretation surrounding the observed color.


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciColor Matching Process

Basis for industrial color standards


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciColor Matching Experiment 1

Image courtesy Bill Freeman


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h




Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h




Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h




Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h




Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h




Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h




Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h




Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciConclusion from Color Matching

� Three primaries are sufficient for most people to reproduce arbitrary colors� The human eye normally contains only three types of

color receptors, called cone cells� Each color receptor responds to different ranges of the

color spectrum. � Humans respond to the light stimulus via a three-

dimensional sensation, which generally can be modeled as a mixture of three primary colors.


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciCaveat: Context Matters !

Figure courtesy ofD. Forsyth


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciPrinciple of Trichromaticity

∑

∑

=

=

=

===

=

3

1

321

1

enough are primaries 3 imply that sexperiment matchingColor

(B) nm 44444 (G), nm 32526 (R), nm 16645

:Example weights.are primaries, are

:generalIn

iii

ii

m

iii

PwT

.P.P.P

wP

PwT


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciGrassman’s Laws

� For color matches:� symmetry: U=V <=>V=U� transitivity: U=V and V=W => U=W� proportionality: U=V <=> tU=tV� additivity: if any two (or more) of the statements

U=V, W=X, (U+W)=(V+X) are true, then so is the third

� These statements are as true as any biological law. They mean that additive color matching is linear.

Forsyth & Ponce


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciMeasure color by color-matching paradigm

� Pick a set of 3 primary color lights.� Find the amounts of each primary, e1, e2, e3, needed to match

some spectral signal, t.� Those amounts, e1, e2, e3, describe the color of t. If you have

some other spectral signal, s, and s matches t perceptually, then e1, e2, e3 will also match s, by Grassman’s laws.

� Why this is useful—it lets us:� Predict the color of a new spectral signal� Translate to representations using other primary lights.


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciHow to compute the color match for any color

signal for any set of primary colors

� Pick a set of primaries,

� Measure the amount of each primary, needed to match a monochromatic light, at each spectral wavelength (pick some spectral step size). These are called the color matching functions.

)(),(),( 321 λλλ ppp

)(),(),( 321 λλλ ccc

)(λtλ


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciColor matching functions for a particular set of

monochromatic primariesp1 = 645.2 nmp2 = 525.3 nmp3 = 444.4 nm

Foundations of Vision, by Brian Wandell, Sinauer Assoc., 1995


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciColor Spaces

� Use color matching functions to define a coordinate system for color.

� Each color can be assigned a triple of coordinates with respect to some color space (e.g. RGB).

� Devices (monitors, printers, projectors) and computers can communicate colors precisely.


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciSuppose you use one set of primaries and I

use another?

� We address this in 2 ways:� Learn how to translate between primaries

� Standardize on a few sets of favored primaries.


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciNTSC color components: Y, I, Q

−−−=

B

G

R

Q

I

Y

312.0523.0211.0

322.0274.0596.0

114.0587.0299.0

� Mathematically, color space conversions of linear color spaces are basis transformations

� Basis transformations can be expressed in terms of a matrix multiplication


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciNTSC - RGB


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciHue Saturation Value

� Attempt to describe the colors’ perceptual relationship

� Related to RGB

� Value: from black to white

� Hue: dominant color (red, orange, etc)

� Saturation: from gray to vivid color

� Exampes:RGB HSV (1, 0, 0) (0°, 1, 1) red(0, 0, 0.5) (240°, 1, 0.5) dark blue


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciRGB <> HSV

� Converting RGB values to HSV


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciLightness and Color Constancy


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciHuman Color Constancy

� Color constancy: hue and saturation

� Lightness constancy: gray-level

� Humans can perceive� Color a surface would have under white light (surface

color)

� Color of reflected light (separate surface color from measured color)

� Color of illuminant (limited)


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciLand’s Mondrian Experiments

� Squares of color with the same color radiance yield very different color perceptions

Photometer: 1.0, 0.3, 0.3 Photometer: 1.0, 0.3, 0.3

Audience: “Red” Audience: “Blue”White light Colored light

Blue

Red

Blue

Red


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciSpecularities

Real scene:

Figure courtesy ofSing Bing Kang

Removed specularities:


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciSummary Color

� Color is not that straight forward as one mighthave guessed

� Color is a significant industry with conferences, standards bodies, etc


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciSelected Bibliography

Vision and Art : The Biology of Seeingby Margaret Livingstone, David H. HubelHarry N Abrams; ISBN: 0810904063 208 pages (May 2002)

Vision Science

by Stephen E. PalmerMIT Press; ISBN: 0262161834760 pages (May 7, 1999)

Billmeyer and Saltzman's Principles of Color Technology, 3rd Editionby Roy S. Berns, Fred W. Billmeyer, Max SaltzmanWiley-Interscience; ISBN: 047119459X304 pages 3 edition (March 31, 2000)


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ciOther color references

� Reading: � Chapter 6, Forsyth & Ponce

� Chapter 4 of Wandell, Foundations of Vision, Sinauer, 1995 has a good treatment of this.

Credits: Bill Freeman


Com

pute

r V

isio

n fo

r H

uman

-Com

pute

r In

tera

ctio

n

Res

earc

h G

roup

, Uni

vers

itätK

arls

ruhe

(T

H)

cv:h

ci

End of Lecture

Documents

Visuelle Perzeption für Mensch- Maschine Schnittstellen · Visuelle Perzeption für Mensch-Maschine Schnittstellen Vorlesung, ... Forsyth and Jean Ponce ... (Georgia Institute of