Fourier Analysis of Video Signals

8/12/2019 Fourier Analysis of Video Signals

1/29

Fourier Analysis of Video Signals

Frequency Response of the HVS

Yao Wang

Polytechnic University, Brooklyn, [email protected]


2/29

Yao Wang, 2003 Frequency Domain Analysis

Outline

Fourier transform over multidimensional space Continuous space FT (CSFT) Discrete space FT (DSFT) Sampled space FT (SSFT)

Frequency domain characterization of video signals

Spatial frequency Temporal frequency Temporal frequency caused by motion

Frequency response of the HVS

Spatial frequency response Temporal frequency response and flicker Spatio-temporal response Smooth pursuit eye movement

Video sampling a brief discussion


3/29

Yao Wang, 2003 Frequency Domain Analysis 3

Continuous Space Signals

K-D Space Signals

Convolution

Example function Delta function

k K R x x x = ],...,,[),( 21xx

yyyxxx d hhk R = )()()(*)(


4/29


Continuous Space Fourier Transform

(CSFT) Forward transform

Inverse transform

Convolution theorem

xxf xf d jk R

T c = )2exp()()(

f xf f x d jk R

T c= )2exp()()(

)(*)()()(

)()()(*)(

f f xx

f f xx

cc

cc

H h

H h


5/29


Continuous Space Systems

General system over K-D continuous space

Linear and Space-Shift Invariant (LSI) System

LSI systems can be completely described by itsimpulse response

k RT = xxx )),(()(

( ))()()()(*)()(

)()(

f f f xxx

xx

ccc H h

T h==

=

)())((

))()(()()(

00

22112211

xxxx

xxxx+=+

+=+

T

T


6/29 Yao Wang, 2003 Frequency Domain Analysis 6

Discrete Space Signals

K-D Space Signals

Convolution

Example function Delta function

K K Z nnn = ],...,,[),( 21nn

= K Z

hhm

mmnnn )()()(*)(



Discrete Space Fourier Transform

(DSFT) Forward transform

Inverse transform

Convolution theorem

{ })2/1,2/1(, : periodlFundamenta1of periodwithdimensioneachin periodicis)(

)2exp()()(

=

=

k K

d

R

T d

f I

j K

f

f

nf nf n

= K

I

T d d j

f

f nf f n )2exp()()(

)(*)()()(

)()()(*)(

f f nn

f f nn

d d

d d

H h

H h


8/29 Yao Wang, 2003 Frequency Domain Analysis

Frequency domain characterization of

video signals Spatial frequency

Temporal frequency Temporal frequency caused by motion



Spatial Frequency

Spatial frequency measures how fast the image

intensity changes in the image plane Spatial frequency can be completely characterized by

the variation frequencies in two orthogonal directions

(e.g horizontal and vertical) f x : cycles/horizontal unit distance f y : cycles/vertical unit distance

It can also be specified by magnitude and angle ofchange

)/arctan(,22 x y y xm f f f f f =+=



Illustration of Spatial Frequency



Angular Frequency

ee)cycle/degr (f 180

f f

(degree)180

n)h/2d(radia(radian))2/arctan(2

s s

hd

d h

d h

==

==

Problem with previous defined spatial frequency:Perceived speed of change depends on the viewing distance.



Temporal Frequency

Temporal frequency measures temporal variation

(cycles/s) In a video, the temporal frequency is spatial position

dependent, as every point may change differently

Temporal frequency is caused by camera or objectmotion



Temporal Frequency

caused by Linear Motion



)(

:frequencytemporalandspatial,motion, betweenRelation

)(),(),,(

),(),,(

0

0

y y x xt

y y x xt y xt y x

y x

f v f v f

f v f v f f f f f f

t v yt v xt y x

+=

++=

=

The temporal frequency of the image of a moving object depends on motion aswell as the spatial frequency of the object.

Example: A plane with vertical bar pattern, moving vertically, causes notemporal change; But moving horizontally, it causes fastest temporal change

Relation between Motion, Spatial and

Temporal Frequency

isat time patternimage the),,(is

0at patternimagetheAssume ).,(speedwithmovingobjectanConsider

0 t y x

t vv y x

=



Illustration of the Relation


16/29



Temporal Response

200

100

50

20

10

5

2

12 5 10 20 50

Frequency (Hz)

C o n

t r a s t s e n s i

t i v

i t y

0.06 trolands

850 trolands9300 trolands

7.1 trolands77 trolands

0.65 trolands

Figure 2.5 The temporal frequency response of the HVS obtained by a visualexperiment. Different curves represent the responses obtained with different meanbrightness levels, B, measured in trolands. The horizontal axis represents the ickerfrequency f , measured in Hz. Reprinted from D. H. Kelly, Visual responses totime-dependent stimuli. I. Amplitude sensitivity measurements, J. Opt. Soc. Am.(1961) 51:42229, by permission of the Optical Society of America.

Critical flicker frequency: The

lowest frame rate at which the eyedoes not perceive flicker.

Provides guideline for determiningthe frame rate when designing a

video system.

Critical flicker frequency dependson the mean brightness of thedisplay:

60 Hz is typically sufficient forwatching TV.

Watching a movie needs lower

frame rate than TV



Spatial Response

100

50

C o n

t r a s t s e n s i

t i v

i t y

20

10

5

2

0.21

0.5 1Spatial frequency (cpd)

2 5 10

Figure 2.6 The spatial frequencyresponse of the HVS, obtained by a visualexperiment. The three curves result fromdifferent stabilization settings used toremove the effect of saccadic eyemovements. Filled circles were obtainedunder normal, unstablized conditions;open squares, with optimal gain settingfor stabilization; open circles, with thegain changed about 5 percent. Reprintedfrom D. H. Kelly, Motion and vision. I.Stabilized images of stationary gratings, J. Opt. Soc. Am. (1979), 69:126674, bypermission of the Optical Society of America.



Spatiotemporal Response

0.3

3

1 3 10

Spatial frequency (cpd)

(a)

30

10

30

100

300

C

o n

t r a s t s e n s i

t i v

i t y

0.3

3

1 3 10

Temporal frequency (Hz)

(b)

30

10

30

100

300

C

o n

t r a s t s e n s i

t i v

i t y

Figure 2.7 Spatiotemporal frequency response of the HVS. (a) Spatial frequency responsesfor different temporal frequencies of 1 Hz (open circles), 6 Hz (lled circles), 16 Hz (opentriangles), and 22 Hz (lled triangles). (b) Temporal frequency responses for different spatialfrequencies of 0.5 cpd (open circles), 4 cpd (lled circles), 16 cpd (open triangles), and 22 cpd(lled triangles). Reprinted from J. G. Robson, Spatial and temporal contrast sensitivityfunctions of the visual systems, J. Opt. Soc. Am. (1966), 56:114142, by permission of theOptical Society of America.

The reciprocal relationbetween spatial and

temporal sensitivity wasused in TV system design:

Interlaced scan providestradeoff between spatialand temporal resolution.



Smooth Pursuit Eye Movement

Smooth Pursuit: the eye tracks moving objects

Net effect: reduce the velocity of moving objects on the retinal plane,so that the eye can perceive much higher raw temporal frequenciesthan indicated by the temporal frequency response.

y y x xt

y y x xt t

y x

y y x xt

y x

vvvv f

f v f v f f

vv

f v f v f

vv

===

++=

+=

~,~ if 0~

)~~(~

:)~,~(atmovingiseye when theretinaat thefrequencytemporalObserved

)(

:),(atmovingisobjectn themotion wheobject bycausedfrequencyTemporal


21/29

(b)

S p a t i a l f r e q u e n c y ( c p d ) T

e m p o

r a l f r e q u e

n c y ( H z

) C

o n

t r a s t s e n s i

t i v i t y

0.15

0

010

20

30

510

1520

0.1

0

0.05

(a)

S p a t i a l f r e q u e n c y ( c p d )

0.15

0.1

0

0.05

0

010

20

30

510

15

20 T e m p o

r a l f r e q u e n c y ( H z

)

(c)

S p a t i a l f r e q u e n c y ( c p d )

0.15

0.1

0

0.05

0

010

20

30

510

15

20 T e m p o

r a l f r e q u e n c y ( H z

)

Figure 2.8 Spatiotemporal response of the HVS under smooth pursuit eye movements:(a) without smooth pursuit eye movement; (b) with eye velocity of 2 deg/s; (c) with eyevelocity of 10 deg/s. Reprinted from Girod, B. Motion compensation: visual aspects, accuracy,and fundamental limits. In Sezan, M. I., and R. L. Lagendijk, eds., Motion Analysis and ImageSequence Processing, Boston: Kluwer Academic Publishers, 1993, 12652, by permission of Kluwer Academic Publishers.


22/29

Yao Wang, 2003 Frequency Domain Analysis

Video Sampling A Brief Discussion

Review of Nyquist sampling theorem in 1-D

Extension to multi-dimensions Prefilter in video cameras Interpolation filter in video displays


23/29


Nyquist Sampling Theorem in 1-D

Given a band-limited signal with maximum frequency f max, it canbe sampled with a sampling rate f

s>=2 f

max. The original

continuous signal can be reconstructed (interpolated) from thesamples exactly, by using an ideal low pass filter with cut-offfrequency at f s /2.

Practical interpolation filters: replication (sample-and-hold, 0th

order), linear interpolation (1 st order), cubic-spline (2 nd order) Given the maximally feasible sampling rate f s, the original signal

should be bandlimited to f s /2, to avoid aliasing. The desired

prefilter is an ideal low-pass filter with cut-off frequency at f s /2. Prefilter design: Trade-off between aliasing and loss of high

frequency


24/29


Extension to Multi-dimensions

If the sampling grid is aligned in each dimension (rectangular in2-D) and one performs sampling in each dimension separately,the extension is straightforward: Requirement: f s,i


25/29


26/29


Video Cameras

Sampling mechanism All perform sampling in time Film cameras capture continuous frames on film Analog video cameras sample in vertical but not horizonal

direction, arrange the resulting horizontal scan lines in a 1-Dcontinuous signal

Digital cameras sample in both horizontal and vertical direction,yielding pixels with discrete 3-D coordinates

Sampling frequency (frame rate and line rate) Depending on the maximum frequency in the underlying signal, the

human visual thresholds, as well as technical feasibility and cost Prefilter

Controlled by temporal exposure, scanning beam, etc. Digital cameras may capture at higher sampling rates and then

implement explicit filtering before converting to lower resolution


27/29


Typical Camera Response

Temporal prefilter: the value read out at any frame is

the average of the sensed signal over the exposuretime

Spatial prefilter: the value read out at any pixel is aweighted integration of the signal in a small windowsurrounding it, called the aperture, can be

approximated by a box average or a 2-D Gaussianfunction


28/29


Video Display

The display device presents the analog or digital video on thescreen to create the sensation of continuously varying signal inboth time and space.

With CRT, three electronic beams strike red, green, and bluephosphors with the desired intensity at each pixel location. No

explicit interpolation filters are used. Spatial filtering determinedby the size of the scanning beam, temporal filtering determinedby the decaying time of the phosphors.

The eye performs the interpolation task: fuses discrete frames

and pixels as continuously varying, if the temporal and spatialsampling rates are sufficiently high.


29/29

Homework 2

Reading assignment:

Chapter 2 Chapter 3, Section 3.4

Written assignment

Prob. 2.1,2.2,2.5,2.6,2.7

Documents

Fourier Analysis of Video Signals