Some Historical Remarks on

Sampling Theorem

Dept. of Electrical and Computer Engineering, Boston University, Boston, USA

Radomir S. Stanković, Jaakko T. Astola, Mark G. Karpovsky

Dept. of Computer Science, Faculty of Electronics, Niš, Serbia

Tampere Int. Center for Signal Processing, Tampere University of Technology, Tampere, Finland

1. Newton’s law of force 2. The law of universal gravitation

3. The second law of thermodynamics 4. Maxwell’s equations

5. The Navier-Stokes equation Newton’s second law for fluids. 6. The Stefan-Boltzmann law 7. Relativity - Einstein’s formula 8. The Lorentz transformation 9. The Schrodinger wave equation 10. Shannon’s sampling theorem

Motivation – Why to It Discuss Further?

IEEE Potentials, December 96-January 97, 39-40

Yet Another Review?

We hope NOT due to analysis of previous Reviews

Butzer, P.L., Stens. L., "Sampling theory for not necessarily band-limited functions – A historical overview", SIAM Review, Vol. 34, No. 1, 1992, 40-53.

Higgins, J.R., ”Five short stories about the cardinal series”, Bull. Amer. Math. Soc., Vol. 12, No. 1, 1985, 54-89.

Jerri, A., "The Shannon sampling theorem – Its various extensions and applications, A tutorial review", Proc. IEEE, Vol. 65, No. 11, 1977, 1565-1596.

Luke, H.D., "The origins of the sampling theory", IEEE Communications Magazine, Vol. 37, No. 4, 1999, 106-108.

Sangwine, S.J., Whitehouse, J.E., "The sampling theorem – a tutorial", Proc. of the IEE Colloquium on Mathematical Aspects of Digital Signal Processing, February 10, 1994, 1/1 - 1/6.

Unser, M., "Sampling-50 years after Shannon", Proc. IEEE, Vol. 88, No. 4, 2000, 569-587.

Vaidyanathan, P.P., "Generalizations of the sampling theorem - seven decades after Nyquist", IEEE Trans. Circuits and Systems, Vol. 48, No. 9, 2001, 1094-1109.

Meijering, E., "A chronology of interpolation: From ancient astronomy to modern signal and image processing", Proc. IEEE, Vol. 90, No. 3, 2002, 319-342.

Reviews on The Sampling Theorem

Novelity in the Paper and Outline The novelity in the paper resides in

Group-theoretic approach to Sampling Theorem

Real line

Localy Compact Abelian Groups Finte groups, Abelian, non-Abelian

Derivation of the Sampling theorem on dyadic groups from the Sampling theorem on LCA groups

Dyadic group

Disucssion of Sampling theorem on finite groups Emphasis on unifrom derivation and unified discussions

Contributors The history of technology is a continual succession of ideas, and not just a discrete list of major accomplishments.

Donald R. Mack

Practitioners

Theoreticians not noticing or emphasizing applications in Signal processing

Theoreticians working purposely for applications

Luke, H.D.

Subject of The Sampling Theorem

1. Sampling

A band limited function is completely determined by its samples.

2. Reconstructing

Recovering function values from samples for a function fulfilling the sampling conditions.

Prehistory - Lagrange

In 1765, J.L. Lagrange has shown that a periodic function, fulfilling conditions to be representable by a trigonometric series, can be expressed as a linear combination of a constant and n sine and n cosine terms.

The knowledge of function values at 2n+1 equidistant points within a period is sufficient to represent uniquely a periodic function with assumed properties. This statement can be viewed as a sampling theorem for bandlimited periodic functions.

Joseph-Louis, Comte de Lagrange

Prehistory - Cauchy In 1841, A.L. Cauchy has shown an interpolation formula for an M-bandlimited function, i.e., a function representable by a complex Fourier series with no more than |n| < M coefficients.

Augustin Louis Cauchy

12,)(||

2 +== ∑≤

nNecxfMn

inxn

π

∑−

=

−

−

=

1

0 sin

)1(1)sin()(N

m

m

NmxN

mfN

xnxfπ

π

A. Cauchy, “Mémoire sur diverses formules d’analyze,” Comptes Rendus des Séances de l’Académie des Sciences, Vol. 12, No. 6, 1841, 283–298.

H. S. Black, Modulation Theory, New York, Van Nostrand, 1953.

Borel, E., “Sur l’ interpolation,” Comptes Rendus des Séances de l’Académie des Sciences, Vol. 124, No. 13, 1897, 673–676.

Prehistory - Borel

Sampling part of the Sampling Theorem

Félix Édouard Justin Émile Borel E. Borel, “Mémoire sur les séries divergentes,” Annales Scientifiques de l’École Normale Supérieure, Ser. 2, Vol. 16, 1899, 9–131.

∑∞

−∞= −−=

k

kk

kzazzG )1(sin)(

ππ

solves the interpolation problem

∞<∈= ∑≠0

/,,)(k

kn kaifZnanG

Distortionless transmission of telegraphic (digital) signals

Nyquist, H., "Certain factors affecting telegraph speed", Bell Syst. Tech. J., Vol. 3, Apr. 1924, 324-346.

This work was presented at the Winter Convention of the A. I. E. E., New York, NY, February 13–17, 1928.

Harry Nyquist

Nyquist, H., "Certain topics in telegraph transmission theory", Trans. AIEEE, Vol. 47, Feb. 1928, 617-664.

Nyquist and the Sampling Theorem

Conclusions

Nyquist considered a different problem, the sampling theorem is a statment dual to that of Nyquist.

There is a sense to consider the sampling theorem for discrete signals.

Knowledge of data at the minimum subarea of the domain of definition sufficient to describe a function completely under some conditions provided.

An interpretation of the Sampling Theorem

Edmund Taylor Whittaker

Whittaker, E.T., "On the functions expansions of the interpolation theory", Proc. Roy. Soc. Edinburgh, Vol. 35, 1915, 181-194.

The problem of determining a function passing through the points (a+kw,f(a+kw)), where k ∈ Z, w – complex number.

Smoothest possible interpolation without singularities and rapid oscillations for given tabular values of f(x).

∑∞

∞− −−

−−+=

)(

)(sin)()(

kwaxw

kwaxwkwafxC π

π Sinc function

E.T. Whittaker

John Macnaughten Whittaker Whittaker, J.M., "The "Fourier" theory of the cardinal function", Proc. Edinburgh Math. Soc., 2, 1927-1929, 169-176.

Whittaker, J.M., "On the cardinal function of interpolation theory", Proc. Edinburgh Math. Soc., Vol. 1, 1929, 412-46.

Whitaker, J.M., Interpolation Function Theory, Cambridge Tracts in Mathematics and Mathematical Physics, No. 33, Cambridge University Press, Chapt. IV, 1935.

Weak version of the sampling theorem, not included in

J.M. Whittaker

Ogura, K., "On a certain transcedental integral function in the theory of interpolation", Tohoky Math. J., 17, 1920, 64-72.

Kinnosuke Ogura

Ogura, K., "On some central difference formulas of interpolation", Tohoky Math. J., Vol. 17, 1920, 232-241.

Sangaku

The Sampling Theorem and a Proof of It Ogura was the first who stated the sampling theorem and traced the way of a rigorous proof of it.

For a proof Ogura recommended to use the results from Lindelof, E., Les calcul des Residues, Gautier-Villard, Paris, 1905.

E. Lindelof Yosan Wasan

Ogura Lindelof

Владимир Александрович Котельников

Kotelnikov, V.A., "On the carrying capacity of the "ether" and wire in telecommunications", Material for the First All-Union Conference on Questions of Communications, Izd. Red.Upr. Svyazi RKKA, Moscow, 1933.

Any function F(t) which consists of frequencies from 0 to f1 periods per second may be represented by the following series

∑∞

∞− −

−

=

1

11

2

2sin

)(

fkt

fktw

DTF k

where k-integer, w1=2πf1, Dk – constant which depends on F(t).

"Электричество", Энергетический институт Академии Наук СССР

A.V. Kotelnikov

Rejected the paper with comment ”far from engineering needs”.

Any function F(t), which consists of frequencies from 0 to f1, can be transmitted continuously with an arbitrary accuracy, by means of numbers sent at intervals of 1/2f1 seconds. Indeed, by measuring of the value F(t) at t=n/2f1 (n integer), we get F(n/2f1)=Dnw1. Since all terms of the series (2) for this value of t tend to zero, except the term for k=n, as it can be easily established by calculation of the indefinite point, equals Dnw1. In this way, after each 1/2f1 we can determine the next Dk. When these Dk are transmitted in a row at each 1/2f1 sec., we can from (2) reconstruct F(t) termwise to any degree of accuracy.

Shannon

Claude Elwood Shannon Theorem 13, page 34 Let f(t) contain no frequencies over W. Then,

,)2(

)2(sin)( ∑∞

∞− −−

=nWt

nWtXtf n ππ

where .2

=

WnfX n

Denis Gabor Theorem 13 has been given previously in other forms by mathematicians (Whittaker) but in spite of its evident importance seems not to have appeared explicitly in the literature of communication theory.

Nyquist, (1924, Bennett, 1941) however, and more recently Gabor, (1946) have pointed out that approximately 2TW numbers are sufficient, basing their arguments on a Fourier series expansion of the function over the time interval T.

Gabor

Herbert P. Raabe Raabe, H., "Untersuchungen an der Wechselzeitigen Mehrfachübertragung (Multiplexübertragung)", Elektrische Nachrichtentechnik, Vol. 16, 1939, 213-228.

Royal Techical Highschool in Berlin around 1890 where H. Raabe graduated in 1936 and received Dr.-Ing. Degree, summa cum laude, in 1939.

Benett, W.R., ”Time division multiplex systems”, Bell. Syst. Tech. J., Vol. 20, 1941, 199.

Karl Kupfmuller "The time law allows comparison of the capacity of each transfer method with various known methods. On the other hand it indicates the limits that the development of technology must stay within. One interesting question for example is where the lower limit for k lies. The answer is acquired by at least one power change being needed to achieve one signal. So the frequency range must be at least so wide that the settling time becomes less than the duration of a signal, and from this comes k=1/2. So we can never get below this value, no matter how technology develops."

Karl Kupfmuller

Isao Someya Someya, I., Hakei Denso (Waveform Transmission), Shykyo,Tokyo, 1949.

Wetson, J.D., "A note on the theory of communication", Philos. Mag., 303, Vol. 40, 1949, 449-453.

Wetson, J.D., "The cardinal series in Hilbert space", Proc. Cambridgh. Philos. Soc., Vol. 45, 1949, 335-341.

The Theorem in Western Litearature

Sampling Theorem for Duration-Limited Signals

Rxxf ∈),( [ ]baxxf ,,0)( ∉= cannot be band-limited Duration (time) limited functions

Bandlimited functions Rxxf ∈),( Sf(w) = 0, w > w0

Functions of bounded variation f(x), x∈R, f(x) ∈R ],[ bax ∈ ∑ −+

iii

Pxfxf )()(sup 1

supremum over all partitions P = x0,...,xn of [a,b]

- total variation

Bounded variation, if total variation is finite

Charles-Jean Baron de la Vallée Poussin

De la Valléee Poussin, Ch.-J., "Sur la convergence des formulaes d'interpolation entre ordonees équidistantes", Bull. Cl. Sci. Acad. Roy. Belg., 4, 1908, 319-410.

Interpolation formula for time-limited functions

∑∈ −

−=

),[ )()(sin)()(

ba k

kkm

kxm

xmfxFα α

αα

nmZkmk

k =±±=∈= ,,...2,1,0,πα or ,...3,2,1,2/1 =∈+= Nnnm

C.-J. de la Vallée Poussin

A generalization of the Lagrange interpolation formula for infinite number of nodes

A counterpart of the Riemman localization principle or Fourier integrals in the case of Fm.

Under the additional condition f(b) = 0 besides f(x) = 0, for x ≠ [a, b], the interpolation function Fm can be viewed as a discrete version of the Dirichlet convolution integral, a particular form of the Fourier inversion integral, and the behaviour of Fm for is similar to that of the Fourier inversion integral for f

m → ∞

Extensions by P.L. Butzer & P.L. Stens

Maria Theis

Theis, M., Über eine Interpolations formel von de la Vallée Poussin", Math. Z., 3, 1919, 93-113.

For the convergence of Fm(x) to f(x), besides continuity of f(x) on [a,b], f(x) should be of bounded variation. For convergence of Fm(x) for any continuous function f(x), Theis used the kernel function

2)/(sin)( xxx ππφ =- a counterpart of Féjer method in summation of Fourier series

Lipót Fejér

Functions of Bounded Variation

Consider a function f that is Riemann integrable over any finite interval of R and f(x)/x is of bounded variation in (N, ∞) and (-∞,N) for some N > 0. If f is continuous at x0∈R and of bounded variation in a neighborhood of x0, then

∑∞

−∞=∞→ −

−=

k k

kkm xm

xmfxf)(

)(sin)(lim)(0

00 α

αα

Whittaker, J.M., "The "Fourier" theory of the cardinal function", Proc. Edinburgh Math. Soc., 2, 1927-1929, 169-176.

Not necessarily vanishing outside a finite interval [a,b]

Sampling of non-band-limited functions Weiss, P., "An estimate of the error arising from misapplication of the sampling theorem", Notices Amer. Math. Soc., 10,1963, 351.

Brown, J.L., "On the error in reconstructing a non-limited function by means of the bandapass sampling theorem", J. Math. Annal. Appl., 18, 1967, 75-84, Erratum, same journal 21,1968, 699.

Butzer, P.L., Splettstosser, E., "A sampling theorem for duration limited functions with error estimates", Information and Control, Vol. 34, 1977, 55-65.

Butzer, P.L., Splettstosser, E., "Sampling principle for duration limited signals and dyadic Walsh analysis", Information Science, Vol. 14, 1978, 93-106.

Butzer, P.L., Splettstosser, W., Index of Papers on Signal Theory, 1972-1989, Lehrstuhl A fur Mathematik, Aachen University of Technology, Aachen, Germany, 1990.

Locally Compact Abelian Groups Notation

G – additive locally compact Abelian group Γ – dual group of G H – discrete subgroup of G with the discrete annihilator

Hywyw ∈∀==Λ ,1),(| χ

Baire measurable subset Ω of Γ which contains a single element from each coset of Λ

)( Λ+∩Ω w contains a single point for each w ∈ Γ

René-Louis Baire

Integers Dk or Xn in Kluvanek and Shannon notation

Sampling Theorem on LCA Groups If f ∈ L2(G) and its Fourier transform Sf (w) =0 for almost all w∉Ω, then f is almost everywhere equal to a continuous function, and if f is a continuous function,

where this series converge both uniformly on G and in the norm in L2(G). Further,

The function φ in (1) is defined as

where mΓ is the Haar measure on G and χ (x,w) are group characters of G.

∑∈

−=Hy

yxyfxf )()()( φ

∑∈

=Hy

yff 22 )(

(1)

,)(),()( ∫Ω

Γ= wdmwxx χφ

Kluvánek, I., "Sampling theorem in abstract harmonic analysis", Mat. Fiz. Časopis Sloven. Akad., Vied. 15, 1965, 43-48.

Igor Kluvánek

Igor Kluvánek

)./()(sin)(,

,,...2,,0,,2...,,,),(),,(

xxxgetwehwithhhhhHlyconsequent

andGIf

ααφπα

αα

==−−=−=Ω∞−∞=Γ=

α>=∞−∞∈ wforwSandLf f 0)(),(2If we get the sampling theorem on R.

Walsh-Fourier Analysis [ )∞=∈ + ,0Rx ,...2,1,0)(,1,0,2

)(±±=∈∈= ∑

∞

−=

− ZxNxxx ixNi

ii

ZqPppxRxD q ∈=∈=∈= ++ ,,...2,1,0,2/|

+∈ Dx

unique representation select finite expansion If

+∉ Dx

+∈ Ryxxy ,),,(ψ - generalized Walsh functions

∫=r

dssxrxJ0

),(),( ψ - Walsh-Dirichlet kernel

nr 2= <≤

=−

otherwisex

xJnn

n

,0,20,2

)2,([ )

+∈

=⊕−−

otherwisessx

sxJnn

n

,0,)1(2,2,1

)2,1(

y - sequency

dyadic rational numbers

Dyadic Sampling Theorem

If f and its Walsh-Fourier spectrum Sf(w) belong to L1(R+)

f continuous on R+\D+ f continuous from the right on D+

Sf (w) = 0 for w ≥ 2n, n ∈ Z, Z – the set of integers, i.e.,

+

∞

=

∈⊕

= ∑ RxsxJsfxf n

sn ),2,1(

2)(

0

f is sequency limited, then

F. Pichler S. Kak

P.L. Butzer

Pichler, F.R., ”Sampling theorem with respect to Walsh- Fourier analaysis”, Appendix B in Reports Walsh Functions and Linear System Theory, Elec. Eng., Dept., Univ. of Maryland, College Park, May 1970

Dyadic and LCA Groups

dual group the additve group G’ of the dyadic field

Walsh-Fourier analysis

domain group the set of generalized Walsh functions Ψ(λ,x)

A discrete subgroup H = s/2n, s = 0,1,..., n ∈ Z The annihilator Λ for H is isomorphic to the sequences

1,0,...,0,...,0,..., ∈=Λ − ik λλ

For every x ∈ H 01 =∑ −n

nn xλ and Ψ(λ,x)=1

Ω=[0,2n)

Dyadic Sampling Theorem – A Proof ∫

∈

==−

−

n

otherwisexif

dwxwxn

n2

0 ,0),2,0(,1

),(2)( ψφDefine the reconstruction function as

From Kluvánek theorem ∫=n

dwxwwSxf f

2

0

),()()( φ

For every ))1(2,2[ +∈ −− kkx nn ,...2,1,)(2)()1(2

2∫

+−

−

=∈=k

k

n

n

n

Pkduufxf

Thus, f is constant on all the intervals [2-nk, 2-n(k+1)), k ∈ P

Then, +−−

∞

=

∈+

= ∑ Rxxkkkfxf nn

kn ),))(1(2,2[

2)(

0ρ

the characteristic function of [a,b)

∉∈

=),[,0),,[,1

))(,[baxifbaxif

xbaρ

Since for k∈ P

⊕=+∈

=+−−

−− )2,1(,0

)),1(2,2[,1)))(1(2,2[ kxJ

otherwisekkxif

xkk nnn

nnρ

Sampling in Finite Walsh Analysis

Band-limiting of time-limited signals and vice versa

Important research topics

Le Dinh, C. T., Le, P., Goulet, R., "Sampling expansions in discrete and finite Walsh-Fourier analysis", Proc. 1972 Symp. Applic. Walsh Functions, Washington, D.C., USA, 265-271.

Le Dinh, C. T., Goulet, R., "Time-sequency-limited signals in finite Walsh transforms", IEEE Trans. Inform. Theory, Vol. 20, No. 2, 1974, 274-276.

Determining minimum number of points to define a function under some conditions provided

In discrete analysis

Characterization of signals by duration and bandwith

Applications dealing with large sets of discrete and finite signal samples (compression, multiplexing)

Finite Walsh Sampling Theorem

f(x), x ∈ Bn = 0,1,...,2n-1

f(x) is M-sequency band-limited (MBL) if Sf(w)=0 for w > M

Walsh-Fourier kernel

∑ ∑−

=

−

=

−=⊕==1

0

1

0

1),(),()(M

w

R

rM NMrxMxwwalxd δ

An MBL function f(x) can be completely reconstructed from its M values

∑−

=

− ⊕=1

0

1 )()()(M

pM pxdpfMxf

⊕ componentwise addition modulo 2

M-sequency limited bandpass functions (MBP)

Sampling on Finite Abelian Groups

i

m

iGG

1

0

−

=×= 110

1

0

..., −

−

=

≤≤≤= ∏ m

m

ii ggggg

∏=

−− −∈==k

iim mkgMp

01 1,...,1,0,

∑ ∑−

=

−

=

− ===1

0

1

0

1* /),(),()(M

w

R

kM MgRxkMMxwxd δχ

° - group operation on G, χ – group character,

* - complex conjugate, δ(x)=δx,0 – Kronecker delta

f is MBL on G if Sf(w) = 0 for w ≥ M, then

∑ ∑ ∑−

=

−

=

−

=

−−−−−

==

1

0

1

0

1

0

11111 )(()()()()(M

r

M

r

R

kM xrkMMpfMxrdrfMxf δ

Sampling on Finite non-Abelian Groups

i

m

iGG

1

0

−

=×= 110

1

0

..., −

−

=

≤≤≤= ∏ m

m

ii ggggg

Mi - number of irreducible unitary representations Rw,i of Gi

∏=

−− <=k

iim KMMM

01,

Γ – dual object of G, |Γ | = K

f is M-band limited (MBL) on G if Sf(w) = 0 for w ≥ M

∑−

=

=1

0))(()(

M

wwwM xTrrxd R

∑−

=

−−=1

0

11 )()()(Q

uM uxdufQxf ∏

=−−=

k

uimgQ

01 k- fixed when selecting M

Rw(x) xy ≠ yx

Instead of Conclusions

Group

Real line R

Locally compact Abelian

Dyadic

Finite dyadic Finite Abelian Finite non-Abelian

Sampling Theorem on Groups

K. Kupfmuller, 1931 H. Nyquist, 1924 V.A. Kotelnikov, 1933 H. Raabe, 1938 K. Ogura, 1920

Band-limited Time-limited C.-J. de la Vallée Poussin, 1908 M. Theis, 1919 Weiss, P., 1963 Brown, J.L., 1967 Butzer, P.L., 1977 further

C.E. Shannon, 1948 I. Someya, 1949 J.D. Wetson, 1949 E.T. Whittaker, 1915 J.M. Whittaker, 1924

I. Kluvánek, 1965 L.J. Fogel, 1959

F. Pichler, 1970 S. Kak, 1970 P.L. Butzer, 1978

Le Dinh, C. T., Le, P., Goulet, R., 1972

Signals

Acknowledgment

Thanks are due to Mrs. Marju Taavetti

and Mrs. Pirkko Ruotsalainen

of Tampere Univeristy of Technology for the help in collecting the literature.