Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal

Information Signal ProcessingInformation Signal ProcessingJoseph A. OJoseph A. O’’SullivanSullivan

Electronic Systems and Signals Research LaboratoryCenter for Security Technologies

Department of Electrical and Systems Engineering Washington University

[email protected]://essrl.wustl.edu/~jao

Supported by: ONR, NSF, NIH, Boeing Foundation, DARPASpecial thanks to Naveen Singla

J. A. O’Sullivan. 05/25/2004Information Signal Processing

2

CollaboratorsCollaborators

Jasenka BenacMichael D. DeVoreAndrew (Lichun) LiClayton MillerLee MontagninoRyan Murphy Natalia SchmidNaveen SinglaBrandon WestoverShenyu Yan

G. James BlaineRoger ChamberlainMark FranklinDaniel R. FuhrmannRonald S. IndeckChenyang LuPierre Moulin, UIUCMarcel MullerRobert PlessDavid G. PolitteChrysanthe PrezaAndrew Singer, UIUC Donald L. SnyderBruce R. WhitingJeffrey F. Williamson, VCULihao Xu

Faculty Current and Former Students


3

OutlineOutline

• DSP ISP- Signal processing - Time, space, samples- Information, distortion, transmission

• Graphical Models- Data models - Computational and communication models- X-ray CT imaging- Iterative decoding

• Message Passing EM Algorithms- Graphical models- Projections

• Applications Revisited• Speculation on Trends• Conclusions


4

OutlineOutline

• DSP ISP- Data models, computational models, algorithms- Central role of information

• Graphical Data Models- X-ray CT imaging- Iterative decoding

• Message Passing EM Algorithms



5

Signal Processing

Information TheoryComputation andCommunication

FFTFFT

MultiresolutionMultiresolutionanalysisanalysis


6

X(0)

X(1)

X(2)

X(3)

X(4)

X(5)

X(6)

X(7)

x(0)

x(4)

x(2)

x(6)

x(1)

x(5)

x(3)

x(7)

FFT


7

Signal Processing

Information Theory

Numerical analysisNumerical analysisProcessors: parallel, Processors: parallel,

ASIC, etc.ASIC, etc.Systolic architecturesSystolic architecturesFFTWFFTW

Computation andCommunication


8

Signal Processing

Information Theory

Numerical analysisNumerical analysisProcessors and Processors and

architecturesarchitecturesFFT, FFTWFFT, FFTWTransversal filtersTransversal filters

MRAMRA

Complexity theoryComplexity theoryGraphical modelsGraphical models

KalmanKalman filtersfiltersCompression Compression



9


Information Signal ProcessingInformation Signal Processing

Signal Processing

Information Theory

ComplexityComplexity--constrained processingconstrained processingSignal processing on graphsSignal processing on graphsDistributed signal processingDistributed signal processingDistributed information theoryDistributed information theoryDistributed computation and Distributed computation and

communicationcommunicationOptimal information extraction, Optimal information extraction,

communication, computationcommunication, computation


10

Distributed sensing,communication, computation

““The architecture for a The architecture for a fully netted maritime forcefully netted maritime force””


11

Wireless Sensor NetworksWireless Sensor Networks

http://www.greatduckisland.net/index.php

Great Duck Island Habitat


12


13

Information Signal ProcessingInformation Signal Processing• Measure information quantitatively

– Entropy rates, mutual information, relative entropy– Rate-distortion theory

• Graphical data models– Forward likelihood models for data– Priors, penalties, and multiresolution models– Abstractions to random graphs

• Constrained implementations– Parameterize implementations: communication rates,

computational complexity, computation time

• Principled approach to complexity-constrained signal modeling, data analysis, and algorithm development


14

OutlineOutline






15

Graphical Data ModelsGraphical Data Models

•• Model 1: Model 1: y = Hsy = Hs– y is n × 1, s is m × 1,

and H is n × m– yj depends on sk if hjk ≠ 0– Defines a graphical model

•• Model 2:Model 2:

– Neighborhood structure– Bipartite graph model

•• Model 3:Model 3:– RVs on edges of graph

k

j∏ ∈=

jkj jksypp ))(,|()|( sy

∏ ∏∈

=k kj

kjk sxpp)(

)|()|( sx

∏ ∈=j

jkj jkxypp ))(,|()|( xy


16

Tomography

S

D

Nonrandom Graphs

Line integrals through patientQuantization point spread function

weights on edges of graphHelps organize computations

Siemens Somotom Emotion


17

Computational ModelsComputational Models

System model accounts for:• Information extraction

problem definition• Compression of sensor data• Network throughput• Processor cycles per

instruction• Size of processor local

memory• Communication bandwidth

of each link• etc.


18

Computational ModelsComputational Models

• Local resources plus remote• Communicate observation as well as classification

- Human in the loop- Remote contribution to classification when available

• Dynamic resource availability• Sequence of partial classifications (an,θn)


19

Progress: Computational Graph Progress: Computational Graph Same as Data Model GraphSame as Data Model Graph

•• Message passing algorithmsMessage passing algorithms– Pearl’s belief propagation– Iterative decoding

» Turbo-codes, parallel concatenated codes» Low density parity check codes» Repeat-accumulate codes, serial concatenated codes

– Iterative equalization and decoding

•• ExpectationExpectation--Maximization (EM) AlgorithmsMaximization (EM) Algorithms– Graphical models– General problem– Gaussian, Poisson (emission tomography, transmission

tomography)– Abstract examples on random graphs


20

⎟⎟⎟⎟⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜⎜⎜⎜⎜

⎝

⎛

=

10011010101000010101010100011010000101010101100110010100100101010010100010100010101101001010100110

H Regular (3,6) n=14

Random Graphs

Comment: LDPC parity check matrix


21

⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜

⎝

⎛

=

001010100000100100000010010001001000000100010001100000001100001000100010010001010000000010000101

H Regular (2,3) n=12

Random Graphs


22

⎟⎟⎟⎟⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜⎜⎜⎜⎜

⎝

⎛

=

10011010101000010101010100011010000101010110101010010100100101010010100010100010110101001001010110

H Irregular n=14

Random Graphs


23

Tomography

S

D


24

k

j

νjk

µjk

µ

µ µ

µ

νν

),(

),()1(

')1(

)('

)1(

stateg

statefmkjk

mjk

mjkj

mjk

++

+

=

=

µν

νµ

Message Passing AlgorithmsMessage Passing Algorithms


25

TomographyS

D


26


27

Iterative Decoding Message PassingIterative Decoding Message Passing

∑∈

+ =kjk

mjk

mjk

\)('

)('

)1( νµ

∏∈

++

−=jkj

mkjz

mjk k

\)('

)1('

)1(

2tanh)1(

2tanh

µν

Codeword Bit Nodes

zk

xj

Check Nodes

jkµ

jkν


28

Iterative Decoding Message PassingIterative Decoding Message Passing

-6

-5

-4

-3

-2

-1

0

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8

SNR [dB]

Bit

erro

r rat

e, L

og B

ase

10 ISI-freeBIAWGN Capacity

[10000,5000] regular (3,6) matrix[10000,5000] regular (3,6) matrix


29

OutlineOutline






30

ML Problem: { }xsxxys

dpp∫ )|()|(lnmax

EM Algorithm: xsxxy

syxsyxs

dpp∫ ⎥

⎦

⎤⎢⎣

⎡ ΦΦ

Φ )|()|(),|(ln),|(minmin

EM AlgorithmEM Algorithmsk

yj

Hidden Data

Incomplete Data

xjk


31

VariationalVariational RepresentationsRepresentations•• Convex Decomposition LemmaConvex Decomposition Lemma. Let f be convex. Then

• Special Case: f is ln

• Basis for EM; see also De Pierro, Lange, Fessler

∑

∑ ∑≥=

≤

iii

i ii

irii

rr

xfrxf

0,1

)()( 1

⎭⎬⎫

⎩⎨⎧

=Φ≥ΦΦ=

ΦΦ−=⎟

⎠

⎞⎜⎝

⎛

∑

∑∑∈Φ

iii

i i

ii

ii q

q

1,0:

lnminln

P

P


32

EM AlgorithmEM Algorithm

∫=Φ +

')|'()'|()|()|(),|(

)(

)()()1(

xsxxysxxysyx

dpppp

m

mmm

xsxsyxss

dp, mmm )|(ln)|(argmax )()1()1( ∫ ++ Φ=

Assume the factorizations (Model 3):

∏ ∏∈

=k kj

kjk sxpp)(

)|()|( sx

∏ ∈=j

jkj jkxypp ))(,|()|( xy

These computations become local and thus message passing

In general, these are global computations


33

Message Passing EM AlgorithmMessage Passing EM Algorithm

∫ ∏∏∫ ∏∏

∈∈

≠∈∈+

∈

∈=Φ

)(''

)('

)('''

'),(''

)('

)('''

1(

)|())(',|(

)|())(',|(),|(

jkjk

jk

mkjkjkj

kkjkjk

jk

mkjkjkj

(m)jjk

)m

dxsxpjkxyp

dxsxpjkxypyx

KK

KK

K

Ks

jkkjkm

jjkm

s

mk dxsxp,yxs

k

)|(ln)|(argmax )()1()1( ∫ ++ Φ= s

sk

yj

Input Data

Measured Data

xjk

)( jkxΦ

ks


34

GaussianGaussian--MAPMAP

)2

,0(~

),0(~;

0 Iw

IswHsy

NN

PN+=

∑∈

+

+=

)(

)(

0

)1( ˆ

|)(|2|)(|

ˆkj

ljk

lk x

jNkP

Ps

⎟⎟⎠

⎞⎜⎜⎝

⎛−+= ∑

∈

+++

)('

)1('

)1()1( ˆ|)(|

1ˆˆjk

lkj

lk

ljk sy

jsx

sk

yj

Input Data

Measured Data

xjk

jkx

ks


35

GaussianGaussian--MAPMAP[10000,5000] regular (3,6) matrix[10000,5000] regular (3,6) matrix


36

Emission TomographyEmission Tomography

y ~ Poisson(Hλ)λ: Mean of emitted photons

∑∈

+ =)(

)()(

)1( ˆ|)(|

ˆˆkj

mj

mkm

k qk

λλ

∑∈

=

)('

)('

)(

ˆˆ

jk

mk

jmj

yq

λ

k

yj

Pixels

Measured Data

jqkλ


37

Emission TomographyEmission Tomography[10000,5000] regular (3,6) matrix[10000,5000] regular (3,6) matrix


38

Emission TomographyEmission Tomography[50000,1000] regular (3,150) matrix[50000,1000] regular (3,150) matrix


39

Transmission TomographyTransmission Tomography

y ~ Poisson(I0exp(-Hλ))λ: photon attenuation ⎥

⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡−=

∑∑

∈

∈+

)(

)()()()1( ln1ˆˆ

kj

mj

kjj

mk

mk q

y

zλλ

⎥⎦

⎤⎢⎣

⎡−= ∑

∈ )(

)(0

)( ˆexpjk

mk

mj Iq λ

k

yj

Pixels

Measured Data

jqkλ


40

[10000,5000] regular (3,6) matrix[10000,5000] regular (3,6) matrix

Transmission TomographyTransmission Tomography


41

Comments on DetailsComments on Details

•• Information geometry basisInformation geometry basis

•• Easily extended to arbitrary Easily extended to arbitrary HH

•• Low density Low density sparsesparse

•• Constraints in iterative decoding vs. forward Constraints in iterative decoding vs. forward modelmodel

•• Performance is intimately connected to Performance is intimately connected to graphgraph


42

OutlineOutline• DSP ISP

- Data models, computational models, algorithms- Central role of information



• Applications Revisited- Iterative decoding- X-ray CT imaging

• Speculation on Trends• Conclusions


43

Science and technologyScience and technologypotentially yield potentially yield

6 Tb/in6 Tb/in22

courtesy R. S. courtesy R. S. IndeckIndeck

Patterned Magnetic Media


44

Advanced Recording MediaAdvanced Recording MediaBluBlu--Ray DiscRay Disc

Next-generation Optical Disc Video Recording Format


45

2D Intersymbol Interference2D Intersymbol Interference

111111

11111111

21

21

11211

ΛΛΛ

ΜΟΜΜΟ

ΛΛΛ

kkkk

k

xxx

xxxx

1111110

110

12

1111110

100100

+++++

+

+

+

+

kkkkkk

kkkkkk

k

kk

k

rrrrrrrr

rrrrrrrr

ΛΛΛΛ

ΜΟΜΜΟΜΜ

ΛΛΛΛΛw(i,j)

⎟⎟⎠

⎞⎜⎜⎝

⎛=

25.05.05.01

h ⊕

Includes Guard Band

jijijijijiji wxxxxr ,1,11,,1,, 25.05.05.0 ++++= −−−−

Singla et al., “Iterative decoding and equalization for 2-D recording channels,” IEEE Trans. Magn., Sept. 2002.


46

Full Graph Message PassingFull Graph Message Passing

Measured Data Nodes (r)

Codeword Bit Nodes (x)

Check Nodes (z)

jijijijijiji wxxxxr ,1,11,,1,, 25.05.05.0 ++++= −−−−⎟⎟⎠

⎞⎜⎜⎝

⎛=

25.05.05.01

h


47


•• Codeword bit nodes to check nodesCodeword bit nodes to check nodes

•• Check nodes to codeword bit nodes Check nodes to codeword bit nodes

∑∑∈

−→

∈

−→→ +=

zxNz

lxz

xNm

lxm

lzx LLL

\)('

)1('

)(

)1()(

∏∈

−→→ −=

xzNx

lzxz

lxz LL

\)('

)1('

)(

2tanh)1(

2tanh


48

•• Codeword bit nodes to measured data nodesCodeword bit nodes to measured data nodes

•• Measured data nodes to codeword bit nodesMeasured data nodes to codeword bit nodes

∑∑∈

→∈

−→→ +=

)(

)(

\)('

)1('

)(

xNz

lxz

mxNm

lxm

lmx LLL

})\)(':({ )('

)( xmNxLfL lmx

lxm ∈= →→



49

Full Graph Message Passing

-6

-5

-4

-3

-2

-1

0

0 0.5 1 1.5 2 2.5 3

SNR [dB]

Bit

erro

r rat

e, L

og B

ase

10

ISI-freeFull Graph_50

Full Graph Message PassingFull Graph Message Passing[10000,5000] regular (3,6) matrix[10000,5000] regular (3,6) matrix


50

Full Graph AnalysisFull Graph Analysis

Length 4 cycles present which degrade performance Length 4 cycles present which degrade performance of messageof message--passing algorithm passing algorithm

x(i+2,j) x(i+2,j+1)

x(i+1,j) x(i+1,j+1)

x(i,j) x(i,j+1)

x(i+2,j+2)

x(i+1,j+2)

x(i,j+2)

r(i+1,j+1)

r(i,j+1)r(i,j)

r(i+1,j)

FromCheckNodes

Kschischang et al., “Factor graphs and the sum-product algorithm,” IEEE Trans. Inform. Theory, Feb. 2001.


51

Ordered Subsets Message Passing Ordered Subsets Message Passing From Imaging From Imaging –– Data set is grouped into subsets to Data set is grouped into subsets to increase rate of convergence of image increase rate of convergence of image reconstruction algorithmsreconstruction algorithms

For Decoding For Decoding –– Measured data is grouped into Measured data is grouped into subsets to eliminate short length cycles in the subsets to eliminate short length cycles in the Channel ISI graphChannel ISI graph

H. M. Hudson, and R. S. Larkin, “Accelerated image reconstruction using ordered subsets of projection data,” IEEE Trans. Medical Imaging, Dec. 1994


52

Labeling of data nodes into 4 subsetsLabeling of data nodes into 4 subsetsFor each iteration use data nodes of one label onlyFor each iteration use data nodes of one label only

Labeled ISI GraphLabeled ISI Graph

J. A. O’Sullivan, and N. Singla, “Ordered subsets message-passing,” Int’l Symp. Inform. Theory, Yokohama, Japan 2003.


53

Ordered Subsets Message Passing

-6

-5

-4

-3

-2

-1

0

0 0.5 1 1.5 2 2.5 3SNR [dB]

Bit

erro

r rat

e in

log1

0

ISI-freeOrdered Subsets_200Full Graph_50

[10000,5000] regular (3,6) matrix[10000,5000] regular (3,6) matrixOrdered Subsets Message PassingOrdered Subsets Message Passing


54

CT Imaging in Presence of High CT Imaging in Presence of High Density Attenuators (J. Williamson, PI)Density Attenuators (J. Williamson, PI)

Brachytherapy applicators After-loading colpostats

for radiation oncology

Cervical cancer: 50% survival rateDose prediction important

Object-Constrained Computed Tomography (OCCT)


55

Filtered Back ProjectionFiltered Back Projection

Truth FBP

FBP: inverse Radon transform


56

Transmission TomographyTransmission Tomography• Source-detector pairs indexed by y; pixels indexed by x• Data d(y) Poisson, means g(y:µ), log likelihood function

• Mean unattenuated counts I0, mean background β• Attenuation function µ(x,E), E energies

• Maximize over µ or ci; equivalently minimize I-divergence

)(),(),(exp),():(

):():(ln)()):(|(

0 yExxyhEyIyg

ygygydgdl

E x

y

βµµ

µµµ

+⎟⎟⎠

⎞⎜⎜⎝

⎛−=

−=⋅

∑ ∑

∑

∈

∈

X

Y

∑=

=I

iii ExcEx

1)()(),( µµ


57

MaximumMaximum--Likelihood Likelihood Minimum IMinimum I--divergencedivergence

• Poisson distribution

• Poisson distributed data loglikelihood function

• Maximization over µ equivalent to minimization of I-divergence

λλ

λ

λλ

λ λ

+−=

−−==

== −

kkkkI

kkkNP

ek

kNPk

ln)||(

!lnln)(ln!

)(

)(),(),(exp),():(

):()():(

)(ln)()):(||(

):():(ln)()):(|(

0 yExxyhEyIyg

ygydyg

ydydgdI

ygygydgdl

E x

y

y

βµµ

µµ

µ

µµµ

+⎟⎠

⎞⎜⎝

⎛−=

+−=⋅

−=⋅

∑ ∑

∑

∑

∈

∈

∈

X

Y

Y


58

Maximum Likelihood Maximum Likelihood Minimum IMinimum I--DivergenceDivergence

Difficulties: log of sum, sums in exponent

)()()(),(exp),():(

):()():(

)(ln)()):(||(

):():(ln)()):(|(

10 yExcxyhEyIyg

ygydyg

ydydgdI

ygygydgdl

E x

I

iii

y

y

βµµ

µµ

µ

µµµ

+⎟⎟⎠

⎞⎜⎜⎝

⎛−=

+−=⋅

−=⋅

∑ ∑ ∑

∑

∑

∈ =

∈

∈

X

Y

Y


59

Interpretation: Compare predicted data to measured data via ratio of backprojectionsUpdate estimate using a normalization constant

Comments: choice for constants; monotonic convergence;linear convergence; fixed points satisfy Kuhn-Tucker conditions; constraints easily incorporated

)(~)(ˆ

ln)(

1)(ˆ)(ˆ)(

)()()1(

xbxb

xZxcxc l

i

li

i

li

li −=+

New Alternating Minimization AlgorithmNew Alternating Minimization Algorithmfor Transmission Tomographyfor Transmission Tomography


60David G. PolitteOctober 31, 2002

Mini CT, AM Iteration 0000001
























































79

Our Plans in CT ImagingOur Plans in CT Imaging

•• CT CT MultirowMultirow SinogramSinogram data: data: 1408 1408 ×× 768 768 ×× nndd ×× nnzz

– where nd is the number of detector rows and nz is the number of gantry rotations

•• Fully 3Fully 3--D Implementations for D Implementations for Quantitative CTQuantitative CT

•• SpeedSpeed--up: Ordered Subsets, up: Ordered Subsets, MultigridMultigrid Methods, Parallel Methods, Parallel Implementations on Clusters Implementations on Clusters of PCsof PCs

•• Future: PETFuture: PET--CT CT Siemens Somotom Emotion


80

Slide and data from R. Laforest and M. Mintun.

PETCT-211


81

PETCT-165

Slide and data from R. Laforest and M. Mintun.


82

Additional Algorithm/DetectorAdditional Algorithm/DetectorModel DevelopmentModel Development

•• RegularizationRegularization•• Energy integrating detectors Energy integrating detectors •• Finite detector size, better source modelFinite detector size, better source model•• Finite pixel, Finite pixel, voxelvoxel sizesize•• Average integral or average exponentialAverage integral or average exponential

(arithmetic vs. geometric average)(arithmetic vs. geometric average)•• Partial volume effectsPartial volume effects•• MotionMotion•• ScatteringScattering•• Limited angle tomographyLimited angle tomography•• Region of interestRegion of interest•• Scanner implementations: beam hardening Scanner implementations: beam hardening

correction, sampling, etc.correction, sampling, etc.

∫ ),( EyEdN


83


Signal Processing

Information Theory

ComplexityComplexity--constrained processingconstrained processingSignal processing on graphsSignal processing on graphsDistributed signal processingDistributed signal processingDistributed information theoryDistributed information theoryDistributed computation and Distributed computation and

communicationcommunicationOptimal information extraction, Optimal information extraction,

communication, computationcommunication, computation


84

Limits of Information TheoryLimits of Information Theory•• Information theory provides bounds on performance Information theory provides bounds on performance

of communication, compression, and data analysisof communication, compression, and data analysis– Channel coding theorem (capacity)– Entropy, rate-distortion theory– Fisher information

•• Open Problems in Information TheoryOpen Problems in Information Theory– Broadcast channel p(y1,y2|x) capacity region of achievable (R1,R2)

» Depends only on p(y1|x) and p(y2|x); degraded channel known– Distributed source compression achievable (R1,R2, D1,D2)

•• Algorithmic information theory (Algorithmic information theory (KolmogorovKolmogorovcomplexity)complexity)

p(x,yp(x,y))XXnn

YYnn

XXnn

ffyy

ffxxYYnngg ^

^


85

Speculation on FutureSpeculation on Future•• Distributed compressionDistributed compression

– Compression with side information– Analogies to information embedding– Reduced communication rates

•• Broadcast channel models Broadcast channel models – Appropriate for motes– Communication at different rates using a common signal– Reduced communication rates

•• Tradeoffs in communication and computationTradeoffs in communication and computation

•• Mobile computing: cheap Mobile computing: cheap expensive expensive cheapcheap– mobile base station + network mobile


86

ConclusionsConclusions

•• Information Signal ProcessingInformation Signal Processing—— DSP, Information Theory, Computation and DSP, Information Theory, Computation and

CommunicationCommunication

•• Role of Graphical ModelsRole of Graphical Models•• Message Passing EM AlgorithmsMessage Passing EM Algorithms•• Iterative Equalization and DecodingIterative Equalization and Decoding•• XX--Ray CT ImagingRay CT Imaging


87


88

Future WorkFuture Work

Signal Processing

Information Theory

Fast algorithmsFast algorithms

Optimal communicationOptimal communication

Distributed information Distributed information theory theory



89

MultigridMultigrid and and MultiresolutionMultiresolutionComment: NIBIB ProposalComment: NIBIB Proposal

•• MotivationMotivation– Speed of computations– Multiresolution capabilities– Regularization with adaptive resolution– Complexity regularization

•• C. C. BoumanBouman, Oh, et al., Purdue, Oh, et al., Purdue•• Surrogate function viewSurrogate function view

– Original function difficult to minimize directly– Upper bound using a convex function– Minimize upper bound

•• New ideasNew ideas– Guaranteed monotonicity in multigrid– Multigrid alternating minimization algorithms

•• DifficultyDifficulty– Dual representations in parameters and means


90

MultigridMultigrid BasisBasis

)()()(),(exp),():(

):()():(

)(ln)()):(||(

10 yExcxyhEyIyg

ygydyg

ydydgdI

E x

I

iii

y

βµµ

µµ

µ

+⎟⎠

⎞⎜⎝

⎛−=

+−=⋅

∑ ∑ ∑

∑

∈ =

∈

X

Y

[ ] ( )sother term

)()(exp)(

1)(ˆ)()()(~)||(

)()()(

)(),(exp)()(),(exp),(

)()()()(),(exp),(),(

),(),(),(),(ln),()||(minmin

)1(

1

)1()()(

1

)1(

1

)(0

10

+

∆−+∆+≤

⎟⎟⎠

⎞⎜⎜⎝

⎛∆−⎟

⎠

⎞⎜⎝

⎛−

=⎟⎟⎠

⎞⎜⎜⎝

⎛−=

+−=

+

∈ =

+

∈ =

+

∈ =

∈ =

∈∈

∑∑

∑∑∑ ∑

∑ ∑

∑∑

xcxZxZ

xbxcxcxbqpI

xcxZxZ

ExyhExcxyhEyI

ExZxZxcxyhEyIEyq

EyqEypEyqEypEypqpI

kii

x

I

i ii

ki

kii

k

x

I

i

kii

i

i

x

I

ii

ki

x

I

ii

i

ii

y Epq

X

XX

X

YL

µµ

µ

Restrict updates to a subsetAll inequalities hold


91

DSP DSP ISPISP

•• SamplingSampling– Uniform shifts xn = x(t-nT)– Filtering: Fourier transform (FFT) – linearity, implied stationarity– Time space space-time

•• DistributedDistributed– Signals– Sensing– Computation

∑−

=

−=1

0

/2N

n

Nknjnk exX π

∑−

=

−=1

0

||,2N

n

Nnkjnk exX π

X(0)X(1)X(2)X(3)X(4)X(5)X(6)X(7)

x(0)x(4)x(2)x(6)x(1)x(5)x(3)x(7)


92


93


94


95


96


97

DSP DSP ISPISP

•• Digital Signal ProcessingDigital Signal Processing

•• DiscreteDiscrete--Time Signal ProcessingTime Signal Processing

•• ‘‘DistributedDistributed’’ Signal ProcessingSignal Processing

•• Information Theory Information Theory Bits, BitBits, Bit--Rates, DistortionRates, Distortion

•• ‘‘DistributedDistributed’’ Information TheoryInformation Theory

•• Information Signal ProcessingInformation Signal Processing


98

Signal Processing

Information Theory Computations


99

Signal Processing

Information Theory Computations

Documents

Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal