Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal...

Preview:

Citation preview

Information Signal ProcessingInformation Signal ProcessingJoseph A. OJoseph A. O’’SullivanSullivan

Electronic Systems and Signals Research LaboratoryCenter for Security Technologies

Department of Electrical and Systems Engineering Washington University

jao@wustl.eduhttp://essrl.wustl.edu/~jao

Supported by: ONR, NSF, NIH, Boeing Foundation, DARPASpecial thanks to Naveen Singla

J. A. O’Sullivan. 05/25/2004Information Signal Processing

2

CollaboratorsCollaborators

Jasenka BenacMichael D. DeVoreAndrew (Lichun) LiClayton MillerLee MontagninoRyan Murphy Natalia SchmidNaveen SinglaBrandon WestoverShenyu Yan

G. James BlaineRoger ChamberlainMark FranklinDaniel R. FuhrmannRonald S. IndeckChenyang LuPierre Moulin, UIUCMarcel MullerRobert PlessDavid G. PolitteChrysanthe PrezaAndrew Singer, UIUC Donald L. SnyderBruce R. WhitingJeffrey F. Williamson, VCULihao Xu

Faculty Current and Former Students

J. A. O’Sullivan. 05/25/2004Information Signal Processing

3

OutlineOutline

• DSP ISP- Signal processing - Time, space, samples- Information, distortion, transmission

• Graphical Models- Data models - Computational and communication models- X-ray CT imaging- Iterative decoding

• Message Passing EM Algorithms- Graphical models- Projections

• Applications Revisited• Speculation on Trends• Conclusions

J. A. O’Sullivan. 05/25/2004Information Signal Processing

4

OutlineOutline

• DSP ISP- Data models, computational models, algorithms- Central role of information

• Graphical Data Models- X-ray CT imaging- Iterative decoding

• Message Passing EM Algorithms

• Applications Revisited• Speculation on Trends• Conclusions

J. A. O’Sullivan. 05/25/2004Information Signal Processing

5

Signal Processing

Information TheoryComputation andCommunication

FFTFFT

MultiresolutionMultiresolutionanalysisanalysis

J. A. O’Sullivan. 05/25/2004Information Signal Processing

6

X(0)

X(1)

X(2)

X(3)

X(4)

X(5)

X(6)

X(7)

x(0)

x(4)

x(2)

x(6)

x(1)

x(5)

x(3)

x(7)

FFT

J. A. O’Sullivan. 05/25/2004Information Signal Processing

7

Signal Processing

Information Theory

Numerical analysisNumerical analysisProcessors: parallel, Processors: parallel,

ASIC, etc.ASIC, etc.Systolic architecturesSystolic architecturesFFTWFFTW

Computation andCommunication

J. A. O’Sullivan. 05/25/2004Information Signal Processing

8

Signal Processing

Information Theory

Numerical analysisNumerical analysisProcessors and Processors and

architecturesarchitecturesFFT, FFTWFFT, FFTWTransversal filtersTransversal filters

MRAMRA

Complexity theoryComplexity theoryGraphical modelsGraphical models

KalmanKalman filtersfiltersCompression Compression

Computation andCommunication

J. A. O’Sullivan. 05/25/2004Information Signal Processing

9

Computation andCommunication

Information Signal ProcessingInformation Signal Processing

Signal Processing

Information Theory

ComplexityComplexity--constrained processingconstrained processingSignal processing on graphsSignal processing on graphsDistributed signal processingDistributed signal processingDistributed information theoryDistributed information theoryDistributed computation and Distributed computation and

communicationcommunicationOptimal information extraction, Optimal information extraction,

communication, computationcommunication, computation

J. A. O’Sullivan. 05/25/2004Information Signal Processing

10

Distributed sensing,communication, computation

““The architecture for a The architecture for a fully netted maritime forcefully netted maritime force””

J. A. O’Sullivan. 05/25/2004Information Signal Processing

11

Wireless Sensor NetworksWireless Sensor Networks

http://www.greatduckisland.net/index.php

Great Duck Island Habitat

J. A. O’Sullivan. 05/25/2004Information Signal Processing

12

J. A. O’Sullivan. 05/25/2004Information Signal Processing

13

Information Signal ProcessingInformation Signal Processing• Measure information quantitatively

– Entropy rates, mutual information, relative entropy– Rate-distortion theory

• Graphical data models– Forward likelihood models for data– Priors, penalties, and multiresolution models– Abstractions to random graphs

• Constrained implementations– Parameterize implementations: communication rates,

computational complexity, computation time

• Principled approach to complexity-constrained signal modeling, data analysis, and algorithm development

J. A. O’Sullivan. 05/25/2004Information Signal Processing

14

OutlineOutline

• DSP ISP- Data models, computational models, algorithms- Central role of information

• Graphical Data Models- X-ray CT imaging- Iterative decoding

• Message Passing EM Algorithms

• Applications Revisited• Speculation on Trends• Conclusions

J. A. O’Sullivan. 05/25/2004Information Signal Processing

15

Graphical Data ModelsGraphical Data Models

•• Model 1: Model 1: y = Hsy = Hs– y is n × 1, s is m × 1,

and H is n × m– yj depends on sk if hjk ≠ 0– Defines a graphical model

•• Model 2:Model 2:

– Neighborhood structure– Bipartite graph model

•• Model 3:Model 3:– RVs on edges of graph

k

j∏ ∈=

jkj jksypp ))(,|()|( sy

∏ ∏∈

=k kj

kjk sxpp)(

)|()|( sx

∏ ∈=j

jkj jkxypp ))(,|()|( xy

J. A. O’Sullivan. 05/25/2004Information Signal Processing

16

Tomography

S

D

Nonrandom Graphs

Line integrals through patientQuantization point spread function

weights on edges of graphHelps organize computations

Siemens Somotom Emotion

J. A. O’Sullivan. 05/25/2004Information Signal Processing

17

Computational ModelsComputational Models

System model accounts for:• Information extraction

problem definition• Compression of sensor data• Network throughput• Processor cycles per

instruction• Size of processor local

memory• Communication bandwidth

of each link• etc.

J. A. O’Sullivan. 05/25/2004Information Signal Processing

18

Computational ModelsComputational Models

• Local resources plus remote• Communicate observation as well as classification

- Human in the loop- Remote contribution to classification when available

• Dynamic resource availability• Sequence of partial classifications (an,θn)

J. A. O’Sullivan. 05/25/2004Information Signal Processing

19

Progress: Computational Graph Progress: Computational Graph Same as Data Model GraphSame as Data Model Graph

•• Message passing algorithmsMessage passing algorithms– Pearl’s belief propagation– Iterative decoding

» Turbo-codes, parallel concatenated codes» Low density parity check codes» Repeat-accumulate codes, serial concatenated codes

– Iterative equalization and decoding

•• ExpectationExpectation--Maximization (EM) AlgorithmsMaximization (EM) Algorithms– Graphical models– General problem– Gaussian, Poisson (emission tomography, transmission

tomography)– Abstract examples on random graphs

J. A. O’Sullivan. 05/25/2004Information Signal Processing

20

⎟⎟⎟⎟⎟⎟⎟⎟⎟

⎜⎜⎜⎜⎜⎜⎜⎜⎜

=

10011010101000010101010100011010000101010101100110010100100101010010100010100010101101001010100110

H Regular (3,6) n=14

Random Graphs

Comment: LDPC parity check matrix

J. A. O’Sullivan. 05/25/2004Information Signal Processing

21

⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟

⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜

=

001010100000100100000010010001001000000100010001100000001100001000100010010001010000000010000101

H Regular (2,3) n=12

Random Graphs

J. A. O’Sullivan. 05/25/2004Information Signal Processing

22

⎟⎟⎟⎟⎟⎟⎟⎟⎟

⎜⎜⎜⎜⎜⎜⎜⎜⎜

=

10011010101000010101010100011010000101010110101010010100100101010010100010100010110101001001010110

H Irregular n=14

Random Graphs

J. A. O’Sullivan. 05/25/2004Information Signal Processing

23

Tomography

S

D

J. A. O’Sullivan. 05/25/2004Information Signal Processing

24

k

j

νjk

µjk

µ

µ µ

µ

νν

),(

),()1(

')1(

)('

)1(

stateg

statefmkjk

mjk

mjkj

mjk

++

+

=

=

µν

νµ

Message Passing AlgorithmsMessage Passing Algorithms

J. A. O’Sullivan. 05/25/2004Information Signal Processing

25

TomographyS

D

J. A. O’Sullivan. 05/25/2004Information Signal Processing

26

J. A. O’Sullivan. 05/25/2004Information Signal Processing

27

Iterative Decoding Message PassingIterative Decoding Message Passing

∑∈

+ =kjk

mjk

mjk

\)('

)('

)1( νµ

∏∈

++

−=jkj

mkjz

mjk k

\)('

)1('

)1(

2tanh)1(

2tanh

µν

Codeword Bit Nodes

zk

xj

Check Nodes

jkµ

jkν

J. A. O’Sullivan. 05/25/2004Information Signal Processing

28

Iterative Decoding Message PassingIterative Decoding Message Passing

-6

-5

-4

-3

-2

-1

0

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8

SNR [dB]

Bit

erro

r rat

e, L

og B

ase

10 ISI-freeBIAWGN Capacity

[10000,5000] regular (3,6) matrix[10000,5000] regular (3,6) matrix

J. A. O’Sullivan. 05/25/2004Information Signal Processing

29

OutlineOutline

• DSP ISP- Data models, computational models, algorithms- Central role of information

• Graphical Data Models- X-ray CT imaging- Iterative decoding

• Message Passing EM Algorithms

• Applications Revisited• Speculation on Trends• Conclusions

J. A. O’Sullivan. 05/25/2004Information Signal Processing

30

ML Problem: { }xsxxys

dpp∫ )|()|(lnmax

EM Algorithm: xsxxy

syxsyxs

dpp∫ ⎥

⎤⎢⎣

⎡ ΦΦ

Φ )|()|(),|(ln),|(minmin

EM AlgorithmEM Algorithmsk

yj

Hidden Data

Incomplete Data

xjk

J. A. O’Sullivan. 05/25/2004Information Signal Processing

31

VariationalVariational RepresentationsRepresentations•• Convex Decomposition LemmaConvex Decomposition Lemma. Let f be convex. Then

• Special Case: f is ln

• Basis for EM; see also De Pierro, Lange, Fessler

∑ ∑≥=

iii

i ii

irii

rr

xfrxf

0,1

)()( 1

⎭⎬⎫

⎩⎨⎧

=Φ≥ΦΦ=

ΦΦ−=⎟

⎞⎜⎝

∑∑∈Φ

iii

i i

ii

ii q

q

1,0:

lnminln

P

P

J. A. O’Sullivan. 05/25/2004Information Signal Processing

32

EM AlgorithmEM Algorithm

∫=Φ +

')|'()'|()|()|(),|(

)(

)()()1(

xsxxysxxysyx

dpppp

m

mmm

xsxsyxss

dp, mmm )|(ln)|(argmax )()1()1( ∫ ++ Φ=

Assume the factorizations (Model 3):

∏ ∏∈

=k kj

kjk sxpp)(

)|()|( sx

∏ ∈=j

jkj jkxypp ))(,|()|( xy

These computations become local and thus message passing

In general, these are global computations

J. A. O’Sullivan. 05/25/2004Information Signal Processing

33

Message Passing EM AlgorithmMessage Passing EM Algorithm

∫ ∏∏∫ ∏∏

∈∈

≠∈∈+

∈=Φ

)(''

)('

)('''

'),(''

)('

)('''

1(

)|())(',|(

)|())(',|(),|(

jkjk

jk

mkjkjkj

kkjkjk

jk

mkjkjkj

(m)jjk

)m

dxsxpjkxyp

dxsxpjkxypyx

KK

KK

K

Ks

jkkjkm

jjkm

s

mk dxsxp,yxs

k

)|(ln)|(argmax )()1()1( ∫ ++ Φ= s

sk

yj

Input Data

Measured Data

xjk

)( jkxΦ

ks

J. A. O’Sullivan. 05/25/2004Information Signal Processing

34

GaussianGaussian--MAPMAP

)2

,0(~

),0(~;

0 Iw

IswHsy

NN

PN+=

∑∈

+

+=

)(

)(

0

)1( ˆ

|)(|2|)(|

ˆkj

ljk

lk x

jNkP

Ps

⎟⎟⎠

⎞⎜⎜⎝

⎛−+= ∑

+++

)('

)1('

)1()1( ˆ|)(|

1ˆˆjk

lkj

lk

ljk sy

jsx

sk

yj

Input Data

Measured Data

xjk

jkx

ks

J. A. O’Sullivan. 05/25/2004Information Signal Processing

35

GaussianGaussian--MAPMAP[10000,5000] regular (3,6) matrix[10000,5000] regular (3,6) matrix

J. A. O’Sullivan. 05/25/2004Information Signal Processing

36

Emission TomographyEmission Tomography

y ~ Poisson(Hλ)λ: Mean of emitted photons

∑∈

+ =)(

)()(

)1( ˆ|)(|

ˆˆkj

mj

mkm

k qk

λλ

∑∈

=

)('

)('

)(

ˆˆ

jk

mk

jmj

yq

λ

k

yj

Pixels

Measured Data

jqkλ

J. A. O’Sullivan. 05/25/2004Information Signal Processing

37

Emission TomographyEmission Tomography[10000,5000] regular (3,6) matrix[10000,5000] regular (3,6) matrix

J. A. O’Sullivan. 05/25/2004Information Signal Processing

38

Emission TomographyEmission Tomography[50000,1000] regular (3,150) matrix[50000,1000] regular (3,150) matrix

J. A. O’Sullivan. 05/25/2004Information Signal Processing

39

Transmission TomographyTransmission Tomography

y ~ Poisson(I0exp(-Hλ))λ: photon attenuation ⎥

⎥⎥

⎢⎢⎢

⎡−=

∑∑

∈+

)(

)()()()1( ln1ˆˆ

kj

mj

kjj

mk

mk q

y

zλλ

⎥⎦

⎤⎢⎣

⎡−= ∑

∈ )(

)(0

)( ˆexpjk

mk

mj Iq λ

k

yj

Pixels

Measured Data

jqkλ

J. A. O’Sullivan. 05/25/2004Information Signal Processing

40

[10000,5000] regular (3,6) matrix[10000,5000] regular (3,6) matrix

Transmission TomographyTransmission Tomography

J. A. O’Sullivan. 05/25/2004Information Signal Processing

41

Comments on DetailsComments on Details

•• Information geometry basisInformation geometry basis

•• Easily extended to arbitrary Easily extended to arbitrary HH

•• Low density Low density sparsesparse

•• Constraints in iterative decoding vs. forward Constraints in iterative decoding vs. forward modelmodel

•• Performance is intimately connected to Performance is intimately connected to graphgraph

J. A. O’Sullivan. 05/25/2004Information Signal Processing

42

OutlineOutline• DSP ISP

- Data models, computational models, algorithms- Central role of information

• Graphical Data Models- X-ray CT imaging- Iterative decoding

• Message Passing EM Algorithms

• Applications Revisited- Iterative decoding- X-ray CT imaging

• Speculation on Trends• Conclusions

J. A. O’Sullivan. 05/25/2004Information Signal Processing

43

Science and technologyScience and technologypotentially yield potentially yield

6 Tb/in6 Tb/in22

courtesy R. S. courtesy R. S. IndeckIndeck

Patterned Magnetic Media

J. A. O’Sullivan. 05/25/2004Information Signal Processing

44

Advanced Recording MediaAdvanced Recording MediaBluBlu--Ray DiscRay Disc

Next-generation Optical Disc Video Recording Format

J. A. O’Sullivan. 05/25/2004Information Signal Processing

45

2D Intersymbol Interference2D Intersymbol Interference

111111

11111111

21

21

11211

ΛΛΛ

ΜΟΜΜΟ

ΛΛΛ

kkkk

k

xxx

xxxx

1111110

110

12

1111110

100100

+++++

+

+

+

+

kkkkkk

kkkkkk

k

kk

k

rrrrrrrr

rrrrrrrr

ΛΛΛΛ

ΜΟΜΜΟΜΜ

ΛΛΛΛΛw(i,j)

⎟⎟⎠

⎞⎜⎜⎝

⎛=

25.05.05.01

h ⊕

Includes Guard Band

jijijijijiji wxxxxr ,1,11,,1,, 25.05.05.0 ++++= −−−−

Singla et al., “Iterative decoding and equalization for 2-D recording channels,” IEEE Trans. Magn., Sept. 2002.

J. A. O’Sullivan. 05/25/2004Information Signal Processing

46

Full Graph Message PassingFull Graph Message Passing

Measured Data Nodes (r)

Codeword Bit Nodes (x)

Check Nodes (z)

jijijijijiji wxxxxr ,1,11,,1,, 25.05.05.0 ++++= −−−−⎟⎟⎠

⎞⎜⎜⎝

⎛=

25.05.05.01

h

J. A. O’Sullivan. 05/25/2004Information Signal Processing

47

Full Graph Message PassingFull Graph Message Passing

•• Codeword bit nodes to check nodesCodeword bit nodes to check nodes

•• Check nodes to codeword bit nodes Check nodes to codeword bit nodes

∑∑∈

−→

−→→ +=

zxNz

lxz

xNm

lxm

lzx LLL

\)('

)1('

)(

)1()(

∏∈

−→→ −=

xzNx

lzxz

lxz LL

\)('

)1('

)(

2tanh)1(

2tanh

J. A. O’Sullivan. 05/25/2004Information Signal Processing

48

•• Codeword bit nodes to measured data nodesCodeword bit nodes to measured data nodes

•• Measured data nodes to codeword bit nodesMeasured data nodes to codeword bit nodes

∑∑∈

→∈

−→→ +=

)(

)(

\)('

)1('

)(

xNz

lxz

mxNm

lxm

lmx LLL

})\)(':({ )('

)( xmNxLfL lmx

lxm ∈= →→

Full Graph Message PassingFull Graph Message Passing

J. A. O’Sullivan. 05/25/2004Information Signal Processing

49

Full Graph Message Passing

-6

-5

-4

-3

-2

-1

0

0 0.5 1 1.5 2 2.5 3

SNR [dB]

Bit

erro

r rat

e, L

og B

ase

10

ISI-freeFull Graph_50

Full Graph Message PassingFull Graph Message Passing[10000,5000] regular (3,6) matrix[10000,5000] regular (3,6) matrix

J. A. O’Sullivan. 05/25/2004Information Signal Processing

50

Full Graph AnalysisFull Graph Analysis

Length 4 cycles present which degrade performance Length 4 cycles present which degrade performance of messageof message--passing algorithm passing algorithm

x(i+2,j) x(i+2,j+1)

x(i+1,j) x(i+1,j+1)

x(i,j) x(i,j+1)

x(i+2,j+2)

x(i+1,j+2)

x(i,j+2)

r(i+1,j+1)

r(i,j+1)r(i,j)

r(i+1,j)

FromCheckNodes

Kschischang et al., “Factor graphs and the sum-product algorithm,” IEEE Trans. Inform. Theory, Feb. 2001.

J. A. O’Sullivan. 05/25/2004Information Signal Processing

51

Ordered Subsets Message Passing Ordered Subsets Message Passing From Imaging From Imaging –– Data set is grouped into subsets to Data set is grouped into subsets to increase rate of convergence of image increase rate of convergence of image reconstruction algorithmsreconstruction algorithms

For Decoding For Decoding –– Measured data is grouped into Measured data is grouped into subsets to eliminate short length cycles in the subsets to eliminate short length cycles in the Channel ISI graphChannel ISI graph

H. M. Hudson, and R. S. Larkin, “Accelerated image reconstruction using ordered subsets of projection data,” IEEE Trans. Medical Imaging, Dec. 1994

J. A. O’Sullivan. 05/25/2004Information Signal Processing

52

Labeling of data nodes into 4 subsetsLabeling of data nodes into 4 subsetsFor each iteration use data nodes of one label onlyFor each iteration use data nodes of one label only

Labeled ISI GraphLabeled ISI Graph

J. A. O’Sullivan, and N. Singla, “Ordered subsets message-passing,” Int’l Symp. Inform. Theory, Yokohama, Japan 2003.

J. A. O’Sullivan. 05/25/2004Information Signal Processing

53

Ordered Subsets Message Passing

-6

-5

-4

-3

-2

-1

0

0 0.5 1 1.5 2 2.5 3SNR [dB]

Bit

erro

r rat

e in

log1

0

ISI-freeOrdered Subsets_200Full Graph_50

[10000,5000] regular (3,6) matrix[10000,5000] regular (3,6) matrixOrdered Subsets Message PassingOrdered Subsets Message Passing

J. A. O’Sullivan. 05/25/2004Information Signal Processing

54

CT Imaging in Presence of High CT Imaging in Presence of High Density Attenuators (J. Williamson, PI)Density Attenuators (J. Williamson, PI)

Brachytherapy applicators After-loading colpostats

for radiation oncology

Cervical cancer: 50% survival rateDose prediction important

Object-Constrained Computed Tomography (OCCT)

J. A. O’Sullivan. 05/25/2004Information Signal Processing

55

Filtered Back ProjectionFiltered Back Projection

Truth FBP

FBP: inverse Radon transform

J. A. O’Sullivan. 05/25/2004Information Signal Processing

56

Transmission TomographyTransmission Tomography• Source-detector pairs indexed by y; pixels indexed by x• Data d(y) Poisson, means g(y:µ), log likelihood function

• Mean unattenuated counts I0, mean background β• Attenuation function µ(x,E), E energies

• Maximize over µ or ci; equivalently minimize I-divergence

)(),(),(exp),():(

):():(ln)()):(|(

0 yExxyhEyIyg

ygygydgdl

E x

y

βµµ

µµµ

+⎟⎟⎠

⎞⎜⎜⎝

⎛−=

−=⋅

∑ ∑

X

Y

∑=

=I

iii ExcEx

1)()(),( µµ

J. A. O’Sullivan. 05/25/2004Information Signal Processing

57

MaximumMaximum--Likelihood Likelihood Minimum IMinimum I--divergencedivergence

• Poisson distribution

• Poisson distributed data loglikelihood function

• Maximization over µ equivalent to minimization of I-divergence

λλ

λ

λλ

λ λ

+−=

−−==

== −

kkkkI

kkkNP

ek

kNPk

ln)||(

!lnln)(ln!

)(

)(),(),(exp),():(

):()():(

)(ln)()):(||(

):():(ln)()):(|(

0 yExxyhEyIyg

ygydyg

ydydgdI

ygygydgdl

E x

y

y

βµµ

µµ

µ

µµµ

+⎟⎠

⎞⎜⎝

⎛−=

+−=⋅

−=⋅

∑ ∑

X

Y

Y

J. A. O’Sullivan. 05/25/2004Information Signal Processing

58

Maximum Likelihood Maximum Likelihood Minimum IMinimum I--DivergenceDivergence

Difficulties: log of sum, sums in exponent

)()()(),(exp),():(

):()():(

)(ln)()):(||(

):():(ln)()):(|(

10 yExcxyhEyIyg

ygydyg

ydydgdI

ygygydgdl

E x

I

iii

y

y

βµµ

µµ

µ

µµµ

+⎟⎟⎠

⎞⎜⎜⎝

⎛−=

+−=⋅

−=⋅

∑ ∑ ∑

∈ =

X

Y

Y

J. A. O’Sullivan. 05/25/2004Information Signal Processing

59

Interpretation: Compare predicted data to measured data via ratio of backprojectionsUpdate estimate using a normalization constant

Comments: choice for constants; monotonic convergence;linear convergence; fixed points satisfy Kuhn-Tucker conditions; constraints easily incorporated

)(~)(ˆ

ln)(

1)(ˆ)(ˆ)(

)()()1(

xbxb

xZxcxc l

i

li

i

li

li −=+

New Alternating Minimization AlgorithmNew Alternating Minimization Algorithmfor Transmission Tomographyfor Transmission Tomography

J. A. O’Sullivan. 05/25/2004Information Signal Processing

60David G. PolitteOctober 31, 2002

Mini CT, AM Iteration 0000001

J. A. O’Sullivan. 05/25/2004Information Signal Processing

61David G. PolitteOctober 31, 2002

Mini CT, AM Iteration 0000002

J. A. O’Sullivan. 05/25/2004Information Signal Processing

62David G. PolitteOctober 31, 2002

Mini CT, AM Iteration 0000005

J. A. O’Sullivan. 05/25/2004Information Signal Processing

63David G. PolitteOctober 31, 2002

Mini CT, AM Iteration 0000010

J. A. O’Sullivan. 05/25/2004Information Signal Processing

64David G. PolitteOctober 31, 2002

Mini CT, AM Iteration 0000020

J. A. O’Sullivan. 05/25/2004Information Signal Processing

65David G. PolitteOctober 31, 2002

Mini CT, AM Iteration 0000050

J. A. O’Sullivan. 05/25/2004Information Signal Processing

66David G. PolitteOctober 31, 2002

Mini CT, AM Iteration 0000100

J. A. O’Sullivan. 05/25/2004Information Signal Processing

67David G. PolitteOctober 31, 2002

Mini CT, AM Iteration 0000200

J. A. O’Sullivan. 05/25/2004Information Signal Processing

68David G. PolitteOctober 31, 2002

Mini CT, AM Iteration 0000500

J. A. O’Sullivan. 05/25/2004Information Signal Processing

69David G. PolitteOctober 31, 2002

Mini CT, AM Iteration 0001000

J. A. O’Sullivan. 05/25/2004Information Signal Processing

70David G. PolitteOctober 31, 2002

Mini CT, AM Iteration 0002000

J. A. O’Sullivan. 05/25/2004Information Signal Processing

71David G. PolitteOctober 31, 2002

Mini CT, AM Iteration 0005000

J. A. O’Sullivan. 05/25/2004Information Signal Processing

72David G. PolitteOctober 31, 2002

Mini CT, AM Iteration 0010000

J. A. O’Sullivan. 05/25/2004Information Signal Processing

73David G. PolitteOctober 31, 2002

Mini CT, AM Iteration 0020000

J. A. O’Sullivan. 05/25/2004Information Signal Processing

74David G. PolitteOctober 31, 2002

Mini CT, AM Iteration 0050000

J. A. O’Sullivan. 05/25/2004Information Signal Processing

75David G. PolitteOctober 31, 2002

Mini CT, AM Iteration 0100000

J. A. O’Sullivan. 05/25/2004Information Signal Processing

76David G. PolitteOctober 31, 2002

Mini CT, AM Iteration 0200000

J. A. O’Sullivan. 05/25/2004Information Signal Processing

77David G. PolitteOctober 31, 2002

Mini CT, AM Iteration 0500000

J. A. O’Sullivan. 05/25/2004Information Signal Processing

78David G. PolitteOctober 31, 2002

Mini CT, AM Iteration 1000000

J. A. O’Sullivan. 05/25/2004Information Signal Processing

79

Our Plans in CT ImagingOur Plans in CT Imaging

•• CT CT MultirowMultirow SinogramSinogram data: data: 1408 1408 ×× 768 768 ×× nndd ×× nnzz

– where nd is the number of detector rows and nz is the number of gantry rotations

•• Fully 3Fully 3--D Implementations for D Implementations for Quantitative CTQuantitative CT

•• SpeedSpeed--up: Ordered Subsets, up: Ordered Subsets, MultigridMultigrid Methods, Parallel Methods, Parallel Implementations on Clusters Implementations on Clusters of PCsof PCs

•• Future: PETFuture: PET--CT CT Siemens Somotom Emotion

J. A. O’Sullivan. 05/25/2004Information Signal Processing

80

Slide and data from R. Laforest and M. Mintun.

PETCT-211

J. A. O’Sullivan. 05/25/2004Information Signal Processing

81

PETCT-165

Slide and data from R. Laforest and M. Mintun.

J. A. O’Sullivan. 05/25/2004Information Signal Processing

82

Additional Algorithm/DetectorAdditional Algorithm/DetectorModel DevelopmentModel Development

•• RegularizationRegularization•• Energy integrating detectors Energy integrating detectors •• Finite detector size, better source modelFinite detector size, better source model•• Finite pixel, Finite pixel, voxelvoxel sizesize•• Average integral or average exponentialAverage integral or average exponential

(arithmetic vs. geometric average)(arithmetic vs. geometric average)•• Partial volume effectsPartial volume effects•• MotionMotion•• ScatteringScattering•• Limited angle tomographyLimited angle tomography•• Region of interestRegion of interest•• Scanner implementations: beam hardening Scanner implementations: beam hardening

correction, sampling, etc.correction, sampling, etc.

∫ ),( EyEdN

J. A. O’Sullivan. 05/25/2004Information Signal Processing

83

Computation andCommunication

Signal Processing

Information Theory

ComplexityComplexity--constrained processingconstrained processingSignal processing on graphsSignal processing on graphsDistributed signal processingDistributed signal processingDistributed information theoryDistributed information theoryDistributed computation and Distributed computation and

communicationcommunicationOptimal information extraction, Optimal information extraction,

communication, computationcommunication, computation

J. A. O’Sullivan. 05/25/2004Information Signal Processing

84

Limits of Information TheoryLimits of Information Theory•• Information theory provides bounds on performance Information theory provides bounds on performance

of communication, compression, and data analysisof communication, compression, and data analysis– Channel coding theorem (capacity)– Entropy, rate-distortion theory– Fisher information

•• Open Problems in Information TheoryOpen Problems in Information Theory– Broadcast channel p(y1,y2|x) capacity region of achievable (R1,R2)

» Depends only on p(y1|x) and p(y2|x); degraded channel known– Distributed source compression achievable (R1,R2, D1,D2)

•• Algorithmic information theory (Algorithmic information theory (KolmogorovKolmogorovcomplexity)complexity)

p(x,yp(x,y))XXnn

YYnn

XXnn

ffyy

ffxxYYnngg ^

^

J. A. O’Sullivan. 05/25/2004Information Signal Processing

85

Speculation on FutureSpeculation on Future•• Distributed compressionDistributed compression

– Compression with side information– Analogies to information embedding– Reduced communication rates

•• Broadcast channel models Broadcast channel models – Appropriate for motes– Communication at different rates using a common signal– Reduced communication rates

•• Tradeoffs in communication and computationTradeoffs in communication and computation

•• Mobile computing: cheap Mobile computing: cheap expensive expensive cheapcheap– mobile base station + network mobile

J. A. O’Sullivan. 05/25/2004Information Signal Processing

86

ConclusionsConclusions

•• Information Signal ProcessingInformation Signal Processing—— DSP, Information Theory, Computation and DSP, Information Theory, Computation and

CommunicationCommunication

•• Role of Graphical ModelsRole of Graphical Models•• Message Passing EM AlgorithmsMessage Passing EM Algorithms•• Iterative Equalization and DecodingIterative Equalization and Decoding•• XX--Ray CT ImagingRay CT Imaging

J. A. O’Sullivan. 05/25/2004Information Signal Processing

87

J. A. O’Sullivan. 05/25/2004Information Signal Processing

88

Future WorkFuture Work

Signal Processing

Information Theory

Fast algorithmsFast algorithms

Optimal communicationOptimal communication

Distributed information Distributed information theory theory

Computation andCommunication

J. A. O’Sullivan. 05/25/2004Information Signal Processing

89

MultigridMultigrid and and MultiresolutionMultiresolutionComment: NIBIB ProposalComment: NIBIB Proposal

•• MotivationMotivation– Speed of computations– Multiresolution capabilities– Regularization with adaptive resolution– Complexity regularization

•• C. C. BoumanBouman, Oh, et al., Purdue, Oh, et al., Purdue•• Surrogate function viewSurrogate function view

– Original function difficult to minimize directly– Upper bound using a convex function– Minimize upper bound

•• New ideasNew ideas– Guaranteed monotonicity in multigrid– Multigrid alternating minimization algorithms

•• DifficultyDifficulty– Dual representations in parameters and means

J. A. O’Sullivan. 05/25/2004Information Signal Processing

90

MultigridMultigrid BasisBasis

)()()(),(exp),():(

):()():(

)(ln)()):(||(

10 yExcxyhEyIyg

ygydyg

ydydgdI

E x

I

iii

y

βµµ

µµ

µ

+⎟⎠

⎞⎜⎝

⎛−=

+−=⋅

∑ ∑ ∑

∈ =

X

Y

[ ] ( )sother term

)()(exp)(

1)(ˆ)()()(~)||(

)()()(

)(),(exp)()(),(exp),(

)()()()(),(exp),(),(

),(),(),(),(ln),()||(minmin

)1(

1

)1()()(

1

)1(

1

)(0

10

+

∆−+∆+≤

⎟⎟⎠

⎞⎜⎜⎝

⎛∆−⎟

⎞⎜⎝

⎛−

=⎟⎟⎠

⎞⎜⎜⎝

⎛−=

+−=

+

∈ =

+

∈ =

+

∈ =

∈ =

∈∈

∑∑

∑∑∑ ∑

∑ ∑

∑∑

xcxZxZ

xbxcxcxbqpI

xcxZxZ

ExyhExcxyhEyI

ExZxZxcxyhEyIEyq

EyqEypEyqEypEypqpI

kii

x

I

i ii

ki

kii

k

x

I

i

kii

i

i

x

I

ii

ki

x

I

ii

i

ii

y Epq

X

XX

X

YL

µµ

µ

Restrict updates to a subsetAll inequalities hold

J. A. O’Sullivan. 05/25/2004Information Signal Processing

91

DSP DSP ISPISP

•• SamplingSampling– Uniform shifts xn = x(t-nT)– Filtering: Fourier transform (FFT) – linearity, implied stationarity– Time space space-time

•• DistributedDistributed– Signals– Sensing– Computation

∑−

=

−=1

0

/2N

n

Nknjnk exX π

∑−

=

−=1

0

||,2N

n

Nnkjnk exX π

X(0)X(1)X(2)X(3)X(4)X(5)X(6)X(7)

x(0)x(4)x(2)x(6)x(1)x(5)x(3)x(7)

J. A. O’Sullivan. 05/25/2004Information Signal Processing

92

J. A. O’Sullivan. 05/25/2004Information Signal Processing

93

J. A. O’Sullivan. 05/25/2004Information Signal Processing

94

J. A. O’Sullivan. 05/25/2004Information Signal Processing

95

J. A. O’Sullivan. 05/25/2004Information Signal Processing

96

J. A. O’Sullivan. 05/25/2004Information Signal Processing

97

DSP DSP ISPISP

•• Digital Signal ProcessingDigital Signal Processing

•• DiscreteDiscrete--Time Signal ProcessingTime Signal Processing

•• ‘‘DistributedDistributed’’ Signal ProcessingSignal Processing

•• Information Theory Information Theory Bits, BitBits, Bit--Rates, DistortionRates, Distortion

•• ‘‘DistributedDistributed’’ Information TheoryInformation Theory

•• Information Signal ProcessingInformation Signal Processing

J. A. O’Sullivan. 05/25/2004Information Signal Processing

98

Signal Processing

Information Theory Computations

J. A. O’Sullivan. 05/25/2004Information Signal Processing

99

Signal Processing

Information Theory Computations

Recommended