21055823 (1) this is a statistical signal processing complementary for index clarification detailed image analysis of grandiose field pacants

7/30/2019 21055823 (1) this is a statistical signal processing complementary for index clarification detailed image analysis of

1/11

Statistical SignalProcessingDetection, Estimation, and Time Series Analysis

Louis L. ScharfUniversity of Colorado at Boulder

with Cedric Demeure collaborating on Chapters 10 and 11

TTADDISON-WESLEY PUBLISHING COMPANYReading, M assach usetts Menlo Park, California New YorkDon Mills, On tario W okingham, England Am sterdamBonn Sydney Singapo re Tokyo M adrid San Jua n


2/11

Contents

CHAPTER 1Introduction 11.1 Statistical Signal Processing and Related Topics 11.2 The Structure of Statistical Reasoning 31.3 A Detection Problem 41.4 An Estimation Problem 71.5 A Time Series Problem 91.6 Notation and Terminology 11

Probability Distributions 11Linear Models 12References and Comments 18Problems 19

C H A P T E RRudiments of Linear Algebraand Multivariate Normal Theory 232.1 Vector Spaces 24

Euclidean Space 24Hilbert Space 25Matrices 252.2 Linear Indepen dence 25

Gram Determinant 26Sequences of Gram Determinants 27Cholesky Factors of the Gram Matrix 29

IX


3/11

4x Contents

2 .3 QR Factors 31Gram -Schmidt Procedure 32Househ older Transformation 33Given s Transformation 362.4 Linear Subspaces 37Basis 38Direct Subsp aces 38Unicity 38Dim ension, Ran k, and Nullity 39Linear Equations 40Decomposition of R" 41

2.5 Hermitian Matrices 42The Eigenvalues of a Herm itian Matrix Are Real 42The Eigenvectors of a Herm itian M atrix Are Orthogonal 42Herm itian M atrices Are Diago nalizable 42

2.6 Singular Value Deco mpo sition 43Range and Null Space 45Low Rank Approximation 45Reso lution (or De com position ) of Identity 46

2.7 Projections, Rotations, and Pseudoinv erses 46Projections 47Rotations 48Pseudoinverse 49Orthogonal Representations 49

2.8 Quadratic Forms 512.9 M atrix Inversion Formulas 522.10 The M ultivariate Norm al Distribution 55

Cha racteristic Function 56The Bivariate Norm al Distribution 57Linea r Transformations 59Ana lysis and Synthes is 60Diagonalizing Transformations 61

2.11 Quadratic Forms in M VN Random Variables 62Quadratic Forms Using Projection M atrices 64Asymptotics 66

References and Com men ts 66Problems 67

CHAPTER OSufficiency and MVUB E stim ato rs 773.1 Sufficiency 78

Discrete Ran dom Variables 79Continuous Random Variables 82No nsing ular Transform ations and Sufficiency 85


4/11

Contents xi

3.2 M inimal and Co mp lete Sufficient Statistics 85Minimality 87Completeness 88Unbiasedness 88Com pleteness Ensures M inimality 88Summary 89

3.3 Sufficiency and Co m pleten ess in the Exp one ntial Fam ily 89Sufficiency 89Completeness 90

3.4 S ufficiency in the Lin ear Statistical M ode l 91Recursive Computation of the Sufficient Statisticin the Line ar Statistical M ode l 91Partitioned M atrix Invers e 92Sufficient Statistic 92W hite Noise 93

3.5 Sufficiency in the Co m pon ents of Variance M ode l 933.6 Minimum Variance Unb iased (MV UB ) Estimators 94

Interpretation 96References and Com me nts 97Problems 97

CHAPTERNeyman-Pearson Detectors 1034.1 Classifying Tests 1044.2 The Testing of Binary Hy pothes es 105

Size or Proba bility of False Alarm 105Pow er or De tection Proba bility 106Receiver Operating Characteristics (ROC ) 107

4.3 The Neyman-Pearson Lem ma 107Choosing the Threshold 108Interpretation 109Geom etrical Properties of the RO C Curve 109North-by-N orthwest: Birdsall 's Insight 1104.4 The M ultivariate Norm al M odel 111Uncomm on M eans and Com mon Covariance 111Linea r Statistical M odel 115Com mon Means and Uncom mon Covariances 116Uncom mon M eans and Uncom mon Variances 117

4.5 Binary Com mun ication 1174.6 Sufficiency 1214.7 The Testing of Com posite Binary Hy potheses 123

Uniformly M ost Powerful Test 1244.8 Invariance 127

Invariance of an Hy pothesis Testing Problem 128


5/11

Contents

Invariant Tests and M axim al Invariant Statistics 132Uniformly M ost Powerful Invarian t Test 135Reduction by Sufficiency and Invarian ce 1354.9 M atched Filters (Normal) 1364.1 0 CFAR M atched Filters (f) 1404.11 M atched Subspace Filters O r) 1454.12 CFAR Matched Subspace Filters (F ) 148

A Com parative Summ ary and a Partial Ordering of Performa nce 1494.1 3 Signal Design 153

Signal Design for De tection 153Con strained Signal Des ign 1544.1 4 Detectors for Gaussian Rando m Signals 157Likelihood Ratios and Qu adratic Detec tors 157Orthogona l Deco mp osition 158Distribution of Log Likelihoo d 158Rank Red uction 1604.15 Linear Disc riminan t Func tions 162

Linear Discrim ination 163An Extremization Problem 163M aximizing Divergence 164References and Com me nts 166Problems 167Appendix: The t, % 2, and F Distributions 174

Central y2 174Central t 175Central F 175Noncentral \2 176Noncentral t illNoncentral F 178Size and Pow er 178

CHAPTER DBayes Detectors 1795.1 Bayes Risk for Hy pothe sis Testing 180

Loss 181Risk 181Bay es Risk 1835.2 Bayes Tests of Simp le Hy pothe ses 183

5.3 M inimax Tests 185Risk Set 185Bay es Tests 187M inimax and M aximin Tests 188Com puting M inimax Tests 189Least Favorable Prior 190


6/11

Contents xiii

5.4 Bayes Tests of M ultiple Hyp otheses 1915.5 M -Orthog onal Signals 1935.6 Com posite M atched Filters and Asso ciative M em ories 198

Application to Asso ciative M em ories 201Summary 2025.7 Likelihood Ra tios, Posterior Proba bilities, and Od ds 202

Bayes Tests 2035.8 Balan ced Tests 204References and Co mm ents 205Problems 206

CHAPTER OMaximum Likelihood Estimators 2096.1 M axim um Likelihood Principle 210

Random Parameters 2136.2 Sufficiency 2166.3 Invariance 2176.4 The Fisher Matrix and the Cram er-Rao Bo und 221Cramer-Rao Bound 222Concentration Ellipses 225Efficiency 226Cram er-Rao Boun d for Functions of Param eters 229Numerical Maximum Likelihood and the StochasticFisher Matrix 2306.5 Nuisan ce Parameters 2316.6 Entropy, Likelihood , and Non linear Least Squares 233 -

Entropy 233Likelihood 234Nonlinear Least Squares 234Comments 2356.7 The M ultivariate Normal M odel 2356.8 The Linear Statistical M odel 2386.9 M od e Identification in the Lin ear Statistical M ode l 239

Maxim um Likelihood Equations 240The Fisher Information M atrix 2416.10 M axim um Likelihood Identification of AR M A Param eters 242M aximum Likelihood Equations 245The Projector P( a) 245Interpretations 246KiSS 247The Fisher Information M atrix 248M ode Identification 2506.11 M aximu m Likelihood Identification of a Signal Subsp ace 252


7/11


8/11

Contents ' xvCHAPTER OMinimum Mean-Squared Error Estimators 3238.1 Conditional Expectation and Orthogo nality 3248.2 M inimu m M ean-Squared Error Estimators 3268.3 Linear Minimu m Mean-Squared Error (LMM SE) Estimators 327

Wiener-Hopf Equation 327Sum mary and Interpretations 3288.4 Low -Rank W iener Filter 3308.5 Linear Prediction 3318.6 The Kalm an Filter 333

Prediction 333Estimation 335Covariance Recursions 336The Kalman Recursions 3378.7 Low -Rank Approx imation of Rand om Vectors 337

Interpretation 338Order Selection 3398.8 Optimum Scalar Quantizers 339

Scalar Quan tizers 340Designing the Op timum Quan tizer 342

8.9 Optimum Block Quan tizers 3468.10 Rank Redu ction and Rate Distortion 349References and Com ments 351Problems ' 352

CHAPTER 9Least Squares 3599.1 The Linear M odel 360

Interpretations 360The Normal Error M odel 364

9.2 Least Squares Solu tions 365Projections 365Signal and Orthogo nal Sub spaces 366Orthogonality 3689.3 Structure of Sub spaces in Least Squ ares Prob lems 371

9.4 Singular Value Decom position 372Synthesis Representation 373Analysis Representations 3739.5 Solving Least Squares Problems 374Cholesky-Factoring the Gram M atrix 37427?-Factoring the M ode l M atrix 375Singular Value Decom position 376


9/11

xvi Contents

9.6 Performance of Least Squares 377Posterior M odel 377Performance 378

9.7 Goo dness of Fit 378Statistician's Pythagorean Theo rem 3799.8 Improvement in SNR 3799.9 Rank Reduction 3809.10 Order Selection in the Linea r Statistical M odel 3819.11 Sequential Least Squ ares 3849.12 Weighted Least Squares 3869.13 Constrained Least Squares 387

Interpretations 388Condition Adjustment 3889.14 Q uadratic M inimization with Linear Constraints 3899.15 Total Least Squ ares 3929.16 Inverse Problems and Und erdetermined Least Squares 393

Characterizing the Class of Solutions 394Minimum -Norm Solution 395Reducing Rank 395Bayes 397Max imum Entropy 398Image Formation 398Newton-Raphson 400

9.17 M ode Identification in the Linea r Statistical M od el 4019.18 Identification of Autoregressive M oving Average Signals and System s 4029.19 Linear Prediction and Prony's M ethod 405

Modified Least Squares 405 Linear Prediction 406Prony's M ethod 4079.20 Least Squares Estimation of Structured Correlation M atrices .409Linear Structure 409Toeplitz M atrix 410Low-Rank Matrix 411Orthonormal Case 413

M ore on the Orthonorm al Case 413References and Co mm ents 415Problems 415

CHAPTER 1 ULinear Prediction 42310.1 Stationary Time Series 424

Wold Representation 426Kolmog orov Representation 427Filtering Interpretations 42 8


10/11

C onte nts . xvii

10.2 Stationary Prediction Theo ry 429Prediction Error Variance 430Prediction Error Variance and Poles and Zero s 431Spectral Flatness 43 3Filtering Interpretation s 43 4

10.3 M aximu m Entropy and Linear Prediction 43610.4 Nonstationary Linear Prediction 438

Synthesis 440Nonstationary Innovations Representation 441Analysis 441Non stationary Predictor Represen tation 44210.5 Fast Algorithms of the Levinson Type 442

Interpretation 446Backward Form 447Filtering Interpretation s 44 7AR Synthesis Lattice 448AR MA Lattice 45010.6 Fast Algorithms of the Schur Type 45110.7 Least Squares Theory of Linear Prediction 452

QR Factors and Sliding W indows 456Sum mary and Interpretations 45910.8 Lattice Algo rithms 460

Initialization 462Recursions for k{ A62Solving for of 462Algebraic Interpretations 463Lattice Interpretations 46 310.9 Prediction in Autoregressive M oving Average Time Series 464

Stationary State-Space Rep resentations 464M arkovian State-Space M odel 466Non stationary (or Innovations) State-Space Represe ntations; 468

10.10 Fast Algorithms of the M orf-Sidhu-Kailath Type 47010.11 Linear Prediction and Likelihood Formu las 47210.12 Differential PC M 47 3References and Co mm ents 475Problems 480

C H A P T E R 1 1Modal Analysis 4 8 311.1 Signal M odel 484

Modal Decom posit ion 485ARM A Impulse Response 486Linear Prediction 488


11/11

xviii Contents

11.2 The Original Prony M ethod 489Filter Coefficients 489Modes 490

11.3 Least Squares Prony M ethod 491Choice of the Data M atrix 492Fast Algorithm s 492Solving for the M ode W eights 493Solving for the Filter Coefficients 49 311.4 Maximu m Likelihood and Exact Least Squares 493

Com pressed Likelihood 49411.5 Total Least Squares 49511.6 Principal Com ponen t M ethod (Tufts and Ku m aresan) 496

Information Criteria 49 6Overrating 497Order Selection 498Real Arithmetic 49811.7 Pisarenko and M US IC M ethods 499

The Pisarenko Method 500MUSIC 50211.8 Pencil Methods 50311.9 A Frequency-Dom ain Version of Prony's Method (Kumaresan) 505

Divided Differences 506Solving for A(z) 506Solving for B(z) 507References 508

Index 515

Documents

21055823 (1) this is a statistical signal processing complementary for index clarification detailed image analysis of grandiose field pacants