Second Edition
USA
This edition first published 2013 © 2013, John Wiley & Sons, Ltd
First Edition published in 1998 © 1998, John Wiley & Sons, Ltd.
Registered office John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, United Kingdom
For details of our global editorial offices, for customer services and for information about how to apply for permission to reuse the copyright material in this book please see our website at www.wiley.com.
The right of the author to be identified as the author of this work has been asserted in accordance with the Copyright, Designs and Patents Act 1988.
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by the UK Copyright, Designs and Patents Act 1988, without the prior permission of the publisher.
Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books.
Designations used by companies to distinguish their products are often claimed as trademarks. All brand names and product names used in this book are trade names, service marks, trademarks or registered trademarks of their respective owners. The publisher is not associated with any product or vendor mentioned in this book.
Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. It is sold on the understanding that the publisher is not engaged in rendering professional services and neither the publisher nor the author shall be liable for damages arising herefrom. If professional advice or other expert assistance is required, the services of a competent professional should be sought.
MATLAB® is a trademark of The MathWorks, Inc. and is used with permission. The MathWorks does not warrant the accuracy of the text or exercises in this book. This book’s use or discussion of MATLAB® software or related products does not constitute endorsement or sponsorship by The MathWorks of a particular pedagogical approach or particular use of the MATLAB® software.
Library of Congress Cataloging-in-Publication Data
Farhang-Boroujeny, B. Adaptive filters : theory and applications / Behrouz Farhang-Boroujeny. – Second edition.
pages cm Includes bibliographical references and index. ISBN 978-1-119-97954-8 (cloth) 1. Adaptive filters. 2. Adaptive signal processing. I. Title. TK7872.F5F37 2013 621.3815′324 – dc23
2012050968
A catalogue record for this book is available from the British Library.
ISBN: 978-1-119-97954-8
Set in 10/12 Times by Laserwords Private Limited, Chennai, India
Contents
Preface xvii
Acknowledgments xxi
1 Introduction 1 1.1 Linear Filters 1 1.2 Adaptive Filters 2 1.3 Adaptive Filter Structures 3 1.4 Adaptation Approaches 7
1.4.1 Approach Based on Wiener Filter Theory 7 1.4.2 Method of Least-Squares 8
1.5 Real and Complex Forms of Adaptive Filters 9 1.6 Applications 9
1.6.1 Modeling 9 1.6.2 Inverse Modeling 11 1.6.3 Linear Prediction 15 1.6.4 Interference Cancellation 20
2 Discrete-Time Signals and Systems 28 2.1 Sequences and z-Transform 28 2.2 Parseval’s Relation 32 2.3 System Function 33 2.4 Stochastic Processes 35
2.4.1 Stochastic Averages 35 2.4.2 z-Transform Representations 37 2.4.3 The Power Spectral Density 38 2.4.4 Response of Linear Systems to Stochastic Processes 41 2.4.5 Ergodicity and Time Averages 44 Problems 44
3 Wiener Filters 48 3.1 Mean-Squared Error Criterion 48 3.2 Wiener Filter – Transversal, Real-Valued Case 50
viii Contents
3.3 Principle of Orthogonality 55 3.4 Normalized Performance Function 57 3.5 Extension to Complex-Valued Case 58 3.6 Unconstrained Wiener Filters 61
3.6.1 Performance Function 61 3.6.2 Optimum Transfer Function 64 3.6.3 Modeling 66 3.6.4 Inverse Modeling 69 3.6.5 Noise Cancellation 74
3.7 Summary and Discussion 79 Problems 80
4 Eigenanalysis and Performance Surface 90 4.1 Eigenvalues and Eigenvectors 90 4.2 Properties of Eigenvalues and Eigenvectors 91 4.3 Performance Surface 104
Problems 112
5 Search Methods 119 5.1 Method of Steepest Descent 120 5.2 Learning Curve 126 5.3 Effect of Eigenvalue Spread 130 5.4 Newton’s Method 131 5.5 An Alternative Interpretation of Newton’s Algorithm 133
Problems 135
6 LMS Algorithm 139 6.1 Derivation of LMS Algorithm 139 6.2 Average Tap-Weight Behavior of the LMS Algorithm 141 6.3 MSE Behavior of the LMS Algorithm 144
6.3.1 Learning Curve 146 6.3.2 Weight-Error Correlation Matrix 149 6.3.3 Excess MSE and Misadjustment 151 6.3.4 Stability 153 6.3.5 The Effect of Initial Values of Tap Weights
on the Transient Behavior of the LMS Algorithm 155
6.4 Computer Simulations 156 6.4.1 System Modeling 156 6.4.2 Channel Equalization 158 6.4.3 Adaptive Line Enhancement 163 6.4.4 Beamforming 165
6.5 Simplified LMS Algorithms 167 6.6 Normalized LMS Algorithm 170 6.7 Affine Projection LMS Algorithm 173 6.8 Variable Step-Size LMS Algorithm 177
Contents ix
6.9 LMS Algorithm for Complex-Valued Signals 179 6.10 Beamforming (Revisited) 182 6.11 Linearly Constrained LMS Algorithm 186
6.11.1 Statement of the Problem and Its Optimal Solution 186 6.11.2 Update Equations 187 6.11.3 Extension to the Complex-Valued Case 188 Problems 190 Appendix 6A: Derivation of Eq. (6.39) 205
7 Transform Domain Adaptive Filters 207 7.1 Overview of Transform Domain Adaptive Filters 208 7.2 Band-Partitioning Property of Orthogonal Transforms 210 7.3 Orthogonalization Property of Orthogonal Transforms 211 7.4 Transform Domain LMS Algorithm 213 7.5 Ideal LMS-Newton Algorithm and Its Relationship with TDLMS 215 7.6 Selection of the Transform T 216
7.6.1 A Geometrical Interpretation 216 7.6.2 A Useful Performance Index 220 7.6.3 Improvement Factor and Comparisons 221 7.6.4 Filtering View 224
7.7 Transforms 229 7.8 Sliding Transforms 230
7.8.1 Frequency Sampling Filters 230 7.8.2 Recursive Realization of Sliding Transforms 231 7.8.3 Nonrecursive Realization of Sliding Transforms 234 7.8.4 Comparison of Recursive and Nonrecursive Sliding Transforms 238
7.9 Summary and Discussion 242 Problems 243
8 Block Implementation of Adaptive Filters 251 8.1 Block LMS Algorithm 252 8.2 Mathematical Background 255
8.2.1 Linear Convolution Using the Discrete Fourier Transform 255 8.2.2 Circular Matrices 257 8.2.3 Window Matrices and Matrix Formulation of the
Overlap-Save Method 258 8.3 The FBLMS Algorithm 260
8.3.1 Constrained and Unconstrained FBLMS Algorithms 261 8.3.2 Convergence Behavior of the FBLMS Algorithm 262 8.3.3 Step-Normalization 263 8.3.4 Summary of the FBLMS Algorithm 264 8.3.5 FBLMS Misadjustment Equations 266 8.3.6 Selection of the Block Length 266
8.4 The Partitioned FBLMS Algorithm 267 8.4.1 Analysis of the PFBLMS Algorithm 269 8.4.2 PFBLMS Algorithm with M > L 271
x Contents
8.4.3 PFBLMS Misadjustment Equations 274 8.4.4 Computational Complexity and Memory Requirement 274 8.4.5 Modified Constrained PFBLMS Algorithm 276
8.5 Computer Simulations 276 Problems 279 Appendix 8A: Derivation of a Misadjustment Equation for the BLMS
Algorithm 285 Appendix 8B: Derivation of Misadjustment Equations for the FBLMS
Algorithms 288
9 Subband Adaptive Filters 294 9.1 DFT Filter Banks 295
9.1.1 Weighted Overlap–Add Method for Realization of DFT Analysis Filter Banks 296
9.1.2 Weighted Overlap–Add Method for Realization of DFT Synthesis Filter Banks 297
9.2 Complementary Filter Banks 299 9.3 Subband Adaptive Filter Structures 303 9.4 Selection of Analysis and Synthesis Filters 304 9.5 Computational Complexity 307 9.6 Decimation Factor and Aliasing 308 9.7 Low-Delay Analysis and Synthesis Filter Banks 310
9.7.1 Design Method 310 9.7.2 Filters Properties 312
9.8 A Design Procedure for Subband Adaptive Filters 313 9.9 An Example 316 9.10 Comparison with FBLMS Algorithm 318
Problems 319
10 IIR Adaptive Filters 322 10.1 Output Error Method 323 10.2 Equation Error Method 327 10.3 Case Study I: IIR Adaptive Line Enhancement 332
10.3.1 IIR ALE Filter, W(z) 333 10.3.2 Performance Functions 334 10.3.3 Simultaneous Adaptation of s and w 335 10.3.4 Robust Adaptation of w 337 10.3.5 Simulation Results 337
10.4 Case Study II: Equalizer Design for Magnetic Recording Channels 343 10.4.1 Channel Discretization 344 10.4.2 Design Steps 345 10.4.3 FIR Equalizer Design 345 10.4.4 Conversion from FIR into IIR Equalizer 347 10.4.5 Conversion from z Domain into s Domain 348
Contents xi
Problems 352
11 Lattice Filters 355 11.1 Forward Linear Prediction 355 11.2 Backward Linear Prediction 357 11.3 Relationship Between Forward and Backward Predictors 359 11.4 Prediction-Error Filters 359 11.5 Properties of Prediction Errors 360 11.6 Derivation of Lattice Structure 362 11.7 Lattice as an Orthogonalization Transform 367 11.8 Lattice Joint Process Estimator 369 11.9 System Functions 370 11.10 Conversions 370
11.10.1 Conversion Between Lattice and Transversal Predictors 371 11.10.2 Levinson–Durbin Algorithm 372 11.10.3 Extension of Levinson–Durbin Algorithm 374
11.11 All-Pole Lattice Structure 376 11.12 Pole-Zero Lattice Structure 376 11.13 Adaptive Lattice Filter 378
11.13.1 Discussion and Simulations 380 11.14 Autoregressive Modeling of Random Processes 383 11.15 Adaptive Algorithms Based on Autoregressive Modeling 385
11.15.1 Algorithms 386 11.15.2 Performance Analysis 390 11.15.3 Simulation Results and Discussion 394 Problems 400 Appendix 11A: Evaluation of E[ua(n)xT(n)K(n)x(n)uT
a (n)] 407 Appendix 11B: Evaluation of the parameter γ 408
12 Method of Least-Squares 410 12.1 Formulation of Least-Squares Estimation for a
Linear Combiner 411 12.2 Principle of Orthogonality 412 12.3 Projection Operator 415 12.4 Standard Recursive Least-Squares Algorithm 416
12.4.1 RLS Recursions 416 12.4.2 Initialization of the RLS Algorithm 418 12.4.3 Summary of the Standard RLS Algorithm 419
12.5 Convergence Behavior of the RLS Algorithm 421 12.5.1 Average Tap-Weight Behavior of the RLS Algorithm 422 12.5.2 Weight-Error Correlation Matrix 422 12.5.3 Learning Curve 423
xii Contents
12.5.4 Excess MSE and Misadjustment 426 12.5.5 Initial Transient Behavior of the RLS Algorithm 427 Problems 430
13 Fast RLS Algorithms 433 13.1 Least-Squares Forward Prediction 434 13.2 Least-Squares Backward Prediction 435 13.3 Least-Squares Lattice 437 13.4 RLSL Algorithm 440
13.4.1 Notations and Preliminaries 440 13.4.2 Update Recursion for the Least-Squares Error Sums 443 13.4.3 Conversion Factor 444 13.4.4 Update Equation for Conversion Factor 446 13.4.5 Update Equation for Cross-Correlations 447 13.4.6 RLSL Algorithm Using A Posteriori Errors 450 13.4.7 RLSL Algorithm with Error Feedback 450
13.5 FTRLS Algorithm 453 13.5.1 Derivation of the FTRLS Algorithm 454 13.5.2 Summary of the FTRLS Algorithm 458 13.5.3 Stabilized FTRLS Algorithm 458 Problems 460
14 Tracking 463 14.1 Formulation of the Tracking Problem 463 14.2 Generalized Formulation of LMS Algorithm 464 14.3 MSE Analysis of the Generalized LMS Algorithm 465 14.4 Optimum Step-Size Parameters 469 14.5 Comparisons of Conventional Algorithms 471 14.6 Comparisons Based on Optimum Step-Size Parameters 475 14.7 VSLMS: An Algorithm with Optimum Tracking Behavior 477
14.7.1 Derivation of VSLMS Algorithm 477 14.7.2 Variations and Extensions 478 14.7.3 Normalization of the Parameter ρ 480 14.7.4 Computer Simulations 480
14.8 RLS Algorithm with Variable Forgetting Factor 485 14.9 Summary 486
Problems 488
15 Echo Cancellation 492 15.1 The Problem Statement 492 15.2 Structures and Adaptive Algorithms 495
15.2.1 Normalized LMS (NLMS) Algorithm 496 15.2.2 Affine Projection LMS (APLMS) Algorithm 499 15.2.3 Frequency Domain Block LMS Algorithm 501 15.2.4 Subband LMS Algorithm 502
Contents xiii
15.3 Double-Talk Detection 512 15.3.1 Coherence Function 512 15.3.2 Double-Talk Detection Using the Coherence Function 513 15.3.3 Numerical Evaluation of the Coherence Function 513 15.3.4 Power-Based Double-Talk Detectors 517 15.3.5 Numerical Results 518
15.4 Howling Suppression 521 15.4.1 Howling Suppression Through Notch Filtering 521 15.4.2 Howling Suppression by Spectral Shift 521
15.5 Stereophonic Acoustic Echo Cancellation 524 15.5.1 The Fundamental Problem 526 15.5.2 Reducing Coherence Between x1(n) and x2(n) 528 15.5.3 The LMS-Newton Algorithm for Stereophonic Systems 532 Appendix 15A: Multitaper method 542 Appendix 15B: Derivation of the Two-Channel Levinson–Durbin
Algorithm 549
16 Active Noise Control 551 16.1 Broadband Feedforward Single-Channel ANC 553
16.1.1 System Block Diagram in the Absence of the Secondary Path S1(z) 554
16.1.2 Filtered-X LMS Algorithm 555 16.1.3 Convergence Analysis 555 16.1.4 Adding the Secondary Path S1(z) 557
16.2 Narrowband Feedforward Single-Channel ANC 559 16.2.1 Waveform Synthesis Method 560 16.2.2 Adaptive Notch Filters 569
16.3 Feedback Single-Channel ANC 573 16.4 Multichannel ANC Systems 577
16.4.1 MIMO Blocks/Transfer Functions 578 16.4.2 Derivation of the LMS Algorithm for MIMO Adaptive Filters 579 Appendix 16A: Derivation of Eq. (16.46) 582 Appendix 16B: Derivation of Eq. (16.53) 583
17 Synchronization and Equalization in Data Transmission Systems 584 17.1 Continuous Time Channel Model 585 17.2 Discrete Time Channel Model and Equalizer Structures 589
17.2.1 Symbol-Spaced Equalizer 590 17.2.2 Fractionally Spaced Equalizer 591 17.2.3 Decision Feedback Equalizer 592
17.3 Timing Recovery 593 17.3.1 Cost Function 593 17.3.2 The Optimum Timing Phase 595
xiv Contents
17.3.3 Improving the Cost Function 598 17.3.4 Algorithms 600 17.3.5 Early-Late Gate Timing Recovery 600 17.3.6 Gradient-Based Algorithm 604
17.4 Equalizers Design and Performance Analysis 606 17.4.1 Wiener–Hopf Equation for Symbol-Spaced Equalizers 606 17.4.2 Numerical Examples 613
17.5 Adaptation Algorithms 617 17.6 Cyclic Equalization 618
17.6.1 Symbol-Spaced Cyclic Equalizer 618 17.6.2 Fractionally Spaced Cyclic Equalizer 625 17.6.3 Alignment of s(n) and x(n) 627 17.6.4 Carrier and Timing Phase Acquisition and Tracking 627
17.7 Joint Timing Recovery, Carrier Recovery, and Channel Equalization 628 17.8 Maximum Likelihood Detection 629 17.9 Soft Equalization 631
17.9.1 Soft MMSE Equalizer 633 17.9.2 Statistical Soft Equalizer 635 17.9.3 Iterative Channel Estimation and Data Detection 641
17.10 Single-Input Multiple-Output Equalization 643 17.11 Frequency Domain Equalization 645
17.11.1 Packet Structure 646 17.11.2 Frequency Domain Equalizer 647 17.11.3 Packet Structure for Fast Tracking 648 17.11.4 Summary 649
17.12 Blind Equalization 649 17.12.1 Examples of Kurtosis 651 17.12.2 Cost Function 652 17.12.3 Blind Adaptation Algorithm 654 Problems 654
18 Sensor Array Processing 659 18.1 Narrowband Sensor Arrays 660
18.1.1 Array Topology and Parameters 660 18.1.2 Signal subspace, noise subspace, and spectral factorization 662 18.1.3 Direction of Arrival Estimation 665 18.1.4 Beamforming Methods 670
18.2 Broadband Sensor Arrays 678 18.2.1 Steering 679 18.2.2 Beamforming Methods 680
18.3 Robust Beamforming 683 18.3.1 Soft-Constraint Minimization 686 18.3.2 Diagonal Loading Method 688 18.3.3 Methods Based on Sample Matrix Inversion 690 Problems 692
Contents xv
19 Code Division Multiple Access Systems 695 19.1 CDMA Signal Model 695
19.1.1 Chip-Spaced Users-Synchronous Model 696 19.1.2 Chip-Spaced Users-Asynchronous Model 698 19.1.3 Fractionally Spaced Model 699
19.2 Linear Detectors 699 19.2.1 Conventional Detector: The Matched Filter Detector 700 19.2.2 Decorrelator Detector 700 19.2.3 Minimum Mean-Squared Error (Optimal) Detector 701 19.2.4 Minimum Output Energy (Blind) Detector 703 19.2.5 Soft Detectors 707
19.3 Adaptation Methods 707 19.3.1 Conventional Detector 707 19.3.2 Decorrelator Detector 707 19.3.3 MMSE Detector 708 19.3.4 MOE Detector 708 19.3.5 Soft Detectors 709 Problems 709
20 OFDM and MIMO Communications 711 20.1 OFDM Communication Systems 711
20.1.1 The Principle of OFDM 711 20.1.2 Packet Structure 714 20.1.3 Carrier Acquisition 716 20.1.4 Timing Acquisition 717 20.1.5 Channel Estimation and Frequency Domain Equalization 717 20.1.6 Estimation of Rhh and Rνν 720 20.1.7 Carrier-Tracking Methods 721 20.1.8 Channel-Tracking Methods 730
20.2 MIMO Communication Systems 730 20.2.1 MIMO Channel Model 732 20.2.2 Transmission Techniques for Space-Diversity Gain 732 20.2.3 Transmission Techniques and MIMO Detectors
for Space-Multiplexing Gain 737 20.2.4 Channel Estimation Methods 741
20.3 MIMO–OFDM 743 Problems 743
References 746
Index 761
Preface
This book has grown out of the author’s research work and teaching experience in the field of adaptive signal processing as well as signal processing applications to a variety of communication systems. The second edition of this book, while preserving the presentation of the basic theory of adaptive filters as in the first edition, expands significantly on a broad range of applications of adaptive filters. Six new chapters are added that look into various applications of adaptive filters.
This book is designed to be used as a text to teach graduate-level courses in adaptive filters at different levels. It is also intended to serve as a technical reference for practicing engineers.
A typical one-semester introductory course on adaptive filters may cover Chapters 1, 3–6, and 12, and the first half of Chapter 11, in depth. Chapter 2, which contains a short review of the basic concepts of the discrete-time signals and systems, and some related concepts from random signal analyses, may be left as self-study material for students. Selected parts of the rest of this book may also be taught in the same semester, or, broader range of chapters may be used for a second semester course on advanced topics and applications.
In the study of adaptive filters, computer simulations constitute an important supple- mental component to theoretical analyses and deductions. Often, theoretical developments and analyses involve a number of approximations and/or assumptions. Hence, computer simulations become necessary to confirm the theoretical results. Apart from this, com- puter simulation turns out to be a necessity in the study of adaptive filters for gaining an in-depth understanding of the behavior and properties of the various adaptive algorithms. MATLAB® from MathWorks Inc. appears to be the most commonly used software simula- tion package. Throughout this book, MATLAB® is used to present a number of simulation results to clarify and/or confirm the theoretical developments. The programs as well as data files used for generating these results can be downloaded from the accompanying website of this book at www.wiley.com/go/adaptive_filters
Another integral part of this text is exercise problems at the end of chapters. With the exception of the first few chapters, two kinds of exercise problems are provided in each chapter:
1. The usual problem exercises. These problems are designed to sharpen the readers’ skill in theoretical development. They are designed to extend results developed in the text and illustrate applications to practical problems. Solutions to these
xviii Preface
problems are available to instructors on the companion website to the book: www.wiley.com/go/adaptive_filters
2. Simulation-oriented problems. These involve computer simulations and are designed to enhance the readers’ understanding on the behavior of the different adaptive algo- rithms that are introduced in the text. Most of these problems are based on the MATLAB® programs, which are provided on the accompanying website. In addition, there are also other (open-ended) simulation-oriented problems, which are designed to help the readers to develop their own programs and prepare them to experiment with practical problems.
The book assumes that the reader has some background of discrete-time signals and systems (including an introduction to linear system theory and random signal analysis), complex variable theory, and matrix algebra. However, a review of these topics is provided in Chapters 2 and 4.
This book starts with a general overview of adaptive filters in Chapter 1. Many examples of applications such as system modeling, channel equalization, echo cancellation, and antenna arrays are reviewed in this chapter. This follows with a brief review of discrete- time signals and systems in Chapter 2, which puts the related concepts in a framework appropriate for the rest of this book.
In Chapter 3, we introduce a class of optimum linear systems collectively known as Wiener filters. Wiener filters are fundamental to the implementation of adaptive filters. We note that the cost function used to formulate the Wiener filters is an elegant choice, leading to a mathematically tractable problem. We also discuss the unconstrained Wiener filters with respect to causality and duration of the filter impulse response. This study reveals many interesting aspects of Wiener filters and establishes a good foundation for the study of adaptive filters for the rest of this book. In particular, we find that, in the limit, when the filter length tends to infinity, a Wiener filter treats different frequency components of underlying processes separately. Numerical examples reveal that when the filter length is limited, separation of frequency components may be replaced by separation of frequency bands within a good approximation. This treatment of adaptive filters, which is pursued throughout this book, turns out to be an enlightening engineering approach for the study of adaptive filters.
Eigenanalysis is an essential mathematical tool for the study of adaptive filters. A thorough treatment of this topic is covered in the first half of Chapter 4. The second half of this chapter gives an analysis of the performance surface of transversal Wiener filters. This is followed by search methods, which are introduced in Chapter 5. The search methods discussed in this chapter are idealized versions of the statistical search methods that are used…