8/3/2019 1 EDT lect1

1/30

1

EC 6307

Estimation and Detection Theory

Monday: 2 -3 PM

Tuesday: 10.15 11.15 AM

Thursday: 11.15 12.15

Friday: 8 9 AM

8/3/2019 1 EDT lect1

2/30

What is EC 6307 About?

Goal: Get useful information out of messy data

Strategy: Formulate probabilistic model of datax, which depends on

underlying parameter(s) U

Terminology depends on parameter space:

Detection (simple hypothesis testing):

U {0,1}, i.e. 0= target absent, 1= target present

Classification (multi-hypothesis testing):

U {0,1,,M}, i.e. U {DC-9, 747, F-15, MiG-31}

Estimation

U Rn 2

8/3/2019 1 EDT lect1

3/30

Ex: Positron Emission Tomography

Simple, traditional linear DSP-based approach

Raw DataStatistical Estimate

Advanced, estimation-theoretic approach

3

8/3/2019 1 EDT lect1

4/30

Tasks of Statistical Signal Processing

1. Create statistical model for measured data

2. Find fundamental limitations on our ability to perform

inference on the data

Cramr-Rao bound, Chernov bound, etc.

3. Develop an optimal (or suboptimal) estimator

4. Asymptotic analysis (i.e., assume we have lots and lots of data)

of estimator performance to see if it approaches boundsderived in (2)

5. Do simulations and experiments comparing algorithm

performance to lower bounds and competing algorithms4

8/3/2019 1 EDT lect1

5/30

First engineering textbook on the topic

Some homework problems are masters thesis topics!!!

Unfortunately, showing its age

All emphasis on continuous time problems, as might be

implemented with analog computers

Ex: Discrete-time Kalman filtering relegated to a homework

problem!!!

Detection, Estimation, and Modulation Theory,

Part I, Harry L. Van Trees, 1968

5

8/3/2019 1 EDT lect1

6/30

excellent tutorial and research reference book on estimation theory.

The theory is illustrated with very concrete examples; the examples

give an under-the-hood insight into the solution of some common

estimation problems in signal processing.If you're a statistician, you

might not like this book. If you're an engineer, you will like it.

Amazon review

The theory is explained well and motivated, but what makes the book

great are the examples. There are many worked examples and they are

chosen to make things very clear. Amazon review

Fundamentals of Statistical Signal Processing,Volume 1:

Estimation Theory, Steven Kay, 1993

6

8/3/2019 1 EDT lect1

7/30

Pedagogical Question:

What to do first?

View 1: Detection first, then Estimation (Van Trees et. al)

Detection Theory is easier; introduces concepts used in

Estimation Theory in a simple context

View 2: Estimation first, then Detection (Kay)

Detection Theory just a special case of estimation theory

Detection problems with unknown parameters are easier

to think about if youve already seen estimation theory

View 1 seems more common, but well take View 2 7

8/3/2019 1 EDT lect1

8/30

8

Read Ch 1 & 2 in :Fundamentals of Statistical Signal

Processing,Volume 1: Estimation Theory by Steven Kay

for the Mondays class

8/3/2019 1 EDT lect1

9/30

9

Introduction to Estimation

8/3/2019 1 EDT lect1

10/30

An Example Estimation Problem: DSB Receiver

10

8/3/2019 1 EDT lect1

11/30

11

Discrete-Time Estimation Problem

These days, we almost always work with samples of the

observed signal (signal plus noise)

Our Thought Model: Each time you observe x[n] it

contains same s[n] but different realization of noise

w[n], so the estimate is different each time.

8/3/2019 1 EDT lect1

12/30

12

Our Job: Given finite data setx[0], x[1], x[N-1],

find estimator functions that map data into estimates

8/3/2019 1 EDT lect1

13/30

13

PDF of Estimate

Since estimates are RVs, we describe them with a PDF

8/3/2019 1 EDT lect1

14/30

14

General Mathematical Statement of EstimationProblem:

What captures all the statistical information needed for

an estimation problem ?

8/3/2019 1 EDT lect1

15/30

15

Need the N-dimensional PDF of the data, parameterized by

8/3/2019 1 EDT lect1

16/30

16

Ex. Estimating a DC Level in Zero Mean AWGN

Consider a single data point is observed

8/3/2019 1 EDT lect1

17/30

17

Soin this case, the parameterization changes the data

PDF would mean

8/3/2019 1 EDT lect1

18/30

18

Ex. Modeling Data with Linear Trend

We propose signal and noise models as:

Signal Model: Linear TrendNoise Model: AWGNwith zero mean

8/3/2019 1 EDT lect1

19/30

19

Typical Assumptions for Noise Model

W and G is always easiest to analyze

Usually assumed unless you have reason to believe

otherwise

Whiteness is usually the first assumption removed

Gaussian is less often removed due to CLT

Zero Mean is a nearly universal assumption

Most practical cases have zero mean

8/3/2019 1 EDT lect1

20/30

20

Variance of noise doesnt always have to be known to

make an estimate

BUT,must know to assess expected goodness of the

estimate

Usually perform goodness analysis as a function of

noise variance (or SNR= Signal-to-Noise Ratio)

Noise variance sets the SNR level of the problem

8/3/2019 1 EDT lect1

21/30

21

Classical vs. Bayesian Estimation Approaches

If we view (parameter to estimate) as Non-Random

Classical Estimation

Provides no way to include a priori information about

If we view (parameter to estimate) as Random

Bayesian EstimationAllows use of some a priori PDF on

8/3/2019 1 EDT lect1

22/30

22

The first part of the course: Classical Methods

Minimum Variance,Maximum Likelihood, LeastSquares

Second part of the course: Bayesian Methods

MMSE, MAP,Wiener filter,Kalman Filter

Third & fourth parts of the course: Detection theory

8/3/2019 1 EDT lect1

23/30

23

Assessing Estimator Performance

Can do this only when the value of is known:

- Theoretical Analysis, Simulations, Field Tests, etc.

Thus it has a PDF of its own and that PDF completely

displays the quality of the estimate.

Often, just capture quality through mean and variance

of the estimator

8/3/2019 1 EDT lect1

24/30

24

8/3/2019 1 EDT lect1

25/30

25

Equivalent View of Assessing Performance

Completely describe estimator quality with error PDF:

p(e)Desire:

8/3/2019 1 EDT lect1

26/30

26

Example: DC Level in AWGN

Model:

PDF of an individual data sample

8/3/2019 1 EDT lect1

27/30

27

Uncorrelated Gaussian RVs are Independent

so the joint PDF is the product of the individual PDFs:

8/3/2019 1 EDT lect1

28/30

28

8/3/2019 1 EDT lect1

29/30

29

8/3/2019 1 EDT lect1

30/30

30

Theoretical Analysis vs. Simulations

Ideally wed like to be always be able to theoretically

analyze the problem to find the bias and variance of the

estimator

Theoretical results show how performance depends on

the problem specifications

But sometimes we make use of simulations

to verify that our theoretical analysis is correct

sometimes cant find theoretical results