Upload
redcard53
View
242
Download
1
Embed Size (px)
DESCRIPTION
ESTIMATION THEORY
Citation preview
Estimation theory
Johan E. Carlson
Dept. of Computer Science, Electrical and Space EngineeringLuleå University of Technology
Lecture 1
1 / 26
Outline1. General course information
2. Introduction (Chapter 1)2.1. Estimation theory in Signal Processing
2.2. Problem formulation
2.3. Assessing estimator performance
3. Minimum variance unbiased estimation (Chapter 2)3.1. Unbiased estimators
3.2. Existence of the MVUB estimator
3.3. Finding the MVUB estimator
3.4. Extension to a vector parameter
2 / 26
Notes
Notes
General course informationCourse web page:http://staff.www.ltu.se/~johanc/estimation_theory/
Text book:Steven M. Kay, Fundamentals of Statistical Signal Processing:Estimation Theory, Vol 1. Prentice Hall, 1993.ISBN10: 0133457117Examination:
Completion of theoretical homework assignments (writtensolutions to be handed in to me).Completion of computer assignments (short lab reports to me).
3 / 26
Estimation in Signal Processing
Modern estimation theory is central to many electrical system, e.g.Radar, Sonar, AcousticsSpeechImage and videoBiomedicine (Biomedical engineering)Communications (Channel estimation, synchronization, etc.)Automatic controlSeismology
4 / 26
Notes
Notes
Estimation in Signal Processing
Example: Ultrasonic characterization of thin multi-layeredmaterials
Estimate thickness of layersLocate flaws/cracks in internal layers
5 / 26
Estimation in Signal Processing
Simply put, given an observed N -point data set
{x[0], x[1], . . . , x[N − 1]}
which depends on an unknown parameter θ, wee define theestimator
θ = g(x[0], x[1], . . . , x[N − 1]),
where g is some function.
6 / 26
Notes
Notes
Mathematical problem formulation
Since data are inherently random, we will describe them in termsof probability density functions (PDF), i.e.
p(x[0], x[1], . . . , x[N − 1]; θ),
where semicolon (;) denotes that the PDF is parameterized by theunknown parameter θ.
The estimation problem is thus to find (infer ) the value of θ from theobservations. The PDF should be chosen so that
It takes into account any prior knowledge or constraintsIt is mathematically tractable
7 / 26
Mathematical problem formulation
Example: The Dow-Jones index
0 20 40 60 80 1002800
2900
3000
3100
3200
Day number
Dow
−Jo
nes
aver
age
It appears it is "on average increasing".
8 / 26
Notes
Notes
Mathematical problem formulation
Example: The Dow-Jones indexA reasonable model could then be
x[n] = A+Bn+ w[n], n = 0, 1, . . . , N − 1,
where w[n] is white Gaussian noise (WGN), i.e. each sample ofw[n] has the PDF N (0, σ2), and is uncorrelated with all the othersamples. The unknown parameters can arranged in the vectorθ = [A B]T . The PDF of x[n] then is
p(x;θ) =1
(2πσ2)N2
exp
[− 1
2σ2
N−1∑n=0
(x[n]−A−Bn)2],
where x = [x[0], x[1], . . . , x[N − 1]]T
9 / 26
Mathematical problem formulation
Example: The Dow-Jones indexThe straight line assumption is consistent with data. A modelsthe offset and B models the linear increase over time.The choice of the Gaussian PDF makes the modelmathematically tractable.Here the parameters are assumed to be unknown, butdeterministic.One could also assume that θ is random, but constrained, sayA is in [2800, 3200], and that it is uniformly distributed in thisinterval.This would lead to a Bayesian approach, andthe joint PDF will be
p(x;θ) = p(x|θ)p(θ)
10 / 26
Notes
Notes
Assessing estimator performance
Consider the following signal (a realization of a DC voltagecorrupted by noise).
0 20 40 60 80 100−1
0
1
2
3
n
x[n]
A realistic signal model would then be
x[n] = A+ w[n],
where w[n] is N (0, σ2).11 / 26
Assessing estimator performance
How to estimate A?
A reasonable estimator would be the sample mean
A =1
N
N−1∑n=0
x[n]
How close will A to A?Are there any better estimators than the sample mean?
12 / 26
Notes
Notes
Assessing estimator performance
Another estimator could be
A = x[0]
Intuitively, this should not perform as well, since we’re notusing all available data.But, for any given realization of x[n] it might actually be closerto the true A than the sample mean.So, how do we assess the performance?We need to consider the estimators from a statisticalperspective!Let’s consider E(A) and var(A)
13 / 26
Assessing estimator performance
For the first estimator
E(A) = E
(1
N
N−1∑n=0
x[n]
)
=1
N
N−1∑n=0
E(x[n])
= A
For the second estimator
E(A) = E(x[0]
= A
14 / 26
Notes
Notes
Assessing estimator performance
For the first estimator
var(A) = var
(1
N
N−1∑n=0
x[n]
)
=1
N2
N−1∑n=0
var(x[n])
=1
N2Nσ2
=σ2
N
For the second estimator
var(A) = var(x[0])
= σ2
15 / 26
Assessing estimator performance
So, the expected value of both estimators, E(A) = A, i.e. theyare both unbiased.The variance of the second estimator is σ2, which is largerthan the variance of the first estimator.It appears that indeed the sample mean is a better estimatethan x[0]!
16 / 26
Notes
Notes
Minimum variance unbiased (MVUB) estimation
Let’s start by considering estimation of unknown butdeterministic parameters.We will restrict the search for estimators to those who onaverage yield the true parameter value, i.e. to unbiasedestimators.For all possible unbiased estimators, we will then look for theone with the minimum variance, i.e. the minimum varianceunbiased estimator (MVUB).
17 / 26
Unbiased estimatorsAn estimator is said to be unbiased if
E(θ) = θ,
for all possible values of θ.
If θ = g(x), this means that
E(θ) =
∫g(x)p(x;θ)dx = θ,
for all θ.
18 / 26
Notes
Notes
The minimum variance criterionLet’s look at a natural optimality criterion, known as the meansquare error (MSE)
mse(θ) = E[(θ − θ)2
]
19 / 26
The minimum variance criterionUnfortunately, this criterion often leads to unrealizable estimators,since
mse(θ) = E
{[(θ − E(θ)
)+(E(θ)− θ
)]2}= var(θ) +
[E(θ)− θ
]2= var(θ) + b2(θ),
which shows that the MSE depends both on the variance of theestimator and on the bias. If the bias depends on the parameteritself, we’re in trouble!
Let’s restrict ourselves to search only forunbiased estimators!
20 / 26
Notes
Notes
Existence of the MVUB estimatorDoes an MVUB estimator always exist?No!
21 / 26
Existence of the MVUB estimatorA counterexample of to the existence
Assume that we have to independent observations x[0] and x[1]with PDF
x[0] ∼ N (θ, 1)
x[1] ∼{N (θ, 1), θ ≥ 0N (θ, 2), θ < 0
22 / 26
Notes
Notes
Existence of the MVUB estimatorThe two estimators
θ1 =1
2(x[0] + x[1])
θ2 =2
3x[0] +
1
3
can easily be shown to be unbiased. The variances are
var(θ1) =1
4(var(x[0]) + var(x[1]))
var(θ2) =4
9var(x[0]) +
1
9var(x[1])
23 / 26
Existence of the MVUB estimatorAs a result (looking back at the PDFs), we have that
var(θ1) =
{1836 , θ ≥ 02436 , θ < 0
var(θ2) =
{2036 , θ ≥ 02436 , θ < 0
So, for θ ≥ 0, the minimum variance is 18/36 (estimator 1) and forθ < 0 it is 24/36 (estimator 2). Hence, no single estimator will havethe uniformly minimum variance for all θ.
24 / 26
Notes
Notes
Finding the MVUB estimator
There is no (known) universal method that will always producethe MVUB estimator.There are some possible approaches though
Determine the Cramer-Rao lower bound (CRLB) and check ifsome estimator satisfies it (Chapter 3 and 4).Apply the Rao-Blackwell-Lehmann-Scheffe (RBLS) theorem(Chapter 5).Further restrict the estimators to be also linear. Then find theMVUB estimator within this class. (Chapter 6).
25 / 26
Extension to a vector parameter
If θ = [θ1, θ2, . . . , θp]T is a vector of unknown parameters, we say
that an estimator is unbiased if
E(θi) = θi,
for i = 1, 2, . . . , p. By defining
E(θ) =
E(θ1)
E(θ2)...
E(θp)
,we can define un unbiased estimator as having the property
E(θ) = θ.
26 / 26
Notes
Notes