CERTIFICATE

A PROJECT REPORT ON

SPEECH PROCESSING USING KALMAN FILTER

Submitted In The Partial Fulfillment Of The Requirement For

The Award Of The Degree Of

BACHELOR OF TECHNOLOGY

In

ELECTRONICS AND COMMUNICATION

ENGINEERING

Submitted By

G.ADITYA D.PRAVEEN VARMA

(05981A0414) (05981A0427)

Y.S.V.J.V.PRASAD BABU G.PRAVEEN RAJ

(05981A0413) (05981A0436)

Under The Esteemed Guidance of

Sri K.PAVAN KUMAR ,[M.Tech]

Assistant Prof, ECE Dept.

DEPARTMENT OF ELECTRONICS AND COMMUNICATION ENGINEERING

RAGHU ENGINEERING COLLEGE

(Affiliated to JNTU-KAKINADA)

Dakamarri, Bheemunipatnam Mandal,

Visakhapatnam - 531 162

2008 - 2009

RAGHU ENGINEERING COLLEGE

(Affiliated to JNTU KAKINADA)

VISAKHAPATNAM – 531 162

CERTIFICATE

This is to certify that the Project Work Entitled “SPEECH PROCESSING USING

KALMAN FILTER” is a bonafide work done by G.ADITYA (05981A0414),

D.PRAVEEN VARMA (05981A0427), G.PRAVEEN RAJ (05981A0436),

Y.S.V.J.V.PRASAD BABU (05981A0413) in the partial fulfillment of the requirement

for the award of the Degree Of Bachelor of Technology in ELECTRONICS AND

COMMUNICATION ENGINEERING of RAGHU ENGINEERING COLLEGE

(Affiliated to JNTU Kakinada) Visakhapatnam during the academic year 2008-2009.

INTERNAL GUIDE HEAD OF THE DEPARTMENT

SRI.K.PAVAN KUMAR,[M.Tech]. Prof. B.S.N.MURTHY Phd.

Assistant Professor Department of ECE

Department of ECE Raghu Engineering College

Raghu Engineering College

EXTERNAL EXAMINER

ACKNOWLEDGEMENT

ACKNOWLEDGEMENT

We take this unique opportunity to express our abysmal sense of gratitude

indebtedness and heartful thanks to our guide “Sri.K.PAVAN KUMAR”, Assistant

Professor, Raghu Engineering College, Visakhapatnam, for his able guidance,

sustained interesting supervision with constructive criticism, valuable suggestions and

steady encouragement throughout the course of investigation and preparation of

dissertation.

We would also like to thank “Sri.K.Raghu”, Chairman of Raghu

Engineering College, Visakhapatnam for providing us all the facilities required for

successful completion of our Project and we are also extremely grateful to

“Prof.B.S.N.MURTHY”, Head Of The Department, of Electronics and

Communications and “Prof.B.V.RAMANA MURTHY”, Principal, Raghu

Engineering College, Visakhapatnam for giving us an opportunity and for their constant

encouragement.

We are highly thankful to all the staff of Electronics and Communication Dept.,

Raghu Engineering College, Visakhapatnam for their timely observations and

suggestions during the course of investigation and completion of project work as part of

the academic programme and for affording all the facilities to carry out the project.

Though we cannot be unmindful of the innumerable friends and classmates who rendered

assistance of hundred kinds.

CONTENTS

CONTENTS

Acknowledgement………………………………………………………..

List of tables………………………………………………………………

List of figures……………………………………………………………..

Abstract…………………………………………………………………..

Chapter 1. Introduction…………………………………………………1

1.1 General Introduction………………………………………………….. 1

1.2 Motivation……………………………………………………………..2

1.3 Aims of Thesis………………………………………………………....3

1.4 Overview of Thesis…………………………………………………….3

Chapter 2. Speech Processing…………………………………………….5

2.1 What is Speech Coding? ………………………………………………...5

2.2 Sampling……………………………………………………………….. 10

Chapter 3. Introduction to Digital Filters……………………………….12

3.1 Analog and Digital Filters…………………………………………….....12

3.2 Advantages of using digital filters…………………………………….....13

3.3 Operation of digital filters……………………………………………......14

3.4 Examples of simple digital filters………………………………………...15

3.5 Order of a digital filter…………………………………………………...18

3.6 Digital filter coefficients…………………………………………………19

3.7 Recursive and non-recursive filters……………………………………. .20

3.7.1 Example of a recursive filter……………………………….....21

3.7.2 Order of a recursive (IIR) digital filter……………………….22

3.7.3 Coefficients of recursive (IIR) digital filters…………….........23

3.8 Wiener Filter……………………………………….................................24

3.9 Least Mean Square (LMS) Filter……………….......................................24

3.9.1 Similarities between Wiener and LMS……………………....25

3.10 Kalman Filter………………………………………………………....25

3.11 Relative Advantages of Kalman Filter………………………………..25

3.12 Underlying dynamic system model…………………………………..26

Chapter 4. Kalman Filter……………………………………………… 29

4.1 Introduction…………………………………………………………… 29

4.2 What is a Kalman Filter? …………………………………………….. 30

4.3 How It Came to Be Called a Filter……………………………………. 31

4.4 Its Mathematical Foundations………………………………………… 32

4.5 What it is used for? …………………………………………………… 32

4.6 Impact of Kalman Filtering on Technology…………………………... 34

4.7 The Kalman filter- advantages and disadvantages……………………. 35

4.7.1 Advantages………………………………………………… 35

4.7.2 Disadvantages……………………………………………… 35

4.8 Process to be Estimated………………………………………………. 36

4.9 Computational Origins of the Filter…………………………………... 37

4.10 Probabilistic Origins of the Filter……………………………………. 39

4.11 The Discrete Kalman Filter Algorithm……………………………… 40

4.12 Filter Parameters and Tuning………………………………………... 43

Chapter 5. Implementation of Kalman filter to Speech……………… 44

5.1 Overview of this chapter……………………………………………… 44

5.2 Implementation………………………………………………………. 45

5.3 Flow chart design of Simulation……………………………………… 49

Chapter 6. Simulated Results & Discussions………………………….. 50

6.1 Overview of this chapter……………………………………………… 50

6.2 Simulated Results of Kalman filter…………………………………… 51

6.3 Coefficients at Different Iterations…………………………………….. 54

6.4 Results of Speech Samples……………………………………………. 58

6.5 Cross Correlation………………………………………………………67

Chapter 7. Conclusion…………………………………………………. 72

7.1 Final Thoughts………………………………………………………… 72

7.2 Future Developments…………………………………………………. 73

Appendix……………………………………………………………….. 74

Bibliography…………………………………………………………… 79

LIST OF TABLES 3.1 Coefficients of filters…………………………………………………….. 20

4.1 Examples of Estimation Problems………………………………………… 33

4.2 Time Update Equations…………………………………………………… 41

4.3 Measurement Update Equations………………………………………….. 41

6.1 Different Coefficients at Different Iterations……………………………… 55

6.2 Parameters of Q tuned to the particular speech samples…………………… 67

LIST OF FIGURES 2.1 Process of speech coding……………………………………………………… 6

2.2 Speech Synthesizing Device…………………………………………………… 7

2.3 Process of Speech Recognition………………………………………………… 8

2.4 Speech Enhancement…………………………………………………………. 10

3.1 Block diagram of filter……………………………………………………….. 12

3.2 Process of digital filter………………………………………………………... 13

4.1 Foundational concepts in Kalman filtering…………………………………… 32

4.2 The ongoing discrete Kalman filter cycle…………………………………….. 41

4.3 A complete picture of the operation of the Kalman filter……………………… 42

6.1 1st Kalman coefficient………………………………………………………… 52

6.2 2nd Kalman coefficient………………………………………………………... 52

6.3 3rd Kalman coefficient……………………………………………………….. 53

6.4 4th Kalman coefficient……………………………………………………….. 53

6.5 5th Kalman coefficient……………………………………………………….. 54

6.6 1st Kalman coefficient with different values at different iterations………….. 56

6.7 2nd Kalman coefficient with different values at different iterations…………. 56

6.8 3rd Kalman coefficient with different values at different iterations………….. 57

6.9 4th Kalman coefficient with different values at different iterations………….. 57

6.10 5th Kalman coefficient with different values at different iterations…………. 58

6.11 Input Speech Samples of s180.od………………………………………….. 59

6.12 Output Speech Samples of s180.od………………………………………… 59

6.13 Input Speech Samples of s680.od…………………………………………... 60

6.14 Output Speech Samples of s680.od………………………………………… 60

6.15 Input Speech Samples of s1180.od………………………………………… 61

6.16 Output Speech Samples of s1180.od………………………………………. 61

6.17 Zoom In of first 1000 iterations of Fig 5.13………………………………… 63

6.18 Zoom In of first 1000 iterations of Fig 5.14………………………………… 63

6.19 Zoom In of first 500 iterations of Fig 5.15…………………………………. 64

6.20 Zoom In of first 500 iterations for Fig 5.16………………………………… 64

5.21 Output Speech Samples of s680.od after tuning Q…………………………. 65

5.22 Output Speech Samples of s1180.od after tuning Q………………………... 66

5.23 Cross Correlation of s180.od………………………………………………. 69

5.24 Cross Correlation of 680.od………………………………………………... 69

5.25 Cross Correlation of s1180.od……………………………………………… 70

ABSTRACT

ABSTRACT

Speech processing is used widely in every day’s applications that most people

take for granted, such as network wirelines, cellular telephony, telephony system and

telephone answering machines. Due to its popularity and increasing of demand,

engineers are trying various approaches of improving the process. One of the methods

for improving is trying on different methods of filtering techniques. Thus, this instigates

an introduction of a filtering technique known as Kalman filtering.

In the early days, Kalman filtering was very popular in the research field of

navigation because of its magnificent accurate estimation characteristic. Since then,

electrical engineers manipulate its advantages to useful purpose in speech processing.

Consequently, today it had become a popular filtering technique for estimating and

resolving redundant errors containing in speech.

The objective of this thesis is to generate a reconstructed output speech signal

from the input signal involving the application of a Kalman filter estimation technique.

In this thesis, Kalman filter is used to estimate the parameters of the autoregressive

(AR) process and represented in the state-space domain, in which the speech signal is

modeled as.

The results of this speech coding technique are demonstrated and obtained with

the aid of MATLAB. Accurate estimations by the Kalman filter on speech is simulated

and presented. Comparison between the assigned values and the estimated values of the

parameters is described. The reconstructed speech and the input speech will also be

compared and discussed.

CHAPTER 1

1

CHAPTER-1

INTRODUCTION

1.1 General Introduction

Speech is a form of communication in every day life. It existed since human

Civilizations began and even till now, speech is applied to high

technological telecommunication systems. A particular field which I personally feel will

excel will be speech signal processing in the world of telecommunications. As

applications like cellular and satellite technology are getting popular among mankind,

human beings tend to demand more advance technology and are in search of

improved applications.

For this reason, researchers are looking closely into the four generic

attributes of speech coding. They are complexity, quality, bit rate and delay. Other

issues like robustness to transmission errors, multistage encoding/decoding, and

accommodation of non-voice signals such as in-band signaling and voice band

modem data play an important role in coding of speech as well .

In order to understand these processes, both human and machine speech has

to be studied carefully on the structures and functions of spoken language: how we

produce and perceive it and how speech technology may assist us in communication.

Therefore in this thesis, we will be looking more into speech processing

with the aid of an interesting technology known as the Kalman Filter. Presently, this

technique is not widely used in the field of signal processing, however it is a potential

nominee to be considered. More details on Speech Processing and the Kalman filter will

be explained in the later chapters of this thesis report.

2

1.2 Motivation

Speech processing has been a growing and dynamic field for more than two

decades and there is every indication that this growth will continue and even

accelerate. During this growth there has been a close relationship between the

development of new algorithms and theoretical results, new filtering techniques are also

of consideration to the success of speech processing.

One of the common adaptive filtering techniques that are applied to speech

is the Wiener filter. This filter is capable of estimating errors however at only very

slow computations. On the other hand, the Kalman filter suppresses this

disadvantage.

As widely known to the world, Kalman filtering techniques are used on

GPS (Global Positioning System) and INS (Inertial Navigation System). Nonetheless,

they are not widely used for speech signal coding applications. According to, the

reason why Kalman filter is so popular in the field of radar tracking and navigating

system is that it is an optimal estimator, which provides very accurate estimation of the

position of either airborne objects or shipping vessels.

Due to its accurate estimation characteristic, electrical engineers are

picturing the Kalman filter as a design tool for speech, whereby it can estimate and

resolve errors that are contained in speech after passing through a distorted channel.

Due to this motivating fact, there are many ways a Kalman filter can be tuned to suit

engineering applications such as network telephony and even satellite phone

conferencing.

3

Knowing the fact that preserving information, which is contained in speech, is of

extreme importance, the availability of signal filters such as the Kalman filter is of great

importance.

1.3 Aims of Thesis

In this thesis, the primary goal is to design a MATLAB based simulator for

processing of speech together with the aid of the Kalman filtering technique and to

obtain a reconstructed speech signal, which is similar to the input speech signal. To

achieve these results, sample speeches were obtained. These were modeled as an

autoregressive (AR) process and represented in the state-space domain by the Kalman

filter. The original speech signal and the reconstructed speech signal obtained from the

output of the filter were compared. The idea of this comparison is to pursue an output

speech signal, which is similar to the original one. It was concluded that Kalman filtering is

a good constructing method for speech.

1.4 Overview of Thesis

The above serves as a general introduction on how widely speech technology is

applied to telecommunications. The motivation behind this research and the aims of this

thesis are noted.

Chapter 2 describes the applications of speech processing. Some

descriptions of these applications are also covered. Sampling theorem is also included in

order to fit the purpose of this thesis.

Chapter 3 describes the about the types of digital Filters, its working. Some

descriptions of filter coefficients and a brief introduction to Wiener filter, Least Mean

Square Filter and Kalman Filter

4

Chapter 4 firstly provides an informal introduction to the Kalman filter

followed by some applications, which use the Kalman filter and the reasons behind using

it. The chapter will be wrapped up with theory and functions of this filter.

Chapter 5 illustrates the implementation of Kalman filtering to speech. A

flowchart of the design and implementation of the simulations is also portrayed.

Chapter 6 shows the simulations and results obtained from the software

MATLAB. Thereafter, comparisons and discussions were covered according to

the simulations.

The thesis concludes with a Chapter 7 in which some ideas for

improvement and possible areas for future research work are given.

CHAPTER 2

5

CHAPTER-2

SPEECH PROCESSING 2.1 What is Speech Processing? The term speech processing basically refers to the scientific

discipline concerning the analysis and processing of speech signals in order to achieve

the best benefit in various practical scenarios. The field of speech processing is, at

present, undergoing a rapid growth in terms of both performance and applications.

This is stimulated by the advances being made in the field of microelectronics,

computation and algorithm design.

The signals are usually processed in a digital representation whereby

speech processing can be seen as the intersection of digital signal processing and

natural language processing.

Speech processing can be divided into the following categories:

Speech recognition, which deals with analysis of the linguistic content of a speech

signal.

Speech coding, a specialized form of data compression, is important in the

telecommunication area.

Voice analysis for medical purposes, such as analysis of vocal loading and

dysfunction of the vocal cords.

Speech synthesis: the artificial synthesis of speech, which usually means

computer-generated speech.

6

Speech enhancement: enhancing the perceptual quality of a speech signal by

removing the destructive effects of noise, limited capacity recording equipment,

impairments, etc

Speech Coding

Fig.2.1. Process of speech coding

Speech coding or compression is the field concerned with compact

digital representations of speech signals for the purpose of efficient transmission

or storage. The central objective is to represent a signal with a minimum number of

bits while maintaining perceptual quality. Current applications for speech and

audio coding algorithms include cellular and personal communications networks

(PCNs), teleconferencing, desktop multi- media systems, and secure communications.

Speech Synthesis

The process that involves the conversion of a command sequence or

input text (words or sentences) into speech waveform using algorithms and

previously coded speech data is known as speech synthesis. The inputting of text

can be processed through by keyboard, optical character recognition, or from a

previously stored database.

7

Fig.2.2.Speech Synthesizing Device

A speech synthesizer can be characterized by the size of the speech

units they concatenate to yield the output speech as well as by the method used to

code, storeand synthesize the speech. If large speech units are involved, such as

phrases and sentences, high-quality output speech (with large memory

requirements) can be achieved. On the contrary, efficient coding methods can be

used for reducing memory needs, but these usually degrade speech quality.

Speech Recognition

Speech or voice recognition is the ability of a machine or program to

recognize and carry out voice commands or take dictation. On the whole, speech

recognition involves the ability to match a voice pattern against a provided or

acquired vocabulary. A limited vocabulary is mostly provided with a product and the

user can record additional words. On the other hand, sophisticated software has the

ability to accept natural speech.

8

Fig.2.3. Process of Speech Recognition

Speech information can be observed and processed only in the form

of sound Waveforms. It is an essential for speech signal to be reconstructed properly.

Moreover, for this to be process in a discrete Kalman filter, sampling plays a critical

role. In the next section of this chapter, we will take a look at how sampling is done.

9

Speaker recognition It is the computing task of validating a user's claimed identity using

characteristics Extracted from their voices. There is a difference between speaker

recognition (recognizes who is speaks) and speech recognition (recognizing what is

being said). These two terms are frequently confused, as is voice recognition. Voice

recognition is a synonym for speaker, and thus not speech, recognition. In addition, there

is a difference between the act of authentication (commonly referred to as speaker

verification or speaker authentication) and identification.

Speaker recognition has a history dating back some four decades and uses the

acoustic features of speech that have been found to differ between individuals. These

acoustic patterns reflect both anatomy (e.g., size and shape of the throat and mouth)

and learned behavioral patterns (e.g., voice pitch, speaking style). Because speaker

verification has earned speaker recognition its classification as a "behavioral biometric."

Voice analysis

It is the study of speech sounds for purposes other than linguistic content,

such as in speech recognition. Such studies include mostly medical analysis of the

voice i.e. phoniatrics, but also speaker identification. More controversially, some

believe that the truthfulness or emotional state of speakers can be determined using

Voice Stress Analysis or Layered Voice Analysis.

Speech Enhancement

What is speech enhancement? Enhancement means the improvement in the

value or quality of something. When applied to speech, this simply means the

improvement in intelligibility and/or quality of a degraded speech signal by using signal

processing tools. By speech enhancement, it refers not only to noise reduction but also

to dereverberation and separation of independent signals. Since this field is

fundamental for research in the applications of digital signal processing, it is also

of great interest to the industry which is always looking for new solutions that are both

effective and practical.

10

Fig.2.4. Speech Enhancement

Speech enhancement is an area of speech processing where the goal is to

improve the intelligibility and/or pleasantness of a speech signal. The most common

approach in speech enhancement is noise removal, where we, by estimation of noise

characteristics, can cancel noise components and retain only the clean speech signal. The

basic problem with this approach is that if we remove those parts of the signal that

resemble noise, we are also bounded to remove those parts of the speech signal that

resemble noise. In other words, speech enhancement procedures, often inadvertently,

also corrupt the speech signal when attempting to remove noise. Algorithms must

therefore compromise between effectiveness of noise removal and level of distortion in

the speech signal.

2.2 Sampling The purpose of sampling is to transform an analog signal that is

continuous in time to a sequence of samples discrete in time. The signals we use in

the real world, such as our voices, are called "analog" signals. In order to process

these signals in computers, most importantly it must be converted to "digital" form.

While an analog signal is continuous in both time and amplitude, a digital signal is

discrete in both time and amplitude. Since in this thesis, speech will be processed

through a discrete Kalman filter, it is necessary for converting the speech

signal from continuous time to discrete time; hence this process is described as

sampling.

11

The value of the signal is measured at certain intervals in time. Each

measurement is referred to as a sample. Once the continuous analog speech signal is

sample data frequency f, the resulting discrete signal will have more frequency

components than the analog signal.

To be precise, the frequency components of the analog signal are

repeated at the sample rate. Explicitly, in the discrete frequency response they are

seen at their original position, and also centered around +/- f, and +/- 2 f, etc.

In order to ensure that the signal still preserve the information, it is necessary to

sample at a higher rate greater than twice the maximum frequency of the signal.

This is known as the Nyquist rate. The Sampling Theorem states that a signal can be

exactly reconstructed if it is sampled at a frequency f, where f > 2fm where

fm is maximum frequency in the signal.

On the other hand, if the signal is sampled at a frequency that is lower that the

Nyquist rate. This signal when converted back into a continuous time signal, will

exhibit a phenomenon called aliasing. Aliasing is the presence of distortion

components in there constructed signal. These components were not present

when the original signal was sampled. In addition, some of the frequencies in the

original signal may be lost in there constructed signal. Aliasing occurs due to the

fact of overlapping of signal frequencies, which occurs when the sampling

frequency is too low. Frequencies "fold" around half the sampling frequency -

which is why this frequency is often referred to as the folding frequency.

CHAPTER 3

12

CHAPTER-3

INTRODUCTION TO DIGITAL FILTERS

3.1 Analog and Digital Filters

In signal processing, the function of a filter is to remove unwanted parts of

the signal, such as random noise, or to extract useful parts of the signal, such as the

components lying within a certain frequency range.

The following block diagram illustrates the basic idea.

Fig 3.1 block diagram of filter

There are two main kinds of filter, analog and digital. They are quite

different in their physical makeup and in how they work.

An analog filter uses analog electronic circuits made up from

components such as resistors, capacitors and op amps to produce the required

filtering effect. Such filter circuits are widely used in such applications as noise

reduction, video signal enhancement, graphic equalizers in hi-fi systems, and many other

areas.

There are well-established standard techniques for designing an analog filter

circuit for a given requirement. At all stages, the signal being filtered is an

electrical voltage or current which is the direct analogue of the physical quantity (e.g.

a sound or video signal or transducer output) involved.

A digital filter uses a digital processor to perform numerical calculations

on sampled values of the signal. The processor may be a general-purpose computer

such as a PC, or a specialized DSP (Digital Signal Processor) chip.

13

The analog input signal must first be sampled and digitized using an

ADC (analog to digital converter). The resulting binary numbers, representing

successive sampled values of the input signal, are transferred to the processor,

which carries out numerical calculations on them. These calculations typically

involve multiplying the input values by constants and adding the products together. If

necessary, the results of these calculations, which now represent sampled values of

the filtered signal, are output through a DAC (digital to analog converter) to convert

the signal back to analog form. Note that in a digital filter, the signal is represented by a

sequence of numbers, rather than a voltage or current. The following diagram shows the

basic setup of such a system.

Fig 3.2 Process of digital filter

3.2 Advantages of using digital filters

The following list gives some of the main advantages of digital over analog filters.

1. A digital filter is programmable, i.e. its operation is determined by a program

stored in the processor's memory. This means the digital filter can easily be changed

without affecting the circuitry (hardware). An analog filter can only be changed by

redesigning the filter circuit.

2. Digital filters are easily designed, tested and implemented on a general-purpose

computer or workstation.

14

3. The characteristics of analog filter circuits (particularly those containing active

components) are subject to drift and are dependent on temperature. Digital filters do

not suffer from these problems, and so are extremely stable with respect both to

time and temperature.

4. Unlike their analog counterparts, digital filters can handle low frequency

signals accurately. As the speed of DSP technology continues to increase, digital filters

are being applied to high frequency signals in the RF (radio frequency) domain,

which in the past was the exclusive preserve of analog technology.

5. Digital filters are very much more versatile in their ability to process signals in a

variety of ways; this includes the ability of some types of digital filter to adapt to

changes in the characteristics of the signal.

6. Fast DSP processors can handle complex combinations of filters in parallel or

cascade (series), making the hardware requirements relatively simple and compact in

comparison with the equivalent analog circuitry.

3.3 Operation of digital filters

In this section, we will develop the basic theory of the operation of

digital filters. This is essential to an understanding of how digital filters are

designed and used. Suppose the "raw" signal which is to be digitally filtered is in

the form of a voltage waveform described by the function v =x (t) where t is time.

This signal is sampled at time intervals h (the sampling interval). The sampled value at

time t = ih is

Thus the digital values transferred from the ADC to the processor can be

represented by the sequence

Note that the sampled values xn+1, xn+2 etc. are not available, as the haven't

15

happened yet! The digital output from the processor to the DAC consists of the

sequence of values

In general, the value of yn is calculated from the values x0, x1, x2, x3, ... , xn. The way

in which the y's are calculated from the x's determines the filtering action of the digital

filter.

3.4 Examples of simple digital filters

The following examples illustrate the essential features of digital filters.

1. UNITY GAIN FILTER:

yn = xn

Each output value yn is exactly the same as the corresponding input value xn:

y0 = x0

y1 = x1

y2 = x2

... Etc

This is a trivial case in which the filter has no effect on the signal.

2. SIMPLE GAIN FILTER:

yn = Kxn (K = constant)

This simply applies a gain factor K to each input value:

y0 = Kx0

y1 = Kx1

y2 = Kx2

... Etc

K > 1 makes the filter an amplifier, while 0 < K < 1 makes it an attenuator.

K < 0 corresponds to an inverting amplifier. Example (1) above is the

special case where K = 1.

16

3. PURE DELAY FILTER:

yn = xn-1

The output value at time t = nh is simply the input at time t = (n-1)h, i.e. the

signal is delayed by time h:

y0 = x-1

y1 = x0

y2 = x1

y3 = x2

... Etc

Note that as sampling is assumed to commence at t = 0, the input value x-1 at t = -h

is undefined. It is usual to take this (and any other values of x prior to t = 0) as

zero.

4. TWO-TERM DIFFERENCE FILTER:

yn = xn - xn-1

The output value at t = nh is equal to the difference between the current input xn

and the previous input xn-1:

y0 = x0 - x-1

y1 = x1 - x0

y2 = x2 - x1

y3 = x3 - x2

... etc

i.e. the output is the change in the input over the most recent sampling interval h.

The effect of this filter is similar to that of an analog differentiator circuit.

5. TWO-TERM AVERAGE FILTER:

yn = (xn + xn-1) / 2

The output is the average (arithmetic mean) of the current and previous input:

17

y0 = (x0 + x-1) / 2

y1 = (x1 + x0) / 2

y2 = (x2 + x1) / 2

y3 = (x3 + x2) / 2

... etc

This is a simple type of low pass filter as it tends to smooth out high-frequency

variations in a signal. (We will look at more effective low pass filter designs later).

6. THREE-TERM AVERAGE FILTER:

yn = (xn + xn-1 + xn-2) / 3

This is similar to the previous example, with the average being taken of the current

and two previous inputs:

y0 = (x0 + x-1 + x-2) / 3

y1 = (x1 + x0 + x-1) / 3

y2 = (x2 + x1 + x0) / 3

y3 = (x3 + x2 + x1) / 3

... etc

As before, x-1 and x-2 are taken to be zero.

7. CENTRAL DIFFERENCE FILTER:

yn = (xn - xn-2) / 2

This is similar in its effect to example (4). The output is equal to half the change

in the input signal over the previous two sampling intervals:

y0 = (x0 - x-2) / 2

y1 = (x1 - x-1) / 2

y2 = (x2 - x0) / 2

y3 = (x3 - x1) / 2

... etc

18

3.5 Order of a digital filter

The order of a digital filter can be defined as the number of previous inputs

(stored in the processor's memory) used to calculate the current output.

This is illustrated by the filters given as examples in the previous section.

Example (1):

yn = xn

This is a zero order filter, since the current output yn depends only on the

current input xn and not on any previous inputs.

Example (2):

yn = Kxn

The order of this filter is again zero, since no previous outputs are required to give

the current output value.

Example (3):

yn = xn-1

This is a first order filter, as one previous input (xn-1) is required to calculate yn.

(Note that this filter is classed as first-order because it uses one previous input, even

though the current input is not used).

Example (4):

yn = xn - xn-1

This is again a first order filter, since one previous input value is required to

give the current output.

Example (5):

yn = (xn + xn-1) / 2

The order of this filter is again equal to 1 since it uses just one previous input value.

19

Example (6):

yn = (xn + xn-1 + xn-2) / 3

To compute the current output yn, two previous inputs (xn-1 and xn-2) are needed; this

is therefore a second-order filter.

Example (7):

yn = (xn - xn-2) / 2

The filter order is again 2, since the processor must store two previous inputs in

order to compute the current output. This is unaffected by the absence of an explicit

xn-1 term in the filter expression.

The order of a digital filter may be any positive integer. A zero-order filter (such

as those in examples (1) and (2) above) is possible, but somewhat trivial, since it

does not really filter the input signal in the accepted sense.

3.6 Digital filter coefficients

All of the digital filter examples given in the previous section can be written in the

following general forms:

Zero order: yn = a0xn

First order: yn = a0xn + a1xn-1

Second order: yn = a0xn + a1xn-1 + a2xn-2

Similar expressions can be developed for filters of any order.

The constants a0, a1, a2, ... appearing in these expressions are called the filter

coefficients. The values of these coefficients determine the characteristics of a

particular filter.

20

The table opposite gives the values of the coefficients of each of the filters given as

examples in the previous section.

Table 3.1 : coefficients of filters

3.7 Recursive and non-recursive filters

For all the examples of digital filters discussed so far, the current output

(yn) is calculated solely from the current and previous input values (xn, xn-1, xn-2, ...).

This type of filter is said to be non-recursive. recursive filter is one which in addition

to input values also uses previous output values. These, like the previous input

values, are stored in the processor's memory.

The word recursive literally means "running back", and refers to the

fact that previously calculated output values go back into the calculation of the latest

output. The expression for a recursive filter therefore contains not only terms

involving the input values (xn, xn-1, xn-2, ...) but also terms in yn-1, yn-2, ...

From this explanation, it might seem as though recursive filters require

more calculations to be performed, since there are previous output terms in the filter

expression as well as input terms.

In fact, the reverse is usually the case. To achieve a given frequency

response characteristic using a recursive filter generally requires a much lower order

Example Order a0 a1 a2

1 0 1 - -

2 0 K - -

3 1 0 1 -

4 1 1 -1 -

5 1 1/2

1/2 -

6 2 1/3

1/3

1/3

7 2 1/2 0 -1

/2

21

filter, and therefore fewer terms to be evaluated by the processor, than the

equivalent non-recursive filter. This will be demonstrated later.

FIR and IIR filters

Some people prefer an alternative terminology in which a non-

recursive filter is known as an FIR (or Finite Impulse Response) filter, and

a recursive filter as an IIR (or Infinite Impulse Response) filter. These

terms refer to the differing "impulse responses" of the two types of filter

The impulse response of a digital filter is the output sequence

from the filter when a unit impulse is applied at its input. (A unit impulse is a very

simple input sequence consisting of a single value of 1 at time t = 0, followed by

zeros at all subsequent sampling instants).

An FIR filter is one whose impulse response is of finite duration. An

IIR filter is one whose impulse response (theoretically) continues for ever, because

the recursive (previous output) terms feed back energy into the filter input and keep

it going. The term IIR is not very accurate, because the actual impulse responses of

nearly all IIR filters reduce virtually to zero in a finite time. Nevertheless, these

two terms are widely used.

3.7.1 Example of a recursive filter A simple example of a recursive digital filter is given by

yn = xn + yn-1

In other words, this filter determines the current output (yn) by adding the current

input (xn) to the previous output (yn-1).

Thus:

y0 = x0 + y-1

y1 = x1 + y0

y2 = x2 + y1

y3 = x3 + y2

... etc

Note that y-1 (like x-1) is undefined, and is usually taken to be zero.

22

Let us consider the effect of this filter in more detail. If in each of the above

expressions we substitute for yn-1 the value given by the previous expression, we

get the following:

y0 = x0 + y-1 = x0

y1 = x1 + y0 = x1 + x0

y2 = x2 + y1 = x2 + x1 + x0

y3 = x3 + y2 = x3 + x2 + x1 + x0

... etc

Thus we can see that yn, the output at t = nh, is equal to the sum of the

current input xn and all the previous inputs. This filter therefore sums or

integrates the input values, and so has a similar effect to an analog integrator

circuit.

This example demonstrates an important and useful feature of recursive

filters: the economy with which the output values are calculated, as compared

with the equivalent non-recursive filter. In this example, each output is

determined simply by adding two numbers together.

For instance, to calculate the output at time t = 10h, the recursive filter uses the

expression

y10 = x10 + y9

To achieve the same effect with a non-recursive filter (i.e. without using

previous output values stored in memory) would entail using the expression

y10 = x10 + x9 + x8 + x7 + x6 + x5 + x4 + x3 + x2 + x1 + x0

This would necessitate many more addition operations, as well as the storage of

many more values in memory.

3.7.2 Order of a recursive (IIR) digital filter

The order of a digital filter was defined earlier as the number of previous

inputs which have to be stored in order to generate a given output. This definition

is appropriate for non-recursive (FIR) filters, which use only the current and

previous inputs to compute the current output.

23

In the case of recursive filters, the definition can be extended as follows:

“The order of a recursive filter is the largest number of previous input or

output values required to compute the current output.”

This definition can be regarded as being quite general: it applies both to FIR

and IIR filters. For example, the recursive filter discussed above, given by the

expression

yn = xn + yn-1

is classed as being of first order, because it uses one previous output value (yn-1),

even though no previous inputs are required.

In practice, recursive filters usually require the same number of previous

inputs and outputs. Thus, a first-order recursive filter generally requires one

previous input (xn-1) and one previous output (yn-1), while a second-order

recursive filter makes use of two previous inputs (xn1 and xn-2) and two

previous outputs (yn-1 and yn-2); and so on, for higher orders.

Note that a recursive (IIR) filter must, by definition, be of at least first order; a

zero-order recursive filter is impossible.

3.7.3 Coefficients of recursive (IIR) digital filters

From the above discussion, we can see that a recursive filter is basically

like a non-recursive filter, with the addition of extra terms involving previous

outputs (yn-1, yn-2etc.).

A first-order recursive filter can be written in the general form yn = (a0xn + a1xn-1 - b1yn-1) / b0

Note the minus sign in front of the "recursive" term b1yn-1, and the factor (1/b0)

applied to all the coefficients. The reason for expressing the filter in this way is

that it allows us to rewrite the expression in the following symmetrical form:

b0yn + b1yn-1 = a0xn + a1xn-1

24

In the case of a second-order filter, the general form is

yn = (a0xn + a1xn-1 + a2xn-2 - b1yn-1 - b2yn-2) / b0

An alternative "symmetrical" form of this expression is

b0yn + b1yn-1 + b2yn-2 = a0xn + a1xn-1 + a2xn-2

Note the convention that the coefficients of the inputs (the x's) are denoted by

a's, while the coefficients of the outputs (the y's) are denoted by b's.

The most commonly used filter is Wiener Filter, Least Mean Square

Filter and Kalman Filter. Let us study in detail about them :

3.8 Wiener Filter:

The goal of the Wiener filter is to filter out noise that has

corrupted a signal. It is based on a statistical approach. Typical filters are

designed for a desired frequency response. However, the design of the

Wiener filter is from a different angle. One is assumed to have knowledge of

the spectral properties of the original signal and the noise, and one seeks the LTI

filter whose output would come as close to the original signal as possible.

Wiener filters are characterized by the following:

1. Assumption: signal and (additive) noise are stationary linear stochastic processes

with known spectral characteristics or known autocorrelation and cross-correlation

2. Requirement: the filter must be physically realizable, i.e. causal (this requirement

can be dropped, resulting in a non-causal solution)

3. Performance criterion: minimum mean-square error (MMSE)

3.9 Least Mean Square (LMS) Filter

Least mean squares (LMS) algorithms are used in adaptive filters

to find the filter coefficients that relate to producing the least mean squares of the

error signal (difference between the desired and the actual signal).

25

It is a stochastic gradient descent method in that the filter is only adapted based

on the error at the current time. It was invented in 1969 professor BernardWidrow.

3.9.1 Similarities between Wiener and LMS

The Least mean squares filter solution converges to the Wiener filter

solution, assuming that the unknown system is LTI and the noise is stationary.

Both filters can be used to identify the impulse response of an unknown system,

knowing only the original input signal and the output of the unknown system. By

relaxing the error criterion to reduce current sample error instead of minimizing the

total error over all of n, the LMS algorithm can be derived from the Wiener

filter.

3.10 Kalman Filter

The Kalman filter is an efficient recursive filter that estimates the state

of a linear dynamic system from a series of noisy measurements. It is used in

a wide range of engineering applications from radar to computer vision, and is

an important topic in control theory and control systems engineering. Together

with the linear quadratic regulator (LQR), the Kalman filter solves the linear

quadratic Gaussian control problem (LQG). The Kalman filter, the linear

quadratic regulator and the linear quadratic Gaussian controller are solutions to

what probably are the most fundamental problems in control theory.

3.11 Relative Advantages of Kalman Filter

Below are some advantages of the Kalman filter, comparing with another

famous filter known as the Wiener Filter, which this filter was popular

before the introduction of Kalman filter.

The Kalman filter algorithm is implementable on a digital computer, which this

was replaced by analog circuitry for estimation and control when Kalman filter

was first introduced. This implementation may be slower compared to

analog filters of Wiener; however it is capable of much greater accuracy.

26

The Wiener filter implementation in analog electronics can operate at much

higher effective throughput than the (digital) Kalman filter.

Stationary properties of the Kalman filter are not required for the

deterministic dynamics or random processes. Many applications of

importance include non stationary stochastic processes.

The Wiener filter does not require finite-dimensional stochastic process models

for the signal and noise

The Kalman filter is compatible with state-space formulation of optimal

controllers for dynamic systems. It proves useful towards the 2 properties of

estimation and control for these systems.

The Kalman filter requires less additional mathematical preparation to learn

for the modern control engineering student, compared to the Wiener filter.

Necessary information for mathematically sound, statistically-based decision

methods for detecting and rejecting anomalous measurements are provided through

the use of Kalman filter.

Kalman filters are based on linear dynamical systems discretised in the

time domain. They are modeled on a Markov chain built on linear operators

perturbed by Gaussian noise. The state of the system is represented as a vector

of real numbers. At each discrete time increment, a linear operator is applied

to the state to generate the new state, with some noise mixed in, and optionally

some information from the controls on the system if they are known. Then,

another linear operator mixed with more noise generates the visible outputs

from the hidden state.

3.12 Underlying dynamic system model

27

The Kalman filter may be regarded as analogous to the hidden

Markovmodel, with the key difference that the hidden state variables take

values in a continuous space (as opposed to a discrete state space as in the

hidden Markov model). Additionally, the hidden Markov model can represent

an arbitrary distribution for the next value of the state variables, in contrast to the

Gaussian noise model that is used for the Kalman filter. There is a strong

duality between the equations of the Kalman Filter and those of the hidden

Markov model. A review of this and other models is given in Roweis and

Ghahramani (1999).

In order to use the Kalman filter to estimate the internal state of a process

given only a sequence of noisy observations, one must model the process in accordance

with the framework of the Kalman filter. This means specifying the matrices Fk, Hk, Qk,

Rk, and sometimes Bk for each time-step k as described below.

Model underlying the Kalman filter. Circles are vectors, squares are matrices, and stars

represent Gaussian noise with the associated covariance matrix at the lower right.

The Kalman filter model assumes the true state at time k is evolved from the

state at (k − 1) according to

where

Fk is the state transition model which is applied to the previous state xk−1;

Bk is the control-input model which is applied to the control vector uk;

28

wk is the process noise which is assumed to be drawn from a zero mean

multivariate normal distribution with covariance Qk.

At time k an observation (or measurement) zk of the true state xk is made according to

Where Hk is the observation model which maps the true state space into the

observed space and vk is the observation noise which is assumed to be zero mean

Gaussian white noise with covariance Rk.

The initial state, and the noise vectors at each step {x0, w1, ..., wk, v1 ... vk} are all

assumed to be mutually independent.

Many real dynamical systems do not exactly fit this model; however,

because the Kalman filter is designed to operate in the presence of noise, an

approximate fit is often good enough for the filter to be very useful. Variations on the

Kalman filter described below allow richer and more sophisticated models.

CHAPTER 4

CHAPTER-4

KALMAN FILTER

4.1 Introduction

In 1960, R.E. Kalman published his famous paper describing a recursive

solution to the discrete-data linear filtering problem. Since that time, the Kalman filter

has been the subject of extensive research and application, particularly in the area of

autonomous or assisted navigation. Chapter 3 describes a "friendly" introduction to the

general idea of the Kalman filter, while a more complete introductory discussion.

The filter is a mathematical procedure which operates through a prediction and correction

mechanism. In essence, this algorithm predicts a new state from its previous estimation

by adding a correction term proportional to the predicted error. In this way, this error

is statistically minimized. This filter is the main algorithm to estimate dynamic systems

specified in state-space form. A Kalman filter is simply an optimal recursive data

processing algorithm. If we focus on the word optimal, its definition depends on the

criteria chosen to evaluate. A feature is called optimum if the Kalman filter incorporates

all the information provided. It processes all the measurements available, regardless the

precision, to estimate the current value of the interest variables, using:

Knowledge of the system and the measurement devices.

Statistic description of the system noises, measurements of errors and the

uncertainty of the dynamics models.

Any information available about the initials conditions of the variables under

study.

A Kalman filter would be built to combine all these data and with the

knowledge of some dynamic systems to generate the best estimation of the interest

variable.

29

Theoretically, the Kalman Filter is an estimator for what is called the “linear-

quadratic problem”, which focuses on estimating the instantaneous “state” of a linear

dynamic system perturbed by white noise. Statistically, this estimator is optimal with

In practice, this Kalman Filter is one of the greater discoveries in the history of

statistical estimation theory and possibly the greatest discovery in the twentieth century. It

has enabled mankind to do many things that could not have been done without it, and it has

In a more dynamic approach, controlling of complex dynamic systems such as

continuous manufacturing processes, aircraft, ships or spacecraft, are the most immediate

applications of Kalman filter. In order to control a dynamic system, one needs to know what

it is doing first. For these applications, it is not always possible or desirable to measure

every variable that you want to control, and the Kalman filter provides a means for inferring

the missing information from indirect (and noisy) measurements. Some amazing things that

the Kalman filter can do is predicting the likely future courses of dynamic systems that

people are not likely to control, such as the flow of rivers during flood, the trajectories of

From a practical standpoint, these are the perspectives that this section will present:

It is only a tool – It aids mankind in solving problems, however, it does not solve

any problem all by itself. This is however not a physical tool, but a mathematical

one, which is made from mathematical models. In short, essentially tools for the

mind. They help mental work become more efficient, just like mechanical tools,

which make physical work less tedious. Additionally, it is important to understand

It is a computer program - It uses a finite representation of the estimation problem,

which is a finite number of variables; therefore this is the reason why it is said to be

4.2 What is a Kalman Filter?

respect to any quadratic function of estimation errors .

become as indispensable as silicon in the makeup of many electronic systems.

celestial bodies or the prices of traded commodities.

its use and function before one can apply it effectively .

30

these variables are real numbers with infinite precision, some problems do happen.

This is due from the distinction between finite dimension and finite information, and

the distinction between “finite” and “manageable” problem sizes. On the practical

side when using Kalman filtering, the above issues must be considered along with

It is a complete statistical characterization of an estimation problem - This is a

complete characterization of the current state of knowledge of the dynamic system,

including the influence of all past measurements. The reason behind why it is much

more than an estimator is because it propagates the entire probability distribution of

the variables it is tasked to estimate. These probability distributions are also useful

In a limited context, it is a learning method. The estimation problem is modeled

in a way that distinguishes between phenomena (what one is able to observe) and

noumena (what is really going on). Above that, the state of knowledge about the

noumena is that one can deduce from the phenomena. That state of knowledge is

represented by probability distributions, which represent knowledge of the real

world. Thus this cumulative processing of knowledge is considered a learning

process. It is a fairly simple process, however quite effective in many applications

It might seem strange that the term ``filter'' would apply to an estimator. More

commonly, a filter is a physical device for removing unwanted fractions of mixtures.(The

word felt comes from the same medieval Latin stem, for the material was used as a filter for

liquids.) Originally, a filter solved the problem of separating unwanted components of gas-

liquid-solid mixtures. In the era of crystal radios and vacuum tubes, the term was applied to

analog circuits that ``filter'' electronic signals. These signals are mixtures of different

frequency components, and these physical devices preferentially attenuate unwanted

frequencies.

4.3 How It Came to Be Called a Filter

“ideally suited to digital computer implementation” . However, assuming that

the theory .

for statistical analyses and the predictive design of sensor systems .

31

This concept was extended in the 1930s and 1940s to the separation of ``signals''

from ``noise,' both of which were characterized by their power spectral

densities.Kolmogorov and Wiener used this statistical characterization of their probability

distributions in forming an optimal estimate of the signal, given the sum of the signal and

noise.

With Kalman filtering the term assumed a meaning that is well beyond the

original idea of separation of the components of a mixture. It has also come to include

the solution of an inversion problem, in which one knows how to represent the measurable

variables as functions of the variables of principal interest. In essence, it inverts this

functional relationship and estimates the independent variables as inverted functions of the

dependent (measurable) variables. These variables of interest are also allowed to be

dynamic, with dynamics that are only partially predictable.

Figure 1.1 depicts the essential subjects forming the foundations for Kalman filtering

theory. Although this shows Kalman filtering as the apex of a pyramid, it is itself but

part of the foundations of another discipline--``modern'' control theory--and a

proper subset of statistical decision theory.

Although, the applications of Kalman filtering encompass many fields, its use as

Kalman Filtering

Least squares

Least

Mean

squares

Stochastic

systems

Probability

Theory

Dynamic

systems

4.4 Its Mathematical Foundations

Fig. 4.1 Foundational concepts in Kalman filtering

4.5 What it is used for?

a tool is mainly for two purposes: estimation and performance analysis of estimators.

32

Role 1: Estimating the State of Dynamic Systems what is a dynamic system? Almost

everything, if you are picky about it. Except for a few fundamental physical constants, there

is hardly anything in the universe that is truly constant. The orbital parameters of the

asteroid Ceres are not constant, and even the ``fixed'' stars and continents are moving.

Nearly all physical systems are dynamic to some degree. If one wants very precise estimates

of their characteristics over time, then one has to take their dynamics into consideration.

The problem is that one does not always know their dynamics very precisely either.

Given this state of partial ignorance, the best one can do is express our ignorance more

precisely-using probabilities. The Kalman filter allows us to estimate the state of dynamic

possible sensor types that might be used in estimating the state of the corresponding

dynamic systems. The objective of design analysis is to determine how best to use these

sensor types for a given set of design criteria. These criteria are typically related to

estimation accuracy and system cost.

TABLE 4.1 Examples of Estimation Problems

systems with certain types of random behavior by using such statistical information. A few

examples of such systems are listed in the second column of Table 4.1.

Role 2: The Analysis of Estimation Systems. The third column of Table 4 .1 lists some

33

Since the Kalman filter uses a complete description of the probability of its estimation

errors in determining the optimal filtering gains, this probability distribution may be used in

assessing its performance as a function of the “design parameters” of the following

types of sensors to be used;

locations and orientations of the various sensor types with respect to the system to be

estimated;

allowable noise characteristics of the sensors;

pre-filtering methods for smoothing sensor noise;

data sampling rates for the various sensor types and

The level of model simplification for reducing implementation requirements.

A system designer is able to assign an “error budget” to subsystems of an

estimation system, which this is allowed by the analytical capability of the Kalman filter

formalism. Moreover, it can trade off the budget allocations to optimize cost or other

From the standpoint of those involved in estimation and control problems, at least,

this has to be considered the greatest achievement in estimation theory of the twentieth

century. Many of the achievements since its introduction would not have been possible

without it. It was one of the enabling technologies for the Space Age, in particular. The

precise and efficient navigation of spacecraft through the solar system could not have been

done without it.

The principal uses of Kalman filtering have been in ``modern'' control systems, in

the tracking and navigation of all sorts of vehicles, and in predictive design of estimation

and control systems. These technical activities were made possible by the introduction of

the Kalman filter.

4.6 Impact of Kalman Filtering on Technology.

estimation systems:

measures of performance while achieving a required level of estimation accuracy.

34

4.7 The Kalman filter: advantages and disadvantages

4.7.1 Advantages

It avoids the influence of possible structural changes on the result. The

recursive estimation starts from an initial sample and updates the estimations by

adding a new observation until the end of the data. This implies that the most recent

coefficients estimation is affected by the distant history; in presence of structural

changes the data series can be cut. Like this, the Kalman filter, like other recursive

methods, uses all the series history but with one advantage: it tries to estimate a

stochastic path of the coefficients instead of a deterministic one. In this way it

solves the possible estimation cut when structural changes happen.

The Kalman filter uses the least square method to recursively generate a state

estimator on k moment, which is unbiased minimum and variance linear. This filter is

in equal terms with Gauss Markov theorem and this gives to Kalman filter its

enormous power to solve a wide range of problems on statistic inference.

The filter is distinguished by its skill to predict the state of a model in the past,

present and future, although the exact nature of the modeled system is unknown.

The dynamic modeling of a system is one of the key features which distinguish

the Kalman method.

4.7.2 Disadvantages Among the filter disadvantages we can find that it is necessary to know the

initial conditions of the mean and variance state vector to start the recursive algorithm.

There is no general consent over the way of determinate the initial conditions. The

Kalman filter development, as it is found on the original document, is supposed a wide

knowledge about probability theory, specifically with the Gaussian condition for the

random variables, which can be a limit for its research and application. It is developed

for autoregressive models, the results are conditioned to the past information of the

variable under study. In this sense the prognostic of the series over the time represents

the inertia that the system actually has and they are efficient just for short time term.

35

After going through some of the introduction and advantages of using Kalman

filter, we will now take a look at the process of this magnificent filter. The information

addresses of a general problem of trying to estimate the state of a discrete-time

controlled process that is governed by a linear stochastic difference equation:

with a measurement that is

The random variables and represent the process and measurement noise

(respectively). We assume that they are independent of each other, white, and with

normal probability distributions

Ideally, the process noise covariance and measurement noise covariance

matrices are assumed to be constant, however in practice, they might change with each

time step or measurement.

In the absence of either a driving function or process noise, the matrix A in

4.8 Process to be Estimated

below is understood and analyzed from. The process commences with the

. (4.1)

. (4.2)

. (4.3)

. (4.4)

the difference equation (4.1) relates the state at the previous time step to the state at

36

the current step . In practice, A might change with each time step, however here it is

assumed constant. The matrix B relates the optional control input to the state

the measurement, . In practice might change with each time step or measurement,

however we assume it is constant.

Let say we define (note: the "super minus") to be our priori state

estimate at step , given knowledge of the process prior to step , and (note:

without the "super minus") to be our posteriori state estimate at step , given

measurement . The priori (the one with “super minus”) and posteriori estimate errors

can be defined as:

The priori estimate error covariance is then

and the posteriori estimate error covariance is

After deriving the equations for the Kalman filter, the goal is to find an equation

that computes a posteriori state estimate , as a linear combination of a priori estimate

and a weighted difference between an actual measurement and a measurement

section “The Probabilistic Origins of the Filter” found below.

4.9 Computational Origins of the Filter

x. which is a matrix in the measurement equation (4.2) which relates the state to

. (4.5)

. (4.6)

prediction as shown below in (4.7). Some justification for (4.7) is given in the

37

. (4.7)

The difference of in (4.7) is called the measurement innovation, or the

residual. It reflects the discrepancy between the actual measurement and the

predicted measurement . When the residual has a value of zero means, then the two

are in complete agreement.

The gain, K or otherwise known as the blending factor, minimizes the posteriori

error covariance in (4.6) and is a matrix in (4.7). This minimization can be

accomplished by first substituting (4.7) into the above definition for , after which

substituting into (4.6), performing the indicated expectations, taking the derivative of

the trace of the result with respect to K, setting that result equal to zero, and then

solving for . One form of the resulting K that

minimizes (4.6) is given by (4.8) as follow:

. (4.8)

By looking at (1.8), it can be seen that as the measurement error covariance

approaches zero, the gain K weights the residual more heavily. Specifically,

.

On the other hand, as the priori estimate error covariance approaches zero,

the gain K will weight the residual less heavily. Specifically,

.

38

Another way of describing the weighting by K is that as the measurement error

covariance approaches zero, the actual measurement will be “depended” on more

and more, whereas the predicted measurement is depended on less and less.

On the other hand, as the priori estimate error covariance approaches zero,

the actual measurement is depended on less and less, and the predicted measurement

is depended on more and more.

This section is a short section describing the justification as mentioned in the

previous section for (4.7). This justification is rooted in the probability of a priori

estimate conditioned on all prior measurements (Bayes' rule). For now it is suffice

to point out that the Kalman filter maintains the first two moments of the state

distribution,

The posteriori state estimate of (4.7) reflects the mean (the first moment) of the

state distribution it is normally distributed if the conditions of (4.3) and (4.4) are met.

The posteriori estimate error covariance of (4.6) reflects the variance of the state

distribution (the second non-central moment) . In other words,

.

4.10 Probabilistic Origins of the Filter

39

4.2

This section will begin with a broad overview, covering the "high-level"

operation of one form of the discrete Kalman filter. After presenting this high-level

view, I will narrow the focus to the specific equations and their use in this discrete

version of the filter.

How does the Kalman filter works? Firstly, it estimates a process by using a

form of feedback control loop whereby the filter estimates the process state at some

time and then obtains feedback in the form of (noisy) measurements. As such, these

equations for the Kalman filter fall into two groups: “Time Update equations” and

“Measurement Update equations” .

The responsibilities of the time update equations are for projecting forward (in

time) the current state and error covariance estimates to obtain the priori estimates for

the next time step. The measurement update equations are responsible for the

feedback i.e. for incorporating a new measurement into the priori estimate to obtain

an improved posteriori estimate.

The time update equations can also be thought of as “predictor” equations,

while the measurement update equations can be thought of as “corrector” equations.

By and large, this loop process of the final estimation algorithm resembles that of a

predictor-corrector algorithm for solving numerical problems just like the one shown in

4.11 The Discrete Kalman Filter Algorithm

Fig .

40

Table

Fig

Time Update (“Predict”)

Measurement Update (“Correct”)

Courtesy of [13]

As the time update projects the current state estimate ahead in time, the

measurement update adjusts the projected estimate from the time update by an actual

measurement at that particular time. The specific equations for the “time” and

. (1.9)

. (1.10)

and covariance, estimates forward from time step to step . As mentioned

earlier, the matrixes A and B are from (1.1), while is from (1.3). Initial conditions for

the filter are discussed in the earlier section.

. (1.11)

. (1.12)

. (1.13)

: Measurement Update Equations

“measurement” updates are presented below in Table .2 and Table 4.3 .

Once again, notice how the time update equations in Table 3.2 project its state, x

: The ongoing discrete Kalman filter cycle.

Table : Time Update Equations

4.2

4.2

4.3

41

FigFig :

4.3Fig TableTable

4Table

(4.8)

(4.12) (4.7)

(4.13)

(4.12)

(4.11)

By referring to 2, it is obvious that the first task during the measurement

update is to compute the Kalman gain, . By comparing in the table above and

in the previous section, notice the equations are the same. Next, is to actually

measure the process in order to obtain , and then to generate a posteriori state

estimate, by incorporating the measurement as in . Once again, notice the

repeated equation of here and for completeness. Finally, the last step is to

obtain a posteriori error covariance estimate via .

Thus, after each time and measurement update pair, this loop process is repeated

to project or predict the new time step priori estimates using the previous time step

posteriori estimates. This recursive nature is one of the very appealing features of the

Kalman filter it makes practical implementations much more feasible than (for

example) an implementation of a Wiener filter [1] which is designed to operate on all of

the data directly for each estimate. Instead, the Kalman filter recursively conditions the

current estimate on all of the past measurements. The high-level diagram of Fig 3.1 is

which offers a much more complete and clear picture of the operation of the Kalman

filter.

A complete picture of the operation of the Kalman filter,

combining the high-level diagram of .

Courtesy of [13]

combined with the equations from and , in as shown below, 4.1 4.2

4.34.1

42

Fig:

The measurement noise covariance is usually measured before the operation

of the filter when it comes to the actual implementation of Kalman filter. Generally,

measuring the measurement noise covariance is practically possible due to the fact

that the necessary requirement to measure the process noise covariance (while

operating the filter), therefore it should be possible to take some off-line sample

measurements in order to determine the variance of the measurement noise.

As for determining of the process noise covariance , it will be generally more

difficult. This is due to the reason that the process to be estimated is unable to be

directly observed. Sometimes a relatively simple (poor) process model can produce

acceptable results if one "injects" enough uncertainty into the process via the selection

of . (Certainly, one would hope that the process measurements are reliable).

In either case, whether or not a rational basis is chosen for the parameters,

superior filter performance (statistically speaking) can be obtained by tuning the filter

parameters and .

In closing, under conditions where and are in fact constant, both the

estimation error covariance and the Kalman gain will stabilize quickly and then

remain constant (see the filter update equations in ). If this is the case, these

parameters can be pre-computed by either running the filter off-line, or for example by

determining the steady-state value of as described in .

4.12 Filter Parameters and Tuning

4.2

43

CHAPTER 5

CHAPTER-5

IMPLEMENTATION OF KALMAN

FILTER TO SPEECH

5.1 Overview of this chapter

As mentioned in Chapter 1, the idea of this project is to reconstruct a speech

signal using the Kalman filter technique. This speech signal will be modeled as an AR

process and represented in the state-space domain. In striving for this goal, certain

parameters have to be taken into consideration. This chapter gives a logical account of

the design processes involved in the implementation of the MATLAB simulation

program.

We are going to assume white noise. This means we have the same noise

power for all frequencies. We are going to calculate the coefficients of the noise added

to the original signal, at the beginning of our algorithm and then use them for all the

development; that is, the coefficients of the noise are not going to change as the

differences between the noise signal and the final signal change.

Actually, the state-space models and the Kalman filter models are often

used as synonymous. The estimation and control of the problems of this methodology

are based on stochastic models, assuming errors in the measures.

The state-space representation using Kalman filter is calculated through a

recursive procedure. The optimum estimator of the state vector for each moment n is

based on the information available until this moment. This estimator is optimum

because it minimizes the mean square error.

44

From a statistical point of view, many signals such as speech exhibit large

amounts of correlation. From the perspective of coding or filtering, this correlation can

kNi

ii

k wza

y

∑−=

=−

111

yk 1 k-1 2 k-2 N k-N k

where,

k Number of iterations;

yk current input speech signal sample;

yk–N (N-1)th sample of speech signal;

aN Nth Kalman filter coefficient;

and wk excitation sequence (white noise).

In order to apply Kalman filtering to the speech expression shown above, it must

be expressed in state space form as

k k-1 w k

k k

5.2 Implementation

(5.1)

= a y + a y + ……. + a y + w (5.2)

H = X H + (5.3) y = g H (5.4)

be put to good use . The all pole, or autoregressive (AR), signal model is often used

for speech. From Crisafulli et al , the AR signal model is introduced as:

Equation (5.1) can also be written in this form as shown below:

45

where

=

−

0100

00100001

121

L

MMOMM

L

L

L NN aaaa

X

=

+−

−

−

1

2

1

Nk

k

k

k

k

y

yyy

HM

;

=

0

00

M

k

k

w

w ;

g = [1 0 … 0].

X is the system matrix, Hk consists of the series of speech samples; w k is the

excitation vector and g, the output vector. The reason of (k-N+1)th iteration is due to the

state vector, Hk, consists of N samples, from the kth iteration back to the

(k-N+1)th iteration.

The above formulations are suitable for the Kalman filter. As mentioned in the

guide in implementing Kalman filter to speech, I denote the following steps within the

loop of the filter. Define matrix HTk-1 as the row vector:

T

k-1 k-1 k-2 k-N

and zk k

kT

k-1 k w k

(5.5)

= y . Then (5.2) and (5.6) yield

previous chapter, the Kalman filter functions in a looping method. Referring to as a

H = -[y y … y ] (5.6)

z = H X + (5.7)

46

where Xk will always be updated according to the number of iterations, k.

Note that when the k = 0, the matrix Hk-1 is unable to be determined. However,

when the time zk is detected, the value in matrix Hk-1 is known. The above purpose is

thus sufficient enough for defining the Kalman filter, which consists of:

Xk kT

k-1 k-1 k k

where

=

10000100

00100001

L

L

MMOMM

L

L

I

k k-1 k-1T

k-1 k-1 k-1-1

where Kk is the Kalman gain matrix,

Pk-1 is the a priori error covariance matrix,

R is measurement noise covariance.

and

Pk k-1 k-1 k-1T

k-1 k-1 k-1-1 T

k-1 k-1

where Pk is the a posteriori error covariance matrix;

and .

=

10000100

00100001

L

L

MMOMM

L

L

Q

= [ I – K H ] X + K z (5.8)

with K = P H [ H P H + R ] (5.9)

= P - P H [H P H + R] .H P + Q (5.10)

47

Thereafter the reconstructed speech signal, Yk after Kalman filtering will be

Yk 1 k-1 2 k-2 N k-N k

Since the value of yk is the input at the beginning of the process, there will be no

problem forming HTk-1. In that case a question rises, how is Yk formed? The parameters

wk and { are determined from application of the Kalman filter to the input speech

signal y

Niia 1} =

k. That is in order to construct Yk, we will need matrix X that contains the

Kalman coefficients and the white noise, wk which both are obtained from the

estimation of the input signal. This information is enough to determine HHk-1

where

=

+−

−

−

−

−

1

3

2

1

1

Nk

k

k

k

k

Y

YYY

HHM

Thus, forming the equation (5.11) mentioned above.

= a Y + a Y + ……. + a Y + w (5.11)

formed in a manner similar to (5.2):

48

This section lists the flow chart design of simulation of this thesis project. Attached is as follows:

No.

For Detail, refer

End

Construct Output Speech

Yes. End of Iteration.

Kalman Filter

Input Speech

Samples

Initialize

Start

to Fig 3.2.

5.3 Flow chart design of Simulation 49

CHAPTER 6

CHAPTER-6

SIMULATED RESULTS & DISCUSSIONS

6.1 Overview of this chapter

This chapter includes the simulated results of the implementation discussed

from the previous chapter, as well as discussion of these results. These simulations that

we are about to present include the results of a 5th

order Kalman filtering, which basically

show the successful operation of a simulated Kalman filter in MATLAB. This will

follow by some comparisons of reconstructed output speech signals and the input

signals, which were given by my supervisor. The aim is to employ Kalman filtering in

order to obtain output speech signal similar to the input ones.

The measurement noise covariance R is usually measured before the

operation of the filter when it comes to the actual implementation of Kalman filter.

Generally, measuring the measurement noise covariance R is practically possible due

to the fact that the necessary requirement to measure the process noise covariance Q

(while operating the filter), therefore it should be possible to take some off-line sample

Measurements in order to determine the variance of the measurement noise.

Here three different speech samples will be presented, namely: s180, s680

and s1180. All 3 samples will employ a 5th

order Kalman filter to reconstruct their

output speech. With all 3 speeches, the initiate settings of Process Noise Covariance,

Q = 1 x 10-3

and Measurement Noise Covariance, R = 0.1 are used.

50

6.2 Simulated Results of Kalman filter

The reason for employing the discrete Kalman filter is due to its very accurate

estimation capability. The simulated results presented in this section will prove the

above statement. The following results that are about to be displayed indicate the

estimates of 5 Kalman coefficients of a 5th

order Kalman filter.

The following results are obtained by setting 5 Kalman coefficients, -0.8, 0.2,

-0.6, 0.7 and –0.4. This is followed by an input of random generated noise.

yk = (-0.8)*yk-1 + (0.2)*yk-2 + (-0.6)*yk-3 + (0.7)*yk-4 + (-0.4)*yk-5 + wk (6.1)

The idea of this approach is to prove that Kalman filter functions properly in

MATLAB. As you can see from the results shown below from Fig 6.1 to Fig 6.5, the

Kalman filter generates a very close set of coefficients, which are similar to those,

of the coefficients took several hundreds of iterations in order to stabilize to the required

value. The reason for this is that Kalman filter works in a loop fashion as mentioned in

chapter 4. In order to obtain a more accurate estimate, it will need to go through more

“predict” and “correct” procedures. (Note: As the order of the Kalman filter increases, it

will take a longer duration to generate accurate coefficient estimates.)

selected above in (6.1). As you can see from the following results, accurate estimation

51

Fig 6.1 1st

Kalman coefficient.

Fig 6.2 2nd

Kalman coefficient.

52

Fig 6.3 3rd

Kalman coefficient.

Fig 6.4 4th

Kalman coefficient.

53

Fig 6.5 5th

Kalman coefficient.

6.3 Coefficients at Different Iterations

In the previous section, it has been proven that the Kalman filter estimates the

correct coefficients according to the initialized values set. Another test was conducted to

prove that the discrete Kalman filter is capable of adapting to random changes in the

coefficients at different iterations. By doing so, it will prove that the Kalman filter can

be successfully applied to time varying signals such as speech. In this case, a 5th

order

Kalman filter will have 5 groups of coefficients, however each group of coefficients

have different values set at different iterations.

54

The 5 groups of coefficients are shown in Table 6.1 and the results can be found

in Fig 6.6 to Fig 6.10. By and large, the plots produced satisfactory results. Thus, these

results suggest that the discrete Kalman filter should work well when applied to speech.

(Note that the results of the coefficients do not come exactly in line to the values set.

This factor will be discussed and explained in the next section.)

Table 6.1 Different Coefficients at Different Iterations.

Coefficient/

Iterations

1st

Set

2nd

Set

3rd

Set

4th

Set

5th

Set

1 to 2000 -0.7 0.3 -0.6 0.7 -0.5

2001 to 4000 -0.4 0.5 -0.2 0.5 -0.2

4001 to 6000 -0.5 0.3 -0.5 0.4 -0.6

6001 to 8000 -0.6 0.5 -0.4 0.2 -0.4

8001 to 10000 -0.4 0.6 -0.1 0.3 -0.2

55

Fig 6.6 1st

Kalman coefficient with different values at different iterations.

Fig 6.7 2nd

Kalman coefficient with different values at different iterations.

56

Fig 6.8 3rd

Kalman coefficient with different values at different iterations.

Fig 6.9 4th

Kalman coefficient with different values at different iterations.

57

Fig 6.10 5th

Kalman coefficient with different values at different iterations.

6.4 Results of Speech Samples

In this section, 3 different speech samples will be presented, namely: s180, s680

5th

and s1180. All 3 samples will employ a order Kalman filter to reconstruct their

output speech. With all 3 speeches, the initiate settings of Process Noise Covariance, Q

= 1 x 10-3

and Measurement Noise Covariance, R = 0.1 are used. The input and output

reconstructed speech signals are given in the following figures.

As you can see, with the current settings of Q and R stated above, s180 and

s1180 speeches are able to reconstructed accurately after several hundred iterations. On

the other hand, s680 is unable to be reconstructed satisfactorily. In order for s680 to be

constructed satisfactorily, we try tuning the parameters of Q and R.

58

Fig 6.11 Input Speech Samples of s180.od.

Fig 6.12 Output Speech Samples of s180.od.

59

Fig 6.13 Input Speech Samples of s680.od.

Fig 6.14 Output Speech Samples of s680.od.

60

Fig 6.15 Input Speech Samples of s1180.od.

Fig 6.16 Output Speech Samples of s1180.od.

61

As you can see from the speech signal in Fig 6.13 & Fig 6.14, the output speech

is not well reconstructed. However if we zoom in to the first few iterations, in this case,

taking the first 1000 iterations, the output speech has a similar structure to the input

signal (Shown in Fig 6.17 & Fig 6.18). The cause of this occurrence is the estimation

noise of the Kalman filter. As shown in Fig 6.14, the magnitude of the speech signal is

comparative high to its input. Due to the scale of the magnitude, it is unable to clearly

determine the output speech signal from Fig 6.14.

There is another phenomenon that I discovered when simulating this result. The

arrows in Fig 6.17 & Fig 6.18 show that the Kalman filter when estimating, needs some

time in order to stabilize and estimate the reconstructed output speech signal with the

given input speech signal. Notice that the filter starts to stabilize after 100 iterations.

Although the output speech signal of s1180 is able to reconstruct properly after

several hundred iterations, there is some improvement required for the initiate few

iterations (Observed Fig 6.16). Notice that in Fig 6.19 & Fig 6.20, as shown on the next

page, the structure of the output speech signal is similar to the input speech. The arrows

show the difference, as the Kalman filter needs to stabilize in order to give an optimal

estimation. Note that it also has a similar problem as s680, however the magnitude of

the signal is comparatively low this time.

62

Fig 6.17 Zoom In of first 1000 iterations of Fig 6.13.

Fig 6.18 Zoom In of first 1000 iterations of Fig 6.14.

63

Fig 6.19 Zoom In of first 500 iterations of Fig 6.15.

Fig 6.20 Zoom In of first 500 iterations for Fig 6.16.

64

After acknowledging the problems, solutions will be the next to come in. Notice

superior filter performance or in other words, optimal estimation could be obtained. The

reason for this action is to quickly stabilize both the estimation error covariance and

the Kalman gain

iteration stage.

so that the estimation can process effectively at a much earlier

First, we will try tuning the parameter Q for speech s680, from the previous

value of 1 x 10-3

to a new value of 1 x 10-6

. The reconstructed speech signal is as shown

in Fig 6.21 (Compare with the input signal in Fig 6.13).

Fig 6.21 Output Speech Samples of s680.od after tuning Q.

in the last section of Chapter 4 , it was mentioned that by tuning parameters Q and R,

65

Next, lets try obtaining an improved output signal for s1180. In order for s1180

to have a similar output speech signal, Q is tuned from 1 x 10-3

to 1 x 10-8

. The result of

this is shown below in Fig 6.22 (Compare with its input signal in Fig 6.15).

Fig 6.22 Output Speech Samples of s1180.od after tuning Q.

66

Different orders of Kalman filter have also been tested and simulated for the

values of Q. The results of tuning Q are shown below in Table 6.2.

Table 6.2 Parameters of Q tuned to the particular speech samples.

As observed from the above table, the tuning of parameter Q is dependent on

individual speech samples. The trend flows in a way that as the number of orders

increase; Q will be tuned to a smaller value in order to construct adequate results.

Parameter R had also been tuned, however the effect is found to be negligible.

6.5 Cross Correlation

Cross Correlation is actually a method for measuring the similarity of two

waveforms based upon the amount of common components contained within the two

waveforms [26]. The purpose of this technique is to compare the differences in the time

frame of two signals. Before we move on to the results of cross correlation between the

input speech signals and the output speech signals, let us briefly take a look at cross

Speech Samples

/ Orders of Kalman filter

s180

s680

s1180

2nd

Order 10-1

10-4

10-5

3rd

Order 10-3

10-5

10-7

4th

Order 10-3

10-6

10-7

5th

Order 10-3

10-6

10-8

10th

Order 10-7

10-7

10-9

correlation. The following elucidation was studied and analyzed from .

67

Cross correlation of f1 (t) and f2 (t) is defined as:

t 2

∫t1 f1 (t)f2 (t)dt

C 12 = (6.2) t 2 t 2 2 2

∫t1 f1 (t)dt ∫t1

f2 (t)dt

The magnitude of the integral in the numerator of (6.2) is an indication of the

similarity of those two signals. If,

t 2

∫ f (t)f (t)dt = 0 t1 1 2

It means that the two signals will have no similarity over the time interval (t1, t2).

In general, the cross correlation function will be modified accordingly to

∞ C12 (τ ) = ∫−∞

f1(t)f2 (t − τ )dt (6.3)

Then, C12 will be a measurement of the similarity of two particular signals over

an entire interval (-∞, ∞), whereby τ is the time shift parameter. This integral determines

whether f2 is shifted in time relative to f1.

After having a clear picture of cross correlation between two signals, let us

examine the cross correlation results obtained for the 3 speech samples. (As shown in

Fig 6.23 to Fig 6.25) The conditions for the following cross correlation tests are as

follows: R = 0.1 and the values of Q are from the 5th

Order row of Table 6.2.

68

Fig 6.23 Cross Correlation of s180.od.

Fig 6.24 Cross Correlation of 680.od.

69

Fig 6.25 Cross Correlation of s1180.od.

According to the results above, all three samples have output speech signals

almost identical to their input speech signals. The magnitude of the plots stated with

arrows indicate that they are all close to 1.0 which also signify that they are almost

identical at the point where i = 5000 or in other words, no time shift.

N

∑ x ( j∆ τ ) y ( j∆ τ − k ∆ τ ) j = 0 (6.4) C (k ∆ τ ) =

∑ x 2

( j∆ τ ) ∑ y 2

( j∆ τ − k ∆ τ )

where x Æ the input speech signal;

y Æ the output speech signal;

C Æ the measurement of cross correlation;

j Æ the state of iterations. (-5000 to 5000);

N Æ total iterations.

k Æ time shift between x and y.

70

The x-axes of the plots represent the point of time frame of the output signals

measured, whereby i = 0 indicates the extreme time shift at the left (k=-4999) and i =

10000, the extreme time shift at the right (k=5000). By and large, all 3 plots imply that

the input and output speech signals are similar and there is no time shift between the

input and output signals.

71

CHAPTER 7

CHAPTER-7

CONCLUSION

7.1 Final Thoughts

In this thesis, an implementation of employing Kalman filtering to speech

processing had been developed. As has been previously mentioned, the purpose of this

approach is to reconstruct an output speech signal by making use of the accurate

estimating ability of the Kalman filter. True enough, simulated results from the previous

chapter had proven that the Kalman filter indeed has the ability to estimate accurately.

Furthermore, the results have also shown that Kalman filter could be tuned to

provide optimal performance. With the introduction of tuning parameters Q & R, output

speech signals can be obtained similar to the input speech signals. Additional testing on

different orders of the Kalman filter when applied to speech had also been conducted.

As a consequence, parameter Q has to be tuned in order to meet the objective.

However, parameter R is of superfluous to be tuned.

Moreover, a test for cross correlation had also been conducted during this thesis

for measuring the similarity of the input and output speech signals. This test is of

necessity for the reason that different signals are bound to be similar but not identical.

By and large, this thesis has been quite successful in terms of achieving the

objectives. Consequently, perception on signal processing and Kalman filter had also

been treasured throughout the process. Most importantly, the skill in time management

applied during the research of this thesis had been developed.

72

7.2 Future Developments

The future of Kalman filtering on Speech Processing seems reasonably bright.

During the process of this thesis, many issues have been found to be potential topics for

further research work. For that reason, the following issues were raised for further

developments:

• Speech Compression: The technique of Kalman filtering can be applied to

speech coding using Autoregressive (AR) modeling. Since compression is the major

issue here, optimal compression cannot be achieve if the entire speech signal used.

The best approach is to extract the excitation sequence (white noise) otherwise known as

the non-redundant information, which contains the core information of the entire speech

signal. Moreover, it is said to beneficial for compression. After which, this white noise

can be process through a quantizer and ready to be encoded into suitable bit rates.

• Quality of speech: Human speech is however difficult to artificially produce.

This implies that the quality of speech is yet another major issue. Quality factors to

be considering which will affect the speech are some how complex. For instance,

tandem connections, robustness to acoustic inputs, robustness to digital transmission

errors as well as delay of transmission are all important factors for thorough investigation.

However, this requires tremendous effort of research to be put into.

73

APPENDIX

APPENDIX

Appendix - A

This section lists out the notations that have been used in this thesis project.

represents the estimate of at time n given observations up to, and

including time m.

The state of the filter is represented by two variables:

, the estimate of the state at time k given observations up to and including

time k;

, the error covariance matrix (a measure of the estimated accuracy of the state

estimate).

The Kalman filter has two distinct phases: Predict and Update. The predict phase uses

the state estimate from the previous time step to produce an estimate of the state at the

current time step. In the update phase, measurement information at the current time step

is used to refine this prediction to arrive at a new, (hopefully) more accurate state

estimate, again for the current time step.

Predict

Predicted state

Predicted estimate covariance

74

Update

Innovation or measurement

residual

Innovation (or residual)

covariance

Optimal Kalman gain

Updated state estimate

Updated estimate covariance

The formula for the updated estimate covariance above is only valid for the optimal

Kalman gain. Usage of other gain values requires a more complex formula found in

the derivations section.

Invariants

If the model is accurate, and the values for and accurately reflect the

distribution of the initial state values, then the following invariants are preserved: all

estimates have mean error zero

where E[ξ] is the expected value of ξ, and covariance matrices accurately reflect the

covariance of estimates

75

Appendix - B

The posterior estimate covariance matrix:

Starting with our invariant on the error covariance Pk|k as above

substitute in the definition of

and substitute

and

and by collecting the error vectors we get

Since the measurement error vk is uncorrelated with the other terms, this becomes

by the properties of vector covariance this becomes

which, using our invariant on Pk|k-1 and the definition of Rk becomes

76

This formula (sometimes known as the "Joseph form" of the covariance update

equation) is valid no matter what the value of Kk. It turns out that if Kk is the optimal

Kalman gain, this can be simplified further as shown below.

Kalman gain derivation

The Kalman filter is a minimum mean-square error estimator. The error in the posterior

state estimation is

We seek to minimize the expected value of the square of the magnitude of this

vector, . This is equivalent to minimizing the trace of the posterior

estimate covariance matrix . By expanding out the terms in the equation above and

collecting, we get:

The trace is minimized when the matrix derivative is zero:

Solving this for Kk yields the Kalman gain:

This gain, which is known as the optimal Kalman gain, is the one that

yields MMSE estimates when used.

77

Simplification of the posterior error covariance formula

The formula used to calculate the posterior error covariance can be simplified when the

Kalman gain equals the optimal value derived above. Multiplying both sides of our

Kalman gain formula on the right by SkKkT, it follows that

Referring back to our expanded formula for the posterior error covariance,

we find the last two terms cancel out, giving

This formula is computationally cheaper and thus nearly always used in practice, but is

only correct for the optimal gain. If arithmetic precision is unusually low causing

problems with numerical stability, or if a non-optimal Kalman gain is deliberately used,

this simplification cannot be applied; the posterior error covariance formula above must

be used.

78

BIBLIOGRAPHY

BIBLIOGRAPHY

[1] R. G. Brown and P. Y. C. Hwang, Introduction to Random Signals and Applied

Kalman Filtering 2nd

Edition, John Wiley & Sons, Inc., 1992.

[2] A. Gelb, Applied Optimal Estimation, MIT Press, Cambridge, MA, 1974.

[3] M. S. Grewal and A.P. Andrews, Kalman Filtering Theory and Practice, Upper

Saddle River, NJ USA, Prentice Hall, 1993.

[4] Jacobs, O. L. R. Introduction to Control Theory 2nd Edition, Oxford University

Press, 1993.

[5] Julier, Simon and Jeffrey Uhlman. "A General Method of Approximating

Nonlinear Transformations of Probability Distributions," Robotics Research

Group, Department of Engineering Science, University of Oxford, Nov 1995.

[6] S. J. Julier, J. K. Uhlmann, and H. F. Durrant-Whyte, "A New Approach for

Filtering Nonlinear Systems", Proceedings of the 1995 American Control

Conference, Seattle, Washington, pp 1628-32. Available from

http://www.robots.ox.ac.uk/~siju/work/publications/ACC95_pr.zip.

[7] Simon Julier's home page at http://www.robots.ox.ac.uk/~siju/.

[8] Kalman, R. E. 1960. "A New Approach to Linear Filtering and Prediction

Problems," Transaction of the ASME--Journal of Basic Engineering, pp. 35-45,

March 1960.

79

[9] Lewis, Richard, Optimal Estimation with an Introduction to Stochastic Control

Theory, John Wiley & Sons, Inc., 1986.

[10] Maybeck, Peter S. 1979. Stochastic Models, Estimation, and Control, Volume 1,

Academic Press, Inc.

[11] Sorenson, H. W. 1970. "Least-Squares estimation: from Gauss to Kalman,"

IEEE Spectrum, vol. 7, pp. 63-68, July 1970.

[12] A. Gelb, J. F. Kasper, Jr., R. A. Nash, Jr., C. F. Price, and A. A. Sutherlandd, Jr.,

Applied Optimal Estimation, MIT Press, Cambridge, MA 1974.

[13] http://www.cs.unc.edu/~welch/kalman/kalman_filter/kalman-1.html#pgfId-

11839

[14] M.S. Grewal and A.P. Andrews, Kalman Filtering Theory and Practice Using

MATLAB 2nd

edition, John Wiley & Sons, Canada, 2001, pp 1-12.

[15] M.S. Grewal and A.P. Andrews, Kalman Filtering Theory and Practice Using

MATLAB 2nd

edition, John Wiley & Sons, Canada, 2001, pp 16-17.

[16] R.G. Brown and P.Y.C. Hwang, Introduction to Random Signals & Applied

Kalman Filtering with MATLAB EXERCISES & Solutions 3rd

edition, John Wiley

& Sons, Canada, 1997, pp 242-252.

[17] M.S. Grewal and A.P. Andrews, Kalman Filtering Theory and Practice Using

MATLAB 2nd

eition, John Wiley & Sons, Canada, 2001, pp 15-17

[18] R. P. Ramachandran and R. Mammone, Modern Methods of Speech Processing,

Kluwer Academic Publishers, Massachusetts, USA, 1994.

[19] A.N. Ince, Digital Speech Coding, Kluwer Academic Publishers, Massachusetts,

USA, 1992.

[20] A.N. Ince, “Overview of Voice Communications and Speech Processing”,

Digital Speech Coding, Kluwer Academic Publishers, Massachusetts, USA,

1992, pp 2-41

[21] C. Wheddon & R. Linggard, “Low Bit Rate Speech”, Speech and Language

Processing, Chapman and Hall, 1990, pp 31-33.

80

[22] R.V. Cox, “Current Methods of Speech Coding”, International Journal of High

Speed Electronics and Systems, Vol. 8, No. 1, World Scientific Publishing

Company, 1997, pp-13-68.

[23] S. Crisafulli, J.D. Mills, and R.R Bitmead, “Kalman Filtering Techniques in

Speech Coding”. In Proc. IEEE International Conference on Acoustics, Speech, and

Signal Processing, San Francisco, March 1992.

[24] B.D.O. Anderson and J.B. Moore, “The Kalman Filter,” Optimal Filtering,

Prentice Hall Inc., Englewood Cliffs, N.J., 1979, pp. 50-52.

[25] B. P. Lathi, “Signal Comparison: Correlation”, Signal, Systems and

Communication, John Wiley & Sons, USA, 1965.

[26] C. R. Watkins, “Practical Kalman Filtering in Signal Coding”, New Techniques

in Signal Coding, ANU, Dec 1994.

[27] S. Saito and K. Nakata, “Digitization”, Fundamental of Speech Processing”,

Academic Press, Japan, 1985, pp 10-11.

[28] http://www.eas.asu.edu/~speech/education/educ1.html

[29] http://www2.egr.uh.edu/~glover/applets/Sampling/Sampling.html

81