ECE 3800Probabilistic Methods of Signal

and System Analysis

Dr. Bradley J. BazuinWestern Michigan University

College of Engineering and Applied SciencesDepartment of Electrical and Computer Engineering

1903 W. Michigan Ave.Kalamazoo MI, 49008-5329

ECE 3800 2

Course/Lecture Overview

• Syllabus• Personal Intro.• Textbook/Materials Used• Additional Reading• ID and Acknowledgment of Policies

• Chapter 1

ECE 3800 3

Syllabus

• Everything useful for this class can be found on Dr. Bazuin’s web site!– http://homepages.wmich.edu/~bazuinb/

• The class web site is at– http://homepages.wmich.edu/~bazuinb/ECE3800/ECE3800_Sp18.html

• The syllabus …– http://homepages.wmich.edu/~bazuinb/ECE3800/Syl_3800_ABET.pdf– http://homepages.wmich.edu/~bazuinb/ECE3800/Syl_3800.pdf

Dr. Bradley J. Bazuin

• Born and raised in Michigan, Grand Rapids – Forest Hills Northern• Education

– Undergraduate BS in Engineering and Applied Sciences, Extensive Electrical Engineering from Yale University in 1980

– Graduate MS and PhD in Electrical Engineering from Stanford University in 1982 and 1989, respectively.

• Industrial Employment– Part-time ARGOSystems, Inc., Sunnyvale, CA, 1981-1989– Full-time ARGOSystems, Inc., Sunnyvale, CA, 1989-1991– Full-time Radix Technologies, Mountain View, CA, 1991-2000

• Academics– Term-appointed Faculty, WMU ECE Dept. 2000-2001– Tenure track Assistant Professor, WMU ECE Dept. 2001-2007– Interim Department Chair, WMU ECE 2017– Tenured Associate Professor, WMU ECE Dept. 2007-– Department Chair, WMU ECE 2018-

ECE 3800 4

Research and Technical Interests

• Wireless Communications – Physical Layer signal and system implementation– Software Defined Radios (SDR) - USRP & GNU radio– Xilinx with VHDL coding and Graphic processing units (GPU)

• Advanced Digital Signal Processing– Algorithmic techniques for processing detecting, estimating and exploiting signals

(communications, electronics, and sensors).– Multirate signal processing, estimation theory, adaptive signal processing

• CAPE & CASSS– Center for the Advancement of Printed Electronics– Center for Advanced Smart Sensors and Structures

• Sunseeker Solar Team Adviser & WMU Educational Solar Garden Technical Director– Embedded processing systems (TI MSP430 based)– Embedded software (control, monitoring, safety, telemetry)– Energy conversion (solar cells, batteries, super capacitors)

• Collaborative Engineering– Supporting other WMU research activities where I can contribute

ECE 3800 5

ECE 3800 6

Textbook/Materials

Required Textbook• Charles Boncelet, “Probability, Statistics, and Random

Signals," Oxford University Press, 2016, ISBN: 978-0-19-020051-0.

Recommended Material• The MATH Works,

MATLAB Student Version ($99) or CAE Centerhttp://www.mathworks.com/– Learn MATLAB for free

• https://matlabacademy.mathworks.com/ • http://www.mathworks.com/support/learn-with-matlab-tutorials.html

Other Books and Materials

• H. Stark and J.W. Woods, “Probability, Statistics and Random Processes for Engineers, 4th ed.," Prentice-Hall, Inc., Upper Saddle River, NJ, 2012, ISBN: 013-231123-2

– Previous text used for ECE 3800• George R. Cooper and Clare D. McGillem, Probabilistic Methods of Signal

and System Analysis, 3rd ed., Oxford University Press Inc., 1999. ISBN: 0-19-512354-9.

– Previous text used for ECE 3800• Alberto Leon-Garcia, “Probability, Statistics, and Random Processes For

Electrical Engineering, 3rd ed.”, Pearson Prentice Hall, Upper Saddle River, NJ, 2008, ISBN: 013-147122-8.

– Graduate text used for ECE 5820• A. Papoulis, "Probability, Random Variables, and Stochastic Processes,"

McGraw-Hill, 1965. ISBM: 07-048448-1.– My course textbook in graduate school

ECE 3800 7

ECE 3800 8

Identification and Acknowledgement

• Identification for Grade Posting, Acknowledgment of completing prerequisites,Reminder of Course and University Policies, and Acknowledgement and Signature Block

• Please read, provide unique identification, sign and date, and return to Dr. Bazuin.

ECE 3800 Based on materials in the course textbook: Charles Boncelet, “Probability, Statistics, and Random Signals," Oxford University Press, 2016.

9

Course/Text Overview1 PROBABILITY BASICS

1.1 What Is Probability?1.2 Experiments, Outcomes, and Events1.3 Venn Diagrams1.4 Random Variables1.5 Basic Probability Rules1.6 Probability Formalized1.7 Simple Theorems1.8 Compound Experiments1.9 Independence1.10 Example: Can S Communicate With D?1.11 Example: Now Can S Communicate With D?1.12 Computational Procedures

2 CONDITIONAL PROBABILITY2.1 Definitions of Conditional Probability2.2 Law of Total Probability and Bayes Theorem2.3 Example: Urn Models2.4 Example: A Binary Channel2.5 Example: Drug Testing2.6 Example: A Diamond Network

3 A LITTLE COMBINATORICS3.1 Basics of Counting3.2 Notes on Computation3.3 Combinations and the Binomial Coefficients3.4 The Binomial Theorem3.5 Multinomial Coefficient and Theorem3.6 The Birthday Paradox and Message Authentication3.7 Hypergeometric Probabilities and Card Games

4 DISCRETE PROBABILITIES AND RANDOM VARIABLES4.1 Discrete Random Variable and Probability Mass

Functions4.2 Cumulative Distribution Functions4.3 Expected Values4.4 Moment Generating Functions4.5 Several Important Discrete PMFs4.6 Gambling and Financial Decision Making

Tentative Schedule for Exam #1

ECE 3800Based on materials in the course textbook: Charles Boncelet,

“Probability, Statistics, and Random Signals," Oxford University Press, 2016.

10

Course/Text Overview (2)5 MULTIPLE DISCRETE RANDOM

VARIABLES5.1 Multiple Random Variables and PMFs5.2 Independence5.3 Moments and Expected Values5.4 Example: Two Discrete Random Variables5.5 Sums of Independent Random Variables5.6 Sample Probabilities, Mean, and Variance5.7 Histograms5.8 Entropy and Data Compression

6 BINOMIAL PROBABILITIES6.1 Basics of the Binomial Distribution6.2 Computing Binomial Probabilities6.3 Moments of the Binomial Distribution6.4 Sums of Independent Binomial Random Variables6.5 Distributions Related to the Binomial6.6 Parameter Estimation for Binomial and Multinomial

Distributions6.7 Alohanet6.8 Error Control Codes

7 A CONTINUOUS RANDOM VARIABLE7.1 A Continuous Random Variable and Its Density,

Distribution Function, and Expected Values7.2 Example Calculations for a Single Random Variable7.3 Selected Continuous Distributions7.4 Conditional Probabilities for a Continuous Random

Variable7.5 Discrete PMFs and Delta Functions7.6 Quantization7.7 A Final Word

8 MULTIPLE CONTINUOUS RANDOM VARIABLES

8.1 Joint Densities and Distribution Functions8.2 Expected Values and Moments8.3 Independence8.4 Conditional Probabilities for Multiple Random Variables8.5 Extended Example: Two Continuous Random Variables8.6 Sums of Independent Random Variables8.7 Random Sums8.8 General Transformations and the Jacobian8.9 Parameter Estimation for the Exponential Distribution8.10 Comparison of Discrete and Continuous Distributions

Tentative Schedule for Exam #2

ECE 3800 Based on materials in the course textbook: Charles Boncelet, “Probability, Statistics, and Random Signals," Oxford University Press, 2016.

11

Course/Text Overview (3)9 THE GAUSSIAN AND RELATED

DISTRIBUTIONS 9.1 The Gaussian Distribution and Density9.2 Quantile Function9.3 Moments of the Gaussian Distribution9.4 The Central Limit Theorem9.5 Related Distributions9.6 Multiple Gaussian Random Variables9.7 Example: Digital Communications Using QAM

10 ELEMENTS OF STATISTICS10.1 A Simple Election Poll10.2 Estimating the Mean and Variance10.3 Recursive Calculation of the Sample Mean10.4 Exponential Weighting10.5 Order Statistics and Robust Estimates10.6 Estimating the Distribution Function10.7 PMF and Density Estimates10.8 Confidence Intervals

10.9 Significance Tests and p-Values10.10 Introduction to Estimation Theory10.11 Minimum Mean Squared Error Estimation10.12 Bayesian Estimation

11 GAUSSIAN RANDOM VECTORS AND LINEAR REGRESSION11.1 Gaussian Random Vectors11.2 Linear Operations on Gaussian Random Vectors11.3 Linear Regression

12 HYPOTHESIS TESTING12.1 Hypothesis Testing: Basic Principles12.2 Example: Radar Detection12.3 Hypothesis Tests and Likelihood Ratios12.4 MAP Tests

Tentative Schedule for Exam #3

ECE 3800 Based on materials in the course textbook: Charles Boncelet, “Probability, Statistics, and Random Signals," Oxford University Press, 2016.

12

Course/Text Overview (4)

13 RANDOM SIGNALS AND NOISE 13.1 Introduction to Random Signals13.2 A Simple Random Process13.3 Fourier Transforms13.4 WSS Random Processes13.5 WSS Signals and Linear Filters13.6 Noise13.7 Example: Amplitude Modulation13.8 Example: Discrete Time Wiener Filter13.9 The Sampling Theorem for WSS Random Processes

14 SELECTED RANDOM PROCESSES 14.1 The Lightbulb Process14.2 The Poisson Process14.3 Markov Chains14.4 Kalman Filter

Final Exam

Chapter 1: PROBABILITY BASICS

• 1.1 What Is Probability?• 1.2 Experiments, Outcomes, and Events• 1.3 Venn Diagrams• 1.4 Random Variables• 1.5 Basic Probability Rules• 1.6 Probability Formalized• 1.7 Simple Theorems• 1.8 Compound Experiments• 1.9 Independence• 1.10 Example: Can S Communicate With D?• 1.11 Example: Now Can S Communicate With D?• 1.12 Computational Procedures

ECE 3800 Based on materials in the course textbook: Charles Boncelet, “Probability, Statistics, and Random Signals," Oxford University Press, 2016.

13

On to the Course Material