PROBABILITY AND STATISTICS FOR ENGINEERING

Hossein Sameti

Sharif University of TechnologySpring 2008

04/19/23 Sharif University of Technology 2

Text Book

A. Papoulis and S. Pillai, Probability, Random Variables and Stochastic Processes, 4th Edition, McGraw Hill, 2000

Source AT: www.mhhe.com/papoulis 3

TABLE OF CONTENTS

PROBABILITY THEORYLecture – 1 BasicsLecture – 2 Independence and Bernoulli TrialsLecture – 3 Random Variables Lecture – 4 Binomial Random Variable Applications, Conditional Probability Density Function and Stirling’s Formula.Lecture – 5 Function of a Random Variable Lecture – 6 Mean, Variance, Moments and Characteristic FunctionsLecture – 7 Two Random Variables Lecture – 8 One Function of Two Random Variables Lecture – 9 Two Functions of Two Random Variables Lecture – 10 Joint Moments and Joint Characteristic FunctionsLecture – 11 Conditional Density Functions and Conditional Expected Values Lecture – 12 Principles of Parameter Estimation Lecture – 13 The Weak Law and the Strong Law of Large numbers

STOCHASTIC PROCESSESLecture – 14 Stochastic Processes - IntroductionLecture – 15 Poisson ProcessesLecture – 16 Mean square EstimationLecture – 17 Long Term Trends and Hurst PhenomenaLecture – 18 Power SpectrumLecture – 19 Series Representation of Stochastic processesLecture – 20 Extinction Probability for Queues and Martingales

04/19/23 Sharif University of Technology 4

Teaching Assistants:

Hoda Akbari

Course Evaluation:

Homeworks: 10-20%Midterm: 25-30%Final Exam: 50-60%

04/19/23 Sharif University of Technology 5

Probability And Statistics

Source: http://ocw.mit.edu

PROBABILITY THEORY

1.Basics

04/19/23 Sharif University of Technology 7

Random Phenomena, Experiments

Study of random phenomena

Different outcomes

Outcomes that have certain underlying patterns about them

Experiment

- repeatable conditions

Certain elementary events Ei occur in different but completely

uncertain ways.

probability of the event Ei : P(Ei )>=0

04/19/23 Sharif University of Technology 8

Probability Definitions

Laplace’s Classical Definition

- without actual experimentation

- provided all these outcomes are equally likely.

Example

• a box with n white and m red ballselementary outcomes: {white , red}Probability of “selecting a white ball”:

• 1 P a given number is divisible by a prime p

p

04/19/23 Sharif University of Technology 9

Probability Definitions

Relative Frequency Definition- The probability of an event A is defined as

- nA is the number of occurrences of A

- n is the total number of trials

04/19/23 Sharif University of Technology 10

Probability Definitions

Example

1. The probability that a given number is divisible by a prime p:

04/19/23 Sharif University of Technology 11

Counting - Remark

General Product Rule

if an operation consists of k steps each of which can be performed in ni ways (i = 1, 2, …, k), then the entire operation can be performed in ni ways.

- Number of PINs

- Number of elements in a Cartesian product

- Number of PINs without repetition

- Number of Input/Output tables for a circuit with n input signals

- Number of iterations in nested loops

Example

04/19/23 Sharif University of Technology 12

Permutations and Combinations - Remark

If order matters choose k from n:

- Permutations :

If order doesn't matters choose k from n:

- Combinations :

A fair coin is tossed 7 times. What is the probability of obtaining 3 heads? What is the probability of obtaining at most 3 heads?

Example

04/19/23 Sharif University of Technology 13

Example: The Birthday Problem

Suppose you have a class of 23 students. Would you think it likely or unlikely that at least two students will have the same birthday?

It turns out that the probability of at least two of 23 people having the same birthday is about 0.5 (50%).

04/19/23 Sharif University of Technology 14

Axioms of Probability- Basics

The axiomatic approach to probability, due to Kolmogorov, developed through a set of axioms

The totality of all events known a priori, constitutes a set Ω, the set of all experimental outcomes.

04/19/23 Sharif University of Technology 15

Axioms of Probability- Basics

A and B are subsets of Ω .

A B

BA

A B A

BA A

A

04/19/23 Sharif University of Technology 16

Mutually Exclusiveness and Partitions

A and B are said to be mutually exclusive if

A partition of is a collection of mutually exclusive(ME) subsets of such that their union is .

BA

BA

1A2A

nA

iA

jA

, BA

04/19/23 Sharif University of Technology 17

De-Morgan’s Laws

BABABABA ;

A B

BA

A B

BA

A B

BA

A B

04/19/23 Sharif University of Technology 18

Events

Often it is meaningful to talk about at least some of the subsets of as events

we must have mechanism to compute their probabilities.

Example

Tossing two coins simultaneously:

A: The event of “Head has occurred at least once” .

04/19/23 Sharif University of Technology 19

Events and Set Operators

“Does an outcome belong to A or B”

“Does an outcome belong to A and B”

“Does an outcome fall outside A”?

These sets also qualify as events.

We shall formalize this using the notion of a Field.

04/19/23 Sharif University of Technology 20

Fields

A collection of subsets of a nonempty set forms a field F if

Using (i) - (iii), it is easy to show that the following also belong to F.

. then, and If (iii)

then, If (ii)

(i)

FBAFBFA

FAFA

F

04/19/23 Sharif University of Technology 21

Fields

If

then

We shall reserve the term event only to members of F.

Assuming that the probability P(Ei ) of elementary outcomes Ei of Ω are

apriori defined.

The three axioms of probability defined below can be used to assign probabilities to more ‘complicated’ events.

04/19/23 Sharif University of Technology 22

Axioms of Probability

For any event A, we assign a number P(A), called the probability of the event A.

Conclusions:

04/19/23 Sharif University of Technology 23

Probability of Union of to Non-ME Sets

A BA

BA

04/19/23 Sharif University of Technology 24

Union of Events

Is Union of denumerably infinite collection of pairwise disjoint events Ai an event?

If so, what is P(A ) ?

We cannot use third probability axiom to compute P(A), since it only deals with two (or a finite number) of M.E. events.

04/19/23 Sharif University of Technology 25

An Example for Intuitive Understanding

in an experiment, where the same coin is tossed indefinitely define:

A = “head eventually appears”.

Our intuitive experience surely tells us that A is an event.

If

We have:

Extension of previous notions must be done based on our intuition as new axioms.

},,,,,{

tossth theon 1st time for the appears head

1

htttt

nA

n

n

04/19/23 Sharif University of Technology 26

σ-Field (Definition):

A field F is a σ-field if in addition to the three mentioned conditions, we have the following:

- For every sequence of pairwise disjoint events belonging to F, their union also belongs to F

04/19/23 Sharif University of Technology 27

Extending the Axioms of Probability

If Ai s are pairwise mutually exclusive

from experience we know that if we keep tossing a coin, eventually, a head must show up:

But:

Using the fourth probability axiom we have:

.1)( AP

1

,n

nAA

).()(11

nn

nn APAPAP

04/19/23 Sharif University of Technology 28

Reasonablity

In previously mentioned coin tossing experiment:

So the fourth axiom seems reasonable.

04/19/23 Sharif University of Technology 29

Summary: Probability Models

The triplet (, F, P)

is a nonempty set of elementary events

- F is a -field of subsets of .

- P is a probability measure on the sets in F subject the four axioms

The probability of more complicated events must follow this framework by deduction.

04/19/23 Sharif University of Technology 30

Conditional Probability

In N independent trials, suppose NA, NB, NAB denote the number of times events A, B and AB occur respectively.

According to the frequency interpretation of probability, for large N,

Among the NA occurrences of A, only NAB of them are also found among the NB occurrences of B.

Thus the following is a measure of “the event A given that B has already occurred”:

.)( ,)( ,)(N

NABP

N

NBP

N

NAP ABBA

)(

)(

/

/

BP

ABP

NN

NN

N

N

B

AB

B

AB

04/19/23 Sharif University of Technology 31

Satisfying Probability Axioms

We represent this measure by P(A|B) and define:

As we will show, the above definition is a valid one as it satisfies all probability axioms discussed earlier.

,)(

)()|(

BP

ABPBAP

.0)( BP

04/19/23 Sharif University of Technology 32

Satisfying Probability Axioms

04/19/23 Sharif University of Technology 33

Properties of Conditional Probability

Example

In a dice tossing experiment,

- A : outcome is even

- B: outcome is 2.

The statement that B has occurred

makes the odds for A greater

04/19/23 Sharif University of Technology 34

Law of Total Probability

We can use the conditional probability to express the probability of a complicated event in terms of “simpler” related events.

Suppose that

So,

04/19/23 Sharif University of Technology 35

Conditional Probability and Independence

A and B are said to be independent events, if

This definition is a probabilistic statement, not a set theoretic notion such as mutually exclusiveness.

If A and B are independent,

Thus knowing that the event B has occurred does not shed any more light into the event A.

).()()( BPAPABP

).()(

)()(

)(

)()|( AP

BP

BPAP

BP

ABPBAP

04/19/23 Sharif University of Technology 36

Independence - Example

Example

From a box containing 6 white and 4 black balls, we remove two balls at random without replacement.

What is the probability that the first one is white and the second one is black?

?)( 21 BWP

).()|()()( 1121221 WPWBPWBPBWP

,5

3

10

6

46

6)( 1

WP

,9

4

45

4)|( 12

WBP

45

12

9

4

5

3)( 21 BWP

.122121 WBBWBW

04/19/23 Sharif University of Technology 37

Example - continued

Are W1 and B2 independent?

Removing the first ball has two possible outcomes:

These outcomes form a partition because:

So,

Thus the two events are not independent.

,5

2

15

24

5

2

3

1

5

3

9

4

10

4

36

3

5

3

45

4

)()|()()|()( 1121122

BPBBPWPWBPBP

15

4)(

5

3

5

2)()( 1212 WBPWPBP

04/19/23 Sharif University of Technology 38

General Definition of Independence

Independence between 2 or more events:

Events A1,A2, ..., An are mutually independent if, for all possible subcollections of k ≤ n events:

In experiment of rolling a die,A = {2, 4, 6} B = {1, 2, 3, 4}C = {1, 2, 4}.

Are events A and B independent?What about A and C?

Example

Source: http://ocw.mit.edu

04/19/23 Sharif University of Technology 39

Bayes’ Theorem

We have:

Thus,

Also,

).()|()( BPBAPABP

,)(

)(

)(

)()|(

AP

ABP

AP

BAPABP

).()|()( APABPABP

).()|()()|( APABPBPBAP

,)(

)()|(

BP

ABPBAP

04/19/23 Sharif University of Technology 40

Bayesian Updating: Application Of Bayes’ Theorem

Suppose that A and B are dependent events and A has apriori probability of P(A ) .

How does Knowing that B has occurred affect the probability of A?

The new probability can be computed based on Bayes’ Theorm.

Bayes’ Theorm shows how to incorporate the knowledege about B’s occuring to calculate the new probability of A.

04/19/23 Sharif University of Technology 41

Bayesian Updating - Example

Suppose there is a new music device in the market that plays a new digital format called MP∞. Since it’s new, it’s not 100% reliable.

You know that

- 20% of the new devices don’t work at all,

- 30% last only for 1 year,

- and the rest last for 5 years.

If you buy one and it works fine, what is the probability that it will last for 5 years?

Example

Source: http://ocw.mit.edu

04/19/23 Sharif University of Technology 42

Generalization of Bayes’ Theorem

A more general version of Bayes’ theorem involves partition of Ω :

In which,

Represents a collection of mutually exclusive events with assiciated apriori probabilities:

With the new information “B has occurred”, the information about Ai can be updated by the n conditional probabilities:

,)()|(

)()|(

)(

)()|()|(

1

n

iii

iiiii

APABP

APABP

BP

APABPBAP

,1 , niAi

.1 ),( niAP i

04/19/23 Sharif University of Technology 43

Bayes’ Theorem - Example

Example

1. Two boxes, B1 and B2 contain 100 and 200 light bulbs

respectively. The first box has 15 defective bulbs and the second 5. Suppose a box is selected at random and one bulb is picked out.

a) What is the probability that it is defective?

04/19/23 Sharif University of Technology 44

Example - Continued

Suppose we test the bulb and it is found to be defective. What is the probability that it came from box 1?

Note that initially,

But because of greater ratio of defective bulbs in B1 ,this probability

is increased after the bulb determined to be defective..

.8571.00875.0

2/115.0

)(

)()|()|( 11

1 DP

BPBDPDBP

?)|( 1 DBP

;5.0)( 1 BP