Statistics and Probability for Engineers...Statistics and Probability for Engineers Joost Slingerland based (heavily) on notes by Ollie Mason Semester 1, 2018-2019 Joost Slingerland

Statistics and Probability for Engineers

Joost Slingerlandbased (heavily) on notes by Ollie Mason

Semester 1, 2018-2019

Joost Slingerland based (heavily) on notes by Ollie Mason Statistics and Probability for Engineers

Introduction/Motivation

Why study statistics and probability?Apart from the intrinsic interest and beauty, there are many, manyapplications. For example,

Probability is fundamental in Queueing Theory and the designof modern communications systems.

Information Theory, which is key to data compression and thedesign of error correcting codes relies heavily on probability.

Modern methods for information retrieval from the Web andmathematical models of the Web and other complex real worldnetworks are built on probability theory. GOOGLE ranks pagesaccording to the probability that a random surfer visits them!

Control systems in the real world are subject to randomdisturbances and noise - modelling and designing such systemsrequires probability.


Introduction/Motivation

Statistical methods are developed using Probability Theory.

Quality improvement: In comparing different products andprocesses, statistics is key to determining if there is asignificant difference between them.

Determining the reliability, lifetime of products based onsampling relies on statistical methods.

Market research methods depend heavily on statistics.

In planning how to allocate resources (schools, transportfacilities, manpower), statistical methods are required.


Course Outline

Basic Probability

Definitions and basic rules of probability

Counting Techniques

Conditional Probability, Independence and Bayes Theorem

Random Variables RVs

Discrete RVs and probability mass functions

Binomial and Poisson distributions

Markov and Chebyshev Inequalities for Discrete RVs


Course Outline

Random Variables

Continuous RVs - probability density and cumulativedistribution functions

Gaussian (Normal) distribution and applications

Exponential distribution

Gamma and Weibull distributions and applications


Course Outline

Statistics

Introduction to Sampling

Central Limit Theorem

Confidence intervals and Hypothesis Tests for populationmeans and proportions

Chi-square Tests

Correlation and regression - prediction intervals andhypothesis testing


Sample Spaces and Events

We start our investigation of probability with some basic conceptsand definitions. In probability we are always concerned withexperiments in which the outcome is not determined in advance.For example:

Rolling a 6 sided die;

Dealing a hand of 5 cards from a deck of 52;

Measuring the time to failure of a mechanical component.

Measuring rainfall on a given day.

Measuring CO2 emissions from a car or a factory.

Measuring how many people use a rail link between 7am and9am on a given day.



Definition

The sample space of an experiment is the set of all possibleoutcomes of the experiment.

We often denote the sample space by Ω

If we roll an unbiased (fair) 6-sided die, then the sample spaceis Ω = 1, 2, 3, 4, 5, 6.If we toss a fair coin twice, and denote heads by H and tailsby T then the sample space is Ω = HH,HT ,TH,TT.

In all of these examples, the sample space is finite.



In many examples, sample spaces are not finite.

If we toss a fair coin until heads comes up, then the samplespace is H,TH,TTH,TTTH, . . . , . This time the samplespace is infinite. However, the sample space is still countableor discrete.

If we are measuring the emissions from cars in an NCT centreand are interested in how many cars we need to test before wefind one that fails, then our sample space is again countablyinfinite.

If we are measuring the actual emissions in g/km then oursample space would be continuous.

Note that the definition of a sample space depends on what we areinterested in measuring. We may only be interested in whether ornot a car fails an emission test in which case the sample spacewould be Pass,Fail.



In probability theory, we are concerned with quantifying thelikelihood of events. For our purposes, we shall think of events assubsets of the sample space. For example:

Example

If we toss an unbiased coin twice, then the following are examplesof events.

Getting heads twice - HH.Getting at least one tail - HT ,TH,TT.Getting heads exactly once - HT ,TH.

All of these are subsets of the sample spaceΩ = HH,HT ,TH,TT

In some examples, not all subsets of the sample space will beevents.



Example

If we roll a fair 6-sided die twice, then the following are allexamples of events.

Getting two 5’s - (5, 5).Getting two numbers that sum up to 10 -(4, 6), (5, 5), (6, 4).Getting two numbers that differ by 4 -(1, 5), (2, 6), (5, 1), (6, 2).Getting two numbers that sum to 8 -(2, 6), (3, 5), (4, 4), (5, 3), (6, 2).


Events and Sets

Example

Two different models of electric motor are tested for 1,000 hoursand the time to failure of each model is measured. Here oursample space is continuous

(x1, x2) : 0 ≤ x1 ≤ 1000, 0 ≤ x2 ≤ 1000.

The following are examples of events:

Model 1 fails first: (x1, x2)|x1 < x2.A failure occurs before 500 hours: (x1, x2|min(x1, x2) < 500.Both models fail at exactly the same time : (x1, x2)|x1 = x2.

Before defining some basic laws on how to combine events, weneed to review some elementary notions from set theory.


Events and Sets

Events can be combined in a variety of ways corresponding tostandard operations on sets. The three basic operations are union,intersection and complement.

Union E ∪ F is the event consisting of all outcomes that areeither in E or F (or both).

Intersection E ∩ F is the event consisting of all outcomes that arein both E and F .

Complement E c is the event consisting of all outcomes notbelonging to E .

E c is sometimes written as E ′ (e.g. in Montgomery & Runger)

Standard Notations∅ denotes the empty set or the impossible eventE\F denotes E ∩ F c .


Events and Sets

Another way to think about the combinations of events we justdescribed is as follows. If E and F are events, then

Union E ∪ F is the event that E or F or both happen.(Mathematicians would just say E or F happens, butthey mean that both may happen)

Intersection E ∩ F is the event that both E and F happen.

Complement E c is the event that E does not happen.

We can extend the list with more complicated events, for exampleE\F is the event that E happens, but F does not


Events and Sets

Venn diagrams are a useful way to think about events andcombinations of events:

E F

E ∪ F is the entire coloured region;

E ∩ F is the purple shaded region;

(E ∪ F )c is the unshaded region.


Mutually Exclusive Events

Events with E ∩ F = ∅ are said to be mutually exclusive.

EF

G

E and F are mutually exclusive

E and G are not

F and G are not

Another word for mutually exclusive is disjoint


Sets - Some Simple Rules

E ∪ (F ∪ G ) = (E ∪ F ) ∪ G ;

E ∩ (F ∩ G ) = (E ∩ F ) ∩ G ;

E ∩ (F ∪ G ) = (E ∩ F ) ∪ (E ∩ G );

E ∪ (F ∩ G ) = (E ∪ F ) ∩ (E ∪ G ).

de Morgan’s laws Verify using Venn diagrams and with the algebraof sets that

(E ∪ F )c = E c ∩ F c ;

(E ∩ F )c = E c ∪ F c .


Events and Sets

Example

An unbiased coin is tossed twice. Consider the events:

Getting heads twice - E = HH.Getting at least one tail - F = HT ,TH,TT.Getting heads exactly once - G = HT ,TH.

Then E and F are mutually exclusive.E ∪ G = HH,TH,HT. F ∩ G = G .E c = HT ,TH,TT, G c = TT ,HH.E c ∩ G c = TT = (E ∪ G )c .


Events and Sets

E is a subset of F if every element (outcome) in E is also in F .Formally, we write

E ⊆ F ⇐⇒ x ∈ E ⇒ x ∈ F .

E

F

This means that if the event E occurs, then F has occurred.


Events and Sets

Example

A fair 6-sided die is rolled twice. Consider the following events.

Getting two 5’s - E1 = (5, 5).Getting two numbers that sum up to 10 -E2 = (4, 6), (5, 5), (6, 4).Getting two numbers that differ by 4 -E3 = (1, 5), (2, 6), (5, 1), (6, 2).Getting two numbers that sum to 8 -E4 = (2, 6), (3, 5), (4, 4), (5, 3), (6, 2).

Then E1 ⊆ E2. E3 ∩ E4 = (2, 6), (6, 2). E3\E4 = (1, 5), (5, 1).


Axioms of Probability and Interpretations

Axioms of Probability and Interpretations


Classical Probability

The sample space describes all possible outcomes of anexperiment. Events represent sets of outcomes that are of interest.We are now interested in quantitatively describing the likelihood ofdifferent events.


An experiment has N outcomes that are equally likely tooccur;

The event E contains exactly k of these outcomes;

Then the probability of E is P(E ) = kN .

Essentially, in this case the probability of an event is the ratio offavourable outcomes to all possible outcomes .



Example

If the sample space for an experiment is a, b, c , d , e and alloutcomes are equally likely, what is:

P(a, b): Answer: 25

P(c , d , e): Answer: 35

Example

If a card is drawn from a well shuffled pack, what is the probabilitythat:

it is an ace? Answer: 452

it is a spade? Answer: 1352 = 1

4

it is red? Answer: 12 .



Example

If a fair die is rolled twice, what is the probability that the sum ofthe two scores is equal to 10?

Let E be the event that the sum is equal to 10.

The size of the sample space is N = 36.

E = (4, 6), (5, 5), (6, 4) so k = 3.

Therefore P(E ) = 336 = 1

12 .


Problems with Classical Probability

The classical concept of probability has limited applicability. Inmany real situations, not all outcomes are equally likely.

Football matches or horse races;

Lifetime of a light bulb - 100 hours, 1020 hours, lifetime of theuniverse.

Height of a person selected at random from the population.

Wind speed measured at a potential site for a wind farm.

Does a windscreen shatter or not when subjected to a specificstress?

Survey on how people travel to work - bus, cycle, rail, drive,walk, hang-glide, helicoptor.


Frequency Interpretation of Probability

An alternative concept of probability is the relative frequencyinterpretation.

Relative Frequency

The experiment is repeated N times (N is very large);

The event E occurs N(E ) times.

Then the probability of E is (approximately) P(E ) = N(E)N .

The probability of an event is the proportion of time the eventoccurs if the experiment is repeated a large number of times.


Frequency Interpretation of Probability

Example

A particular rail service has run 1000 times and has been on timeon exactly 970 of these times. Using the frequency interpretation ofprobability, what is the probability of this service running on time?

SolutionE - event that the service runs on time. Then P(E ) = 970

1000 = 0.97.

Similarly, if 70 of the last 10,000 integrated circuit chips producedby a factory have had a crack, then the probability of a chip fromthis factory being cracked is 70

10,000 = .007.


Frequency Interpretation

Example

A sample of 150 plastic panels was selected and tested for shockresistance and scratch resistance. The results are summarised inthe table below.

shock resistancehigh low

scratch high 125 12resistance low 7 6

If a panel is selected at random, what is the probability that it has:

high scratch and high shock resistance? Answer: 125150 .

high scratch or high shock resistance? Answer: 144150 .

high scratch and low shock resistance? Answer: 12150 .


Problems with Frequency Interpretation

How large does N have to be?

Is the concept well-defined? If we carry out two separate runsof N experiments, the numbers do not need to be equal.

One approach is to define P(E ) as

limN→∞N(E )

N.

The question then arises if this limit exists and, if so, how dowe calculate it in practice.

The problems with constructing a meaningful theory using eitherthe frequency or classical concepts of probability led to thedevelopment of an axiomatic approach (due to A. N. Kolmogorov).


Axioms of Probability

Sample space We begin with a sample space Ω. Events are subsetsof Ω but not all subsets are necessarily events.

Set of events We call the set of all events F , so E ∈ F ⇒ E ⊂ Ω.Simple combinations of events are also events, so Fmust be closed under complement and undercountable unions and countable intersections. Such aset of events is called a sigma algebra.

Probability measure A probability measure P : F → R is a mapwhich assigns a probability to each event.

Probability measures must satisfy the following axioms.

1 P(E ) ≥ 0 for all events E .

2 P(Ω) = 1. This means that the probability of the entiresample space must equal 1 - something must occur;

3 If events E1,E2,E3, . . ., are disjoint, meaning Ei ∩ Ej = ∅ forall i 6= j , then P(∪∞i=1Ei ) =

∑∞i=1 P(Ei ).


Axioms of Probability

These axioms make sense in terms of the frequency interpretationof probability.

Repeat an experiment N times. The event E occurs k times.Obviously, 0 ≤ k ≤ N. So 0 ≤ P(E ) ≤ 1.(We will soon show that the axioms indeed give P(E ) ≤ 1)

Ω is the collection of all possible outcomes, so the event Ωmust happen each time we repeat the experiment. SoP(Ω) = 1.

If E and F are mutually exclusive and if E occurs kE times, Foccurs kF times, then E ∪ F must occur exactly kE + kFtimes. So

P(E ∪ F ) =kE + kF

N= P(E ) + P(F ).


Mutually Exclusive Events

Example

A fair 6-sided die is rolled twice. What is the probability that thesum of the scores is either strictly greater than 10 or strictly lessthan 5?

Let E denote the event that the sum is strictly greater than10 and F the event that the sum is strictly less than 5.

E = (5, 6), (6, 5), (6, 6),F = (1, 3), (1, 2), (1, 1), (2, 2), (2, 1), (3, 1).P(E ) = 3

36 = 112 , P(F ) = 6

36 = 16 .

As these are mutually exclusive (why?) we can writeP(E ∪ F ) = 1

12 + 16 = 1

4 .


Simple properties of P

We have P(∅) = 0.To see this, take the sequence in axiom 3 to be Ei = ∅ for all i .Clearly this gives Ei ∩ Ej = ∅ ∩ ∅ = ∅. Also, ∪∞i=1Ei = ∅.The axiom then gives P(∅) = P(∅) + P(∅) + P(∅) + . . .Clearly this implies P(∅) = 0.

If E1 ∩ E2 = ∅ then we have P(E1 ∪ E2) = P(E1) + P(E2).This comes from axiom 3 if we take E3 = E4 = . . . = ∅.Similarly for any finite number of mutually exclusive eventsE1, . . .En we have the finite version of axiom 3,P(∪ni=1Ei ) =

∑ni=1 P(Ei ).


Complementary Events

The event E and the event E c = S\E are said to becomplementary. Complementary events are always mutuallyexclusive.

Thus we know that P(E ∪ E c) = P(E ) + P(E c).

But E ∪ E c = S and P(S) = 1.

This means that P(E c) = 1− P(E ) for any event E .

Theorem

For any event E , P(E c) = 1− P(E ).

This also shows that P(E ) ≤ 1 for any event (as expected).


Example

A fair die is rolled twice. What is the probability that the sum ofthe scores is less than or equal to 10?

There are many possible ways that this event can occur.However this event is the complement of the event that thesum is strictly greater than 10.

Let E denote the event that the sum is strictly greater than10. We have previously calculated that P(E ) = 1

12

Hence the answer to our problem is 1− 112 = 11

12 .


Complementary Events

Example

A computer is used to randomly generate 3 digit PINs for access toa building. (0 is allowed as a digit in any position). What is theprobability that a PIN generated in this way contains at least 2distinct digits?

First note that the set of all possible PINs can be identifiedwith the set of integers from 0 to 999. Thus the size of thesample space is 1000.

Let E be the event that a PIN contains the same digitrepeated three times.

The number of ways E can occur is 10. So P(E ) = 1100 .

We are interested in the complement of E . Thus theprobability we want is 1− P(E ) = 99

100 .


More Laws of Probability

Theorem

If the event E := e1, e2, . . . , ek, then

P(E ) = P(e1) + P(e2) + · · ·P(ek).

Events consisting of a single outcome such as e1 are elementaryevents.

Theorem

If the event E is a subset of the event F , E ⊆ F , thenP(E ) ≤ P(F ).


Examples

Example

5 tools are used to produce a certain machine part. If a machinepart is equally likely to have been produced by each tool, what isthe probability that a part selected at random:

Was produced by tool 1 or tool 2?Answer: P(1, 2) = 2

5 .

was produced by tool 2, tool 4 or tool 5?Answer: P(2, 4, 5) = 3

5 .

was not produced by tool 5?Answer: P(5c) = 1− P(5) = 4

5 .


General Addition Formula

How do we calculate P(E ∪ F ) for two events that are notmutually exclusive?

First, let’s write E ∪ F as a combination of mutually exclusiveevents.

E ∪ F = (E ∩ F c) ∪ (E ∩ F ) ∪ (E c ∩ F ).

Thus we can write using the previous result on mutually exclusiveevents:

P(E ∪ F ) = P(E ∩ F c) + P(E ∩ F ) + P(E c ∩ F ).

Adding and subtracting P(E ∩ F ) and we find that

P(E∪F ) = (P(E∩F c)+P(E∩F ))+(P(E∩F )+P(E c∩F ))−P(E∩F )

.


General Addition Formula

ButP(E ) = P(E ∩ F c) + P(E ∩ F )

andP(F ) = (P(E ∩ F ) + P(E c ∩ F ))

Hence the calculation on the last slide gives us the following result.

Theorem

For any two events E and F ,P(E ∪ F ) = P(E ) + P(F )− P(E ∩ F ).


Example

Example

E and F are events and P(E ) = 0.3, P(F ) = 0.4 andP(E ∩ F ) = 0.2. What is

P(E c)?

P(E ∪ F )?

P(E c ∩ F c)?

P(E c ∪ F c)?

P(E ∪ F c)?


Example

Example

E and F are events and P(E ) = 0.3, P(F ) = 0.4 andP(E ∩ F ) = 0.2. What is

P(E c) = 0.7.

P(E ∪ F ) = 0.3 + 0.4− 0.2 = 0.5.

P(E c ∩ F c) = P((E ∪ F )c) = 1− 0.5 = 0.5

P(E c ∪ F c) = 0.8

P(E ∪ F c): P(E ∩ F c) = 0.1 soP(E ∪ F c) = 0.3 + 0.6− 0.1 = 0.8


Examples

Example

A, B and C are events and A and C are mutually exclusive. IfP(A) = 0.2, P(B) = 0.6, P(C ) = 0.5 and P(A ∩ Bc) = 0.1:

What is the probability P(A ∩ C )?Answer As A and C are mutually exclusive,P(A ∩ C ) = P(∅) = 0.

What is the probability P(A ∪ C )?Answer As A and C are mutually exclusive,P(A ∪ C ) = P(A) + P(C ) = 0.7.

What is the probability P(A ∪ B)?Answer P(A ∩ B) = 0.1 so P(A ∪ B) = 0.2 + 0.6− 0.1 = 0.7.

What can you say about P(B ∩ C )?P(B ∩ C ) ≥ 0.2.


Documents

Statistics and Probability for Engineers...Statistics and Probability for Engineers Joost Slingerland based (heavily) on notes by Ollie Mason Semester 1, 2018-2019 Joost Slingerland