Upload
vernon-holt
View
218
Download
0
Tags:
Embed Size (px)
Citation preview
Module #1 contdCenter of a distributionSpread of a distribution
Quartiles5-Number Summary and Boxplot
Outliers
Learning Objectives
By the end of this lecture, you should be able to:
– Recognize how scales, mislabeled axes, etc on charts can be misleading – Describe the two most common statistics to describe the center of a
dataset, and when they should be used– Describe two common statistics used to describe the spread of a
dataset, and when they should be used– Understand boxplots and the 5-number summary– Describe what is meant by an outlier and describe two techniques for
identifying outliers. – Describe and apply the 1.5*IQR rule for outliers
3
Misleading chart through poor choice of scale/axis
4
Death rates from cancer (US, 1945-95)
0
50
100
150
200
250
1940 1950 1960 1970 1980 1990 2000
Years
Death
rate
(per
thousand)
Death rates from cancer (US, 1945-95)
0
50
100
150
200
250
1940 1960 1980 2000
Years
Dea
th r
ate
(per
thou
sand
)
Death rates from cancer (US, 1945-95)
0
50
100
150
200
250
1940 1960 1980 2000
Years
Death
rate
(per
thousand)
A picture is worth a thousand words,
BUT
There is nothing like hard numbers.
Look at the scales.
Scales matterHow you stretch the axes and choose your scales can give a different impression.
Death rates from cancer (US, 1945-95)
120
140
160
180
200
220
1940 1960 1980 2000
Years
Death
rate
(pe
r th
ousan
d)
Outliers• This is a very important topic. • Outliers refer to values that seem somehow ‘extreme’ or well
outside the typical range of values in your dataset.• How to deal with outliers is a very involved subject, and while it
certainly merits much discussion, we will not delve into it too much today.
• Your goal for today is to identify outliers. That is, to develop some ability to look at a number and make a reasonably educated decision as to whether or not that value is an outlier.
• We will discuss two techniques for doing so shortly:– Examination of a histogram– Using the “1.5 * IQR” Rule
5
Describing the center and spread of a distribution
• A distribution is best described through a combination of visuals (e.g. graphs), and numbers.
• Two key numeric descriptions are:– Center: e.g. the mean– Spread (aka Variation)
• Center:– Statistics for describing the center: Mean, Median, Mode
• Mean: Most of us are familiar with the ‘mean’ (average). However, we should typically only use the mean if the dataset has no outliers, and is not highly skewed.
• Median: a better choice for the center of a distribution that has outliers, or is skewed• Mode: Will discuss later
• Spread (Variation)– Statistics for describing the spread: Percentiles, Quartiles, Standard Deviation– We will discuss these shortly
6
7
The mean or arithmetic average
To calculate the average, or mean, add
all values, then divide by the number of
individuals. It is the “center of mass.”
Sum of heights is 1598.3
divided by 25 women = 63.9 inches
58.2 64.059.5 64.560.7 64.160.9 64.861.9 65.261.9 65.762.2 66.262.2 66.762.4 67.162.9 67.863.9 68.963.1 69.663.9
Measure of center: the mean
Heights of 25 women in inches
8
Another measure of center: the median
The median is the midpoint of a distribution—the number such
that half of the observations are smaller and half are larger.
1. Sort observations by size.n = number of observations
______________________________
1 1 0.62 2 1.23 3 1.64 4 1.95 5 1.56 6 2.17 7 2.38 8 2.39 9 2.510 10 2.811 11 2.912 3.313 3.414 1 3.615 2 3.716 3 3.817 4 3.918 5 4.119 6 4.220 7 4.521 8 4.722 9 4.923 10 5.324 11 5.6
n = 24 n/2 = 12
Median = (3.3+3.4) /2 = 3.35
2.b. If n is even, the median is the mean of the two middle observations.
1 1 0.62 2 1.23 3 1.64 4 1.95 5 1.56 6 2.17 7 2.38 8 2.39 9 2.510 10 2.811 11 2.912 12 3.313 3.414 1 3.615 2 3.716 3 3.817 4 3.918 5 4.119 6 4.220 7 4.521 8 4.722 9 4.923 10 5.324 11 5.625 12 6.1
n = 25 (n+1)/2 = 26/2 = 13 Median = 3.4
2.a. If n is odd, the median is observation (n+1)/2 down the list
Survival years for Disease X
9
10
‘Resistant’ is an important term. We say that the median is ‘resistant’ to outliers because the presence of 1 or 2 outliers does not affect the median dramatically. Conversely, the mean is not resistant to outliers.
Consider a series of incomes (in thousands) taken from a graduate classroom:18, 24, 37, 41, 62, 63, 2000
The median income is the middle value in the dataset: $41,000
However, the mean is dramatically higher: $320,000 since the one individual who made $2 million dollars pulls the mean disproportionally in the high direction. As a result, we end up with a ‘center’ value that probably does not truly represent the ‘average’ income of our sample.
So we say that:•The median is resistant to outliers•The mean is not resistant to outliers
11
P
erc
en
t o
f p
eo
ple
dyi
ng
Effect of outliers on the mean and median
4.3x
Without the outliers
2.4x
With the outliers
Note the presence of outliers – those two fortunate people who managed to live several years longer than the others. These two large values moved the mean up from 3.4 to 4.2However, the median , the number of years it takes for half the people to die only went from 3.4 to 3.6. Note that this says that the median is fairly resistant, but not 100% resistant. The median is not sensitive to the size of the outlier, rather, iIt is sensitive to the number of outliers.This is typical behavior for the mean and median. The mean is sensitive to outliers, because when you add all the values up to get the mean the outliers are weighted disproportionately by their large size.However, when you get the median, they are just another two points to count –the actual size of those values does not affect things.
12
Measures of spread / variation Most people intuitively ‘get’ the benefit of knowing the center of a distribution
(e.g. the ‘average’ salary of first-year doctors). However, a piece of data that is sadly neglected but is EVERY bit as important, is the spread of the data (also known as the variation).
Just as there are different ways of describing the center of a distribution (e.g. mean, median, mode), there are different techniques for describing the spread of a distribution.
As with the center, you must know which description of the spread is the best of the most accurate tool for describing the spread.
Common techniques for describing the variation in a dataset: Range: the highest and lowest values in the dataset. Important, but
outliers can give people a highly inaccurate picture (imagine if you looked at the range of salaries).
Quartiles – dividing the range into four Standard Deviation / Variance: this is one of the most effective means
of describing the spread, and a tool that we will come back to constantly throughout this course.
Percentiles and Quartiles
• The xth percentile (e.g. the 38th percentile) is the value at which ‘x’ percent of observations fall below it. – Example: If your height is said to be in the 80th percentile, it means that 80%
of the people measured were shorter than you.
• Two commonly used percentiles are the first quartile and the third quartile. These refer to the 25th and 75th percentiles respectively. – Q1 (first quartile): Refers to the 25th percentile. Ie: 25% of observations are
below this value.– Q2 (second quartile): Refers to the 50th percentile. In other words, the
median!– Q3 (third quartile): Refers to the 75th percentile. Ie: 75% of observations fall
below this value.
13
5-Number Summary and Box Plot
• Once you have divided your dataset into quartiles, you now have one technique for creating a neat little summary. It is called the ‘5 Number Summary’ and is made up of:
– Lowest number– First (lower) quartile– Median (not the mean!)– Third (upper) quartile– Highest number
• Once you have this summary in hand, you can even ‘draw’ it using a simple (but very convenient) plot known as a box plot.
15
M = median = 3.4
Q1= first quartile = 2.2
Q3= third quartile = 4.35
1 1 0.62 2 1.23 3 1.64 4 1.95 5 1.56 6 2.17 7 2.38 1 2.39 2 2.510 3 2.811 4 2.912 5 3.313 3.414 1 3.615 2 3.716 3 3.817 4 3.918 5 4.119 6 4.220 7 4.521 1 4.722 2 4.923 3 5.324 4 5.625 5 6.1
Determining the quartiles:
Start by finding the median. (This is Q2).
Then find the middle value between the lowest
number and the median (excluding the median
itself). This is the first quartile, Q1. It is the
value in the sample that has 25% of the
observations (data points) at or below it.
Then find the middle value between the
median and the highest number. This is the
third quartile, Q3. It is the value in the sample
that has 75% of the data at or below it. (It is
the median of the upper half of the sorted data,
excluding M).
Survival time (years)n=25
16
Determining the Five Number Summary
The five number summary is made up of:1. Minimum number2. Q13. Median (Q2)4. Q35. Maximum number
For this dataset, the summary is: 0.6, 2.2, 3.4, 4.35, 6.1
Again, the five number summary is a good tool for summarizing the center and spread of skewed distributions.
M = median = 3.4
Q1= first quartile = 2.2
Q3= third quartile = 4.35
1 1 0.62 2 1.23 3 1.64 4 1.95 5 1.56 6 2.17 7 2.38 1 2.39 2 2.510 3 2.811 4 2.912 5 3.313 3.414 1 3.615 2 3.716 3 3.817 4 3.918 5 4.119 6 4.220 7 4.521 1 4.722 2 4.923 3 5.324 4 5.625 5 6.1
M = median = 3.4
Q3= third quartile = 4.35
Q1= first quartile = 2.2
25 6 6.124 5 5.623 4 5.322 3 4.921 2 4.720 1 4.519 6 4.218 5 4.117 4 3.916 3 3.815 2 3.714 1 3.613 3.412 6 3.311 5 2.910 4 2.89 3 2.58 2 2.37 1 2.36 6 2.15 5 1.54 4 1.93 3 1.62 2 1.21 1 0.6
Largest = max = 6.1
Smallest = min = 0.6
Disease X
0
1
2
3
4
5
6
7
Yea
rs u
nti
l dea
th
Five-number summary:
min Q1 M Q3 max
The boxplot is a graph of the 5-Number summary
BOXPLOT
0123456789
101112131415
Disease X Multiple Myeloma
Yea
rs u
ntil
deat
h
Comparing box plots for a normal and a right-skewed distribution
Boxplots for skewed data
Boxplots remain
true to the data and
depict clearly
symmetry or skew.
OUTLIERS – Identification of the Outlier
At what point do we typically label a datapoint as an outlier? We will discuss
two methods here:
1.One way is to look at a chart and see if any values appear to be “off the
chart” relative to the large majority of values.
2.Another tool is the “1.5 IQR” Rule for outliers.
20
Alaska Florida
Identifying outlier(s) on a histogram
The overall pattern is fairly
symmetrical except for 2
states that clearly do not
belong to the main trend.
Alaska and Florida have
unusual representation of
the elderly in their
population.
A large gap in the
distribution is typically a
sign of an outlier.
Again, we are NOT currently
interested in what to do with
outliers; merely in how to
identify them.
Identification of outliers using the 1.5 IQR Rule
1. Determine the distance between Q1 and Q3 – this is called the
Interquartile Range, or IQR.
2. Multiply by 1.5
3. Determine the distance from the suspicious data point to the nearest
quartile (Q1 or Q3).
4. Determine the distance between Q1 and Q3, called the interquartile
range, or IQR.
5. We call an observation a suspected outlier if it falls more than 1.5 times
the size of the interquartile range (IQR) below the first quartile or above
the third quartile.
This technique is called the “1.5 * IQR rule for outliers.”
22
Example of the 1.5 IQR Rule
Here is the 5-number summary for the dataset discussed earlier: 0.6, 2.2, 3.4, 4.35, 6.1
Would a value of 7.5 be an outlier? What about 8?•IQR = 4.35-2.2 = 2.15•1.5*IQR = 3.23•For a number to be an outlier on the high side, it must be greater than 4.35 +3.23: 7.58•So, 7.5 would not be considered an outlier by this criteria. However, 8 would.
Q1= first quartile = 2.2
Q3= third quartile = 4.35
1 1 0.62 2 1.23 3 1.64 4 1.95 5 1.56 6 2.17 7 2.38 1 2.39 2 2.510 3 2.811 4 2.912 5 3.313 3.414 1 3.615 2 3.716 3 3.817 4 3.918 5 4.119 6 4.220 7 4.521 1 4.722 2 4.923 3 5.324 4 5.625 5 6.1
23
Remember that a histogram does not give you ALL the data - it is merely a summary (albeit a good one!) of the distribution.
However, to be able to do statistics using specific numbers (e.g. to calculate a 5-number summary) you wold need to see the actual dataset.
For this example, I will provide you with Q1 and Q3:Q1: 19.27Q3: 45.40
IQR = 45.40 – 19.27 = 26.131.5*IQR = 39.2
Any amount more than 84.60 is a suspected outlier.
How to deal with OUTLIERSOutliers are data points that require some thought. The first step is to decide whether a data point should indeed
be labeled as an outlier. We will discuss this momentarily. Once you have decided that it is an outlier, the next
question is what you want to do with it.
There are two options for dealing with outliers – you can include them in your analysis, or you can leave them
out.
•Exclude outliers: Suppose you have a datapoint that is extremely high – and you think it was recorded in error.
In this case, you would not want to include this value in your calculations since values like mean and standard
deviation would be thrown off by this bad datapoint.
•However, if you choose to leave out a datapoint, you MUST include in your paper a discussion of your reasons
for doing so.
•Include outliers: The other option, of course, is to include the outlier(s) in your calculations and analysis. In this
case, you have to decide which statistics to use (mean vs median, etc)
•Discussion question: Suppose we wanted to determine the average height of DePaul students and we use our
class as a sample. However, that particular day, we are being visited by an incoming freshman who just happens
to be the tallest person in the world. Would you include him/her in your analysis?
– I would probably leave him out of the analysis since he does not represent the ‘typical’ DePaul student.
– However, when reporting my decision, I MUST report that I did so, and explain my decision.