CHAPTER 3 –PRODUCING DATA

STAT 200

Women and Love: A Cultural Revolution in Progress (1987)The above book had many interesting results:

84% of women are not satisfied emotionally with their relationships

70% of women married 5 or more years are having sex outside of their marriage

95% of women report forms of emotional and psychological harassment from men with whom they are in love relationships

The book contained many more results and was widely criticized, even making the cover of Time magazine. So why exactly was it criticized? How were these results obtained?

2

Complexity of data collectionTo obtain the data which lead to these conclusions:

The sample was self-selected. Of the 100,000 mailed questionnaires and 4.5% were returned.

The survey contained 127 essay questions. Some questions were vague and others were leading.

To analyze 127 survey questions from 4500 respondents is no doubt a difficult task! More importantly, one could imagine that the 95,500 none-respondents differed substantially from the 4500 respondents, thus making the generalization to the general population questionable.

3

Ways of Obtaining Data

Anecdotal – looking at a single case or a few cases obtained haphazardly.

Available – There may be existing data that could be used to answer the question at hand

Surveys (3.2) Observational Studies (3.1) Experiments (3.1)

4

Vocabulary, objectives and basic concepts for Sampling

Section 1.15

Samples and Population

A Sample is a subset of the population that we use to obtain data. The data is used to make conclusions on the population.

We hope our sample is representative of the population.

A perfect sample would be a scaled down version of the population, mirroring every characteristic of the whole population.

Unfortunately there is no way of knowing if our sample is representative or not.

6

Population Characteristics The objective in sampling is to estimate a parameter which is a numerical characteristic of a population. E.g. the mean expenditures on pharmaceutical supplies by hospitals in Canada, median age, total assets, etc.

The value obtained from a sample used to estimate the parameter is a statistic. E.g. the sample mean, sample median, sample proportion, etc.

The statistic and true population characteristic rarely agree perfectly. The discrepancy is a form of error.

7

Types of errors

Sampling error is a discrepancy which arises from the luck of the draw. E.g. Suppose the population is this class and we’re measuring heart rate based on a sample of size two…

Non-sampling error arises from a systematically non-representative sample. E.g. in the self selective survey, the sample included only voluntary respondents, thus the sample wasn’t representative of the population of interest.

A sample which systematically non-representative is said to be biased. Biased samples are trouble. We’ll spend time discussing this issue later.

8

Example

Suppose we’re trying to estimate the percentage of Canadians above 15 years-old which are in support of putting restrictions on smoking in public.

A phone survey is used and 1000 eligible people respond.

Population: Canadians above 15 years oldSample: 1000 respondents.Parameter: proportion of Canadians in support of public smoking restrictions

Statistic: Sample proportion.

9

A Census

A census is a special type of survey in which the sample is the entire population.

Why would we want to take a sample instead of a census? It’s cheaper and easier It can be more precise A census may not be feasible

Taking samples from populations to make inference about the population is one of the major objectives of statistics.

10

Obtaining a Representative Sample While there is no way of knowing whether or not our sample is representative, we can find ways of obtaining samples which tend to lead to them.

As before, the information of interest will be associated to other variables, some of which we are aware of and others which we don’t even know of.

If polling Canadians on environmental issues, location of the individual is important (e.g. Alberta vs. Vancouver). But other variables such as education and age may play a role as well.

11

Matching the Sample to the Population Perfectville is a city which is an ideal, scaled down version of the population of interest. It doesn’t exist.

It may be tempting, to choose a sample such that it represents as many characteristics of the population as possible. E.g. from a sample of 1000, get 40% from Ontario, 25% from Quebec, etc.

Controlling for the distribution of other variables, like age, income, etc. , to match those of the population can be a lot of work and must be restricted to certain variables.

One of the variables we are not aware of may not be well distributed in the sample, leading to error.

12

Randomization

When we randomize, randomly select individuals/cases/participants from our entire population – like drawing names from a hat.

There are two motivations for randomization: It tends to give samples that have characteristics which are comparable to the population, which minimizes the chance of a non-representative sample.

It plays an important role in the theory used to make inference. i.e. it dictates the math that we’ll use.

In other words, we eliminate many potential causes of bias in our sample and tend to obtain a representative sample.

13

Size Matters…

Even if we randomize, the sample needs to be of a certain size in order to be representative. If it is too small, it could be very misleading. There is theory in the literature on sample size, but it will not be covered here.

It is the size of the sample that matters, not its size relative to the population.

Suppose we have two populations: 1 litre of soup and a million gallons of soup…

A large sample cannot make up for bias.

14

Sampling Methods

Section 1.215

Methods of Sampling

Definitions

3 methods: Simple Random Sampling Stratified Sampling Multistage Sampling

16

Probability Sampling

The difference amongst different samples is called the Sampling Variability.

In order to have a grasp over the sampling variability, we need to be able to characterize the behaviour of the sampling. Probability Sampling is a class of methods in which we know the probability of each sample occurring. These are desirable and every method we consider here are probability sampling methods

Non-Probability samples are done haphazardly. The results which they convey are limited in use.

17

1 – Simple Random Sampling (SRS) As the name suggests, it is the simplest method of sampling (conceptually).

It is used a standard to compare more sophisticated sampling methods

A sample is called a SRS if for a sample of size n, every possible sample of size n has an equal chance of being selected.

18

How to obtain a SRS

1. Define the population and obtain a sampling frame

2. Randomly select n individuals from it (typically with a computer).

Population Sample

19

2 – Stratified Sampling

A population can be divided into groups that we call Strata. Often these are defined by another variable besides the one being measured.

E.g. in the public smoking poll, we may want to divide the population into provinces.

Stratified Sampling consists of taking an SRS in each of the Strata.

The sample which consists of all these simple random samples is a stratified sample.

20

Population

Population Divided in Strata

Stratified Sample

21

Advantages of Stratified Sampling It may be simpler. Making independent samples within each province is simpler than one sample across the entire country.

It leads to more precise results, especially if the strata are homogenous with respect to the characteristic of interest.

Results can later be obtained on each group as well as the population.

We often stratify to ensure that our sample is representative. Even though randomization should take care of this, sometimes a variable is too important and we still wish to control for it. The randomization will take care of other factors.

22

3 – Multistage Sampling

Multistage Sampling involves sampling at more than one level.

At every level a SRS is executed.

The advantage of these is that they are cheaper and easier.

They can also be used when we don’t a have a sampling frame of the units of interest.

The disadvantage is that the results are less precise than SRS.

23

Population divided into clusters

A simple random sample of clusters

24

Examples

In order to study school students and their use of facebook, researchers decide to sample 3 school boards in BC, then 4 high schools within each selected school board, followed by 20 students in these schools.

MSF wishes to assess the mental state of its workers in refugee camps. The use of professionals is required, so they first sample 10 refugee camps and then sample workers from each of the selected camps to undergo testing by the professionals.

25

Three Scenarios

A researcher wishes to estimate the proportion of high school students who consume marijuana and other drugs.

The human resources director for google wants to evaluate employee satisfaction. She would like to evaluate each department individually as well

A Doctor wishes to estimate the age of children in her city. Not having a sampling frame, she purchases a map and divides the city into city blocks and samples 15 of these.

26

Things to watch out for!

Section 1.327

Bias

There are many forms of bias. Sometimes it is impossible to avoid it. Here we’ll define some of the more common sources.

Undercoverage Bias: When the sampling frame doesn’t include certain members of the population. E.g. In phone surveys, a little under 1% of the population in Canada do not own phones. Hence these lead to slight undercoverage.

28

More Bias

Convenience Sampling Bias: When the individuals sampled are obtained by convenience. E.g. Go to bus station and sample passers by.

Voluntary Response Bias: People who participate tend to differ from those who don’t. They typically have stronger opinions in one direction or the other.

E.g. Ask people about the Gaza strip and your sample may give more of an indication of how many people of the Jewish faith and of Palestinian descent compose your population.

29

Finishing Bias

Non-Response Bias: When many people don’t respond or are unreachable. The people who refuse to respond may differ from those who do, hence introducing bias.

Response Bias: Sometimes the respondent can be influenced into answering in a particular way. E.g. people lie about their drug consumption. A question may be leading, etc.

30

Exercises

For the following scenarios, provide:

Population Population Parameter Sample Statistic Sampling Technique Possible biases Suggestions for improvement

31

Cows

A survey is conducted to find the average weight of cows in a region. A list of all farms is available for the region, and 50 farms are selected at random. Then the weights of each cow at the 50 selected farms is recorded.

32

Birds

In the North American Breeding Bird Survey, the region of interest was divided into equal size lots and observers were assigned random lots. At each lot, the count of all birds heard or sighted within a quarter mile radius was recorded. The size of the bird population was of interest.

33

Establishing Association between Variables.

Section 2.134

Studies and Experiments

The objective of these is to determine relationships between two or more variables.

Examples: Does a new drug work better than the current one? Is study time related to grade? Does music make you smarter?

Sometimes we gather the appropriate data by observing a situation (observational) and other times we can control the situation entirely (experiment).

As in Sampling, how we collect our data will heavily influence how we can use it.

35

Causation vs Association

36

Perhaps the most important distinction is whether we can imply causation or only association.

While Causation implies association, association does not imply causation.

Example

Suppose we wanted to investigate the relationship between life expectancy and consumption of smoking.

Method 1: Select 50 non-smokers and 50 smokers. Follow these people until death and then compare the ages at death in both groups.

Method 2: From the deaths which occurred in the last year, select 50 smokers and 50 non-smokers.

Compare age of deaths in both groups. Method 3:

Select 100 people Randomly assign smoking to 50 people and non-smoking to the others.

Repeat step 2 from method 1.

37

Example Continued

All three methods report that smoking reduces life expectancy (causation). Should we trust them?

In the first two methods, the assignment of who smoked and who didn’t was not controlled.

It was simply observed who smoked and who didn’t. Such studies are called Observational Studies.

In the third approach, we decided who smoked or didn’t smoked. Since we controlled the assignment of the independent variable, this is an Experiment.

38

Which is Better39

Observational Methods can be quicker and less expensive.

Observational Studies are often the only plausible option Experiment may be ethically wrong Controlling the independent variable may be impossible.

Experiments allow us to conclude a causational relationship.

CONCLUSION: Experiments should be used whenever possible if establishing causation is the objective

Establishing Causation

Why can’t we establish causation in an observational study?

Since we don’t control who is in which group there may be a fundamental difference in the two groups other than the variable of interest. Hence it is impossible to determine what causes the association.

Observational studies are useful for establishing associations, but cannot demonstrate causal relationships.

40

More Vocabulary41

In observational studies there are always many possible lurking variables or confounders.

Lurking variable: A variable that we are not taking into account but that is associated or the cause of both variables being considered. So the Lurking variable is the cause of the observed association.

Confounding Variable: If two variables cannot be distinguished, then it is difficult to establish which is the cause.

Meditation

Suppose a study shows that regular meditation for 6 months seems to reduce blood pressure. If we compared subjects' January BP to June BP there's the possibility that it was the improving weather allowing more outdoor activity that actually helped (or some combination of warmth, sunshine, activity, and meditation). Here the meditation is confounded with another variable (weather), leaving us unable to figure out what's really going on.

42

Clicker Q743

To study Caffeine and whether it affects test performance, results were looked at for students in the morning and afternoon sections of a class. Upon further review, it was determined that most caffeine drinkers were in the morning section.

A. Class section is a confounderB. Class section is a lurking variableC. We shouldn’t care about having most coffee drinkers in the morning

Definitions and Terms for Experiments

Section 2.244

Testing Aspirin

A side effect of Aspirin is that it thins the blood which is a desirable effect for many people recovering from heart attacks. We would like to verify if Aspirin can help the survival rate of such patients. A group of 150 subjects is collected and they are randomly allocated to three groups (in groups of 50). The first is given sugar pills, the second is given a regular dose of Aspirin and the third a double dose. Survival rates while in the ICU are recorded and comparisons are made.

45

Randomized, Comparative Experiments In an Experiment we control how the explanatory variable of interest is divided among the experimental units. In the Aspirin example, the experimenter was free to determine which individual received which dosage.

Key to the notion of an experiment is the random allocation of subjects/experimental units to the different groups. Note that in experiments we refer to individuals as experimental units as well.

Randomization helps remove the effects of lurking variables and/or confounding factors.

46

Randomized, Comparative Experiments 2

Factor: An explanatory variable that is manipulated in the experiment. These are categorical variables. In our example, the factor is Aspirin (drug).

Levels: The categories of our Factor. In our example, the levels are no dose (sugar pill), regular dose and double dose.

Treatment: There can be more than one factor in an experiment. A treatment is a combination of specific levels given to a participant. Often we only consider one factor, so the levels of the explanatory variables are the treatment. E.g. sugar pill is treatment as is regular dose of Aspirin.

47

More terms

Note that both experiments and observational studies require at least one explanatory variable and at least one response variable.

Control Group: this group serves as a baseline – a reference. In order to determine if there is a ‘treatment effect’ we need to have a reference point.

Treatment Group: the group(s) receiving the active treatment(s).

In our example, the sugar pill group is a control group and the other two groups are treatment groups.

48

Psychology

200 individuals were randomly assigned to attend one of two talks on the interaction between arts and science. The talks were the same, but the people at the first talk were embarrassed at the start of the talk, while those who attended the second were not. Afterwards, people were asked to rate the talk on a scale of 1 to 10. Find the following:

Experimental unit: Factor: Levels: Response: Treatments: Control group:What graphical tool could we use to assess the difference here?

49

Principles of Experimental Design

50

Principles of Experimental DesignRandomization Treatments are randomly assigned to experimental units.

Randomization helps “average out” the effects of extraneous factors that may be present, making the treatment groups comparable except for the treatment received.

For this reason, we can establish a cause and effect relationship between the response and the explanatory variable(s).

51

Principles of Experimental DesignBlocking It might be beneficial to control for variables that are not factors but are believed to have some influence on the response variable. Such variables are called blocking variables. The idea is similar to stratifying in Sampling.

E.g. When dealing with the Aspirin example, gender may have an effect, so it becomes a blocking variable.

Experimental units are first divided into blocks (gender). Random treatment assignment and comparisons between treatments are done within each block separately.

Blocking improves the accuracy of comparisons between treatments by eliminating the effect of the blocking variables on the response variable within each block.

52

Principles of Experimental DesignReplication – of experimental units We must have `Replicates' if we are going to use any statistical methods.

A replicate means that the treatment has been applied to more than one (many) experimental units/subjects.

It is only with replicates that we can obtain an estimate of the variability in the response.

The outcome of an experiment on a single person is called an anecdote (anecdotal evidence).

53

Principles of Experimental DesignReplication – of Experiments It is desirable for the entire experiment to be repeated (or replicated). If results are consistent, it strengthens our conclusions.

Furthermore, if the experiment is repeated in another setting with a different group of individuals, it will broaden the scope of generalization of the results.

Beware of pseudo-replication. Taking 20 measurements on 5 individuals and 1 measurement on 100 individuals both result in 100 measurement, but the sample sizes will be 5 and 100 respectively. The implications here will become clear later when we look closer at the math

54

Principles of Experimental Design - Control We attempt to control variation that would occur due to differences in groups by making the conditions as similar as possible for each group.

We do this to make detecting differences brought upon by the treatment easier to detect.

This can cause problems in generalization. E.g. If we are using plant A for testing a fertilizer, how can we generalize to all plants?

There are variables for which we control and have little effect on generalization. E.g. fixed time to the protocol, using the same nurse to take measurements of interest.

55

Blinding and Placebo

We use blinding and placebos to prevent bias from entering our data.

Knowing the treatment an experimental unit receives can influence the response of both the experimental unit, and the person evaluating them. This is why we use Blinding.

There are 2 sets of people who can influence the conclusions of an experiment:1) The subjects/exp. units and the people who administer the treatments. (they influence the results)2) The people who collect the results, such as physicians, evaluators, administrators, etc.

Sometimes we also consider those who manipulate the results (statisticians)

56

Definitions

An experiment is single blinded if one of the two groups does not know who is getting which treatment.

An experiment is double blinded if both groups do not know who is getting which treatment.

In order to blind the experimental units, a placebo, which is designed to look like a treatment but having no real treatment value, is usually given to the control group if no existing treatment is used as the baseline measurement for comparison. (e.g.) a Placebo is often a sugar-pill in medical studies.

The Placebo Effect happens when a subject from the control group is on the placebo, but they think that they may be on one of the active treatments, and so they end up showing an improvement in their response, simply by believing that they are on a real drug.

57

Experimental Design

Completely Randomized Experiment

Randomized Block Design

Matched Pairs Design

58

When Can We Say That a Treatment Has an Effect?

In the Aspirin example, suppose that those taking aspirin had a survival rate 2% higher than the control group. Is the difference caused by the aspirin or due to luck?

The difference in response between the control and other treatment group1. could be due to chance

2. could be real and caused by the difference in treatment

Statistical theories will be used to discriminate between these two possibilities (see later in the course).

59

Analogy flipping a coin 100 timesWe expect to get 50 heads and 50 tails on average. In practice, the numbers are likely to be different - perhaps 47 heads and 53 tails. The next time you flip a coin 100 times, you may get 55 heads and 45 tails. How different would the numbers have to be than 50-50 before you believe the coin flipped is not fair?

To answer the question properly, we’ll need to develop some statistical methods.

60

When is the difference big enough?

61

Without using formal statistics, we can compare the response variable for the different treatments using boxplots – if the response is quantitative. Remember to compare both the measures of center and spread!

If the difference appears to be due to the treatments rather than luck, we say the difference is statistically significant.

Conclusions

Observational studies are useful for finding associations, but should not be used to determine causal relationships. The reason for this is confounding and lurking variables.

Experiments can be used to determine causal relationships. Mainly because randomization should remove the effects of lurking variables and avoid confounding.

62

Conclusions part deux

Experiments can be improved by blocking on variables we think will affect the response by reducing the sampling variability.

They can also be improved by including blinding to ensure that results will not be affected by various groups involved in the production of the data.

Experiments are not always a plausible option. Forcing people to smoke would not be ethical…

63