Planning A Study

AP Statistics Page 1 of 12 Review: Planning a Study

______________________________ Copyright 2011 Apex Learning Inc. (See Terms of Use at www.apexvs.com/TermsOfUse)

Key terms and Concepts

Before taking the quiz, you need to be able to explain the meanings (and recognize symbols in cases where there is an associated symbol) of each of these terms or concepts. You should also know when and how to use them in statistics problems.

Nearly all of these terms and concepts are defined in the Glossary.

anecdotal data available data bias blocking census cluster sample completely randomized design control control group convenience sample experiment factor generalizability level matched pairs matching observational study placebo placebo effect randomization replication representative sample sample self-selected sample simple random sample sources of bias in surveys (under-coverage, non-response, voluntary response, experimenter

bias, wording of questions) (This is not listed in the Key Terms glossary as a specific term, but the concepts are in Key Terms and in the Lesson 2 Tutorial on Bias in Surveys.) statistical significance

stratified random sample subject systematic random sample three major principals of experimental design treatment group use of a random number generator



Objectives, Example Problems, and Study Tips

Methods of Data collectionExperiments and Surveys

Objective 1 Explain the differences between an observational study and an experiment.

Objective 2 Explain the differences between anecdotal and available data, and between a sample and a census.

Examples 1. Explain the differences between an observational study and an experiment.

2. Describe the difference between a sample and a census.

3. Why is anecdotal data not a good basis for a generalization?

4. How can available data be useful to a researcher? Tips

Be sure you understand the definition of each term. (Check the Key Terms list if you aren't sure.)

Think about this: what's the difference between observing some phenomena and trying to influence it?

Here's an example of anecdotal evidence: "I wouldn't buy a Honda; Aunt Betty bought one, and it fell apart almost immediately." Why is this not a good basis for generalization?

Sometimes you want to know something, but don't have the resources to carry out a study. What could you do to make progress on finding out what you want to know?

Answers: 1. An observational study observes and measures variable of interest, but no treatment is

applied. An experiment does impose some treatment on the subjects to be observed.

2. A census attempts to contact every member of the population. A sample selects, preferably in a random way, a part of the population of interest.

3. Anecdotal data says something about one individual, but absolutely nothing about the population that individual represents. Aunt Betty may simply have bought a lemon. To generalize you need a valid sample.

4. You might consider using available data. Available data may be very good data collected for some other purpose at some time in the past. The library is full of available data (for example, U.S. Census data). There's also a large supply of available data on the Internet. Available data, unlike anecdotal data, can be very useful.

Objective 3 Explain the following terms from the vocabulary of experiments: subject, factor, level, placebo and placebo effect, control and control groups, treatments and treatment groups, bias, matching, matched pairs design, randomization, blocking, replication.

Examples 1. What is the purpose of a control group?



2. Describe a situation in which the placebo effect might be of medical benefit.

3. You want to know if distance learning in statistics is effective. You measure students' knowledge of statistics before and after exposure to a distance-learning course. Is this an example of a matched-pairs design? Why or why not?

4. What is the main purpose of randomization? Of blocking?

5. A study designed to measure the effects of gender (male/female), race (Caucasian/African American/Hispanic/Asian), and place of residence (Northeast, South, Midwest, West) on party preference (Democrat/Republican). Identify the levels and factors in this study.

6. An experiment systematically overestimates the positive effects of a new treatment.

What's going on? Tips

Be sure you understand the definition of each term. One legitimate use of a matched-pairs design is a "before and after" design. For example,

an experiment might give a new pill to a group of volunteers and then measure the "relief" they experience after taking the pill.

Think about what can go wrong if there are no well-defined treatment and control groups. Matched pairs are a form of blocking.

Answers 1. A control group is a group of subjects who receive a placebo or no treatment at all. This

allows us to control for "lurking" variables. It's vital to have a control group because it's the only way we can be sure the treatment causes the measured response. Control is one of the three major principles of statistical design of experiments.

2. Some control group patients get better because of the placebo effect. That is, they improve because they think they've had a treatment designed to make them better. A doctor worried about the effects of a medication on a patient could let the patient believe she is receiving a drug when, in fact, she's receiving a placebo. Of course, ethical considerations must be considered when doing this.

3. This would be a matched-pairs design, since each individual student represents a block on which there are two measurements: an exam score before the treatment and an exam score after the treatment.

4. The major purpose of randomization is to control for systematic bias within treatment groups (favoring one outcome over another because of the way a sample is drawn or groups are assigned). The major purpose of blocking is to reduce variation within treatment groups.

5. There are 3 factors: gender, race, and location. Gender has two levels (male/female), race has four levels, and location also has four levels. Factors and levels refer to explanatory variables only; that is, they refer to the treatment. Party preference is a response variable that has two values; the terms factor and level do not apply.

6. Something in the design of the experiment is biasing the responses so they don't center on the true mean value. Since this is a result of the experiment's design, it's called design bias. Design bias is best controlled by being sure you employ a proper design, since it's



very difficult to know if you are systematically biasing your responses.

Objective 4 Name and describe the three major principals of experimental design.

Example 1. Name the three principals of experimental design, tell why each is important, and

give a short example of how they can be applied in an experiment.

Tip Replication means repeating an experiment. It does not mean running an

experiment once with the treatments applied to many subjects. Though replication is known as one of the three primary principles of experimental design, it often isn't directly a part of any single experiment. However, it's important that an experiment be designed and documented so that another researcher could repeat it. An unrepeatable experiment is often suspect.

Answers 1. Replication is repeating an experiment on different units or under different conditions.

This reduces chance variation in the results, and makes it easier to show that the results can be generalized to the population being studied (or even to other populations). Replication is first accomplished by designing and documenting an experiment so that it can be repeated.

Control is accounting for all other potentially influential variables (lurking variables) so you can be confident that any differences in the response variable between groups is due to the explanatory variable. In experiments this can be accomplished by randomization, matching, and setting up control groups that don't receive the treatment. For example, in a study that looks at the effects of a new medication, subjects can be randomly assigned to a control group that does not receive the treatment or that receives a placebo.

Randomization is the practice of randomly assigning experimental units to treatment and control groups. This ensures that the effects of any lurking variables are cancelled out between groups. It allows you to be confident that differences in response between the two groups are due to the treatment. For example, in a study that looks at the effects of a new medication, subjects can be randomly assigned to a control group that does not receive the treatment or that receives a placebo.

Objective 5 Given a testable idea, describe a completely randomized design, a randomized matched-pairs design, and a randomized block design, and explain which is best and why.

Examples 1. You want to test an anxiety-reduction medication on a group of adults, but are concerned

that men and women might react differently to the treatment. Which of the three designs is most appropriate? Why?

2. You want to use a randomized matched-pairs design to compare weight-loss programs. How might you do this?

3. Describe a completely randomized design for determining if people can distinguish between two diet sodas: Skinny Bubbles and Fitsofizz.



Tips Be sure you understand the definition of each design. One of the most common ways to reduce variability is by blocking. Can you think of a

source of variability you might like to reduce in question A? Matched-pairs can involve either making repeated measurements on the same subjects

or making one set of measurements on matched subjects Answers 1. Certainly, a completely randomized design would work here. However, if you're concerned

that men and women might react differently, then you may have more variability in your responses than you'd like. A randomized-block design would reduce variability by restricting groups to fewer characteristics that contribute to variability. By using a randomized block design, you avoid the problem of having the variation between men and women combined with the variation among the men and the variation among the women. You could put men in one block and women in the other and then, within each group, perform the same randomized comparative study.

2. There are a number of ways to do this, but one way might be the following: Have two

treatment groups matched on a number of characteristics, say: sex, age, weight, height, ethnicity, and occupation. This is an attempt to be sure that the two groups are as alike as possible. Then apply a treatment to each of the groups (the treatment for one of the groups could be no treatment-that is, it would simply be a control group). Then measure the weight loss for each individual within each group and find the difference between that quantity and the weight loss for the matched subject in the other group. Do appropriate inferential statistics on the resulting set of difference scores. Note that no matter how carefully you match individuals, you can never be completely sure you've considered all relevant variables. This is why studies often use some combination of matching and randomization.

Another form of matched pairs is to do before and after tests on the same individuals. This is essentially separating the subjects into blocks of one unit, for example, before and after scores on a test with a treatment between the two testings.

3. Have samples of Skinny Bubbles and Fitsofizz in small sample cups that are coded in such a way that neither the person doing the tasting nor the person conducting the trials is aware of what is in the cup (this is called double-blind). The cups of Skinny Bubbles and Fitsofizz should be arranged in random order (for example, several subjects in a row might taste the same cola) with equal numbers of each. Each subject, in turn, tastes their cola and rates its taste (say, on a 1-10 scale). The mean score for Skinny Bubbles can then be compared statistically with the mean score for Fitsofizz. (Note that it's reasonable to assume here, that there are two samples for comparison, rather than just the one sample in a matched-pairs design.)



Methods of Data Collection - Surveys

Objective 1 Define and explain the advantages and disadvantages of the various types of samples: simple random sample, stratified random sample, convenience sample, systematic sample, cluster sample, and self-selected sample.

Examples 1. You want to survey the students in your high school about their attitudes toward drinking

by teenagers. You want a sample of 15 boys and 15 girls from each grade level. What kind of sample is this, and why would you use such a sample?

2. For the information given in A, describe a procedure that would result in a random sample. Describe a procedure that would result in a systematic sample that would also be a representative sample.

3. Suppose you obtained a convenience sample for your survey by picking the first students you contact at lunch in the lunchroom until you had your sample. Why is, or isn't, this a good idea?

4. Assume that for seniors in Example A, you have randomly chosen one boy and one girl from each of 15 senior homerooms to get your sample. Certainly, before any names have been drawn, each student has an equal chance to be selected for the sample. Is the resulting group of 30 students a simple random sample from the senior class? Explain.

5. A 1988 survey by Shere Hite, Women and Love: A Cultural Revolution in Progress, reported on how women feel about contemporary relationships. One of the findings was that, "84% of women are not emotionally satisfied with their relationships." She sent out 100,000 questionnaires and based her conclusions on those women who chose to respond. What kind of sample has she selected?

Tips

Purely random samples might be desirable but, in practice, they may be impractical. If you're interested in accurately representing the population, other methods might work just fine.

In a simple random sample, each sample of a given size must be equally likely. In a self-selected sample, who is most likely to respond?

Answers 1. This is a stratified sample. If the members were chosen randomly, it would be a stratified

random sample. The strata are grade level and sex of respondent. Therefore there are eight categories, each of which will contain exactly 15 subjects. The 15 senior boys, say, are a random sample from the population of all senior boys. You'd use this kind of sample if you wanted to isolate the attitudes within each category and compare attitudes between categories.

2. To get a random sample within each strata, you could number each of the members of the eight groups sequentially and then use a table of random numbers or a graphing calculator to select the 15 potential respondents. For example, suppose there were 150 senior boys. Number them from 1 to 150 and use the randInt function on your calculator to select 15 for your sample.



Here's how to do it on the TI-83: Go to the MATH PRB menu and select #5, randInt( . Enter randInt(1,150) and press ENTER as many times as necessary to get 15 different random integers from 1 to 150. Here's how to do it on the TI-89: Go to the [2nd] [MATH] 7:Probability menu and select #4, rand(. Enter rand(150) and press ENTER as many times as necessary to get 15 different random integers from 1 to 150. One way to get a systematic sample would be to take an alphabetical list and select names according to some pattern. In the example above, you could randomly enter the list somewhere in the first 10 names and then take each 10th name on the list after that. This is not strictly random (if there were only two Clinton's among the list of senior boys, for example, they couldn't both be in the sample), but will probably be reasonably representative because you're getting a cross-section of the class.

3. This isn't a good idea. Convenience samples may be quite biased. In this case, students who hang around the quad at lunch may be among the more social members of the class and, therefore, have different attitudes toward drinking than the average student.

4. This isn't a simple random sample, but it is a stratified random sample because the sample is stratified by homeroom and gender. To be a simple random sample, any group of students must be equally likely; that is, it's equally likely that you'd get two students from each homeroom, or 30 students from one homeroom.

5. This is a self-selected sample. Self-selected samples are always suspect, since those who choose to return a survey might not accurately represent the population from which they were selected. (In many cases, a self-selected sample is biased toward the population members who feel negatively about an issue.)

Objective 2 Identify sources of bias in surveys: under-coverage, non-response, voluntary response, experimenter bias, wording of questions.

1. A questionnaire is sent to 10,000 subscribers to a magazine on country lifestyles asking them to express their preference for an upcoming presidential election. What is likely to happen in this survey?

2. One year after the Detroit race riots of 1967, interviewers asked a sample of Black residents of Detroit if they felt they could trust most White people, some White people, or none at all. When the interviewer was White, 35% answered "most"; when the interviewer was Black, 7% answered "most" (from Exploring Surveys and Information from Samples, Dale Seymour Publications, 1987, pg. 50). Comment on this finding.

3. Suppose you're interested in opinions toward more rigorous gun laws. Write two versions of a question to determine opinion on this issue, one that might influence the respondent to agree that more rigorous gun laws should be passed and one that might influence the respondent to disagree with this opinion.

Tips

Think about this: In a voluntary response situation, who do you think is most likely to respond?

The reason a double-blind technique is used in experimentation is because of the placebo



effect, or a tendency for people to behave in the way the experimenter wants them to. There is an analogous tendency for people to respond in a survey in a way they think the interviewer would want them to.

Answers 1. First, the voluntary response effect needs to be considered. That is, those who feel most

strongly about the election may be those most likely to return their questionnaires. That's not necessarily the same group as the people who will vote. But even if everybody returned their ballot, the 10,000 subscribers probably aren't a random selection of the population. This could also be a problem of under coverage, since the sampled population was not representative of the wider population.

As a real life example of this, the Literary Digest predicted that Alf Landon would defeat Franklin D. Roosevelt (by a margin of 57% to 43%) in the 1936 presidential election. They sent out 10 million questionnaires to names they obtained from telephone books, vehicle registration lists, club membership lists, and their own subscribers. 2.4 million (a high response rate!) returned them. But Roosevelt won the election, not Landon. It's certainly possible that there was a voluntary response bias at work here. It's also possible, considering the way in which they selected their sample of 10 million, that they were sampling mainly Republicans! Remember, this was at the depth of the depression, and people who had telephones, owned cars, and subscribed to magazines such as Literary Digest were more likely than the general population to have higher incomes and vote for Landon.

2. What's going on here is the tendency for an interviewee to want to please the interviewer. This is a form of interviewer bias. Most people are likely to be more honest about their feelings on a sensitive subject with someone like themselves. In this case, the interviewees were probably following their basic human tendency to tell the interviewer what they thought the interviewer wanted to hear.

3. There are many correct answers to this question. One possibility to influence the respondent to agree that we need stricter gun laws: "Do you agree or disagree with the statement, "Because of the large number of deaths in families because of guns, we should have stricter gun laws." To influence disagreement: "Do you agree or disagree with the following statement, "Because the constitution grants us the right to bear arms, stricter gun laws are not necessary." Note that in both cases, an answer to agree would indicate the desired direction of influence.

Objective 3 Explain how the proper design of experiments and surveys allows researchers to make inferences about a population, and how sampling variation influences this. Examples 1. Why would you prefer to have your experiment or survey be conducted on 100 subjects

rather than 10?

2. Why do we do experiments and surveys? Discuss why proper design is important and use the terms parameter and statistic in your answer.

Answers 1. You'd prefer to have 100 subjects rather than 10 because of the influence of sampling

variability on the results. That is, imagine flipping a fair coin. You wouldn't be too surprised



to get 7 heads in 10 flips of the coin (in fact, there's about a 17% chance getting 7 or more heads in 10 flips of a coin). On the other hand, you'd be very surprised to get 70 heads in 100 flips (there's only about .004% chance of getting 70 or more heads in 100 flips of a fair coin). Having more trials reduces the influence of sampling variability.

2. We do experiments and surveys on samples to gather data (statistics are values computed from sample data) to make generalizations to the larger population from which the samples were drawn (parameters are population values). In short, we gather statistics to make predictions about parameters. Experiments are also used to make comparisons between two populations (for example, is the difference between the average final exam scores for the same class of two different teachers large enough to argue that one teacher is doing a better job of teaching than the other).

Proper design of surveys and experiments is what gives us confidence that our inferences from data are reliable. It's the old computer story of "garbage in, garbage out." If we don't start with proper design, any predictions about populations will be suspect. Failure to randomize treatment and control groups, for example, might generate statistical differences, but we have no way of knowing if those differences were caused by our variable of interest (the treatment variable) or if it's because of some lurking variable that we can't account for.

Objective 4 Generate a random sample using either a table of random numbers or your graphing calculator.

Examples 1. You need to assign 30 people to three groups of 10 people each. Describe a procedure for

doing this. Will the three groups be independent? Why or why not?

2. You need to assign 30 people to three groups so that the groups are independent. Describe a procedure for doing this. Will the three groups each have 10 people each in them?

3. Discuss a procedure (using either a random number table or a TI-83) to simulate drawing a sample of size 30 from a population with p = .4 of having some trait. Will exactly 40% of the samples (12 out of 30) have the trait?

Tips

Independence means that the probability of assigning a member to a group is the same for each subject. Is independence possible if you limit the number in the group? If not, how do you make the assignment as random as possible?

There are a number of random number generators: coins, dice (or other polyhedra), random number tables, numbers drawn out of a hat, calculators, and so on.

Answers: 1. Put the letter A on 10 slips of paper, the letter B on 10 slips of paper, and the letter C on 10

slips of paper. Put all 30 sheets into a container and mix well. Randomly select one of the slips of paper and pair it with any of the subjects. Subjects are placed in groups according to which letter they receive. Continue this until all 30 slips of paper are gone. This does not result in an independent assignment since the probability of being assigned to any given group depends on the numbers already assigned to each group (if 20 people had already been assigned and there were only 3 in group A, the next person has p = 7/10 of being assigned to A; if there were already 7 in group A, the next person has p = 3/10 of being



assigned to A)

2. For each subject, roll a fair die. If the die comes up 1 or 2, the subject is in group A. If the die comes up 3 or 4, the subject is in group B. And, if the die comes up 5 or 6, the subject is in group C. The probability is 1/3 of being assigned to any of the three groups for each new subject. While it's possible that all three groups would end up with the same number of subjects, it's unlikely. Note that if there is a large number of subjects, differences in the numbers of each group will not be a great importancein fact, it will probably be better to know that the groups are independent.

3. Consider the random digits: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9. Consider 0, 1, 2, 3 to be "successes." Find a random starting point in a table of random numbers and count along until you have 30 numbers. Count the number of 0, 1, 2 or 3. This will be the number in your sample that have the trait. While this may be exactly 40%, it probably won't be, due to random variation. What we do know is that, on average, samples of size 30 will have 12 individuals (40%) with the given trait.

Alternatively, use a graphing calculator to generate random integers until you achieve the desired number of successes. Let a "success" be defined as above.

On your TI-83, press [MATH PRB 5 ENTER] to get the "randInt" function on the screen. Then enter, [randInt(0,9,300 STO L1]. This will store 30 random integers in List 1. Now you can count the number of successes. It's even easier if you sort L1 first.

On your TI-89, press [2nd] [MATH] 3:List 1:seq( [ENTER] [2nd] [MATH] 7:Probability 4:rand( [ENTER] 300[)] [,] x [,] 1 [,] 9 [)] [STO->] list1 [ENTER]. On your screen, it should look like this: seq(rand(300),x,1,9) list1. This will store 30 random integers in list1. Now you can count the number of successes. It's even easier if you sort list1 first.



Unit 4 Wrap-Up

Objective 1 Define each key term and concept in this unit.

Objective 2 Explain the significance of each key term and concept in this unit.

Objective 3 Apply concepts you learned in this unit to specific problems.

Tip As you review the key terms, think about how they relate to one another. Play a

game in which you pick a term at random, define it, then pick another term and think how it relates to the first term. The two terms may not seem related at first, but all of the terms you'll learn in this course are somehow related (some distant cousins rather than brothers and sisters).

Example 1. There are many ways in which the concepts from this Unit can be combined into a

specific problem. Here is just one example:

You visit a gambling casino in Reno and tell them that you're a teacher and need some materials, namely dice, for probability experiments. The casino graciously gives you 24 pair of used casino dice. To indicate that they are used, the casino stamps one side of each die with a circle that actually cuts slightly into the die. You're concerned that the marking used to cancel the dice has biased them, so that each face is no longer equally likely. Design an experiment to help you decide if the dice are fair or not.

Answer 1. There are many defensible solutions. One such solution might be: randomly select

one of each pair of dice through a random process such as flipping a coin. Roll each die a set number of times, say 96 (it's handy to have the number of rolls be a multiple of 6). Then statistically compare the number of each face that comes up in the 96 rolls with what you'd expect to get based on probability calculations (you'll learn how to do this later in the course). This method has randomization and sufficient replication, but is lacking a control (unless you argue that the theoretical expectation-what your calculations say you should expect to getis the control.)

Possibly a better solution, since it involves a more overt element of control, would be to use a matched-pairs design that pairs each canceled die with a brand new die. The dice would be randomly paired. That is, the first canceled die would be paired with a randomly selected fair die, and so on). Compare the outcomes of 96 rolls (or whatever number) of each canceled die with 96 rolls of a brand new die. The statistical analysis needed to do these comparisons has not yet been taught in this course so you'd only need to state that a statistical comparison would be needed.



About the Unit Quiz

What to Bring Scratch paper Calculator Approved formula sheet Approved tables You can't have any reference materials other than those specifically mentioned above. You won't be able to ask for help during the quiz.

Hints and Tips for the Free-Response Portion

Show your work. The test corrector won't assume you used proper set up and methods if you reach the correct answer. It's up to you to communicate the methods that you used. Answers alone, without appropriate justification, will receive no credit.

Take your time reading the question. Since we want to see how well you can apply your knowledge to new and somewhat unfamiliar situations, take some time to think about the question. If you don't understand the question, you're unlikely to find the right answer. Read the entire question before beginning to answer.

Most questions will be given in several parts. The answers from one section will often be used in subsequent sections. Missing points in an early section does not mean you'll lose points in subsequent sections. Again, read the entire question to see how the different sections connect to each other.

The calculator. As in the AP Exam, this quiz will test you on how well you know statistics, not on how well you can use your calculator. Be sure you understand the concepts behind the calculator operations. Don't use "calculator-speak" in your answer-the instructor doesn't want to read a set of steps for your graphing calculator! Use your calculator for doing the mechanics, but be sure to clearly communicate your process for solving the problem.

Use Units. If units are given in the problem, make sure that you give them in your answer.

Answer the Question. Finally, be very careful to answer the question asked. Before you move on, read over your answer to make sure you're providing exactly what the question asks for. Generally, an answer to a question you weren't asked will receive no credit.

Documents

Planning A Study