class 7: 10/21/13intro to statistical methods

cont.

                                                                   

• Being wrong in science is fine, and even necessary—as long as scientists recognize that they blew it, report their mistake openly instead of disguising it as a success, and then move on to the next thing—until they come up with the very occasional breakthrough. But as long as careers remain contingent on producing a stream of research that’s dressed up to seem more right than it is, scientists will keep delivering exactly that.

Science is a noble endeavor, but it is also a low-yield endeavor. I’m not sure that more than a very small percentage of medical research is ever likely to lead to major improvements in clinical outcomes and quality of life. We should be very comfortable with that fact. (p. 86) Friedman, David H. (2010, November). Lies, damned lies, and

medical science. The Atlantic, 306(4), 76-86

all researchers must learn the trick and avoid the mistake

• trick: begin with the question and then to figure out the best way to answer that question

• mistake: begin with the method and fit the question to the method

more on models• models should meet three criteria:

– generality, precision, accuracy• models can usually satisfy any two, at the cost of

sacrificing the third.– climatology settles for generality & accuracy– ecologists focusing particular species, for precision

& accuracy– rigorous history & ethnography often give up

generality for precision & accuracy—results can still be important

• Kitcher, Philip. (2012, May 24). The trouble with scientism: Why history and the humanities are also a form of knowledge. The New Republic, 243, 20-25)

• research using –measurement description –statistical analysis

critical for answering certain kinds of important questions

strengths of measurement description • precise descriptions• often efficient—one can make confident

predictions based on relatively small samples—if samples good

• increasingly sophisticated ways of analyzing measurement data

• powerful stat packages now available for desktop computers, e.g, Systat, SPSS, SAS

cautions• measure only what can be measured

– “to replace the unmeasureable with the unmeaningful is not progress” (Achen, 1977, p. 806)

• value precision but realize a precise description may not be an accurate one

• scientific method (drawing inferences from observations) comprises many research methods— strength not from any one specific method

my personal recommendations• whatever your Ph.D. Research Specialization

take at least one stat course, preferably 2 or 3

• whatever your methodological expertise, find people with similar interests but different methodological expertise and work with them—the best research often uses many approaches

• the statistician knows, for example, that in nature there never was a normal distribution, there never was a straight line, yet with normal and linear assumptions, known to be false, he can often derive results which match, to a useful approximation, those found in the real world. (Box, p. 792)

• all models are false, but some are useful. Box, George E. P. (1976). Science and statistics. Journal of the American Statistical Association, 71, 791-799.

further caution• Statistics today is in a conceptual and

theoretical mess. The discipline is divided into two rival camps, the frequentists and the Bayesians, and neither camp offers the tools that science needs for objectively representing and interpreting statistical data as evidence. (pp. 127-128)

Royall, Richard. (2004). The likelihood paradigm for statistical evidence. In M. L. Taper & S. R. Lele (Eds.), The nature of scientific evidence: Statistical, philosophical, and empirical considerations (pp. 199-152). Chicago, IL: University of Chicago.

K ch 19: inferential statistics • inferential statistics allow one to infer the

characteristics of a population from a representative sample– estimate characteristics of population

within a determined range with a given probability

– determine (in general) with a given probability whether effect beyond sampling and chance error exists

• parameters: refer to population• statistics: refer to sample• sampling distribution: descriptive statistic

calculated from repeated sampling• confidence intervals: range that includes

the population value with a given probability

confidence level: • the probability that the interval will contain

the population value: conventionally 68%, 95%, and 99% (2 to 1, 19 to 1, 99 to 1 respectively)

• the wider the interval the more likely it contains the population value (and the less valuable the information)

• hypothesis testing (traditionally takes form of rejecting the null hypothesis, i.e., that there is no effect beyond sampling and chance error)

• alpha level: the risk the result is due to chance; set by the researcher in advance, traditionally .10, .05, .01, .001 (N.B., no good reason for these and not others)

• p-level: the actual probability level found, which is then compared to the alpha level

two-tailed test: • non-directional, puts the alpha level at both

ends. used when one does not expect results in one direction

one-tailed test: • directional, puts alpha level at one end

(determined by researcher). increases probability of finding statistically significant result

common statistical testst test of difference between means• common, simple test for differences

between means of two groupschi-square• common test for categorical data and

frequencies–are cell values different from what

would be expected

chi-square examplesJefferson, Madison Combined: Years in Kindergarten by

SES

poor* not poor total2-year 24 34 581-year 6 65 71 total 30 99 129 chi-square: 19.4 (1 df) p

< .0001

* eligible for free or reduced lunch

Jefferson, Madison Combined: Years in Kindergarten by race

non-white white total2-year 10 48 581-year 9 62 71 total 19 110 129 chi-square: .530 (1 df) p < .5

ANOVA (analysis of variance)• experimental designs where two or more

groups or multiple conditions are being compared (common in psychology and ed psych, and in educational research in general)

• powerful: – accurate measure of error variance– tests significance of each variable as well as

combined effect, – avoids inflation of probabilities problem

(not in K)regression analysis• explains (predicts) variability of a dependent

variable using information about one or more independent variables.

• predicts expected change in dependent variable given specific changes in the independent variable

• not used in educational research as much as ANOVA, but more useful for policy purposes

regression example

achievement*= 77.5 - .80 SES**

*combined math & reading scores, ITBS** percent of low income students

errors of inference• type I error (alpha error): a concern when

theory testing (K, “when validating a finding”)• type II error (beta error): a concern when

theory building (K: “when exploring”)

• decreasing the probability of one type increases the probability of the other

• pointless to talk about Type I or II error absent discussion of what is at stake

cost of type I error in theory testing• dominant theory not challenged• knowledge production stoppedcost of type II error in theory building• possibly important explanations etc. ignored• knowledge production stopped

one of the many challenges the late and great Lee Cronbach (1916-2001) made to the accepted wisdom of the day

statistical power: 1-beta

• increasing statistical power:– increase size of effect (stronger treatment)– increase sample size– reduce variability

statistical & practical significance

• statistical: confidence at a given probability that result is not due to chance

• practical: is the result important enough, big enough, feasible, affordable—all value judgments– if one apple a day keeps the doctor away,

but it takes three grapefruit, then…?

• no statistic or statistical test can make a practical decision

• whether one risks being wrong cautiously (Type I) or wrong incautiously (Type II) cannot be decided absent cost and risk, needs, what’s at stake etc.

• no statistical analysis better than numbers (descriptions) fed into it: garbage in, garbage out

statistical significance refers only to samples from population

• it does not refer to size of effect—ceteris paribus larger effects are more likely to be statistically significant, but with large samples very small effects will be

• if you have the population, any effects are real, no matter the size

no proof in science:• a statistically significant result (assuming

appropriate analysis etc.) does not prove that the hypothesis is true, only that it has escaped disconfirmation

• the more often an hypothesis passes the test and the more demanding the tests it passes, the more certain we can be that we know something—the more we have reduced uncertainty

other terms• parametric: assumes random sampling, from

distribution with known parameters, often normal distribution

• nonparametric: when data do not come from known distribution—often with nominal or ordinal data

• robust test: accurate even when assumptions violated

• effect size: too long and too often ignored—journals now requiring estimates of effect size

thinking

simple statistical way to find out what people may not willing to admit

• ask people to flip coin– if head, answer “head: no answer”– if tail and have done X, answer “head: no answer”– if tail and have not done X, answer “no”

• thus, no’s an estimate of half who have not done x• thus, N minus twice the number of “no’s” gives

estimate of those who have done X

Vogt• regression toward the mean• reliability• self-selection bias• sleeper effect• sociogram• spurious relation (or correlation)• suppressor variable

Sieber & Tolich: 8: Degrees of nondisclosure

1. double-blind study (neither research nor subject knows)

2. researcher knows; subject does not3. researcher knows; subject told she will not

know everything at the beginning, study has been judged ethical, will be debriefed at the end.

• Hawthorne effect: short-lived increase in performance due to an intervention.

• Pygmalion effect: expectation of the researcher produces expected effects. (Rosenthal & Jacobson [1986] strongly criticized for methods)

• I find the discussion on the second degree (pp. 143-147 hard to follow).

• third degree: list on pp. 147-148

• 4 defensible justifications for deception:– data unobtainable if subjects knew real purpose– to achieve stimulus control or random assignment– to study responses to low frequency events– to avoid serious risk

• dehoaxing: explaining procedure, carefully

• desensitizing: returning subject to frame of mind at least as positive and constructive as when subject entered study

Lit review

• review section – review lit, follow explicit and logical

scheme. – 3-5 sections, with subsections, if useful– end sections with a discussion

• discussion section – synthesize the review (discussion of

discussions)

• conclusion section (< 1 p)– address original question(s)

• personal reflections section (1 p)– discuss briefly what you learned in the

process of doing the lit review• references

– make sure all citations in references– make sure all references cited

APA• use first person to talk about yourself, not

third, e.g., “The researcher . . .”• use we (us, our, etc.) only to refer to you and

your co-authors (69-70)• do not italicize Latin abbreviations—e.g., et

al., etc. and so on (only use within parens)• for seriation see 63-65• single quotation marks only within double• periods & commas always inside quotation

marks

• italicize new, technical, or key terms or labels the first time, e.g., “The term peer response . . .” (104-106)

• do not separate compound verbs with comma: “She walked down the block past her house and then turned into the driveway.”

• avoid beginning sentences with “however”• avoid “throat-clearings” to begin sentences,

e.g, furthermore, therefore, also, additionally

colon (80-81)• between a grammatically complete intro

clause and a final clause that illustrates, extends, or amplifies the first. If second clause a complete sentence, capitalize.– Kelly presented two findings: Teachers

preferred . . .• do not use a colon after intro that is not a

complete sentence– The students were Ben, Akiko, Mustafa. . . .

this week free and cheap• under construction