Vonk Cochrans Q 2011 June 7

8/10/2019 Vonk Cochrans Q 2011 June 7

1/38

Jet Vonk

Cochrans Q test

Methodology and Statistics for Linguistic Research(LTR002M10, 2010/2011)

10-6-2011


2/38

Jet Vonk

Content

Types of data Cochrans Q test

Who?

What?

When?

How?

Application?

Cochrans Q test: Example Limitations of the Cochrans Q test

Discussion

References

10-6-2011


3/38

Jet Vonk

Types of data

10-6-2011

nominal


4/38

Jet Vonk

Types of data

10-6-2011

nominal

Nominal data: no specific order, nonumerical meaning, differentiatedby a naming system

Examples:

Men/women

Set of countries


5/38

Jet Vonk

Types of data

10-6-2011

nominal

ordinal


6/38

Jet Vonk

Types of data

10-6-2011

nominal

ordinal

Ordinal data: items have a

specific order on the scale

Examples:

Positions within a company Winner, second and third in a

race


7/38

Jet Vonk

Types of data

10-6-2011

nominal

ordinal

interval


8/38

Jet Vonk

Types of data

10-6-2011

nominal

ordinal

interval

Interval data: ordered,constant scale, but no naturalzero

The intervals between valuesare equal

Cannot be multiplied or

divided

Examples:

Temperature

Dates


9/38

Jet Vonk

Types of data

10-6-2011

nominal

ordinal

interval

ratio


10/38

Jet Vonk

Types of data

10-6-2011

nominal

ordinal

interval

ratio

Ratio data: ordered, constantscale, natural zero

Can be multiplied or divided

Examples:

Age

Height

Weight

Length

Time


11/38

Jet Vonk

Types of data

10-6-2011

nominal

ordinal

interval

ratio

Non-parametric


12/38

Jet Vonk

Types of data

10-6-2011

nominal

ordinal

interval

ratio

Non-parametric

Parametric


13/38

Jet Vonk

Types of data

10-6-2011

nominal

Non-parametric


14/38

Jet Vonk

Types of data

10-6-2011

nominal

2 samples

k samples


15/38

Jet Vonk

Types of data

10-6-2011

nominal

dependent

independent

k samples

2 samples

dependent

independent


16/38

Jet Vonk

Cochrans Q test

10-6-2011

nominal

dependent

independent

k samples

2 samples

dependent

independent


17/38

Jet Vonk

Cochrans Q test: who?

William Gemmell Cochran

(1909 1980)

Cochran, W.G. (1950). The Comparison ofPercentages in Matched Samples.Biometrika, 37, 256-66.

10-6-2011


18/38

Jet Vonk

Cochrans Q test: what?

McNemar (1947) considered the problem of testing thesignificance of the difference between two correlatedsample proportions

Cochran suggested a generalization of the problem in

which there are k(>2) matched samples

10-6-2011


19/38

Jet Vonk

Cochrans Q test: what?

To statistically analyze success rate data

Tests the hypothesis that several related dichotomous

variables have the same mean The variables are measured on the same individual or

on matched individuals

10-6-2011


20/38

Jet Vonk

Cochrans Q test: when?

Nominal data More than 2 samples

Dependent

Binary response: succes (1) versus failure (0)

10-6-2011


21/38

Jet Vonk

Cochrans Q test: how?

Each of k treatments is independently applied to bblocks (or subjects) and each outcome is measured asa success (1) or as a failure (0)

10-6-2011


22/38

Jet Vonk


Hypotheses: H0: treatments are similarly effective

H1: treatments differ in effectiveness

10-6-2011


23/38

Jet Vonk


Test statistic:

k is the number of treatments

X jis the column total for thejth treatment

b is the number of blocks

Xi is the row total for the ithblock

Nis the grand total

For significance level , the critical region is

where 21 ,k 1 is the (1 )-quantile of the chi-square distribution

with k 1 degrees of freedom

10-6-2011


24/38


25/38

Jet Vonk

Cochrans Q test: application?

Cochrans Q is often used for meta-analyses, e.g: Is there a difference in treatments (to test)?

Is there a difference in tasks (to test)?

Is there a difference in materials (to test)?

But also: are used in the same rate or is there a difference?

Methods Materials

Devices

10-6-2011


26/38

Jet Vonk

Cochrans Q test: example

Background: A teacher has been observed on 20 different moments in

his lessons (when he was about to explain something)

Observed on 3 different teaching strategies:

Point out the subject

Correspond with questions of students

Use of a stimulating beginning

Question: does the teacher use these strategies in thesame rate or is there a difference?

10-6-2011


27/38

Jet Vonk


10-6-2011


28/38

Jet Vonk


10-6-2011


29/38

Jet Vonk


10-6-2011

There is a significant difference in the use of teachingstrategies used by this teacher (X2(2) = 13.37, p = .001)


30/38

Jet Vonk


When you find any significant effect, you need to do apost-hoc test (as you do for ANOVA)

For Cochran's Q test: run multiple McNemar's testsand adjust the p values with Bonferroni correction(a method used to address the problem of multiple

comparisons, overcorrects for Type I error)

10-6-2011


31/38

Jet Vonk


10-6-2011


32/38

Jet Vonk


10-6-2011

Bonferroni correction: = 0.05

3 comparisons

0.05/3= 0,0166666666666667

Only the difference between the use of point out thesubject and stimulating beginning is significant


33/38

Jet Vonk

Limitations of the Cochrans Q test

Only determines the occurrence of a change, but does not evaluatesthe extent of change

Possible to do multiple McNemars tests, but no interactioneffect can be measured

10-6-2011


34/38

Jet Vonk




The test is known to be poor at detecting true heterogeneity amongstudies as significant

Meta-analyses often include small numbers of studies, and thepower of the test in such circumstances is low

Because the test is poor at detecting true heterogeneity, a non-significant result cannot be taken as evidence of homogeneity

10-6-2011


35/38

Jet Vonk




The test is known to be poor at detecting true heterogeneity amongstudies as significant

Meta-analyses often include small numbers of studies, and thepower of the test in such circumstances is low

Because the test is poor at detecting true heterogeneity, a non-significant result cannot be taken as evidence of homogeneity

The test does not accommodate a control group, because it is a test

for use with dependent observations

10-6-2011


36/38

Jet Vonk

Discussion

To begin with:

Questions?

10-6-2011


37/38

Jet Vonk

Discussion

Discussion points?

for example:

Is this analysis usefull in Linguistic analyses? Is it applicable in your research (e.g. pilot study)?

10-6-2011


38/38

Jet Vonk

References

Cochran, W.G. (1950). The Comparison of Percentages in Matched Samples.

Biometrika, 37, 256-66.

Higgins, J.P.T., Thompson, S.G., Deeks, J.J., Altman, D.G. (2003). Measuringinconsistency in meta-analyses.BMJ, 327, 557-560.

Pett, M.A. (1997).Nonparametric Statistics for Health Care Research: Statistics for

Small Samples and Unusual Distributions. Thousand Oaks, CA: SAGE Publications.

10-6-2011

Documents

Vonk Cochrans Q 2011 June 7