Is Reliability Validity

8/4/2019 Is Reliability Validity

1/17

1

Reliability and Validity in

ResearchBee Bornheimer, Robin Fitzpatrick,Sarah Lehmann, Matt Pierce, and

Maureen WhalenApril 23, 2008


2/17

2

Believing what you read?

. . . there is a need for reliable and validdata on student learning outcomes.

Validity concerns the degree to whichinferences about students based on their

test scores are warranted.-Cameron, L, SL Wise, and SM Lottridge. 2007. The Development andValidation of the Information Literacy Test. College and research

libraries68 (3):229.


3/17

3

Reliable and Valid Data

The teams combined goal was to produce a valid,reliable, authentic assessment of ICT literacy skills.

The goal of the iSkills assessment is to measure theICT literacy skills of studentshigher scores on theassessment should reflect stronger skills.

-Katz, IR. 2007. Testing Information Literacy in Digital Environments: ETS'siSkills Assessment. Information technology and libraries26 (3):3.


4/17

4

Reliability

In statistics or measurement theory, ameasurement or test is considered reliable if itproduces consistent results over repeatedtestings.

Refers to how well we are measuring whateverit is that is being measured (regardless ofwhether or not it is the right quantity tomeasure).

-D. Rindskopf, Reliability: Measurement. In: Neil J. Smelser and Paul B. Baltes,

Editor(s)-in-Chief, International Encyclopedia of the Social & BehavioralSciences, Pergamon, Oxford, 2001, Pages 13023-13028.(http://www.sciencedirect.com/science/article/B7MRM-4MT09VJ2XN/1/083e3cc0b8b9d4e027b0ba214dcd9fa3 )
http://www.sciencedirect.com/science/article/B7MRM-4MT09VJ2XN/1/083e3cc0b8b9d4e027b0ba214dcd9fa3http://www.sciencedirect.com/science/article/B7MRM-4MT09VJ2XN/1/083e3cc0b8b9d4e027b0ba214dcd9fa3http://www.sciencedirect.com/science/article/B7MRM-4MT09VJ2XN/1/083e3cc0b8b9d4e027b0ba214dcd9fa3http://www.sciencedirect.com/science/article/B7MRM-4MT09VJ2XN/1/083e3cc0b8b9d4e027b0ba214dcd9fa3


5/17

5

Reliability

Unlike the common understanding, inthese contexts reliability does not imply a

value judgment

Your car always starts/doesnt start

Your friend is always/ never late


6/17

6

Classical Test Theory (CTT)

A single trait or skill is being measured

The trait or skill can be defined

All items on the test measure the same trait or

skill

Formula for determining reliability

Test is made more reliable by making it longer

Limitation: reliability depends upon the samplegroup and is not a characteristic of the test

itself.


7/17

7

Generalizability Theory (GT)

Based on analysis of variance

Unlike CTT, GT allows for multiple sourcesof error

The test is designed to account for factorsthat researchers predict will influencescores

Can compute multiple estimates ofreliability


8/17

8

Item Response Theory (IRT)

Like CCT, IRT measures a single trait orskill

Relationship between the score on an

individual test item and the skill/trait canbe measured

Adaptive tests tests can be customized

to the individual test-taker, e.g., the GRE Does not use the traditional concept of

reliability


9/17

9

Observational Studies

Some characteristics cannot be measuredthrough a test

Unobtrusiveness

Multiple sources of error

Reliability depends on the extent to which

observers agree


10/17

10

Validity Evidence

Content Validity: that based on expert ratingsof the items in the test

Construct Validity: that based on the degree towhich ILT scores statistically behave as we

would expect a measure of information literacyto behave.

- Cameron, L, SL Wise, and SM Lottridge. 2007. The Development andValidation of the Information Literacy Test. College and research

libraries68 (3):229.


11/17

11

How can validity be established?

Quantitative studies: measurements, scores, instruments used, research

design

Qualitative studies: ways that researchers have devised to establish

credibility: member checking, triangulation, thickdescription, peer reviews, external audits


12/17

12

How can reliability beestablished?

Quantitative studies?

Assumption of repeatability

Qualitative studies?

Reframe as dependability and confirmability


13/17

13

"Reliability and validity are tools of an essentiallypositivist epistemology. While they may haveundoubtedly proved useful in providing checks andbalances for quantitative methods, they sit uncomfortably

in research of this kind, which is better concerned byquestions about power and influence, adequacy andefficiency, suitability and accountability. "

Watling as cited in Simco & Warin, 1997, as cited in Winter, G. A

comparative discussion of the notion of validity in qualitative andquantitative research. The Qualitative Report 4, nos. 3 and 4, (March2000.). http://www.nova.edu/ssss/QR/QR4-3/winter.html.
http://www.nova.edu/ssss/QR/QR4-3/winter.htmlhttp://www.nova.edu/ssss/QR/QR4-3/winter.htmlhttp://www.nova.edu/ssss/QR/QR4-3/winter.htmlhttp://www.nova.edu/ssss/QR/QR4-3/winter.html


14/17

14

Reliability and Validity

Why do we bother?

Terms used in conjunction with oneanother

Quantitative Research: R & V are treated asseparate terms

Qualitative Research: R & V are often allunder another, all encompassing term

Semi-reciprocal relationship


15/17

15

Reliable

Valid

Not Reliable

Not Valid

Not Valid

Reliability Validity


16/17

16

Winter states . . .

There is no single form, construct or concept that canuniversally be claimed to define or encompass the term.Neither, however, can validity be said to be a discreetlyidentifiable element of any research project, which is

capable of being located at multiple and specific stageswithin research. The concept of validity defiesextrapolation from, or categorization within, any researchproject.

-Winter, G. A comparative discussion of the notion of validity inqualitative and quantitative research. The Qualitative Report 4,nos. 3 and 4, (March 2000.). http://www.nova.edu/ssss/QR/QR4-3/winter.html.
http://www.nova.edu/ssss/QR/QR4-3/winter.htmlhttp://www.nova.edu/ssss/QR/QR4-3/winter.htmlhttp://www.nova.edu/ssss/QR/QR4-3/winter.htmlhttp://www.nova.edu/ssss/QR/QR4-3/winter.htmlhttp://www.nova.edu/ssss/QR/QR4-3/winter.html


17/17

17

Questions

Documents

Is Reliability Validity