1 Reliability and Validity. 2 Not Reliable, Not Valid

1

Reliability and ValidityReliability and Validity

2Not Reliable, Not ValidNot Reliable, Not Valid

3Reliable, Not ValidReliable, Not Valid

4Reliable and ValidReliable and Valid

5

Relationship of Reliability and ValidityRelationship of Reliability and Validity

A measure can be reliable without A measure can be reliable without being valid.being valid.

A measure must be reliable to be valid.A measure must be reliable to be valid.

6 •ValidityValidity

• Types of ValidityTypes of Validity

• ContentContent

• ConstructConstruct

• Criterion-RelatedCriterion-Related


• Content ValidityContent Validity

• To what extent do the items on the test To what extent do the items on the test adequately sample from the intended adequately sample from the intended universe of content?universe of content?

• Judgmental EvidenceJudgmental Evidence• Literature ReviewLiterature Review

• Expert ReviewExpert Review


• Construct ValidityConstruct Validity• To what extent do certain explanatory concepts To what extent do certain explanatory concepts

explain:explain:• covariation in the responses to the items, orcovariation in the responses to the items, or• relationships with other know indicators?relationships with other know indicators?

• Empirical evidenceEmpirical evidence• CorrelationsCorrelations• Factor analysisFactor analysis• Structural equation modelingStructural equation modeling• Item response theoryItem response theory• Known groupsKnown groups


• Criterion-Related ValidityCriterion-Related Validity

• What is the relationship between scores on the What is the relationship between scores on the instrument and some external criterion that instrument and some external criterion that provides a more direct measure of the targeted provides a more direct measure of the targeted characteristic?characteristic?

• TypesTypes• ConcurrentConcurrent

• PredictivePredictive

10 •ReliabilityReliability

•““Does the instrument Does the instrument provide us with accurate provide us with accurate

assessments?”assessments?”


•TheoryTheory

•rr1122

TrueTrue = = ReliabilityReliability

• “ “Proportion of variance in Proportion of variance in scores that can be considered scores that can be considered

‘true’ variance”‘true’ variance”


• Sources of ErrorSources of Error

• Inadequate sampling of itemsInadequate sampling of items• Cronbach’s alpha internal Cronbach’s alpha internal

consistency reliabilityconsistency reliability

• Different occasionsDifferent occasions• Stability reliabilityStability reliability


• Types of ReliabilityTypes of Reliability

• Internal consistencyInternal consistency

• StabilityStability


•Generalizability TheoryGeneralizability Theory

•““Simultaneously examines Simultaneously examines homogeneity of the items and homogeneity of the items and

stability of responses over time”stability of responses over time”

15 •Qualitative ResearchQualitative Research

•Procedures which produce Procedures which produce descriptive data -- peoples’ descriptive data -- peoples’ written or spoken word, or written or spoken word, or observations of behaviorobservations of behavior


• Ethnography - Naturalistic InquiryEthnography - Naturalistic Inquiry

• Study behavior in a natural settingStudy behavior in a natural setting

• Involvement with subjectsInvolvement with subjects

• See the world through their eyesSee the world through their eyes

17 •Research MethodsResearch Methods

• Selecting a Research Selecting a Research StrategyStrategy

• QualitativeQualitative

• QuantitativeQuantitative

18 •Research MethodsResearch Methods

•Philosophical BasePhilosophical Base Logical PositivismLogical Positivism PhenomenologyPhenomenology

•PurposePurpose VerificationVerification DiscoveryDiscovery

•Framework/DesignFramework/Design Preordinate/FixedPreordinate/Fixed Emergent/FlexibleEmergent/Flexible

•ConditionsConditions ControlledControlled Invited InterferenceInvited Interference

•TreatmentTreatment StableStable VariableVariable

Quantitative Quantitative QualitativeQualitative

19 •Research TargetResearch Target

• ProcessProcess ProductProduct

• ImplementationImplementationOutcomeOutcome

• ““Implementation as an Implementation as an independent variable”independent variable”


• Front StageFront Stage• Packaged RealityPackaged Reality

• Back StageBack Stage• RealityReality

• GatekeepersGatekeepers

““Schools are not researchers’ sandbox”Schools are not researchers’ sandbox”

Access and EntryAccess and Entry


• InterviewInterview• What is What is saidsaid

• Document AnalysisDocument Analysis• What is What is writtenwritten

• Participant ObservationParticipant Observation• What is What is donedone

Data CollectionData Collection


• Combination of Methodologies Combination of Methodologies • Multiple Reference PointsMultiple Reference Points

• Increase AccuracyIncrease Accuracy• Increase ConfidenceIncrease Confidence

• Convergence of EvidenceConvergence of Evidence• Among MethodsAmong Methods• Within MethodsWithin Methods

TriangulationTriangulation


• OngoingOngoing At conclusionAt conclusion

• InductiveInductive DeductiveDeductive• Generating TheoryGenerating Theory Testing TheoryTesting Theory

QualitativeQualitative QuantitativeQuantitative

--

Data AnalysisData Analysis


• Reporting ResultsReporting Results• Natural HistoryNatural History

• Thematic AnalysisThematic Analysis

• Social NetworksSocial Networks

• Life HistoryLife History

• Allocation and Distribution of ResourcesAllocation and Distribution of Resources

• Organizational AnalysisOrganizational Analysis

• Critical IncidenceCritical Incidence

25 •Research DesignResearch Design

• Development of EducationDevelopment of EducationResearch MethodsResearch Methods

• Genetics Genetics • Correlation (Galton)Correlation (Galton)

• AgricultureAgriculture• Controlled experiments (Fisher - ANOVA)Controlled experiments (Fisher - ANOVA)


• Complexity of Educational Complexity of Educational ProcessesProcesses

• Experimental research: failure of the Experimental research: failure of the

promisepromise

• Trend toward qualitative methodsTrend toward qualitative methods


• Experimental and Quasi-Experimental Designs for Experimental and Quasi-Experimental Designs for Research Research (1963)(1963)

• Donald CampbellDonald Campbell• Julian StanleyJulian Stanley

• Shift focus from statistics to problems of acquiring Shift focus from statistics to problems of acquiring data in educationdata in education

• Develop benchmarks for research designsDevelop benchmarks for research designs

• Compare research designs with respect to benchmarksCompare research designs with respect to benchmarks


• Research Design FocusResearch Design Focus

• 1960’s - 1970’s: Experimental1960’s - 1970’s: Experimental

• The External Validity of ExperimentsThe External Validity of ExperimentsBracht & Glass, 1968Bracht & Glass, 1968

• 1980 +: Quasi-Experimental1980 +: Quasi-Experimental

• Quasi-Experimentation: Design and Analysis Issues for Quasi-Experimentation: Design and Analysis Issues for Field SettingsField SettingsCook & Campbell, 1979Cook & Campbell, 1979


•Internal ValidityInternal Validity

•To what extent have extraneous variables been To what extent have extraneous variables been controlled so that the observed effect (dependent) controlled so that the observed effect (dependent) can be attributed to the treatment variable can be attributed to the treatment variable (independent)?(independent)?

•Is it reasonably plausible that the outcome Is it reasonably plausible that the outcome (dependent) is created by the treatment (dependent) is created by the treatment (independent)?(independent)?


• Threats to External ValidityThreats to External Validity

• Results generalizable to other:Results generalizable to other:

• People?People?• Settings?Settings?• Independent variables?Independent variables?• Dependent variables?Dependent variables?


• Internal Validity ThreatsInternal Validity Threats

• TestingTesting

• InstrumentationInstrumentation

• Statistical regressionStatistical regression

• Differential selectionDifferential selection

• Experimental treatment diffusionExperimental treatment diffusion


• Internal Validity ThreatsInternal Validity Threats

• Compensatory rivalry by the control Compensatory rivalry by the control groupgroup

• Compensatory equalization of Compensatory equalization of treatmentstreatments

• Resentful demoralization of the control Resentful demoralization of the control groupgroup

33

One Group Pre-PostOne Group Pre-Post

X

•Research DesignResearch Design

34

Pre-Post Comparison GroupPre-Post Comparison Group

X

•Research DesignResearch Design

35

Pre-Post Comparison GroupPre-Post Comparison Groupwith Random Assignmentwith Random Assignment

XRR

Research DesignResearch Design

36

Post-Only Comparison GroupPost-Only Comparison Groupwith Random Assignmentwith Random Assignment

XRR

Research DesignResearch Design

37 •External Validity of ResearchExternal Validity of Research

• ““Generalizability”Generalizability”

• Population ValidityPopulation Validity• Generalize from sample to population?Generalize from sample to population?

• Personalogical variables interact with Personalogical variables interact with treatment?treatment?

38 •External Validity of ResearchExternal Validity of Research

• ““Generalizability”Generalizability”

• Ecological ValidityEcological Validity• Explicit description of treatmentExplicit description of treatment• Multiple treatment interferenceMultiple treatment interference• Hawthorne effectHawthorne effect• Novelty and disruption effectsNovelty and disruption effects• Experimenter effectExperimenter effect• Pretest sensitizationPretest sensitization• Measurement of dependent variableMeasurement of dependent variable

39 •Statistical AnalysisStatistical Analysis

Experimental

Xchange

ComparisonChange Scores

Post Pre = Change Post Pre = Change

Xchange


Repeated Measures ANOVARepeated Measures ANOVARMANOVARMANOVA

ExperimentalExperimental

ComparisonComparison

PrePre PostPost


RMANOVARMANOVASources of VarianceSources of Variance

BetweenBetweenGroupGroup FFgroupgroup

Error (b)Error (b)

WithinWithinTimeTime FFtimetime

Time x GroupTime x Group FFtime x grouptime x group

Error (w)Error (w)

TotalTotal


Research Question

Will there be a difference between groups Will there be a difference between groups with respect to posttest scores after with respect to posttest scores after controlling for initial differences in controlling for initial differences in pretest scores?pretest scores?

Analysis of CovarianceAnalysis of CovarianceANCOVAANCOVA

43

Pre CovariatePre Covariate

PostPostDependentDependent

ExpExp

ControlControl

43

44

Pre CovariatePre CovariatePÕST = b(PRE) + aPÕST = b(PRE) + a


ExpExp

ControlControl

44

45



ExpExp

ControlControl

XXContCont XXExpExp

YYCC

YYEE

MarginalMarginalDistributionDistribution

45

46



ExpExp

ControlControl

XXContCont XXExpExp

YYCC

YYEE

MarginalMarginalDistributionDistribution

XX

46

47 •Effect SizeEffect Size

““The difference between two means expressed The difference between two means expressed in standard deviation units”in standard deviation units”

ES =ES = Mean 1 - Mean 2Mean 1 - Mean 2Standard DeviationStandard Deviation

ESES Qualitative DifferenceQualitative Difference

.20.20 SmallSmall

.50.50 MediumMedium

.80.80 LargeLarge

48 •Statistical Significance: Statistical Significance: Sample Size and Effect SizeSample Size and Effect Size

““I have a meaningful say in designing my job/work.”*I have a meaningful say in designing my job/work.”*

StandardStandard Sign. Sign. EffectEffect Group Group NN Mean Mean DeviationDeviation LevelLevel SizeSize

.000.000 .22.22MalesMales 10,74710,747 3.003.00 1.201.20

FemalesFemales 8,0888,088 3.263.26 1.191.19

MalesMales 111111 3.033.03 1.171.17

FemalesFemales 8181 3.283.28 1.221.22.157.157 .21.21

* Responded to on a 5-point Likert agreement scale.* Responded to on a 5-point Likert agreement scale.

Documents

1 Reliability and Validity. 2 Not Reliable, Not Valid