ReliabilityReliabilityPresentationPresentation

Test-RetestTest-Retest

James Blackwood – AED 615James Blackwood – AED 615Fall Semester 2006Fall Semester 2006

Test-Retest ReliabilityTest-Retest Reliability Test-Retest method of determining Test-Retest method of determining

reliability is accomplished by administering reliability is accomplished by administering a test to a group a test to a group

After a period of time has passed, the same After a period of time has passed, the same test is re-administered to the same group.test is re-administered to the same group.

It is also known as stability reliabilityIt is also known as stability reliability It is used in both qualitative and It is used in both qualitative and

quantitative research (qualitative requires a quantitative research (qualitative requires a different technique of analysis)different technique of analysis)

Reliability CoefficientReliability Coefficient

1.1. After two tests have been administered, a reliability After two tests have been administered, a reliability coefficient is calculated to determine the coefficient is calculated to determine the relationship between the two scores obtained. relationship between the two scores obtained.

2.2. If the same results are obtained from the two tests If the same results are obtained from the two tests then the coefficient = 1.0then the coefficient = 1.0

3.3. The coefficient is influenced by the amount of time The coefficient is influenced by the amount of time that has passed between the administration of the that has passed between the administration of the two tests.two tests.

Reliability Coefficient MeasurementReliability Coefficient Measurement

1.1. The reliability coefficient is expected to be The reliability coefficient is expected to be lower the longer the time interval between lower the longer the time interval between the tests due to the possibility of changes in the tests due to the possibility of changes in the population taking the test.the population taking the test.

2.2. The shorter the time gap, the higher the The shorter the time gap, the higher the correlation; the longer the time gap, the correlation; the longer the time gap, the lower the correlation. lower the correlation.

Test-Retest IssuesTest-Retest Issues There cannot be any measurable change in There cannot be any measurable change in

the construct being measured between the the construct being measured between the two tests.two tests.

This method will not work when measuring a This method will not work when measuring a variable that is not stable in an individual.variable that is not stable in an individual.

Unless the instrument is reliable, Unless the instrument is reliable, relationships with other variables in the study relationships with other variables in the study will not be identified.will not be identified.

Testing..Testing.. You can obtain considerably different You can obtain considerably different

estimates of reliability depending on the estimates of reliability depending on the interval between tests.interval between tests.

For educational research, examination of For educational research, examination of scores over a two to three month period is scores over a two to three month period is sufficient for test-retest reliability verification.sufficient for test-retest reliability verification.

The time interval between the two tests should The time interval between the two tests should always be reported when using test-retest as a always be reported when using test-retest as a measure of reliability. measure of reliability.

Test-Retest EquationTest-Retest Equation

Test-Retest IssuesTest-Retest Issues

1.1. Requires twice the data collectionRequires twice the data collection2.2. Population for the test would need to be Population for the test would need to be

willing to repeat the test (impractical)willing to repeat the test (impractical)3.3. Higher costs due to multiple tests being Higher costs due to multiple tests being

administeredadministered4.4. Only works well when practical (better for Only works well when practical (better for

smaller population rather than a large one)smaller population rather than a large one)5.5. Researcher may not be able to retest the Researcher may not be able to retest the

populationpopulation

Examples of Test-RetestExamples of Test-Retest Educational assessmentEducational assessment Drug testingDrug testing Testing Measurement EquipmentTesting Measurement Equipment Medical EvaluationsMedical Evaluations

Research Literature ExampleResearch Literature ExampleAmerican Journal of Agricultural EconomicsAmerican Journal of Agricultural Economics

Volume 71 - Number 1 (Feb 1989), pp 76-84Volume 71 - Number 1 (Feb 1989), pp 76-84 Test-Retest Reliability of the Contingent Test-Retest Reliability of the Contingent

Valuation Method: A Comparison of Valuation Method: A Comparison of General Population and Visitor ResponsesGeneral Population and Visitor Responses

John B. LoomisJohn B. LoomisAbstract: Abstract:

The reliability of the contingent valuation method is evaluated by resurveying the same general households and The reliability of the contingent valuation method is evaluated by resurveying the same general households and visitors nine months after their original survey. Test-retest correlations on willingness to pay are statistically visitors nine months after their original survey. Test-retest correlations on willingness to pay are statistically significant and ranged from .422 for the general population sample to .782 for the visitor sample. Using a significant and ranged from .422 for the general population sample to .782 for the visitor sample. Using a paired T-test, there was no statistical difference between an individual's first and second reported willingness paired T-test, there was no statistical difference between an individual's first and second reported willingness to pay. Chow tests comparing the original and resurvey willingness-to-pay functions showed no statistical to pay. Chow tests comparing the original and resurvey willingness-to-pay functions showed no statistical difference at the .01 level. Reported willingness to pay is reasonably stable over the time period surveyed. difference at the .01 level. Reported willingness to pay is reasonably stable over the time period surveyed.

ReferencesReferencesFraenkel, J. R., & Wallen, N. E. (2006). Fraenkel, J. R., & Wallen, N. E. (2006). How to design and evaluate research in How to design and evaluate research in

education education (6th ed.). New York: Mc-Graw-Hill.(6th ed.). New York: Mc-Graw-Hill.

Guttman, L. (1946). The test-retest reliability of qualitative data. Guttman, L. (1946). The test-retest reliability of qualitative data. Psychometrika, 11Psychometrika, 11(2), (2),

81-95. Abstract retrieved October 15, 2006 from Springer Link database.81-95. Abstract retrieved October 15, 2006 from Springer Link database.

Loomis, J. B. (1989) Test-retest reliability of the contingent valuation method: A Loomis, J. B. (1989) Test-retest reliability of the contingent valuation method: A

comparison of general population and visitor responses. comparison of general population and visitor responses. American Journal of Agricultural American Journal of Agricultural

Economics, 71Economics, 71(1), 78-84.(1), 78-84.

Trochim, W. M., (2006). Types of reliability. Trochim, W. M., (2006). Types of reliability. Research Methods Knowledge Base. Research Methods Knowledge Base. Retrieved from Retrieved from

http://www.socialresearchmethods.net/kb/reltypes.htm http://www.socialresearchmethods.net/kb/reltypes.htm