View
215
Download
1
Category
Preview:
Citation preview
4
Reliability
Implies that the same data would have been collected each time over repeated tests/ observations.
Would a particular technique (or survey question) yield the same result each time? “Did you go to church last week?” vs. “How
many times have you been to church in your life?”
Reliability does not ensure accuracy. Taken from Babbie, E.
5
Reliability
Problem if interpret questions differently
Poorly worded questions Inconsistent coding: coding errors as
with open-ended questions Lack of definition of key terms
6
Reliability
Poorly worded: Does the library have adequate facilities and equipment for physically disabled students
Better: Can patrons in wheelchairs retrieve books from the browsing collection?
7
Reliability (indicators)
Pretest Repeat question(s) Test/retest Split half and Parallel Interscore or scorer
8
Validity
A term to describe a measure that accurately reflects the concept it is intended to measure.
Which is a more “valid” indicator of intelligence- an IQ score, or number of hours spent studying?
Ultimate validity cannot be proven, but can be supported by face, internal, and external measures.
Babbie, E.
9
Types of Validity
Face validity: The quality of an indicator/ question/ test that makes it a reasonable measure of a variable. Church attendance is an indication of
religiosity. Number of grievances filed is an indicator
of worker morale
10
Internal Validity
Approximate truth about inferences regarding causal relationships
Typically applied to studies using inferential statistics (i.e. quantitative measures) than descriptive or observation studies.
Especially useful for studies assessing affects of programs
Only applicable to the study in question- not generalizable. Why not?
Key question: Whether observed changes can be attributed to your program (the cause) and NOT other possible impacts/ causes.
12
Internal Validity
History or specific eventsHistory or specific events: raises the issue that some variable other than the independent variable accounted for the change in the dependent variable. E.G.: the length of time between conducting the pretest and posttest may have a detrimental effect.
MaturationMaturation: the change results from biological or psychological processes, which occurred over time, and not from the treatment itself. Maturation becomes more a concern the longer the period between the pretest and posttest
PretestingPretesting: may affect the dependent variable. Pretesting may alert participants or educate them about the topic under investigation. Therefore if subjects are administered a posttest, their performance may reflect a marked improvement
Measuring instruments or observational techniquesMeasuring instruments or observational techniques: These—not the treatment—may account for the change in the dependent variable. Further, the validity of study findings may have been influenced by the fact that the evaluators as observers, raters, graders, interviewers, and coders gained experience, became tired, obtained a more complete understanding of the project, or eased their expectations of test subjects
13
Internal Validity (continued)
A nonrandom assignment of subjectsnonrandom assignment of subjects to groups may signify that the groups were dissimilar from the beginning. Therefore any change might be attributed to the differential selection of subjects, rather than the actual treatment.
Statistical regressionStatistical regression refers to the tendency for extreme scores to regress or move toward the common mean of subsequent measures. The assignment of subjects to a particular test group on the basis of extreme views may affect study findings.
14
Internal Validity (continued)
MortalityMortality refers to the possibility that some subjects may have dropped out of the study after completion of the pretest but before the administration of the posttest. In such instances, every effort should be made to identify any common patterns or characteristics to ensure that any difference between a group’s pretest and posttest scores cannot be attributed to the loss of subjects.
InteractionInteraction refers to the fact that more than one of the previous threats might be in play. This is especially likely in those cases where subjects were not randomly assigned to groups and the evaluation was based on existing, intact groups.
15
External Validity
The approximate truth of generalizations drawn from a study.
The degree to which conclusions drawn from your study sample would hold true to other persons in other places at other times
Trochim, W.
17
External Validity
Example: institutions of higher education in Massachusetts: control, highest degree offered, and some characteristics of library (staff number, budget, and volume number)
Return rate? Do respondents differ from non-
respondents as a group?
18
Validity
Content validity (for achievement test): How well does the test sample what the students learned? How well does a standardized test cover what was taught in the information literacy program?
19
Validity (continued)
Criterion-related (predictive) (attitude test to predict performance in a library skills program): Who well does the test predict achievement for college freshmen?
Criterion-related (diagnostic): How well does the test diagnose current problems with library use?
20
Validity (continued)
Construct validity: How well does the test measure comprehension of library use? Does a test on the use of an OPAC really measure effective and efficient use rather than one’s ability to read test items?
22
Qualitative Reliability
Researcher is the “instrument”- how to test for reliability? Provide details of method, and abundance of evidence Provide evidence of qualifications as observer Make assumptions (and possible biases) clear State research questions clearly Use early stages of study to generate focus Observe for an adequate period of time, across a full
range of activities Collect data from multiple sources Save data for reanalysis
23
Qualitative Validity
Depends upon reliablity. Like reliability, asserted by documenting steps Triangulation- data from different sources/ methods Full documentation of data- “chain of evidence” Logical connections between data and conclusions Conscious and deliberate inclusion of data that might not
support thesis Preparedness to entertain alternatives Self-reflection, acknowledgement of own biases Review of preliminary reports by objective observers Awareness of limitations
Gorman and Clayton
24
Qualitative Study: Increasing Reliability and Validty
Inquiry affected by
Results in Account for by:
To lead to For findings that are:
During After
Factor patternings
Non-interpretability
Prolonged engagement
Persistent observation
Peer Debriefing
Triangulation
Member checks
Establish structural corroboration (coherence)
Credibility Plausible
Situational Uniqueness
Non-comparability
Collect thick descriptive data
Do theoretical/ positive sampling
Develop thick description
Transferability Context relevant
-Gorman and Clayton
25
Qualitative Study: Increasing Reliability and Validity cont’d
Inquiry affected by
Results in Account for by:
To lead to For findings that are:
During After
Instrumental changes
Instability Use overlap methods
Use stepwise replication
Leave audit trail
Do dependability audit (process)
Dependability Stable
Investigator Predilections
Bias Do triangulation
Practice reflexivity (audit trail)
Do confirmability audit (product)
Confirmability Investigator-free
-Gorman and Clayton
26
Example
For a sweeping study
When conduct it?
For how long?
How deal with reliability and validity?
Course evaluation
27
References
Babbie, E. (2005). The basics of social research. Belmont, CA: Wadsworth Publishing.
Gorman, G.E. & Clayton, P. (2005). Qualitative research for the information professional: A practical handbook. London: Facet Publishing.
Trochim, W. M. K. (2006). Research methods knowledge base. Retrieved July 8, 2008 from http://www.socialresearchmethods.net/kb/intval.php
http://www.socialresearchmethods.net/kb/external.php
Recommended