CDIS 5400Dr Brenda Louw 2010

Validity Issues in Research Design

Objectives

Demonstrate : an understanding of the importance of

internal and external validity issues in experimental research

use this knowledge to critically evaluate research articles for writing a literature review and for planning EBP

Overview

Introduction Terminology Internal validity Credibility External Validity Transferability and dependability

Readings

Schiavetti, et al.,2011, Chapter 4 Pp 135-150

Introduction

Validity : accurate representation; soundness and strength

Major consideration: have factors that can have an effect on the validity of the data been accounted for and controlled ?

Therefore research designs should address possible threats to validity so that the results can be trusted

Evaluation of research design

Internal validity: concerns the degree to which the design meets two purposes, i.e., providing answers to research questions and controlling variability.

External validity: concerns the degree to which generalizations can be made outside of the scope of the study.

Terminology issues

Internal validity External validity

Credibility Transferability,

dependability

Quantitative research Qualitative research

Control of variability

Researcher and critical reader: Is change in DV caused by the

experimental treatment and not by factors that can mimic the effects of the treatment

Internal validity

The ability of a study to rule out competing explanations for the results other than the effect of the IV on the DV by using experimental control to compensate for confounding variables in a study the (Haynes & Johnson ,2009)

Attributes of studies with High and Low degrees of Internal validity Low internal validity

Weak design that does not fully control for extraneous variables

Vague definitions of variables

Results accounted for by alternative explanations other than the IV

Unsystematic

High internal validity Strong design with

strong control of extraneous variables

Strong definitions of variables

Results explained by effects of IV on the DV with alternative explanations limited by the design

Systematic, attention to detail

Threats to internal validity

History: has some event occurred between 1st and 2nd measurements in addition to the DV

Maturation: changes in Subjects themselves that cannot be controlled by the researcher

Reactive pretest: practice effects –exposure to any type of experience in the early part of the study may influence behavior later in the experiment any type of experience in the early part of the study may influence behavior later in the experiment

Continued…

Instrumentation: Study is only as good as the DV Types of instrumentation:

Electronic devices Rating scales, questionnaires , standardized

tests Sources of error

Quality of the instrument, condition , calibration, interpretive skills

Continued…

Statistical regression: Ss selected on basis of atypically low or high

scores change on a subsequent test so that their scores are somewhat better/poorer than originally

Change not due to treatment but due to scores regressing to more typical mean score

Differential subject selection: Selection of experimental and groups important Differences in Ss in 2 groups may account for

treatment effects rather than treatment itself

Continued…

Differential Ss selection Attrition

Withdrawal of participants IC allows for withdrawing Problematic in follow-up studies Common in survey research : 50%

adequate for analysis and reporting

Credibility –Qualitative research Qualitative designs assure internal validity by

establishing credibility Researcher justified in describing impact of one

variable on another or conclusions that relationship is causal

Threats: Researcher bias :researcher instrument of data

collection employ reflexivity

Researcher reactivity: preconceived notions affect data interpretation Use cross comparisons

External validity

The generalizability of research results from the laboratory to the real world (Haynes & Johnson, 2009)

External validity

Can results from a sample of individuals be applied or generalized to the entire population from which the sample was selected

Generalization grows with cumulative research on a given topic

Limit until evidence indicates validity beyond confines of an individuals study e.g. systematic replication studies

Threats to external validity

Subject selection Are Ss representative of the population to

which researcher wants to generalize? Especially important in between subject

designs To counteract:

Develop specific selection criteria Use random assignment Use intentional matching of Ss Include summary of Ss characteristics

Transferability- Qualitative research Used in qualitative research to describe

to which extent findings are externally valid

Dependability : consistency with which the same finding can be observed in similar circumstances

Reliability

Extent to which a measuring instrument or procedure yields consistent or repeatable results

4 ways of establishing reliability : Inter observer reliability Test-retest reliability Parallel forms reliability Split-half reliability

Inter observer reliability

2 different examiners/observers use the procedure to test the same persons

E.g. Group of participants tested 1x by each SLP on PPVT-3

Obtain 2 scores for each participant Compute correlation coefficient to determine

degree of reliability Strong correlation :different examiners

obtained similar results e.g. high score for 1st examiner and high score when tested by 2nd examiner

Test-retest reliability

Participants tested 2 different times but by same examiner

Strong test-retest reliability : consistent results from 1x to the next when administered by the same person

May compute correlation coefficient for each participant

Important when studying treatment outcomes-change due to treatment and not measurement instability

Parallel forms reliability

Construct/select 2 different but equivalent forms of a measure, comparable test items

Conduct 1st test on all participants, then alter 2nd test on all participants

Each participant has 2 scores-determine correlation coefficient to determine the degree of reliability between the 2 measures

E.g.

EI study : developed Surveillance tool for Communication Development (STCD) ( Decker, Louw, Kritzinger, 2009) for 0-6 month old infant population

Tested 55 infants in a rural hospital in South Africa 0- 3 mos and 3-6 mos screening intervals

STCD Rossetti Infant-Toddler Language Scale ( Rossetti,

1990) Results:

High sensitivity score for both intervals 97.22 % and 96.25%

Use of STCD validate for use in rural hospital

Split-half reliabilityl

Administer test 1x to participants Then divide items into 2 equivalent

forms and compare participants scores on each form

Problematic and depends on how test is constructed

Conclusion

Reliability is a matter of degree Correlation coefficient provides

quantitative information re the level of reliability

Level of reliability depends on how scores are to be used

High scores needed for making decisions re intervention

Lower scores needed when mean of group scores are used

Conclusion

Both degree of validity and reliability reflect the degree to which researcher may have error in measurements

Validity errors reflect bias in the instrument itself

Reliability errors reflect use of the instrument