winnie mucherahball state university

FOUNDATIONS OF QUALITY RESEARCH DESIGN: RELIABILITY & VALIDITY

Literature review

• Systematic identification, location, and analysis of documents containing information related to the research problem

• Reviews are used to guide practice and/or to guide research

• Narrative reviews

• Topic reviews

• Theoretical reviews

• Meta-analyses

(Mills, Airasian, & Gay, 2012)

Types of reviews• Narrative/Traditional Reviews

• Most often conducted when writing dissertations and theses in the social sciences

• Also used in introductory paragraphs of a typical research article

• Provides a brief narrative about previous research on a subject to set the context for the current research topic

Topic reviews• Introductory and investigatory reviews

• Conducted when working on a topic for the first time

• Often includes introductory works, e.g., encyclopedia entries and textbooks

• Criteria for good topic reviews:

• Recency (based on up-to-date sources)

• Importance (built on important sources, quality of the journal, impact factor)

• Breath (sources discuss topic broadly)

Theoretical reviews• Not usually featured in lists of types

of reviews, but are important subtypes

• It’s a version of a traditional/narrative review

• It’s specific purpose is to synthesize established theories by focusing on points of agreement and/or to generate new theories by focusing on gaps

• To either synthesize previous theories or to generate new ones.

Meta-analyses reviews

• Systematic reviews/ Research synthesis

• Systematic- used frequently to refer to evidence-based practical applications

• Research synthesis-often refers to research that is not necessarily tied to practical applications

• Similar: researcher states in advance the procedures for findings, selecting, coding and analyzing the data

• Data enables you to calculate effect size

Effect size• Effect size is aptly named

• It’s a measure of the size of an effect.

• Specifically, it’s a standardized measure

• Standardized measures are often stated in standard deviation units

• Therefore, they can be used to compare and combine results across studies

• Comparing and combining results across studies is the whole point of meta-analysis.

quantitative v. qualitative

• Quantitative research

• Numerical data

• Ex - surveys and tests

• Research plan includes an introduction, method section, data analysis description, and results

• Qualitative research

• Comprehensive, narrative, and visual data

• Ex - interviews and naturalistic observations

• Research plan must be responsive to the context and setting under study

• Mixed-method design is ideal (Mills, Airasian, & Gay, 2012)

correlational v. Experimental

• Correlational research

• Collecting data to determine whether a relation exists between two or more quantifiable variables

• Measured by a correlation coefficient (r)

• Strength of relationship ranges from 0 to 1

• Relationship can be positive or negative (inverse)

• Correlation is not causation

Experimental research

•Random assignment to groups

•Involves IV and DV

•At least one independent variable is manipulated

•Effect of one or more dependent variable(s) observed

•Quantitative measure of the degree of correspondence between two or more respondents

Reliability• It’s the consistency or agreement among

measures

• Consistency of data collection

• Results are more likely to be repeatable if you conduct the experiment all over again (because the sample size is large enough to produce the necessary precision)

• Reliability coefficients generally range from 1.0 for a perfectly reliable measure to 0 for one that is completely inconsistent from one rater/test/observation to the next

• Cronbach’s alpha (α)-estimates internal consistency

(Rumsey, 2005)

Measure of reliability• Cronbach’s alpha (α)

• It’s used when you want to know whether the items in your scale or index are measuring aspects of the same thing

• The “scale if item deleted” feature helps identify items that could be removed or analyzed individually (IRT)

• .70 is usually considered the minimum acceptable level; higher levels are needed when results are used for high-stakes decisions

Types of reliability• Inter-rater reliability-refers to the

consistency of two or more raters

• Test-retest reliability-refers to the consistency of the same test over time or consistency of results on repeated tests

• Internal reliability- refers to the consistency of multiple questions probing aspects of the same concept

Validity• It’s a central issue at all stages of a research

project

• Chief concern is whether the study is set up so that you can reach justifiable conclusions about your topic. This is referred to as Internal Validity

• It addresses the question: Do my conclusions apply to my sample?

• The degree to which differences on a measure are attributable to the manipulation of the independent variable

• This is highest in true experimental studies(Mills, Airasian, & Gay, 2012)

External validity

• The degree to which results will be generalizable and to a certain extent replicable in other settings

• It addresses the question: Do my conclusions apply to anyone else?

• Can you generalize your conclusions beyond the participants in the experiment?

• The answer depends on the quality and the appropriateness of your sample

• Construct validity: are concepts measured in ways that enable us to study what we aim to study?

• Content validity: is the measure thorough or representative of the thing being measured?

Sampling procedures• Population

• collection of all individuals of interests • Sample

• subset of the population we measure • Parameter

• a numeric characterization of the population that is of interest to us

• Statistic• a numeric characterization of the sample

that is an estimate of the population• Since we cannot access population, we don’t

have access to parameter, so we take a sample we can obtain, then we make a numeric measurement, also known as a statistic Coladarci & Cobb,

2014

Contextualizing your research

• Refining the substantive question and developing a plan for collecting relevant data

• Use of existing/new measures: Use Factor Analysis

• FA helps you decide about reliability and validity of your measurements of latent variables and thus how to analyze and interpret them

• FA is simply correlations and associations among items

• Purpose of FA is to improve the measurement of latent variables or constructs that cannot be directly observed

(Coladarci & Cobb, 2014)

Latent variables• Latent variables can only be studied

indirectly by using indicators of observed variables, e.g., in a multi-item measure of traits, the items would be indicators (or observed variables) and clusters of questions identified by the FA would help you identify the factors or latent variables, which are the constructs or concepts you seek in your research. E.g., 15 questions toward a controversial issue

• Efficacy or social tolerance or attitudes

Types of Factor analysis• Exploratory FA and Confirmatory FA

• EFA-used when researchers are looking for interesting patterns among variables

• CFA-used when researchers have theories about the patterns they want to test

• The two are often linked because it is very common to conduct them in sequence-first EFA to refine theories, then CFA to test them.

Conclusion

• Substantive Question ---> Statistical Question ---> Statistical Conclusion ---> Substantive Conclusion

• Substantive Conclusion is a context-based conclusion

references

• Coladarci, T., Cobb, C.D., Minium, E.W. & Clarke, R.C. Fundamentals of Statistical Reasoning in Education.

• Mills, G.E., Airasian, P. & Gay, L.R. 2012. Educational Research: Competencies for analysis and applications. 10th Edition.

• Rumsey, D. 2005. Statistics Workbook for Dummies.