Improving the Quality€¦ · Web view2021. 6. 25. · Quantitative and qualitative methods refer to the type of data being collected (quantitative data involve numeric scores,

EDEF 605:Educational Research

Course Notes by ObjectivesVersion 2.0

ContentsExam 1.................................................................................................................................... 2

1. Science & Scientific Research..........................................................................................2Exam 2..................................................................................................................................24

2. Positivist Research Designs............................................................................................242.1 Research Design Overview.......................................................................................242.2 Experimental and Quasi-Experimental Research and Evaluation Designs.................35

Exam 3..................................................................................................................................473. Interpretive Research Designs.......................................................................................47

Exam 4..................................................................................................................................564. Sampling, Construct Validity & Reliability, Data Analysis & Statistics............................56

4.1 Sampling...................................................................................................................564.2 Construct Validity & Reliability of Measurement Procedures.....................................614.3 Data Analysis & Statistics.........................................................................................66

5. Program Evaluation Models...............................................................................................756. Ethical Considerations.......................................................................................................76

Notes: Objectives in italicized font will not be measured on any course exam. You might want to supplement the basic information presented with additional information and examples from the reading material (or other sources) that might help you better understand and perform the skills indicated in the outcomes.

EDEF605: Educational Research Course Notes Page 2

Exam 1

1. Science & Scientific Research

1.1 Explain why education represents a professional field within the social sciences.

Social Science Research Network (SSRN) defines social science as “…those disciplines that study (a) institutions and functioning of human society and the interpersonal relationships of individuals as members of society; (b) a particular phase or aspect of human society.” (Social Science Research Network, n.d.). Some of the more commonly-recognized fields within the social sciences include:

Accounting Anthropology & Archaeology Cognitive Science Communication Criminal Justice Economics Education Geography History Information Systems

Legal Scholarship Leadership Management Marketing Political Science Psychology Public Health Sociology Women's & Gender Studies

In many ways, education represents the application of psychology in the pursuit of influencing and affecting human behavior.

____________________

Social Science Research Network (n.d.). Social Science. Retrieved from https://www.ssrn.com/index.cfm/en/social-sciences/


1.2 Describe the goal of Education as a professional field within the larger scope of social sciences.

Of course there is no singular, commonly-accepted overall goal for education within the field of social science. Examining the mission statement of the most influential professional organization serving education can help narrow the focus of the profession a bit.

The National Education Associate includes the following core value as part of its overall mission:

“We believe public education is the gateway to opportunity. All students have the human and civil right to a quality public education that develops their potential, independence, and character.” (National Education Association, n.d.)

I argue for a more simplified goal. As stated in 1.1, one way to consider education as a social science is the application of psychology in the pursuit of influencing and affecting human behavior. For example, facilitating learning is a fundamental goal of education. Learning can be defined as the development of new knowledge, skills, or attitudes resulting from an individual’s "external" interaction with her/his environment and/or "internal" interaction between new and previously-existing information or knowledge structures. From a practical standpoint, learning can only be inferred by observing a persistent and permanent change in a person’s behavior.

A simplified way of summarizing this perspective is say that one of the primary goals of education is to facilitate changes in human behavior.

_____________________

National Education Association (n.d.). NEA's Vision, Mission, and Values. Retrieved from: http://www.nea.org/home/19583.htm


1.3 Define theory and differentiate a theory from a model.

[Bhattacherjee Chapters 2, 4]

Theory

A theory is a set of systematically interrelated constructs and propositions intended to explain and predict a phenomenon or behavior of interest, within certain boundary conditions and assumptions. Essentially, a theory is a systemic collection of related propositions. While propositions generally connect two or three constructs, theories represent a system of multiple constructs and propositions. Hence, theories can be substantially more complex and abstract and of a larger scope than propositions or hypotheses.

Theories should explain why things happen, rather than just describe or predict. Note that it is possible to predict events or behaviors using a set of predictors, without necessarily explaining why such events are taking place.

Example:

A well-known theory in education and psychology is constructivism. This theory describes learning as an active process in which learners construct new ideas or concepts based upon their current/past knowledge. Many propositions define constructivism, including (but not limited to):

Learning is the acquisition of new knowledge, skills or attitudes resulting from a learner’s interaction with her/his external and internal (cognitive) environments.

Active learning strategies will facilitate learning more effectively that passive learning strategies. Active learning involves participating in the process of learning by interacting with those elements in the learning environment that communicate meaning (including teachers), and not just passively listening or watching.

Because learners construct their own meaning to the information they receive from external sources, more learners will be more successful in the classroom if their instruction involves individualized opportunities for construction.

Model

A model is a representation of all or part of a system that is constructed to study that system (e.g., how the system works or what triggers the system). While a theory tries to explain a phenomenon, a model tries to represent a phenomenon.

Models are often used by decision makers to make important decisions based on a given set of inputs.


To illustrate the relationship between theories and models, the following diagram is presented. It represents an overview of the relationship between the theories that inform various models within the field of instructional technology:


The model below is known as the “Information Processing Model”. It communicates the relationship between various memory systems and strategies used by humans to process external stimuli (information) received through the senses:

Explanatory


1.4 Identify and describe the attributes of a good theory.

[Bhattacherjee Chapter 4]

Logical consistency

Theoretical constructs, propositions, boundary conditions, and assumptions are logically consistent with each other.

If some of these “building blocks” of a theory are inconsistent with each other (e.g., a theory assumes rationality, but some constructs represent non-rational concepts), then the theory is poor.

Explanatory power

A good theory explains (or predicts) a target phenomenon better than competing theories.

Falsifiability

For theories to be valid, they must be falsifiable. Falsifiability ensures that the theory is potentially disprovable, if empirical data does not match with theoretical propositions, which allows for their empirical testing by researchers. In other words, theories cannot be theories unless they can be empirically testable.

However, note that saying that a theory is falsifiable is not the same as saying that a theory should be falsified. If a theory is indeed falsified based on empirical evidence, then it was probably a poor theory to begin with.

Parsimony

Parsimony addresses the number of variables needed to explain a phenomenon.

In the sciences, parsimony is reflected in the proposition “when presented with two competing ideas (theories, hypotheses, etc.), the simpler one is more likely to be correct.”

This is related to the principle of Occam’s Razor, proposed by William of Ockam in the 14th century:

When presented with competing theories or hypotheses that make the same predictions, one should select the solution with the fewest assumptions.


1.5 Describe how science represents a “way of knowing,” and differentiate between science and other ways of knowing.

The information below complements the information about “ways if knowing” included on pp. 4-7 of Fraenkel, Wallen & Hyun (2015).

Way of Knowing Example

Sensing

Anything we perceive with our senses (including seeing, hearing, touching, tasting, or smelling).

VILLAGER #1: We have found a witch…might we burn her?CROWD: Burn her! Burn!BEDEVERE: How do you know she is a witch?VILLAGER #2: She looks like one.

Sharing Information with Others

Storytelling, gossiping, Facebooking, word-of-mouth… these are all ways of sharing information with others.

BEDEVERE: What makes you think she is a witch?VILLAGER #3: Well, she turned me into a newt.BEDEVERE: A newt?VILLAGER #3: I got better.VILLAGER #1: A witch!CROWD: A witch! A witch! A witch! Witch! Witch!

Being Told WITCH: I'm not a witch. I'm not a witch.BEDEVERE: But you are dressed as one.


Something by an Expert

Opinions from experts, or people we THINK are experts, is a powerful way of knowing.

Logical Reasoning

Logical reasoning allows us to create a new kind of knowledge for ourselves. Logical reasoning is based on the application of logic to a series or chain of related assumptions, leading to new conclusions.

VILLAGER #1: If... she... weighs the same as a duck.. she's made of wood.BEDEVERE: And therefore?VILLAGER #1: A witch!CROWD: A witch! A witch! A witch!


Science

Science can mean many things, but a common element of science is the “scientific method”. This way of knowing is an empirical approach that involves a combination of sensing, logic, predictions, and testing.

VILLAGER #1: If... she... weighs the same as a duck.. she's made of wood.BEDEVERE: And therefore?VILLAGER #1: A witch!CROWD: A witch! A witch! A witch!BEDEVERE: We shall use my largest scales!


1.6 Compare and contrast rationalism and empiricism.


Empiricism is a theory stating that knowledge comes only or primarily from sensory experience. Rationalism is a theory in which the criterion of a truth is not sensory but intellectual and deductive (reasoning). Understanding this distinction helps to better understand what it means to "think scientifically".

According to the Stanford Encyclopedia of Philosophy (2017):

The dispute between rationalism and empiricism concerns the extent to which we are dependent upon sense experience in our effort to gain knowledge. Rationalists claim that there are significant ways in which our concepts and knowledge are gained independently of sense experience. Empiricists claim that sense experience is the ultimate source of all our concepts and knowledge.

For more information about this topic, visit the Stanford Encyclopedia of Philosophy website: https://plato.stanford.edu/entries/rationalism-empiricism/

_________________________

Markie, P. (2017, July 6). Rationalism vs. Empiricism. The Stanford Encyclopedia of Philosophy (Fall 2017 Edition), Edward N. Zalta (ed.). Retrieved from: https://plato.stanford.edu/archives/fall2017/entries/rationalism-empiricism


https://plato.stanford.edu/archives/fall2017/entries/rationalism-empiricism

1.7 Distinguish between definitions and examples of basic and applied science/research.


Basic research generally concerns itself with expanding an existing base of scientific knowledge. Hypotheses in basic education research investigations typically address broad learning and/or instructional theories. Basic research is designed to expand knowledge, refine theories, and is generally driven by curiosity. Basic research is also typically designed to increase understanding of fundamental principles by addressing WHY, WHAT or HOW questions. Also, basic research tends to have no immediate commercial value.

Applied research, on the other hand, includes scientific investigations designed to arrive at solutions to specific problems. Such solutions result in new knowledge that typically has some commercial applications and value. Solutions also have direct value to practitioners.

The following graphic depicts the relationship between learning theories, instructional models, and strategies/methods/resource selection and design presented in 1.3. In this model, research addressing the development and/or refinement of learning theories, theories of human development, instructional theories, and to some extent instructional design models constitute BASIC research.

Conversely, educational research investigating the effectiveness of specific aspects of a design model or instructional strategy is applied in nature. Such efforts reflect examples of APPLIED research.



1.8 Differentiate between the purpose and methods of research versus evaluation.

The purpose of research is to organize, clarify and generate new knowledge. The purpose of evaluation is to collect data during an investigation and make judgments about it.

Most sciences rely on the scientific method to inform the development of research methods. Common research methods are described in detail later in these notes (and the course text). The common methods of evaluation are identical to research methods. Only the purpose is different.

Program evaluation methods such as Kirkpatrick's four-level model, the Logic Model, and the Context/Input/Process/Product (CIPP) model can be different from more common research methods, though any program evaluation can include research strategies if needed. These types of evaluation models are described later in the course notes.


1.9 Identify the characteristics of good research problems and choose the better research problem based on a description of related problems.

The information below complements the information about “good research questions” included on pp. 26-34 of Fraenkel, Wallen & Hyun (2015).

A research problem represents a question about a variable of interest to an investigator. Research questions usually emerge from a review of literature related to the variable (or variables) of interest, from professional practice or work-related contexts, or from personal experiences/history.

A good research problem is:

Feasible

It can be investigated without spending an undue amount of time, energy, money, etc.

Clear

The variable(s) and key words in the research question are understood easily by most people.

Significant

Investigating the problem will contribute important knowledge to the human condition.

Ethical

Investigating the problem will not involve physical or psychological harm or damage to human beings or to the natural or social environment of which they are a part.

The following pairs of statements represent related research problems. The better problem is indicated in bold:

1a. Are heuristic approaches to teaching problem-solving skills more effective than theory-based methods?

1b. Is problem-based learning (PBL) instruction as effective as lecture-based methods for teaching problem-solving skills?

1b is clearer than the 1a (most people don’t know what “heuristic” means).

2a. Are school counseling programs worth it?

2b. How do parents feel about school counseling programs?

2b is more feasible and more clear.


1.10 Differentiate between definitions and examples of inductive versus deductive research.

[Bhattacherjee Chapters 1, 2, 3]

See the information presented in Bhattacherjee pp. 3-4, 14-15 & 20-21.

Perhaps the simplest way to distinguish between deductive and inductive reasoning is to consider the order in which theories, hypotheses, and observations are made/generated/tested between the two approaches to research. Deductive research starts with a theoretical statement leading to a specific hypothesis and then tests to see if it’s true through observation. Inductive research starts with observations, looks for patterns, and then generates hypotheses to be tested.

Meissler, D. (2018, August 28). The Difference Between Deductive and Inductive Reasoning. Retrieved from: https://danielmiessler.com/blog/the-difference-between-deductive-and-inductive-reasoning/


Research Proposition Examples:

Deduction

Mindfulness meditation reduces stress. Reduced levels of stress lead to greater success in the classroom. Therefore, mindfulness meditation leads to greater success in the classroom.

Induction

Some of the students in a classroom being observed demonstrate much more success than others. Most of the students demonstrating success regularly engage in mindfulness meditation. Therefore, mindfulness meditation leads to greater success in the classroom.


1.11 Identify definitions and examples of exploratory, descriptive, and explanatory research.


Exploratory

The objective of exploratory research is to gather preliminary information that will help define problems and suggest hypotheses. Exploratory research is oftenconducted in new areas of inquiry, where the goals of the research include: scoping out the magnitude or extent of a particular phenomenon, problem, or behavior; generating some initial ideas (or “hunches”) about that phenomenon; or testing the feasibility of undertaking a more extensive study regarding that phenomenon.

Descriptive

The objective of descriptive research is to make careful observations and detaileddocumentation of a phenomenon of interest. These observations are generally based on the scientific method (i.e., must be replicable, precise, etc.), making them more valid and reliable than casual observations by untrained people.

Explanatory

The objective of explanatory research is to seek explanations of observed phenomena, problems, or behaviors. While descriptive research examines the what, where, and when of a phenomenon, explanatory research seeks answers to why and how types of questions. It attempts to “connect the dots” in research, by testing hypotheses about cause-and-effect relationships.


1.12 Identify examples of concepts and constructs.


Concepts are objects, classes of objects, object features, and object relations…that can be pointed out and identified.

Examples:

Objects: Pieces of paperClasses of objects: Books, posters, cards, wrapping paperObject features: 8½”, greenObject relations: above, near, below, etc.

Constructs are objects, principles, classes, features, and relations that cannot be identified by pointing them out. They must be defined.

Examples:

“quality”, “energy”, “satisfaction”, “happiness”, “IQ”


1.13 Define operational definition and identify examples of operationally defined constructs.


Operational definitions are statements that define constructs in terms of how they can be empirically measured. Operational definitions generally answer two questions:

"How do you know when you have something?""How do you know how much you have?"

Common examples of constructs in education and counseling along with their operationalization include:

Construct Operational DefinitionAchievement Achievement can be defined as how well a student performs on

assessments. The higher the assessment scores, the higher the achievement.

Student engagement Student engagement can be defined as being on task without prompting. The amount of time on task (and paying attention) versus the amount of time off-task could be a measure of engagement.

Positive attitudes Positive attitudes can be defined as scoring high on an attitude survey, as well as choosing to do something of their own volition that is indicative of the targeted attitude (for example, having a positive attitude about math might be indicated by choosing to do extra no-credit problems or puzzles).

Happiness Happiness can be defined as scoring highly on a “happiness” survey and having a smile on your face.

Self-esteem Self-esteem can be defined by responses to questions on a survey about how worthy, lovable, valuable and capable a person feels.


1.14 Identify the variables in an operational definition.


Operationalization is the process of designing precise measures for abstract theoretical constructs. The precise measures generally represent variables of interest to researchers.

So if a researcher operationally defines self-esteem as how people respond to a collection of questionnaire items that address how worthy, lovable, valuable and capable they feel, then the variable of interest would be “score on self-esteem questionnaire”.


1.15 Differentiate between propositions and hypotheses.


Propositions

Associations postulated between constructs based on deductive logic.

For example, suppose you are interested in studying the effects of mindfulness in the classroom. Based on a review of the literature, you are convinced that mindfulness can lead to better focus and attention and can reduce feelings of anxiety. Based on this information, you formulate the following proposition:

Mindfulness will help students be more successful in school.

“Mindfulness” is a construct, and so is “success in school.”

Hypothesis

You might have heard the term hypothesis defined as “an educated guess”. This is an incomplete representation of what a hypothesis truly represents. A hypothesis is the empirical formulation of propositions, stated as a proposed relationship between an independent variable and dependent variable.

If you were to design an experiment to test a hypothesis about the proposition in the example above, you would first operationally define each of these constructs. You might define “mindfulness” for students as practicing mindfulness meditation for 15 minutes before beginning any work in the classroom. And you might operationally define “success in school” as adequate academic achievement as indicated by assessment scores, and minimal negative classroom behavior issues.

So in this case, a hypothesis would be:

Students performing 15 minutes of mindfulness meditation before class will have higher assessment scores and minimal negative classroom behavior issues compared with students who do not perform mindfulness meditation.

Suppose your experimental design included randomly assigning members of a student population to one of two groups: a mindfulness group that learned and practice mindfulness meditation every morning for 15 minutes, and a control group that did not learn how to practice mindfulness meditation. Throughout the school year, you recorded achievement scores and the number of negative behavior issues for each student in both groups, and then you compared the results between groups.

In this case, mindfulness mediation is the independent variable, and achievement scores/negative behavior issues are the dependent variables.


1.16 Identify the dependent, independent and modifier (if present) variables referenced in a given hypothesis.


Independent variable: The variable that is manipulated, or changed on purpose by researchers. This is the variable that is believed to have some effect on one or more other variables.

Dependent variable: This is the variable that is presumably affected by the independent variable. In an experiment, this is the variable that changes as a result of the independent variable.

Modifier variable: A type of secondary independent variable that might be introduced or identified by researchers to determine if it affects the relationship between the independent and dependent variables.

Examples:

Raising teacher salaries significantly will increase the overall quality of education in poorer communities.

The more time spent playing in early grades, the more academic success students will demonstrate in later grades.

Children who attend free schools are not happier or more successful later in life than children who attend public schools.

Students provided with completely self-directed computer-based learning experiences will not perform better on end-of-course tests than students in more traditional classroom settings.

Students who can use calculators in any math class situation throughout a school year will perform just as well on end-of-year tests as students who do not get to use calculators whenever they want.

Students assigned Homework will exhibit less student learning that comparable students not assigned homework.

A Positive Behavioral Interventions and Support (PBIS) program will improve behaviors of SPED students in middle school.


1.17 Given a hypothesis, determine the most likely extraneous (control) variables that would need to be controlled in a research study designed to collect data that could support or not support the veracity of the hypothesis.


Extraneous variables (control variables) are variable that makes possible an alternative explanation of results; an uncontrolled variable. In experimental studies, extraneous variables are held constant among and between research subjects in all groups so that the chance of the variable influencing the independent variable is the same for all subjects.

For example, suppose you believe that mindfulness practices immediately before exams would improve the overall performance of students in your 5th grade math class. You might test this by including the practices before an exam and comparing scores with previous or subsequent exams when your students did not practice mindfulness. But consider all the variables that would need to be controlled that COULD affect the independent variable (exam scores) in such an investigation: difficulty of exam material, time of day, amount of time afforded students to complete the exam, the number and type of students absent, etc. These variables would need to be controlled in some way in order to exclude their possible effects on the independent variable because you are only interested in the dependent variable’s possible effects (the mindfulness practices).


Exam 22. Positivist Research Designs

2.1 Research Design Overview

2.1.1 Distinguish between positivist and interpretive (antipositivism) research approaches.


Positivism holds that science or knowledge creation should be restricted to what can be observed and measured. Positivism tends to rely exclusively on theories that can be directly tested. Though positivism was originally an attempt to separate scientific inquiry from religion (where the precepts could not be objectively observed), positivism led to empiricism or a blind faith in observed data and a rejection of any attempt to extend or reason beyond observable facts.

Common research methods employed within the positivist paradigm are typically deductive in nature and are often referred as “quantitative” methods. These include experimental designs (pretest-posttest or posttest-only control group designs) as well as Solomon four-group designs. Quasi-experimental designs also fall into this category, including:

• Time-Series Experiment• Equivalent Time-Samples Design• Equivalent Materials Design• Nonequivalent Control Group Design• Counterbalanced Designs• Separate-Sample Pretest-Posttest Design• Separate-Sample Pretest-Posttest Control Group Design• Multiple Time-Series Design • Recurrent Institutional Cycle Design: A "Patched-Up" Design • Regression-Discontinuity Analysis

In social sciences, interpretivism (also known as antipositivism) proposes that the social realm cannot be studied with the scientific method of investigation typically applied to nature, and that social science investigations require a different theory of knowledge. Fundamental to that antipositivist epistemology is the belief that the concepts and language that researchers use in their research shapes their perceptions of the social world they are investigating, studying, and defining.

Interpretive methods typically employ an inductive approach that starts with data and tries to derive a theory about the phenomenon of interest from the observed data. Often times, these methods are incorrectly equated with quantitative and qualitative research. Quantitative and qualitative methods refer to the type of data being collected (quantitative data involve numeric scores, metrics, and so on, while qualitative data includes interviews, observations, and so forth) and analyzed (i.e., using quantitative techniques such as regression or qualitative techniques such as coding). Positivist research uses predominantly


quantitative data but can also use qualitative data. Interpretive research relies heavily on qualitative data but can sometimes benefit from including quantitative data as well.

Typical interpretivist research methods include case studies, action, ethnography and phenomenology research designs.


2.1.2 Describe common activities conducted in the exploration, research design, and research execution phases of a typical positivist research process.


Exploration

This phase includes:

Exploring and selecting research questions for further investigation Examining the published literature in the area of inquiry to understand the

current state of knowledge in that area Identifying theories that may help answer the research questions of interest

Research Design

This process is concerned with creating a blueprint of the activities to take in order to satisfactorily answer the research questions identified in the exploration phase. This includes:

Selecting a research method Operationalizing constructs of interest Devising an appropriate sampling strategy

Research Execution

This research phase includes:

Pilot testing the measurement instruments (the operationalized variables…usually the dependent variables)

Data collection Data analysis


2.1.3 Distinguish between definitions and examples of internal and external validity.


Internal Validity

Also called causality, internal validity examines whether the observed change in a dependent variable is indeed caused by a corresponding change in hypothesized independent variable, and not by variables extraneous to the research context.

External Validity

External validity, or generalizability, refers to whether the observed associations can be generalized from the sample to the population (population validity), or to other people, organizations, contexts, or time (ecological validity).

Internal External

Validity(Relevance)

Did intervention (experimental treatment or instruction) make the difference...or was it something else?

To what extent are results comparable and transferable? Can you generalize to other samples in the population?


2.1.4 Describe the conditions necessary to establish causality.


Three conditions are necessary to establish causality between independent and dependent variables:

1. Covariation of cause and effect (i.e., if cause happens, then effect also happens; and if cause does not happen, effect does not happen

2. Temporal precedence: cause must precede effect in time3. No plausible alternative explanation (or spurious correlation)


2.1.5 Explain common ways to improve internal validity, including: manipulation, elimination, inclusion, statistical control, and randomization strategies.

[Bhattacherjee Chapters 5]

Controls are required to assure internal validity (causality) of research designs and are commonly addressed in five ways: (1) manipulation, (2) elimination, (3) inclusion, (4) statistical control, and (5) randomization.

Manipulation

The researcher manipulates the independent variables in one or more levels (called “treatments”) and compares the effects of the treatments against a control group where subjects do not receive the same treatment.

Elimination

Researchers eliminate extraneous variables by holding them constant across treatments, such as by restricting the study to a single gender or a single socioeconomic status.

Inclusion

Researchers consider the role specific extraneous variables might play in affecting the dependent variable, so they are included in the research design (often as moderator variables). During data analysis, their effects on the dependent variable are estimated. Gender and race are common moderator variables.

Statistical Control

Similar to inclusion, statistical control involves measuring extraneous variables and using them as covariates during the statistical testing process. This differs from inclusion because researchers don’t manipulate the variables (or their occurrence in research groups) in any way…they are simply measured and addressed in the data analysis phase.

Randomization

This technique is perhaps the most powerful way to ensure the effects of extraneous variables are cancelled out between groups in a study. Two types of randomization are: (1) random selection, where a sample is selected randomly from a population, and (2) random assignment, where subjects selected in a non-random manner are randomly assigned to treatment or control groups.


2.1.6 Explain common ways to improve external validity.


The most powerful (and common) way to increase the external validity of a study is to randomly select members of the target population to participate in the study.


2.1.7 Explain common ways to establish construct validity.


Construct validity examines how well a given measurement scale is measuring the theoretical construct that it is expected to measure. There are two general categories of construct validity assessment: theoretical and empirical.

Theoretical Assessment of Construct Validity

The two most common types of theoretical construct validity are face validity and content validity.

Face validity refers to whether an indicator seems to be a reasonable measure of its underlying construct “on its face”. For example, “learner engagement” is a construct that is often studied in classrooms, and the amount of time a student spends on task without being reminded to stay on task seems a reasonable measurement of engagement. Face validity is often established using experts to judge the adequacy of specific types of observations.

Content validity is an assessment of how well a set of measurement itemsmatches with the relevant content domain of the construct that it is trying to measure. For example, if you are interested in measuring the construct “math literacy”, you would need to develop a set of items that address the application of reason and simple numerical concepts, including number sense, basic operation sense (addition, subtraction, multiplication, and division), computational abilities, and measurement basics. Of course, the types of items to develop would depend in the development levels of subjects studied. As with face validity, content validity is often established using experts to judge the adequacy of the collective set of items.

Empirical Assessment of Construct Validity

Empirical assessment of validity examines how well a given measure relates to one or more external criterion, based on empirical observations. This type of validity is called criterion-related validity, which includes four sub-types: convergent, discriminant, concurrent, and predictive validity

Convergent validity refers to the closeness with which a measure relates to (or converges on) the construct that it is purported to measure. Convergent validity can be established by comparing the observed values of one indicator of one construct with that of other indicators of the same construct and demonstrating similarity (or high correlation) between values of these indicators. For example, in order to test the convergent validity of a measure of self-esteem, a researcher may want to show that measures of similar constructs, such as self-worth, confidence, social skills, and self-appraisal are also related to self-esteem, whereas non-overlapping factors, such as intelligence, should not relate.

Discriminant validity refers to the degree to which a measure does not measure (or discriminates from) other constructs that it is not supposed to measure. Discriminant validity is established by demonstrating that indicators of one construct are dissimilar from (i.e., have low correlation with) other constructs.


Predictive validity is the degree to which a measure successfully predicts a future outcome that it is theoretically expected to predict. For instance, how well can standardized college admission test scores (e.g., Scholastic Aptitude Test scores) correctly predict academic success in college (e.g., as measured by college grade point average)?

Concurrent validity examines how well one measure relates to other concrete criteria that are presumed to occur simultaneously.

Concurrent validity and predictive validity are two types of criterion-related validity. The difference between concurrent validity and predictive validity rests solely on the time at which the two measures are administered. Concurrent validity applies to validation studies in which the two measures are administered at approximately the same time. For example, an employment test may be administered to a group of workers and then the test scores can be correlated with the ratings of the workers' supervisors taken on the same day or in the same week. The resulting correlation would be a concurrent validity coefficient. This type of evidence might be used to support the use of the employment test for future selection of employees.

Concurrent validity may be used as a practical substitute for predictive validity. In the example above, predictive validity would be the best choice for validating an employment test, because using the employment test on existing employees may not be a strong analog for using the tests for selection. Reduced motivation and restriction of range are just two possible biasing effects for concurrent validity studies


2.1.8 Given a description of a research investigation (or a published research paper), classify the research methodology employed as one or more (mixed-method) of the following: true experimental, quasi-experimental, correlation, field survey, case, focus group, action, ethnographic or secondary data analysis.


As noted earlier, research designs can be classified into two categories: positivist and interpretive. Positivist designs are meant for theory testing, while interpretive designs are meant for theory building. Positivist designs seek generalized patterns based on an objective view of reality, while interpretive designs seek subjective interpretations of social phenomena from the perspectives of the subjects involved.

Some design examples common to positivist research include:

True experimental

Experimental studies are those that are intended to test cause-effect relationships (hypotheses) in a tightly controlled setting by separating the cause from the effect in time, administering the cause to one group of subjects (the “treatment group”) but not to another group (“control group”), and observing how the mean effects vary between subjects in these two groups. In a true experimental design, subjects must be randomly assigned between each group.

Quasi-experimental

The same as experimental, except subjects are not randomly assigned to treatment or control groups.

Correlation

Correlational research is a type of nonexperimental research in which the researcher measures two variables and assesses the statistical relationship (i.e., the correlation) between them with little or no effort to control extraneous variables.

Field surveys

Field surveys are non-experimental designs that do not control for or manipulate independent variables or treatments but measure these variables and test their effects using statistical methods. Field surveys capture snapshots of practices, beliefs, or situations from a random sample of subjects in field settings through a survey questionnaire or less frequently, through a structured interview.

Action

Action research assumes that complex social phenomena are best understood by introducing interventions or “actions” into those phenomena and observing


the effects of those actions. The researcher’s choice of actions must be based on theory, which should explain why and how such actions may cause the desired change. The researcher then observes the results of that action, modifying it as necessary, while simultaneously learning from the action and generating theoretical insights about the target problem and interventions. This type of research is commonly conducted by practitioners such as teachers or counselors working to improve their own practices on subjects in their care (i.e. students, clients).

Secondary data analysis

Secondary data analysis is an analysis of data that has previously been collected and tabulated by other sources.

Some design examples common to interpretive research include:

Case studies

Case research is an in-depth investigation of a problem in one or more real-life settings (case sites) over an extended period of time. Data may be collected using a combination of interviews, personal observations, and internal or external documents. Though commonly used in interpretive (theory building) research, case studies can be positivist in nature (for hypotheses testing).

Focus groups

Focus group research is a type of research that involves bringing in a small group of subjects (typically 6 to 10 people) at one location and having them discuss a phenomenon of interest for a period of 1.5 to 2 hours. The discussion is moderated and led by a trained facilitator, who sets the agenda and poses an initial set of questions for participants, makes sure that ideas and experiences of all participants are represented, and attempts to build a holistic understanding of the problem situation based on participants’ comments and experiences.

Ethnographic research

Ethnography is an interpretive research design inspired by anthropology. It is predicated on the belief that a research phenomenon must be studied within the context of its culture. In this type of research experience, the researcher (or researchers) is deeply immersed in a certain culture over an extended period of time (8 months to 2 years), and during that period, engages, observes, and records the daily life of the studied culture, and theorizes about the evolution and behaviors in that culture.


2.2 Experimental and Quasi-Experimental Research and Evaluation Designs

2.2.1 Differentiate between treatment and control groups.


In experimental research, some subjects are administered an intervention or experimental stimulus, called a treatment (assigned to the treatment group), while other subjects are not administered such an intervention (the control group).


2.2.2 Categorize descriptions and/or examples of threats to validity as one of the following: History, Maturation, Testing, Instrumentation, Mortality, Regression.


Note: In addition to the threats covered in the course text, a few other common threats are included in the table below. You are only responsible for learning about those listed in the objective.

Threat Descriptions Examples What to do about it?

Subject Characteristics/Selection

Specific qualities of the subjects selected for a study might affect the results of any treatment.

The selection of people for a study may result in the individuals or groups differing (i.e., the characteristics of the subjects may differ) from one another in unintended ways that are related to the variables to be studied. If more than one group is studied, such characteristics can result in inequalities of comparison groups.

A researcher might include in-tact classes as groups in a study. Most educators realize that there are many factors that dictate the class schedules of students, and a certain type of student might be more prone to take a certain class in the morning or the afternoon. High school students in band, for example, might not be able to take math courses in the early morning. The absence of band students in a math class might change the overall profile of the class somewhat, resulting in an unequal class compared to others based on specific characteristics.

Random assignment

Pretest or use other ability index to determine equivalence

Block subjects (put into groups based on similarities), then randomly assign

Ensure enough subjects are included in comparison groups to minimize the effects of specific subject characteristics

History A specific event (or events) that occurs during the treatment phase in addition to the treatment variable(s).

Suppose you are testing the effects of an anti-smoking campaign on teenage subjects, and in the middle of the treatment period a popular movie is released that depicts (among other things) very positive examples of attractive teens smoking. Such an event might mitigate

Design assessment so that only the planned variations occur between groups

Monitor implementation carefully and record events that may affect posttest performance and attitudes


the intended effects of the anti-smoking intervention and subsequent assessments.

Statistical Regression

The tendency of groups with extreme scores to score closer to the mean on a second test.

Whenever a group is selected because of unusually high or low performance on a pretest, it will, on average, score closer to the mean on subsequent testing, regardless of what transpires in the meantime.

Use an equivalent control or comparison group that also has extreme scores

Mortality Differential loss of subjects from comparison groups (also known as attrition).

A researcher might study students, comparing scores on an assessment from the beginning of the year to the end of the year. Some of the students assessed at the beginning of the year will no longer be at the school by the end of the year. Some might move, a few might die (sad), and others might drop out if high school students are studied. This results in a comparison group at the end of the year that is not equal to the group at the beginning (students who move around a lot or drop out might possess characteristics that profoundly affect assessment results).

Use data only from subjects who were present at all experimental sessions or on whom you have a complete set of scores

Check to see if it differs much by treatment. If it does, report it. Also, interview subjects to try and determine why.

Testing The effect of a pretest or repeated testing on subsequent observations.

The use of a pretest in intervention studies sometimes may create a "practice effect" that can affect the results of a study. A pretest can also sometimes affect the way subjects respond to an intervention.

Don’t pretest (this has its pros and cons, of course)

Pretest a group outside the design to get an estimate of initial performance

Use alternate test forms on the pretest and posttest


Use item sampling on the pretest so that each subject takes only a part of the test

Maturation Subjects change during the course of the treatment or even between measurements.

A common example is the effects of maturation on young children as subjects in a study. In a relatively short period of time, some young children can grow and mature in ways that affect how they respond within the study. Concentration levels, cognitive abilities, and even certain motivational factors might naturally change due to growth and maturation.

Compare results to a control group of similar-aged subjects

Minimize the amount of time between the testing and retesting if younger subjects are studied

Instrumentation/Instrument Decay

The test itself may have unintended consequences on how the treatment materials are received, or the test may not be implemented in precisely the same way each time.

A common example of this threat is the introduction of small changes in assessment materials from one subject or group to the next. A researcher might discover a major typographical error or erroneous testing item that is revised during the study and not properly documented.

Don’t pretest Ensure the

implementers have training

Ensure the test is not changed between implementations

Instability The learners may not perform the same on any given day

Suppose you were implementing an assessment as part of a third grade study, and one group received the assessment the day after Halloween. This might impact their performance compared with other times when they (or other groups) completed the assessment.

Ensure a good sized sample of learners is tested

Location The particular locations in which data are collected, or in which an intervention is carried out, may

One group of subjects might complete a study in a room with windows and indirect lighting, while another group is in a windowless room illuminated with

If possible, ensure the same location is used for implementation and data collection experiences


create alternative explanations for any results that are obtained.

buzzing fluorescent lights. Such disparate conditions might have unintended effects!

Subject Attitude The attitude of subjects toward a study (and their participation in it) can create a threat to internal validity.

When subjects are given increased attention and recognition because they are participating in a study, their responses may be affected. This is known as the Hawthorne Effect.

Researchers should attempt to be as unobtrusive as possible throughout the study

Data Collector Characteristics

The researcher collecting data may possess characteristics that affect how subjects respond in the study.

Suppose a classroom observer for a research study is an ex-NFL linebacker. Such an imposing figure might influence how some of the students in the class (particularly athletic males) might behave and perform. The same might be true if the researcher is a young, attractive woman.

Using the same data collectors throughout the study might help reduce the impact of characteristics between different groups of subjects.

Data Collector Bias

The researcher collecting data might unconsciously elicit, observe and record data in a biased way.

A classroom observer might be more prone to notice students of a particular gender or ethnicity. Or a researcher might interview subjects in a manner that skews the data toward a specific hypothesis being tested.

If possible, keep the data collectors unaware of hypotheses being tested.

Use the same data collectors throughout the study. This way the bias will be controlled across groups.

Implementation The circumstances surrounding the implementation of the treatment might have an impact beyond the treatment itself.

Suppose researchers are testing a particular method of teaching, and subjects (students) are assigned to one class with Method A, or another class with Method B. But the teachers are different, and so are the lessons. One teacher might have much more experience teaching in general than the other

Choose an implementation model that includes the same teacher teaching the same material using different methods to different classes of students.

Provide extensive training for the teachers implementing the different methods,


teacher or might just be better in the many ways that affect the efficacy of lessons.

ensuring that all the teachers are adequately and similarly prepared for the implementation.

Researchers can use a number of techniques or procedures to control or minimize threats to internal validity. Essentially, they boil down to four alternatives:

1. Standardizing the conditions under which the study occurs2. Obtaining and using more information on the subjects of the study3. Obtaining and using more information on the details of the study4. Choosing an appropriate design.


2.2.3 Given a description of an investigation employing a pre-experimental research design, classify the design as one of the following:

One-shot case study One-group pretest-posttest Static-group comparison Static-group pretest-posttest

[Bhattacherjee Chapter 10, Campbell & Stanley, 1963]

Experimental Design DescriptionOne-shot case study A weak experimental design involving one group that is

exposed to a treatment, then posttested.One-group pretest-posttest

A weak experimental design involving one group that is pretested, exposed to a treatment, then posttested.

Static-group comparison A weak experimental design that involves at least two nonequivalent groups; one receives a treatment, and both are posttested.

Static-group pretest-posttest

The same as the static-group comparison design, except that both groups are pretested.


2.2.4 Given a description of an investigation employing an experimental research design, classify the design as one of the following:

Randomized posttest-only control group Randomized pretest-posttest control group Randomized Solomon four-group

[Bhattacherjee Chapter 10, Campbell & Stanley, 1963]

Experimental Design DescriptionRandomized posttest-only control group

An experimental design involving at least two randomly formed groups; one group receives a treatment, and both groups are posttested.

Randomized pretest-posttest control group

An experimental design that involves at least two groups; both groups are pretested, one group receives a treatment, and both groups are posttested. For effective control of extraneous variables, the groups should be randomly formed.

Randomized Solomon four-group

An experimental design that involves random assignment of subjects to each of four groups. Two groups are pretested, two are not, one of the pretested groups and one of the un-pretested groups receive the experimental treatment, and all four groups are posttested.


2.2.5 Given a description of an investigation employing a quasi-experimental research design, classify the design as one of the following:

Matching-only Counterbalanced Time-series

[Campbell & Stanley, 1963]

Quasi-experimental Design

Description

Matching-only A research design where experimental and control group subjects are matched but not using random assignment. This design can be useful when, for example, researchers assign control and variable classes, but they can’t control who is assigned to each class.

Counterbalanced A research design where all groups receive all treatments. Each group receives the treatments in a different order, and all groups are posttested after each treatment.

Time-series A research design involving one group that is repeatedly pretested, exposed to an experimental treatment, and repeatedly posttested


2.2.6 Identify the threats to validity that are minimized by the use of specific research designs.

As previously addressed, the use of randomization in experimental designs can help minimize several threats to internal validity. For example, a randomized pretest posttest control group design handles several threats to internal validity, such as maturation, testing, and regression, since these threats can be expected to influence both treatment and control groups in a similar (random) manner.

The following chart was prepared by 2015 Fraenkel, Wallen & Hyun (2015, p. 280). It summarizes the threats to validity that are minimized by specific experimental or quasi-experimental designs:


2.2.7 Given an experimental design research study, critique how well the study minimized threats to the validity of the data collected and results obtained.

The following abstract is for the study A randomized controlled pilot trial of classroom-based mindfulness meditation compared to an active control condition in sixth-grade children (Willoughby et al., 2014):

The current study is a pilot trial to examine the effects of a nonelective, classroom-based, teacher-implemented, mindfulness meditation intervention on standard clinical measures of mental health and affect in middle school children. A total of 101 healthy sixth-grade students (55 boys, 46 girls) were randomized to either an Asian history course with daily mindfulness meditation practice (intervention group) or an African history course with a matched experiential activity (active control group). Self-reported measures included the Youth Self Report (YSR), a modified Spielberger State-Trait Anxiety Inventory, and the Cognitive and Affective Mindfulness Measure –Revised. Both groups decreased significantly on clinical syndrome subscales and affect but did not differ in the extent of their improvements. Meditators were significantly less likely to develop suicidal ideation or thoughts of self-harm than controls. These results suggest that mindfulness training may yield both unique and non-specific benefits that are shared by other novel activities.

This is an example of an implementation threat to validity. Not only are the teachers presumably different, but so are the subjects taught (Asian history versus African history) between the treatment and control groups!


2.2.8 Given a description of an experiment designed to test a stated hypothesis, as well as a list of extraneous variables, identify strategies that could effectively minimize the effects of the extraneous variables on the dependent variable(s).

This objective will be assessed in your research proposal. In the proposal, you must describe how you would control for the effects of all extraneous variables on your identified dependent variable.

To successfully perform this objective, you need to apply skills from the earlier objectives that address extraneous variables and threats to internal validity.

Here is a good example taken from the study described in the abstract for the previous objective:

Hypothesis:

Mindfulness meditation will influence the mental health and affect of middle school children.

Independent variable: Mindfulness meditation

Dependent variable: Mental health and affect

Extraneous variables (and how to control for them):

Extraneous Variable ControlEffects of initial mindfulness meditation training experience

Have subjects in both treatment and control group receive mindfulness training. But treatment group will practice it in the classroom.

Development level (“maturity”) of subjects

Random assignment to treatment or control groups.

Subject studied during the mindfulness sessions

Ensure both treatment and control groups are learning the same subject (Asian OR African history).

Different teachers might have different influences on students

Choose a model that can allow for the same teacher, or treatments administered to both groups at different times (counterbalanced design).


Exam 33. Interpretive Research Designs & Survey Design

3.1 Identify and describe common characteristics of interpretive research models, comparing case, grounded theory, ethnography and phenomenology research designs.


Additional information and examples supporting this objective are presented in the material for objective 2.1.8.

Interpretative Research Model

Description

Narrative Research

Narrative research can be a story of a day in the lives of individuals as told by these individuals. The researcher retells or re-stories this into a narrative chronology, combining views taken from the participant's and the researcher's lives.

Phenomen-ology

In its broadest sense, 'phenomenology' refers to a person's perception of the meaning of an event, as opposed to the event as it exists externally to (outside of) that person.

The focus of phenomenological inquiry is what people experience regarding some phenomenon or other and how they interpret those experiences.

A phenomenological research study is a study that attempts to understand people's perceptions, perspectives and understandings of a particular situation (or phenomenon).

In other words, a phenomenological research study tries to answer the question What is it like to experience such and such?

Grounded Theory

The purpose of a grounded theory study is to generate, or discover, a theory.

A grounded theory is one that is inductively derived from the study of phenomena.

The theory is discovered, developed and provisionally verified through systematic data collection and analysis of data pertaining to that phenomena.

The important thing to remember is that you do not begin with a theory and then attempt to prove it, but rather you begin with an area of study and then what is relevant is allowed to emerge from your research.

Case Studies Case studies are a form of qualitative research where a single


individual or example is studied through extensive data collection.

Ethnography Ethnography is the recording and analysis of a culture or society, usually based on participant-observation and resulting in a written account of a people, place or institution.

Ethnography is an extremely broad area with a great variety of practitioners and methods. However, the most common ethnographic approach is participant observation as a part of field research. The ethnographer becomes immersed in the culture as an active participant and records extensive field notes. As in grounded theory, there is no preset limiting of what will be observed and no real ending point in an ethnographic study.

The table below summarizes the differences between five common qualitative methods.

Method Focus Sample Size Data Collection

Ethnography Context or culture — Observation & interviews

NarrativeIndividual

experience & sequence

1 to 2Stories from individuals & documents

PhenomenologicalPeople who have

experienced a phenomenon

5 to 25 Interviews

Grounded TheoryDevelop a theory from grounded in

field data 20 to 60

Interviews & observations, then

open and axial coding

Case StudyOrganization,

entity, individual, or event

—Interviews, documents,

reports, observations


3.2 Categorize survey response formats as dichotomous, nominal, ordinal, interval, or ratio.


Generally, numerical measurement data used in research studies can be categorized into one of the four following scale types:

Measurement Scale Definition Examples

Nominal (including dichotomous)

Numbers are assigned to categories. Differences between numbers don’t mean anything (no mathematical significance) other than differences in categories. When only two choices are present, it is dichotomous.

religion, occupation, political party preference, class, gender (dichotomous)

Ordinal Numbers represent a hierarchical or rank ordering. Differences between numbers mean something, but differences between numbers are not equal.

level of education, attitude survey responses*

Interval Numbers represent a particular hierarchy and order, and the differences between them are relatively equal, but there is no true zero on the scale.

IQ tests, many other norm-referenced test scores

Ratio Numbers represent a particular hierarchy and order, and the differences between them are equal, and the scale is anchored by a true zero.

number of siblings, wages earned, number of arrests, performance on criterion-referenced tests

*Although there are many types of attitude survey items that might constitute different types of numerical scales, a common type of survey item asks subject to rate their level of agreement with specific statements. Common levels of agreement include “strongly agree,” “agree,” “disagree,” and “strongly disagree.” In this case, the levels of agreement reflect an order (most agreement to least), but the differences between “strongly agree” and “agree” is not necessarily the same as the distance between “agree” and “disagree,” even if you assign interval numerical numbers to each choice (i.e. Strongly Agree = 4, Agree = 3, Disagree = 2, Strongly Disagree = 1).


3.3 Identify and describe the following types of scales commonly used in the design of survey items:

Binary Likert Semantic differential Guttman


Binary Scale

Binary scales are nominal scales consisting of binary items that assume one of two possible values, such as yes or no, true or false, and so on.

Example: Suppose you are studying the effects of mindfulness meditation on student attitudes and performance. As part of your data collection, you ask a series of binary scale items about each subject’s experiences with meditation outside the classroom. Such items might include:

Do you ever meditate on your own at home? Yes NoDoes anybody in your house meditate regularly? Yes No

Likert Scale

This is a very popular rating scale for measuring ordinal data in social science research. This scale includes Likert items that are simply-worded statements to which respondents can indicate their extent of agreement or disagreement, commonly using a four, five or seven-point scale ranging from “strongly disagree” to “strongly agree”.

Example: Using the mindfulness example, you might ask the following Likert scale items following a mindfulness intervention in the classroom:

Strongly Agree Agree

Neither Agree Nor Disagree

Disagree Strongly Disagree

Mindfulness meditation helps me relax.Mindfulness meditation helps me focus on my school work.I will continue mindfulness meditation at home over the summer.


Semantic Differentials

This is a composite (multi-item) scale where respondents are asked to indicate their opinions or feelings toward a single statement using different pairs of adjectives framed as polar opposites. Responses in this scale could be easily translated into ordinal data.

Example: Using the mindfulness example, you might ask the following semantic differential items following a mindfulness intervention in the classroom:

How would you rate participating in mindfulness meditation at the beginning of the school day?

Very Much Somewhat Neither Somewhat Very MuchHelpful Waste of

TimeRelaxing StressfulImportant Useless

Guttman Scale

This composite scale uses a series of items arranged in increasing order of intensity of the construct of interest, from least intense to most intense. This could be considered a collection of binary scale items, or the set of related items could be coded to obtain an interval or ratio score (e.g. weight the value of each “Yes” as the items increase in intensity).

Example: Using the mindfulness example, you might ask the following Guttman scale items following a mindfulness intervention in the classroom:

What are your opinions on the following statements about mindfulness meditation?

I like starting each day in school with mindfulness meditation. Yes NoI would like to practice mindfulness meditation at different times throughout the day, in addition to first think in the morning.

Yes No

I plan to practice mindfulness meditation on my own at home. Yes No


3.4 Use the rules for good survey item design to critique exiting survey items.


Responses obtained in survey research are very sensitive to the types of questions asked. Poorly framed or ambiguous questions will likely result in meaningless responses with very little value. The following rules should be followed when developing or evaluating survey items:

Question should be clear and understandable.

Survey questions should be stated in a very simple language, preferably in active voice, and without complicated words or jargon that may not be understood by a typical respondent. And all questions in the questionnaire should be worded in a similar manner to make it easy for respondents to read and understand them.

Do not use negative language in the questions or statement.

Negatively worded questions, such as “Should first grade students not engage in mindfulness meditation in school?” tend to confuse many responses and lead to inaccurate responses. Such questions should be avoided, and in all cases, avoid double-negatives.

Do not ask ambiguous questions.

Survey questions should not include words or expressions that may be interpreted differently by different respondents. For example, “Are you good in school?” could mean many different things: grades, behavior, cooperation, etc.

Avoid using biased or value-laden words.

Bias refers to any property of a question that encourages subjects to answer in a certain way. For example, a survey item measuring attitudes about gun control should avoid biased words, such as “The government should enact laws to prevent citizens from being able to purchase weapons of murder in local department stores.”

Avoid asking double-barreled questions.

Double-barreled questions are those that can have multiple answers. For example, are you satisfied with the hardware and software provided for your work? In this example, how should a respondent answer if he/she is satisfied with the hardware but not with the software or vice versa? It is always advisable to separate double-barreled questions into separate questions: (1) are you satisfied with the hardware provided for your work, and (2) are you satisfied with the software provided for your work.

Avoid asking questions that are too general.

Sometimes, questions that are too general may not accurately convey respondents’ perceptions. For example, suppose a survey question asks “How do you like school?”


and provide a response scale ranging from “not at all” to “extremely well”. If a person selects “extremely well”, what does he/she mean?

Avoid asking questions that are too detailed.

Don’t write unnecessarily detailed questions that serve no specific research purpose. For example, you might be interested in asking questions about a person’s job experiences as part of a study, but including specific questions asking respondents to list all jobs would probably result in too much unnecessary data collected that would not be used.

Avoid asking questions that are presumptuous.

Nothing in a good question should address a presumption that the responder has about the variable you are measuring. For example, asking “What do you think are the benefits of charter schools?” presumes that the respondent has something favorable to say about charter schools.


3.5 Describe strategies for addressing non-response bias in survey research.


Non-response bias. Survey research is generally notorious for its low response rates. A response rate of 15-20% is typical in a mail survey, even after two or three reminders. If the majority of the targeted respondents fail to respond to a survey, then a legitimate concern is whether non-respondents are not responding due to a systematic reason, which may raise questions about the validity of the study’s results. For instance, dissatisfied customers tend to be more vocal about their experience than satisfied customers and are therefore more likely to respond to questionnaire surveys or interview requests than satisfied customers. Hence, any respondent sample is likely to have a higher proportion of dissatisfied customers than the underlying population from which it is drawn. In this instance, not only will the results lack generalizability, but the observed outcomes may also be an artifact of the biased sample. Several strategies may be employed to improve response rates:

Advance notification

A short letter or email sent in advance to the targeted respondents soliciting their participation in an upcoming survey can prepare them in advance and improve their propensity to respond. The message should state the purpose and importance of the study, mode of data collection (e.g., via email, a phone call, a survey form in the mail, etc.), and appreciation for their cooperation. A variation of this technique may request the respondent to return a postage-paid postcard indicating whether they are willing to participate in the study or not.

Relevance of content

If a survey examines issues of relevance or importance to respondents, then they are more likely to respond than to surveys that don’t matter to them.

Respondent-friendly questionnaire

Shorter survey questionnaires tend to elicit higher response rates than longer questionnaires. Furthermore, questions that are clear, non-offensive, and easy to respond tend to attract higher response rates.

Endorsement

For organizational surveys, it helps to gain endorsement from a senior executive attesting to the importance of the study to the organization. Such endorsement can be in the form of a cover letter or a letter of introduction, which can improve the researcher’s credibility in the eyes of the respondents.

Follow-up requests

Multiple follow-up requests may coax some non-respondents to respond, even if their responses are late.

Interviewer training


Response rates for interviews can be improved with skilled interviewers trained on how to request interviews, use computerized dialing techniques to identify potential respondents, and schedule callbacks for respondents who could not be reached.

Incentives

Response rates, at least with certain populations, may increase with the use of incentives in the form of cash or gift cards, giveaways such as pens or water bottles, entry into a lottery, draw or contest, discount coupons, promise of contribution to charity, and so forth.

Non-monetary incentives:

Businesses, in particular, are more prone to respond to nonmonetary incentives than financial incentives. An example of such a non-monetary incentive is a benchmarking report comparing the business’s individual response against the aggregate of all responses to a survey.

Confidentiality and privacy

Finally, assurances that respondents’ private data or responses will not fall into the hands of any third party, may help improve response rates.


Exam 44. Sampling, Construct Validity & Reliability, Data Analysis & Statistics

4.1 Sampling

4.1.1 Distinguish between a population, sampling frame, and sample.


Population: The group of people for whom you want to generalize the results of your study. In education, it is common to have populations such as elementary students, secondary math students, special education students.

Sampling frame: An accessible section of the target population (usually a list with contact information) from where a sample can be drawn. In an education study, a sampling frame is commonly all the students in a particular school or school division.

Sample: The actual subjects selected to be in the study. These could be randomly selected people from the sampling frame, or accessible units in the sampling frame (such as in-tact classes).


4.1.2 Given a sampling method, determine if it represents a probability or a nonprobability approach.


Probability sampling is a technique in which every unit in the population has a chance(non-zero probability) of being selected in the sample, and this chance can be accuratelydetermined.

All probability sampling have two attributes in common:

Every unit in the population has a known non-zero probability of being sampled The sampling procedure involves random selection at some point

Nonprobability sampling is a sampling technique in which some units of the population have zero chance of selection or where the probability of selection cannot be accurately determined. Typically, units are selected based on certain non-random criteria, such as quota or convenience.


4.1.3 Given a description of a probability sampling method, classify it as an example of simple, systematic, stratified, cluster, or matched-pairs.


Simple random sampling

In this technique, all possible subsets of a population (more accurately, of a sampling frame) are given an equal probability of being selected.

Systematic sampling

In this technique, the sampling frame is ordered according to some criteria and elements are selected at regular intervals through that ordered list. For example, researchers might get a list of all students in an elementary school and rank them based on achievement, and then select every third student to include in a sample.

Stratified sampling

In stratified sampling, the sampling frame is divided into homogeneous and non-overlapping subgroups (called “strata”), and a simple random sample is drawn within each subgroup. For example, each class in an elementary school might be considered a stratum, and the same number of students from each class are then randomly selected to be included in the sample.

Cluster sampling

If you have a population dispersed over a wide geographic region, it may not be feasible to conduct a simple random sampling of the entire population. In such case, it may be reasonable to divide the population into “clusters” (usually along geographic boundaries), randomly sample a few clusters, and measure all units within that cluster.

Matched-pairs sampling

Sometimes, researchers may want to compare two subgroups within one population based on a specific criterion. For example, researchers might identify schools in a state that are high-achieving, and randomly select one or more schools from this group. The same is done for schools that are low-achieving. The randomly selected high and low schools would constitute the sample for the study.


4.1.4 Given a description of a nonprobability sampling method, classify it as an example of convenience, quota, expert or snowball sampling.


Convenience sampling

Also called accidental or opportunity sampling, this is a technique in which a sample is drawn from that part of the population that is close to hand, readily available, or convenient. For instance, if you stand outside a shopping center and hand out questionnaire surveys to people or interview them as they walk in, the sample of respondents you will obtain will be a convenience sample. This is a non-probability sample because you are systematically excluding all people who shop at other shopping centers.

Convenience samples are regularly used in educational settings. For example, many social science research studies are conducted on college campuses using the students enrolled in the college as subjects because they are easy to access (and easy to incentivize in the form of mandatory participation in studies as part of certain classes).

Quota sampling

In this technique, the population is segmented into mutually exclusive subgroups (just as in stratified sampling), and then a non-random set of observations is chosen from each subgroup to meet a predefined quota.

For instance, if the American population consists of 70% Caucasians, 15% Hispanic-Americans, and 13% African-Americans, and you wish to understand their voting preferences in a sample of 98 people, you can stand outside a shopping center and ask people their voting preferences. But you will have to stop asking Hispanic-looking people when you have 15 responses from that subgroup (or African-Americans when you have 13 responses) even as you continue sampling other ethnic groups, so that the ethnic composition of your sample matches that of the general American population.

Expert sampling

This is a technique where respondents are chosen in a non-randommanner based on their expertise on the phenomenon being studied. For instance, in order to understand the impact of a new educational policy such as federally-funded vouchers for private school tuition, researchers might want to sample public school superintendents who are familiar with the impact such a policy might have in schools.

Snowball sampling

In snowball sampling, researchers begin by identifying a few respondents that match the criteria for inclusion in a study. These participants are then asked to recommend others they know who also meet the selection criteria. For instance, researchers were


interested in studying parents who “unschool” their children, they might begin with some known unschoolers and ask them if they know of other unschooling families who might be interested in participating in the study.


4.2 Construct Validity & Reliability of Measurement Procedures

4.2.1 Distinguish between definitions and examples of measurement instruments and procedures reliability estimates and evidence of construct validity.


Reliability is the degree to which the measure of a construct is consistent or dependable. In other words, if we use this scale to measure the same construct multiple times, do we get pretty much the same result every time, assuming the underlying phenomenon is not changing? The internal reliability of a measurement addresses the consistency of the measurements in a study, whereas external reliability addresses the consistency of obtaining the same results (using the same measurements and procedures) in different studies (or outside any study).

As discussed earlier, validity addresses the degree to which measurement instruments and procedures actually measure the construct or other variables being measured. The following table summarizes differences between external and internal reliability and validity.

Note that reliability implies consistency but not accuracy. Validity is accuracy, but you can’t be accurate without being consistent. Research measurement instruments and procedures can be reliable without being valid, but they cannot be valid without also being reliable.

Internal External

Validity(Relevance)

Does the assessment measure what it purports to measure (did experimental treatment or instruction make the difference...or was it something else?)

To what extent are assessment results comparable and transferable (can you generalize to other samples in the population?)

Reliability(Consistency)

Do the same methods yield the same assessment result?

Using the same methods, will you consistently obtain the same assessment results elsewhere?


4.2.2 Given a description of reliability measures applied to a specific set of data, determine whether inter-rater, stability (test-retest), split-half, or internal consistency reliable has been estimated.


Measurement instruments must still be tested for reliability. The following represent common ways to estimate reliability:

Inter-rater reliability

Inter-rater reliability, also called inter-observer reliability, is a measure of consistency between two or more independent raters (observers) of the same construct. Usually, this is assessed in a pilot study, and can be done in two ways, depending on the level of measurement of the construct. If the measure is categorical, a set of all categories is defined, raters check off which category each observation falls in, and the percentage of agreement between the raters is an estimate of inter-rater reliability. If the measure is interval or ratio scaled (e.g., classroom activity is being measured once every 5 minutes by two raters on 1to 7 response scale), then a simple correlation between measures from the two raters can also serve as an estimate of inter-rater reliability.

Split-half reliability

Split-half reliability is a measure of consistency between two halves of a construct measure. For instance, if you have a ten-item measure of a given construct, randomly split those ten items into two sets of five (unequal halves are allowed if the total number of items is odd) and administer the entire instrument to a sample of respondents. Then, calculate the total score for each half for each respondent, and the correlation between the total scores in each half is a measure of split-half reliability

Stability Reliability (test-retest)

The degree to which a learner’s performances on the same assessment administered at two or more different times are similar.

Internal Consistency Reliability

The degree to which items within an assessment are functioning in a consistent manner. If a multiple-item construct measure is administered to respondents, the extent to which respondents rate those items in a similar manner reflects internal consistency. This reliability can be estimated in terms of average inter-item correlation, average item-to-total correlation, or more commonly,Cronbach’s alpha.


4.2.3 Identify the purpose of the standard error of measurement.

The Standard Error of Measurement (SEM) is an estimate of the consistency of an individual’s test performance on other equally-difficult (or easy) tests. It is calculated from the standard deviation of the test scores as well as the reliability of the test itself.

Teachers should always be aware that their tests, as well as those developed by others, are not 100% precise. Any individual score reflects an estimate of ability. The lower the standard error, the more precise the estimate…but it will rarely be zero.

Note: this information is different from the information about standard error presented in Bhattacherjee, Chapter 8. The material in the text addresses the standard deviation of a sampling statistic, but SEM refers to a measure of reliability commonly reported with standardized test results.


4.2.4 Given descriptions of tests conducted to determine evidence of construct validity, determine whether face, content, convergent or discriminant evidence is being measured.


See notes for objective 2.1.7


4.2.5 Describe and identify examples of concurrent and predictive construct validity.


See notes for objective 2.1.7


4.3 Data Analysis & Statistics

4.3.1 Distinguish between definitions and examples of statistics versus parameters.


A parameter is a characteristic of an entire population. It is a numerical or graphic way to summarize data obtained from the population.

A statistic, on the other hand, is a characteristic of a sample from the population. It is a numerical or graphic way to summarize data obtained from a sample.

For example, suppose you are interested in studying the effects of a new educational program on inner-city students in Atlanta. This is your population, and you have no intention of generalizing your findings beyond this population. If you collect data from every inner-city student in Atlanta, you are defining a parameter. Common parameters like this include gender and race break-downs, socio-economic status variables, and even the number of students enrolled in all the schools. These are parameters because they reflect a summary of data from all (or nearly all) the population. But a statistic is a characteristic about the population of inner-city Atlanta students based on data from a sample of this population. For example, you might randomly choose students to interview about attitudes toward home and school. Depending on how many students you selected for your sample and the manner in which they were selected, you might be able to report your statistical findings with confidence that they reflect the population of inner-city students in Atlanta as a whole.


Univariate Descriptive Statistics

Univariate analysis, or analysis of a single variable, refers to a set of statistical techniques that can describe the general properties of one variable.

A descriptive statistic is a summary statistic that quantitatively describes or summarizes features of a collection of information.

4.3.2 Identify characteristics of a “normal curve.”


The distribution curve of a normal distribution is called a normal curve. It is bell shaped, and its mean, median, and mode are typically identical (or nearly so).

The following diagram represents a typical normal distribution, with important corresponding statistical concepts represented:

Characteristics of a normal distribution of univariate data:

The mean, mode and median are close in value and usually at or near the highest peak

68% of the data falls within one standard deviation of the mean. 95% of the data falls within two standard deviations of the mean 99.7% of the data falls within three standard deviations of the mean


4.3.3 Identify definitions and examples of the following measures of central tendency: Mean, Median, Mode, Standard Deviation


The following table presents the most commonly-used descriptive statistics in social science research:

Type Procedure DefinitionCommonSymbols

Measures of Central Tendency

Mean Average score of the entire set or “distribution” of scores

_X, M

Mode Most commonly occurring score in a sample

Median Middle score in a distribution

Measures of Variability

Variance Degree of variability of individual scores in a sample S2

Standard DeviationSquare root of the variance (uses same scale as measurement)

SD

Measures of Relationship Correlation

Coefficient

Degree of correspondence between 2 characteristics of a sample (+1.00=perfect positive relationship, -1.00=perfect negative relationship)

r


Bivariate Descriptive Statistics

Bivariate analysis examines how two variables are related to each other. The most common bivariate statistic is the bivariate correlation (often, simply called “correlation”).

4.3.4 Describe a Correlation Coefficient and corresponding significance level and p-value.


As indicated in the chart for objective 4.3.3, a Correlation Coefficient represents the degree of correspondence between 2 characteristics of a sample.

Degree of correlation is communicated with a value between +1.00 (perfect positive relationship) and -1.00 (perfect negative relationship).

The p value is an estimate of the probability that the calculated degree of correspondence between two variables is due to chance. A p-value of .50 indicates that the likelihood a correlation coefficient calculated is due to chance is 50%.


Inferential Statistics

4.3.5 Identify differences between descriptive and inferential statistics.


A descriptive statistic is a summary statistic that quantitatively describes or summarizes features of a collection of information. Descriptive statistics DESCRIBE a population sample.

Inferential statistics are the statistical procedures that are used to reach conclusionsabout associations between variables. They differ from descriptive statistics in that they are explicitly designed to test hypotheses. Inferential statistics allow researchers to make inferences about the relationship between variables.

Inferential statistics involve a chain of reasoning that connects the observed data to populations of data too large to observe completely. Such statistics allow researchers to make inferences about a population based on data obtained from a sample.

Here are the common inferential statistics used in human subjects research:

Type Procedure DefinitionCommonSymbols

Parametric

t-test Statistical significance* of the difference between two means

Data in this category are commonly numerical, measured along a scale.

F-test Statistical significance of the differences between means in an ANOVA, MANOVA, ANCOVA

F (degrees of freedom), p

Analysis of Variance (ANOVA)

Statistical significance of the difference between two or more means

F, p

Analysis of Covariance (ANCOVA)

Statistical significance of the difference between two or more means across two or more related independent variables

F, p

Multivariate Analysis of Variance (MANOVA)

Same as ANOVA except more than one dependent variable is employed

F, p

Multiple Comparisons (Tukey, Sheffé, Newman-Keuls, etc.)

Compare the differences between two means within an ANOVA, ANCOVA, or MANOVA design

t-test, F, p

NonparametricChi-square Determine if a relationship

exists between two categorical variables

2


A nonparametric statistical technique makes few, if any, assumptions about the nature of the population from which the samples in the study were taken.

Data in this category isn’t usually measured along a numerical

Mann-Whitney U-testWilcoxon signed rank test

Same as t-test except nonparametric data are analyzed

p

scale...it might be classified, ranked, or counted.

Kruskal-Wallis test

Friedman test

Same as ANOVA except nonparametric data are analyzed.

*”Significance” is the probability (p) that any differences are due to chance instead of the independent variable. The alpha ( ) is the level of significance chosen in advance by the researcher.


4.3.6 Distinguish between Type II and Type I errors.

Type I error: The rejection by the researcher of a null hypothesis that is actually true; also called an alpha error. False positive.

Suppose you are not pregnant, but the first time you take a pregnancy test, it indicates you are. This is a false positive.

Type II error: The failure of a researcher to reject a null hypothesis that is really false; also called a beta error. False negative.

Suppose you are pregnant, but the first time you take a pregnancy test, it indicates you are not. This is a false negative.


4.3.7 Identify the purpose of a t-test (one-way ANOVA).


The t-test examines whether the means of two groups are statistically different from each other, or whether one group has a statistically larger (or smaller) mean than the other. The general goal of a t-test is to determine the significance level* of any differences in the calculated descriptive statistics between two groups. T-tests can be conducted on means, proportions, and correlation coefficients.

*The conclusion that results are unlikely to have occurred due to sampling error or "chance"; an observed correlation or difference probably exists in the population.


4.3.8 Identify the purpose of calculating the power of a statistical test.

The power of a statistical test is calculated to determine the likelihood that significant differences observed between groups of subjects do, in fact, really exist. The power of a test is a measure of its ability to avoid a Type II error.


5. Program Evaluation Models

In addition to experimental and quasi-experimental research models, describe the main characteristics of the following program evaluation models:

Kirkpatrick's four-level model Logic Model Context/Input/Process/Product (CIPP) model

For information about these common program evaluation models, review the material in the following online article:

Program evaluation models and related theories: AMEE Guide No. 67

https://www.tandfonline.com/doi/full/10.3109/0142159X.2012.668637


https://www.tandfonline.com/doi/full/10.3109/0142159X.2012.668637

6. Ethical Considerations

The following learning objectives* are included in the material from the Collaborative IRB Training Initiative (CITI) assigned modules for research with human subjects:

An understanding of the historical perspectives, ethical principles and federal regulations associate with the conduct of research with human subjects.

A clear understanding of what constitutes human subjects research and how informed consent must be applied in human subjects research.

Basic information on the regulations and policies governing research with investigational drugs, biologicals and devices and how the findings of The International Commission on Harmonization affect the conduct of research with human subjects around the world.

A basic understanding of the risks to privacy and confidentiality of human subjects who participate in Social and Behavioral research.

An understanding of the special considerations that must be addressed when "Vulnerable Populations" such prisoners, minors, pregnant women and fetuses in utero are used in research activities.

An understanding of how to recognize and avoid conflicts of interest in human subjects research.

New insights into the concept of group harms in vulnerable populations such as minorities and workers in a workplace setting and the use of Community Consultation to prevent injury to special social structures.

An understanding of the special risks facing human subjects when they participate in research conducted over the internet.

A clear understanding of the ethical issues and federal regulations in force during the conduct of Social / Behavioral Research, Records Based Research and Genetics Research with human subjects.

An understanding of the Policies, Regulations and Risks associated with conducting research with children in public school setting.

A clear understanding of the special procedural and regulatory policies for human subjects research at VA research facilities.

*Specific skills associated with these broad objectives will be measured with short quizzes following each module presentation on the CITI website.


Documents

Improving the Quality€¦ · Web view2021. 6. 25. · Quantitative and qualitative methods refer to the type of data being collected (quantitative data involve numeric scores,