Validity Concepts in Research

7/28/2019 Validity Concepts in Research

1/8

Validity Concepts in Research: an Integrative Approach

David Brinberg, University of Maryland

ABSTRACT - The research process is described as the interrelationships of conceptual, methodological, and

substantive domains. The conceptual domain is seen as concepts and ideas in abstract form; the methodological

domain as the designs, strategies, measuring devices, and analytic techniques used to study a phenomenon or

theory; and the substantive domain as the events/processes studied. The specification of each domain and their

interrelations will be discussed. Traditional forms of validity will be presented within the proposed framework and new

forms of validity researchers need to consider will also be derived from this framework[ to ci te ]:

David Brinberg (1982) ,"Validity Concepts in Research: an Integrative Approach", in NA - Advances in Consumer

Research Volume 09, eds. Andrew Mitchell, Ann Abor, MI : Association for Consumer Research, Pages: 40-44.

Advances in Consumer Research Volume 9, 1982 Pages 40-44

VALIDITY CONCEPTS IN RESEARCH: AN INTEGRATIVE APPROACH

David Brinberg, University of Maryland

ABSTRACT -

The research process is described as the interrelationships of conceptual, methodological, and substantive domains.

The conceptual domain is seen as concepts and ideas in abstract form; the methodological domain as the designs,

strategies, measuring devices, and analytic techniques used to study a phenomenon or theory; and the substantive

domain as the events/processes studied. The specification of each domain and their interrelations will be discussed.

Traditional forms of validity will be presented within the proposed framework and new forms of validity researchers

need to consider will also be derived from this framework

INTRODUCTION

As the reader is aware, the concept of validity has taken many forms and meanings to describe different components

of the research process. Traditionally, the research process has been divided into four basic components: (l) the

specification, measurement and/or manipulation of a theoretical variable(s), (2) the selection of a particular type of

research design, (3) the analysis of the data obtained from the study and (4) the interpretation and robustness of the

findings. Associated with each component of the research process are one or more forms and/or meanings of validity

to assess the "adequacy" of the particular research component. For instance, construct validity, convergent, and

discriminant validity for theoretical variables; internal validity for research design; statistical conclusion validity for data

analysis; and external validity for the robustness of research findings.

Give the current state of the art concerning the various forms and meanings of validity issues, several options seem

available to researchers. One option would involve further refinement and specification of current validity concepts.

By selecting this option, researchers would assume that the basic formulations of the validity issues are acceptable

and that research is needed to further articulate the validity concepts. A second option is to invent new forms of

validity to deal with methodological and theoretical developments. Researchers selecting this option would assume

current forms of validity are inadequate to deal with new developments. Thus, new forms of validity would be needed.

A third option would be: (a) to develop a conceptual scheme that explicitly describes the components of the research

process and their interrelationships, and (b) to use this scheme to "locate" or derive traditional forms of validity (e.g.,

construct validity, external validity) and identify new validity issues that researchers need to consider. A researcher

selecting this option would assume that a conceptual scheme would help clarify the research process, the various


2/8

roles of validity in research as well as the relationships among the validity concepts. Finally, a fourth option is to

accept the current state of the arc and conduct no further research.

The option to be selected and expanded on in this paper is the development of a conceptual scheme: (1) that

describes the research process and (2) is used to derive and locate various forms and meanings of validity.

THE RESEARCH PROCESS

Research involves drawing upon elements and relations from three basic domains: (a) a conceptual domain, which

includes concepts and relations considered in abstract form, (b) a methodological domain, which includes instruments

and techniques for obtaining observations and for relating sets of observations; and (c) a substantive domain, which

includes events, processes, and phenomenon in the "real" world.

Any research project must contain elements and relations from each of these domains. Thus, it is not possible to

conduct research, without some method, some concept (or set of concepts), and some event or process. Elements

and relations from each of these domains are not all combined simultaneously. Research generally proceeds by

combining two of the domains, to form some structure, and subsequently incorporating (i.e., integrating) the third

domain with the developed structure. With three domains, there are at least three patterns for combining the

domains. Those three ways represent three distinct research paths; and they pose different advantages and

limitations for the investigator.

First, one can combine elements and relations from the conceptual domain with techniques for obtaining observations

and relating these observations--without yet bringing in events and processes from the third (substantive) domain.

This structure is a plan for doing research, that is, a research design but is not a completed research project until it

has been integrated with the substantive domain. This structure will yield a set of intended measures and a plan for

relating (or comparing) these measures. For instance, a researcher may measure an attitude (i.e., an element from

the conceptual domain) by using a semantic differential scale. In addition, a researcher may be interested in

examining the relations among a set of concepts (e.g., attitudes, norms, and intention) by selecting a particular type

of design structure (e.g., within subject factorial design). It is important to note the combination of these two domains

is not a completed research study until it has been integrated with a particular event or process (i.e., elements and

relations from the substantive domain).

Alternatively, a researcher may combine elements and relations from the conceptual domain with events and

processes from the substantive domain. The outcome of this structuring process may reasonably be called a theory.

The mapping of an element (i.e., single concept) from the conceptual domain with an event from the substantive

domain may be termed a theoretical construct. The mapping of a set of constructs with a process or phenomenon

may he termed a hypothesis. For instance, a researcher may combine the concept of salience with information

retrieval (which is the particular substantive event) in order to form a theoretical construct. If a researcher is interested

in a particular process, for instance, a person's decision to purchase generic prescription drugs, the mapping of the

concepts of norms, attitudes, and self-concept to predict this process may be termed a hypothesis. This theory is not

yet a completed research study until it is integrated with techniques and strategies from the methodological domain.

A third pattern is to combine elements and relations from the methodological domain with events and processes from

the substantive domain. Such activity involves mapping methods for making observations onto observable events,and mapping methods for relating (segregating, aggregating, contrasting) sets of observations or events. We can

designate such method-substance mapping as constructing a body of data. For instance, a researcher may use

concurrent protocol techniques to observe an individual's decision process. However, this body of empirical

information is not a completed research study until it has been connected, that is, construed in terms of, elements

and relations from the conceptual domain.

To summarize, given the three domains, three research structures may be developed--a design formed by the

combination of the conceptual and methodological domains; a theory formed by the combination of the conceptual


3/8

and substantive domains; and a body of data formed by the methodological and substantive domains. Each of these

research structures are not a completed study until they have incorporated the third domain.

One path a researcher may follow is to connect events and/or processes from the substantive domain with a

particular design structure. This path (labelled as Path A in Figure l) is an implementation process. For instance, a

researcher may use a Thurstone scale to measure attitude but to implement this scale, it is necessary to se lect a

particular event (e.g., attitudes toward generic drugs). Another researcher may be interested in examining therelations among attitude, norms and intention and use a within subject factorial design. Implementing this design

would require the researcher to connect it with events and/or processes from the substantive- domain, (e.g., the

purchase of generic drugs; the use of birth control pills).

An alternative research path is to build a theory (i.e., develop a set of constructs and some hypothesis) and test this

theory by selecting a particular method. This path (labelled as Path 8 in Figure 1) may be seen as testing a theory.

For instance, a researcher may use a theoretical construct of salience and use reaction time technology to test this

construct. Another researcher may have a hypothesis concerning a particuLar process (e.g., information overload)

and select a method (e.g., behavioral process technology) to Lest this hypothesis.

A third path's researcher may take is to construct a body of data and subsequently select concepts to explain these

data. This path (labelled as Path C in Figure 1) may be seen as an interpretation or explanation process. Forinstance, a researcher may have collected a body of data using concurrent protocol techniques and then attempt to

interpret the data by selecting concepts to categorize various components of the data.

To summarize, there are three paths a researcher may follow: implementing a research design; testing a theory;

explaining a body of data. In one sense, these paths are all the same in that they involve elements and relations from

each of the three domains. In another sense, they are strived at by 3 different sequence of steps and will reflect the

orientation (style) of a particular researcher or an area of research. Adhering to any single path will have certain

advantages---but will also have certain limitations---as with any mono-strategy approach. Figure 1 summarizes the

relations among the domains, structures, and research paths.

In addition to these two steps of structuring and integrating the three domains, there are activities that occur prior to

as well as subsequent to the conduct of the research study that are important to the research process. These two

stages will be referred to as pre-study and post-study activities respectively.

Pre-study activities

Using the scheme presented in Figure 1, the structuring and integrating steps of the research process presuppose a

selection (sampling) of elements and relations from the three domains. Elements and relations from the conceptual

domain consists of concepts and patterns (conceptual relations) among such concepts. In the substantive domain,

elements are observable events and relations are sets of events or processes. Finally, in the methodological domain,

elements are methods/strategies for making observations and relations are techniques for making comparisons

among a set of observed events or concepts.

FIGURE 1

However, such selection (sampling) from the domains implies that there has been some prior exploration/

analysis/understanding of those domains and the kinds of elements and relations that can be drawn from them. Such

explorations are necessary preconditions for conducting a research project but are generally not considered part of

the research process. For instance, in the methodoLogical domain, this prestudy activity leads to the development

and use of specific cools for making observations (e.g., behavioral process technology) and for comparing the

relations among concepts or events (e.g., three mode factor analysis). Similar activities are conducted in both the

conceptual and methodological domains.
http://www.acrwebsite.org/volumes/v09/09041f01.gifhttp://www.acrwebsite.org/volumes/v09/09041f01.gifhttp://www.acrwebsite.org/volumes/v09/09041f01.gif


4/8

Post-study research activities

After completion of one of the research paths previously presented, there are several crucial activities before a study

can become "useful knowledge." These post-study activities are different for the three different research oaths .

For research path A, the researcher combines the conceptual and methodological domains to form a research design

and subsequently combines this structure with the substantive domain. When the researcher has completed thisprocess, they are still faced with a key problem: the extent to which the findings will generalize to or across other

substantive events. This is the problem of the robustness of findings within the substantive domain.

For research path B, the researcher combines the conceptual and substantive domains to form 3 theory and

subsequently tests this theory by selecting a particular method. Upon completion of this process, the researcher is

still faced with a key problem: the extent to which the findings will generalize across other methods. This is the

problem of robustness within the methodological domain.

For research path C, the researcher combines the methodological and substantive domains to form a body of data

and subsequently attempts to explain these data by selecting a set of concepts. Upon completion of this process, the

researcher is still faced with a key problem: the extent to which the findings will stand up when compared with other

elements and relations from the conceptual domain. This is the problem of the robustness of the concepts used to

explain the body of data.

All three post-research activities deal with the adequacy of the elements and relations sampled from the three

domains--and all three are variations of the general concept of external validity. These various forms of external

validity will be discussed in more detail subsequently.

VARIOUS MEANINGS OF VALIDITY

What I hope to establish here is a systematic ordering of different forms and meanings of validity, in relation to one

another and in relation to parts of the research process, so that the full power of those concepts, and the relations

among them may be more clearly seen. Associated with each stage of the research process is a different

fundamental meaning of validity. In Stage 1, validity cakes on the meaning of value. By this, we mean the

importance/relevance/truth of the concepts, methods, and substance selected for study. A second meaning --validityas correspondence --- is associated with Stage 2 of the research process. Many of the traditional forms of validity

hinge on the correspondence between two "entities." For instance, construct validity deals with the correspondence

between a theoretical concept and a particular event; convergent validity with the correspondence of a concept with

two or more methods to measure the concept. The different forms of validity, internal to the research study, are

organized in terms of two categories: (1) validities associated with the structures formed from the combination of

elements and relations from two of the domains and (2) validities associated with the integration of the structure with

elements and relations from the third domain. A third meaning --- validity as robustness, generalizability,

dependability --- is associated with Stage 3 of the research process. The primary focus with this meaning of validity is

determining the scope of a set of findings. As will be discussed, the particular scope a researcher is concerned with

varies as a function of the research path selected in ScaRe 2.

Stage l: Validity 35 Value

Several researchers (e.g., Kaplan, 1964; Gergen, 1973; Kuhn, 1962) have discussed the influence of values on

research. These values guide the selection of standards to be used in a particular research area. This meaning of

validity --- as value ---- is used to identify the concepts to study; the methods to use; and the events/processes to

examine. Generally, the values used in the selection process are determined by the prevailing paradigm. As Kuhn

(1962) has noted, when paradigms change, values will also change. Given the conceptual scheme Presented earlier,

"paradigm shifts" may be regarded as changes in the values involved in selecting elements and relations from each

of the three domains.


5/8

There are many values associated with each domain. For instance, values generally used in the conceptual domain

are: testable, quantifiable, robust, internally consistent; in the methodological domain: significance testing, accuracy,

repeatable, quantifiable; in the substantive domain: observable, what is "real."

The concepts, methods, and events selected for study have changed over time --- in response to the changing

values. For instance, in the earlier 1900's, introspection was an acceptable technique for the study of human learning

(decision making). When Watsonian behaviorism developed, the technique of introspection was discarded because itwould not yield "valid" (i.e., valuable) data. Note that a shift in what was considered valuable resulted in the

elimination of a particular method. Recently, concurrent protocol analysis has been developed as 3 technique for

studying decision making and many aspects of this technique are similar to introspection. Again, a shift in values has

resulted in the use of an "acceptable" technique to yield "valid" data. A similar effect of values may be found in the

conceptual and substantive domains. For instance, early in this century, an instinct was a valid concept to select.

However, given the development of the logical positivist philosophy, concepts were considered of value if they were

quantifiable. Since no quantifiable index of instinct was developed, this concept was no longer considered valid.

Finally, in the substantive domain, events considered "real" will also change as values change. Prior to the Freudian

revolution, unconscious forces were not considered valid (i.e., real). For research within a Freudian perspective,

unconscious forces are "real" and valid events. However, within much of the traditional information processing

literature, unconscious forces are not considered "real" (i.e., they are not valid events). Table 1 contains an example

of a concept, method, and event considered acceptable (and unacceptable) within the information processing area.The reader should note that in other areas (e.g., astrological science), the concept, method, and event listed may be

seen as valid (i.e., valuable).

TABLE 1

STAGE I. PRIOR VALIDITIES

Stage 2: Validity as Correspondence

The second stage of the research process consists of three alternative paths, each with two major steps. As

mentioned earlier, the first step involves combining the three domains pairwise to form a structure --- design, theory,

body of data. Since elements and relations are selected for each structure, there are six different forms of validity to

consider. This set of six validities may reasonably be called "logical validities" since the threats to these forms of

validity occur independent of the completion of the research study (i.e., integration of the structure with the third

domain). These validity issues can be evaluated prior -to conduct of the research study. The second set of forms of

validity are associated with the integration step in Stage 2. Since elements and relations from the third domain will be

integrated with the structure formed in step l, six different forms of validity need to be considered. The potential

threats to these forms of validity occur during the actual conduct of the research study. All 12 of these forms of

validity might reasonably be called "internal validities" since they are internal to the ongoing research process within a

study. Table 2 contains a list of the various forms of validity associated with each structure as well as with each

integration step. Some of the traditional forms of validity will be presented to highlight the association between the

validity issues and the components of the research process. Space limitations preclude a detailed description of each

form of validity and the associated validity threats. A more detailed discussion of these issues may be found in

Brinberg and McGrath (in press).

TABLE 2

STAGE II. INTERNAL VALIDITIES

The matching of concepts and methods at the elements level (i.e., a design structure) yield a set of intended

measures. The correspondence between the methods and concepts may simply be termed instrument validity, that

is, the extent to which a measure can correspond with a concept or set of concepts. One major threat to the validity of
http://www.acrwebsite.org/volumes/v09/09042t01.gifhttp://www.acrwebsite.org/volumes/v09/09042t01.gifhttp://www.acrwebsite.org/volumes/v09/09042t01.gifhttp://www.acrwebsite.org/volumes/v09/09042t01.gifhttp://www.acrwebsite.org/volumes/v09/09043t02.gifhttp://www.acrwebsite.org/volumes/v09/09043t02.gifhttp://www.acrwebsite.org/volumes/v09/09043t02.gifhttp://www.acrwebsite.org/volumes/v09/09043t02.gifhttp://www.acrwebsite.org/volumes/v09/09043t02.gifhttp://www.acrwebsite.org/volumes/v09/09043t02.gifhttp://www.acrwebsite.org/volumes/v09/09042t01.gifhttp://www.acrwebsite.org/volumes/v09/09042t01.gif


6/8

the instrument is the potential confounding of instrument and concept. Campbell and Fiske (1959) describe a

multitrait-multimethod approach that attempts to identify the degree to which there is instrument-concept confounding.

Combining concepts and methods at the relations level deaLs with the correspondence between the design structure

selected and the relations among the concepts. Traditionally, this form of validity has been called internal validity

(Campbell and Stanley, 1966). However, since we have used the term internal validity to refer to all the forms of

validity internal to the research process, the term comparison validity has been developed to replace the traditionalterm of internal validity. The central notion with this form of validity deals with the extent to which planned

comparisons within the study will permit clear inferences about the relations among the concepts. The threats to

comparison validity include: history, maturation, testing, as well as misspecification of the research design (e.g., Judd

and Kenny, in press).

The other forms of validity within the "logical validities" all deal with correspondence between an element/ relation

from one domain with an element/relation from the second domain. Where appropriate, traditional forms of validity

have been included within this conceptual scheme. However, the definition of the form of validity is derived from the

meaning of validity within this step and stage of the research process, that is, the degree of correspondence between

elements and relations selected from two of the three domains.

The second step of each of the three paths involves connecting elements and relations from the third domain with thestructure formed in the first step. These "integration" validities involve the correspondence between elements and

relations selected from the third domain with the structure formed in step 1. For path A, when a design structure is

connected with processes from the substantive domain, the correspondence deals with the execution of the study.

This form of validity may reasonably be called execution validity. Cook and Campbell (1979) discuss threats to this

form of validity that deal with imperfections in conducting (i.e., executing) the study. For example, demand

characteristics and experimenter biases i.e., cues the subject attends to that are incidental to the main emphasis of

the research, will influence the correspondence between the design and the process selected from the substantive

domain. This "artifact" might potentially threaten the validity of the execution of the design.

In addition, the orientation the subject takes to a particular instrument (e.g., response sets) may influence the validity

of the instrument's use. This form of validity may reasonably be called instrument use validity. It.differs from

instrument validity since the potential threats to the use of the instrument only occur when the instrument is used.

The other forms of validity within the integration step of Stage 2 also deal with the correspondence between the

structure formed in step l and the third domain. The reader should note that the validities associated with the

integration step of the research process deal with issues that arise during the conduct of the study --- not prior to the

study. A more detailed discussion of these forms of validity may be found in Brinberg and McGrath (in Dress).

Stage 3: Validity as Robustness

Given the completion of the structuring and integration steps in Stage 2, there is still a need to assess the robustness

(i.e., scope) of the findings. Traditionally, the robustness of research findings deal with the external validity of the

research conclusions. Since three different paths may be pursued in Stage 2, three different external validity issues

may arise. If a researcher develops a design, and subsequently implements this design by selecting substantive

events and processes, they are concerned with the extent to which the findings will generalize across othersubstantive events. This form of validity may reasonably be termed ecological validity. For instance, a researcher

may use a particular design (e.g., within subject factorial design to study the relations among intention, at titude, and

norms) and select an event (e.g., purchasing clothes) to implement this design. Generally, the researcher is

concerned with the extent to which their findings will be robust (i.e., invariant) across other events selected from the

substantive domain (e.g., other behaviors, respondents, settings).

If a researcher first forms a theory and subsequently selects a method to test this theory, they are generally

concerned with the robustness of their findings across different methods. This form of external validity may


7/8

reasonably be called methodological validity. For instance, a researcher may select different methods --behavioral

process technology and reaction time technology --- to test a theory of information overload. The concern with this

form of validity is the extent to which the findings can be reproduced given two or more different methods. As with the

ecological validity, this form of validity involves repeated sampling from the domain of interest -- in this case, the

methodological domain.

Finally, a researcher may collect a body of data and attempt to explain this structure by selecting a set of concepts.The extent to which a set of concepts is adequate to explain the body of data deals with the third form of external

validity -- explanatory validity. For instance, a researcher may use concurrent protocol techniques to construct a body

of data. The robustness involved in attempting to explain this body of data deals with the extent to which the category

system (i.e., set of concepts) selected from the conceptual domain is adequate to explain the body of data. To assess

the generalizability of a -particular category system, a researcher is likely to develop other category systems, that is,

repeatedly sample from the conceptual domain, to determine which set of concepts provides the most accurate

explanation of the body of data. Table 3 summarizes the three different forms of external validity.

TABLE 3

STAGE III. EXTERNAL VALIDITIES

IMPLICATIONS

Historically, the validity concepts described in this paper seemed to have been developed ad hoc (i.e., without a

theoretical foundation) to assess the adequacy of a new component of the research process that had recently been

"discovered" (e.g., the recent development of statistical conclusion validity to deal with the adequacy of inferences

from various data analytic techniques). A serious limitation of the cumulative work concerning the research process

and the associated forms of validity has been the absence of a theoretical/conceptual organization and explanation of

the interrelations among the components of the research process and with the various validity concepts.

One potentially useful feature of the scheme presented in this paper is that the components of the research

enterprise are explicitly specified and interrelated. This may help researchers come to a better understanding of the

advantages and limitations of certain research "patterns." This conceptual scheme was also used to explicitly

associate the forms and meanings of validity with specific components of research process. Rather than add new

validities to describe the research process, the forms and meanings of validity were derived from the general scheme.

This approach has the advantage of providing a theoretical basis for the different validity concepts. A third issue

addressed was the use of this scheme to highlight similarities and distinctions among the validity concepts as well 35

identify new forms of validity researchers need to consider. This conceptual scheme was also used to further

differentiate some forms of validity into several components (e.R., divide external validity into several components).

Finally, I suggest that this framework can help in planning any given study by highlighting the specific validity issues

most germane to and most problematic for that study. It can also help, I believe, in efforts to integrate and assess the

work in a given research area, by pointing up which aspects of the overall validity question represent the most serious

limitations of the cumulative work in that area.

Main Source:http://www.acrwebsite.org/volumes/v09/09042t01.gif
http://www.acrwebsite.org/volumes/v09/09044t03.gifhttp://www.acrwebsite.org/volumes/v09/09044t03.gifhttp://www.acrwebsite.org/volumes/v09/09044t03.gifhttp://www.acrwebsite.org/volumes/v09/09044t03.gifhttp://www.acrwebsite.org/volumes/v09/09042t01.gifhttp://www.acrwebsite.org/volumes/v09/09042t01.gifhttp://www.acrwebsite.org/volumes/v09/09042t01.gifhttp://www.acrwebsite.org/volumes/v09/09042t01.gifhttp://www.acrwebsite.org/volumes/v09/09044t03.gifhttp://www.acrwebsite.org/volumes/v09/09044t03.gif


8/8

REFERENCES

Brinberg, D. and McGrath, J.E. (in press), A network of validity concepts within the research process. In Brinberg, D.

& Kidder, L. (Ed ;). New Directions for Methodology of Social and Behavioral Science: Forms of Validity in Research.

Campbell, D.T. and Fiske, D.W. (1959) Convergent and discriminant validation by the multitrait-multimethod matrix.

Psychological Bulletin, 30, 81-105.

Campbell, D.T. and Stanley, J.C. (1966) Experimental and quasi-experimental designs for research. Chicago: Rand-

McNally.

Cook, T.S. and Campbell, D.T. (1979) Design and analysis of quasi-experiments for field settings. Chicago: Rand-McNally.

Gergen, K.J. (1973) Social psychology as history. Journal of Personality and Social Psychology, 26, 309-320.

Judd, C.M. and Kenny, D.A. (in DreSS). Research design and research validity. In Brinberg, D. & Kidder, L. (Eds).

New Directions for Methodology of Social and Behavioral Science: Forms of Validity in Research.

Kaplan, A. (1964) The conduct of inquiry. Scranton, Pa. Chandler Pub. Co.

Kuhn, T.S. (1962) The structure of scientific revolutions Chicago: University of Chicago Press.

Documents

Validity Concepts in Research