Generalizability Theory (GT)

Generalizability(G) Generalizability(G) TheoryTheory

Presenter: Saeed Majidi M.A studentE-mail: [email protected]

1/15

Key Terms:Key Terms:In defining G-Theory there will be

several terms introduced and is briefly explained here:

facet A characteristic of a measurement procedure such as task, occasion, observer that is defined as a potential source of measurement error. In fact Facets are similar to the “factors” used in analysis of variance, and may include persons, raters, items/forms, time, and settings among other possibilities

condition The levels of a facet (e.g., task 1, task 2, …, task k).

generalizability (G) study A study specifically designed to provide estimates of the variability of as many possible facets of measurement as economically and logistically feasible considering the various uses a test might be put to.

decision (D) study A decision study uses information from a G study to design a measurement procedure that minimizes error for a particular purpose.

2/15

Key Terms |Introduction | Specifications & Purposes | GT Procedure | GT vs. CTT

Key TermsKey Terms

• universe of admissible observations All possible observations that a test user would take as considerable, acceptable substitutes for the observation in hand.

• universe of generalization The conditions of a facet to which a decision maker wants to generalize.

• universe score (denoted as µp) is defined as the expected value of a person’s observed scores over all observations in the universe of generalization (analogous to a person's "true score" in classical test theory)

• variance component The variance of an effect in a G study.

3/15


IntroductionIntroduction

Generalizability (G) theory is a statistical theory for evaluating the dependability (‘reliability’) of behavioral Measurement (Cronbach, Gleser, Nanda, & Rajaratnam, 1972). It expands classical test theory to include multiple sources of error and explicitly connects measurement operations to the purpose of measurement.


4/15

Specifications & purposesSpecifications & purposes In G theory, Error is separated into pieces,

each of which can be estimated( If we collect the data properly).It lets you to separate the error due to differences in measurement conditions.

For example, four counselors rate drug clients over two sessions on coping. We may want to know the error due to counselors and to sessions. We could ask more specific questions, such as how well does a rating by counselor 1 in session 1 generalize to the average of 4 counselors' ratings in session 2?


5/15

Specifications & purposesSpecifications & purposes G theory has the characterization of the multiple

sources of measurement error. G theory considers both systematic and

unsystematic sources of error variation and disentangles them simultaneously.

G theory assumes only randomly parallel tests sampled from the same universe.

G theory pinpoints the sources of measurement error, disentangles them, and estimates each one.

It quantifies the amount of error caused by each facet and interaction of facets


6/15

G Theory proceduresG Theory procedures


7/15

G Theory proceduresG Theory procedures Generalizability (G) study:In order to evaluate the

dependability of behavioral measurements, a G study is designed to isolate particular sources of measurement error. The facets that the decision maker might want to generalize over (e.g., items, occasions) must be included.

8/15



Universe of Generalization:

The universe of generalization is defined as the set of conditions to which a decision maker wants to generalize. A person's universe score (denoted as µp) is defined as the expected value of his or her observed scores over all observations in the universe of generalization (analogous to a person's "true score" in classical test theory).


9/15

G Theory proceduresG Theory procedures Decomposition of Observed Score:

With data collected in a G study, an observed measurement can be decomposed into a component or effect for the universe score and one or more error components.

The relative magnitudes of the estimated variance components provide information about potential sources of error influencing a behavioral measurement. Statistical tests are not used in G theory; instead, standard errors for variance component estimates provide information about sampling variability of estimated variance components.

10/15



DECISION STUDIES:

Generalizability theory distinguishes a decision (D) study from a G study. The G study is associated with the development of a measurement procedure and the D study uses information from a G study to design a measurement that minimizes error for a particular purpose.

Note 1: In planning a D study, the decision maker defines the universe that he or she wishes to generalize to, called the universe of generalization, which may contain some or all of the facets and conditions in the universe of admissible observations.

Note 2: In the D study, decisions usually will be based on the mean over multiple observations rather than on a single observation.


11/15

G Theory proceduresG Theory proceduresTypes of Decisions and Measurement

Error:G theory recognizes that the decision maker might want to

make two types of decisions based on a behavioral measurement:

12/15


Types of Decisions and Measurement Error:Note: Relative error variance is calculated as the interactions between the facets of our measurement designs and the object of measurement.

Note II: The estimate of reliability that is calculated by Relative Error Variance is called Generalizability Coefficient.

Note III: Absolute error variance is calculated as variance components for the interactions between the facets and the object of measurement.

Note IV: The estimate of reliability that is calculated by Absolute Error Variance is called Phi Coefficient that is generally an index of dependability to further distinguish it from CTT reliability.


13/15

CTT Vs. GTCTT Vs. GT I already mentioned several difference of GT and CTT but here I want to

only pinpoint several important ones: As I mentioned earlier GT extends the framework of classical test theory

in order to take into account the multiple sources of variability that can have an effect on test scores:

The focus of classical test theory (CTT) is on determining error of the measurement. Perhaps the most famous model of CTT is the equation where X is the observed score, T is the true score, and e is the error involved in measurement. Although e could represent many different types of error, such as rater or instrument error, CTT only allows us to estimate one type of error at a time. Essentially it throws all sources of error into one error term. This may be suitable in the context of highly controlled laboratory conditions, but variance is a part of everyday life. In field research, for example, it is unrealistic to expect that the conditions of measurement will remain constant. Generalizability theory acknowledges and allows for variability in assessment conditions that may affect measurements. The advantage of G theory lies in the fact that researchers can estimate what proportion of the total variance in the results is due to the individual factors that often vary in assessment, such as setting, time, items, and raters.


14/15

CTT Vs. GTCTT Vs. GT G theory takes into account how the consistency of

outcomes may change if a measure is used to make absolute versus relative decisions

G-theory allows the investigator to decide which facets will be of relevance to the assessment context of interest. This is referred to as the universe of admissible observations.


15/15

Sources:Sources: [1] Brennan, R.L. (2001). Generalizability Theory, Springer-

Verlag, New York. [2] Cronbach, L.J., Gleser, G.C., Nanda, H. & Rajaratnam, N.

(1972). The Dependability of Behavioral Measurements, Wiley, New York.

[3].Lynch, B. K. & McNamara, T. F. (1998). Using G-theory and Many-facet Rasch measurement in the development of performance assessments of the ESL speaking skills of immigrants. Language Testing 15(2), 158-180. doi: 10.1177-026553229801500202

[4] Shavelson, R.J. & Webb, N.M. (1991). Generalizability Theory: A Primer, Sage Publications, Newbury Park.

[5] Shavelson, R.J. & Webb, N.M. (1981). Generalizability theory: 1973–1980, British Journal of Mathematical and Statistical Psychology 34, 133–166.

Thank you for Thank you for paying attentionpaying attention

Documents

Generalizability Theory (GT)