Quasi-Experiments€¦ · Web viewLecture 13 - Meta-Analysis and Quasi-Experiments From The American Heritage College Dictionary. 3rd Ed. (1997). Houghton Mifflin Company. The analysis

Lecture 13 - Meta-Analysis and Quasi-Experiments

From The American Heritage College Dictionary. 3rd Ed. (1997). Houghton Mifflin Company.

The analysis of analyses.

The units of analysis are not individual participant scores but whole study scores.

Two major functions – identical to the functions of all research, but on a larger scale . . .

1) Descriptive: To summarize results (usually relationships between DVs and IVs) across collections of studies

Summaries of relationships across studies used to be done with narrative reviews.

Problems with narrative reviews:

No systematic way of synthesizing the results of individual studies. Conclusions were variable, not reliable. Two reviewers might review the same studies and arrive at different conclusions.

Part of the problem was due to the tendency to report findings as “significant” or “not significant”.

Such a focus on reporting on “significances” left the impression that scientists didn’t know what they were doing – else why didn’t they all report identical “significance” results?

The age of meta-analysis changed the culture to one of reporting quantitative results for studies, rather than simply “significant” or “not significant”.

The emphasis on quantitative results from individual studies lead to the need to summarize those quantitative results, which is part of meta-analysis.

Meta-Analysis and Quasi-experiments Lecture - 1 11/28/17

Why we need descriptive meta-analyses? Wonderlic <---> Conscientiousness scores from 7 studies conducted at UTC . . .


Example of a “descriptive summarize relationships” meta-analysis:

Connolly, J. J., Chockalingam, Viswesvaran. (2000). The role of affectivity in job satisfaction: a meta-analysis. Personality and Individual Differences, 29, 265-281.

This meta-analysis was related to the dispositional theory of job satisfaction – persons are satisfied on the job because that’s the way they are, not because of the type of job they have or the way they’re treated.

The researchers found mean correlation of .49 between a measure of positive affectivity and job satisfaction across studies. Suggests that up to 25% of variance in job satisfaction could be due to individual differences in affectivity, which, presumably, has nothing to do with the job.

This could have been done within a single organization, but it carries much more weight when done across multiple organizations because the mean across organizations is less likely to be influenced by possible weirdness of one organization.

2) Inferential reasons for meta-analyses: To examine factors that affect sizes of relationships across collections of studies. Such factors are called moderators. (More on moderators in PSY 5130.)

Example of a “moderation” meta-analysis:

Griffith, R. W., Hom, P. W., & Gaertner, S. (2000). A meta-analysis of antecedents and correlates of employee turnover: Update, moderator tests, and research implications for the next millennium. Journal of Management, 26, 463-488.

They found that correlations between Pay and turnover depended on whether the organization offered reward contingencies or not. High pay resulted in low turnover only when reward contingencies were offered. Otherwise the correlation was essentially zero. So presence of reward contingencies moderated the pay -> turnover relationship.

This study could not have been conducted in one organization with only one policy. It required multiple organizations, some with reward contingencies and others without them.

Why not conduct just one big study?

1. Because it is likely not possible for a single researcher to conduct a study as big as the combination of many other studies.

Some meta-analyses have Ns of 10,000 or more.

2. Because the use of multiple studies increases generalizability of the results – external validity.

From an interview with Frank Schmidt . . .


Red’d correlations are significant (p < .05).Such sample to sample variation in sample statistics is common.

“Those advocating it usually argue that a single large N study can provide the same precision of estimation as the numerous smaller N studies that go into a meta-analysis. But with a single large N study, there is no way to estimate SD-rho or SD-delta, the variation of the population parameters. This means there is no way to demonstrate external validity. A meta-analysis based on a large number of small studies based on different populations, measurement scales, time periods, and so on, allows one to assess the effects of these methodological differences.”


6 Steps from Understanding Meta-Analysis. Joseph A. Durlack

1. Formulating the research question

Which effects do we wish to summarize?What relationships do we wish to study?

2. Performing the literature search for articles to include in the M-A.

A. Computerized searches of databases, such as PsychInfoB. Manual Searches of journal tables of contents.C. Searches of reference lists of relevant articles.D. Calls or emails to people doing research in the area.E. Being cognizant of the round-file effect – positive bias in rs due to looking only at published articles.

From McAbee, S. & Oswald, F. L. (2013). The criterion-related validity of personality measures for predicting GPA: A meta-analytic validity competition.Literature SearchA systematic search of the literature on the relationship betweenthe Big Five personality traits and GPA in postsecondary studentpopulations was conducted to identify relevant studies for thecurrent meta-analysis. Articles published between January 1992(coinciding with the publication of the NEO-FFI and NEO-PI–R;Costa & McCrae, 1992; and Goldberg’s, 1992, unipolar Markers)and February 2012 were considered for inclusion. Studies werelocated using several online databases, including PsycINFO,ERIC, and Google Scholar. The following keywords were enteredinto the respective databases to locate articles in various combinations:personality, Big Five, Five Factor Model, academic performance,academic achievement, GPA, and grade. Followingdatabase searches, the following online journal records were manuallysearched, starting from 2005 to online-first publications as ofFebruary 2012: Contemporary Educational Psychology, HumanPerformance, Intelligence, Learning and Individual Differences,Journal of Applied Psychology, Journal of Educational Psychology,Journal of Personality, Journal of Personality and SocialPsychology, Journal of Research in Personality, Personality andIndividual Differences. Finally, the references of previous metaanalysesand large reviews on personality and academic performancewere searched to identify additional studies (Noftle &Robins, 2007; O’Connor & Paunonen, 2007; Poropat, 2009; Richardsonet al., 2012; Trapmann et al., 2007). Article abstracts, andMethod sections as necessary, were examined to determinewhether the study included measures of personality related toacademic performance outcomes (e.g., GPA, course grades).


Hunter, J. E., & Schmidt, F. L. (2004). Methods of Meta-Analysis: Correcting Error and Bias in Research Findings. (2nd Ed). Thousand Oaks, CA: Sage.

3. Coding each study.For each study, identifying which value that study has on each independent variable being investigated .

e.g. Suppose the dependent variable is efficacy of therapy.

For each study, code Type of therapy:

Behavioral or TalkingGroup or individualProfessional therapist or lay person

In the McAbee and Oswald study mentioned above, the study was coded for type of Big Five questionnaire used, among other things. On the left is a table from that article describing the coding.


4. Compute the appropriate effect size for each study.

Most meta-analyses use either . . .

A. A mean difference measure analogous to d = (μ1-μ2)/σ

This measure or measures equivalent to it have been defined for many common research designs. Oftentimes, can compute from reports of t or F.

B. A correlation measure, i.e., Pearson r.

Regardless of whether d or r is computed, you must compute the same measure from each study.

When you publish research, please think of meta-analyses.

Can what you report be used for a meta-analysis?


5. Statistical analysis.

From each study, a measure of effect size is computed. May be more than 1 measure if more than 1 dv was computed.

Descriptive Studies

A weighted mean effect size is computed using the formula

(Illustrated assuming the effect size for study i is ri based on a sample of size Ni.

ΣNiri

r-bar = -------------- should be called the weighted r-barΣNi

The variance of the effect sizes is computed using the formula

ΣNi(ri- r-bar)2

S2r = ---------------------

ΣNi

The mean and variance are adjusted for

A) Reliability of the independent variableB) Reliability of the dependent variableC) Range restriction of the dependent variable.

The variance is also adjusted for

D) Sampling error.

These adjustments are somewhat complex, but doable with a good spreadsheet program..

There are several meta-analysis programs available, some for free.


Skipped this page in 2017

6. Testing for existence of systematic differences between studies Cochran’s Q statistic

NX2

K-1 = ------------------ * S2r

(1-r-bar2)2

If the X2 is not significant, then it will be assumed that the effect sizes came from only one population of studies and the mean is an estimate of the mean effect size of that population of studies.

If the X2 is significant, then it must be assumed that there are two or more populations represented by the group of studies, and some attempt to distinguish among them must be made.

Cochran, W. G. (1954). The combination of estimates from different experiments. Biometrics, 10, 101–129.

Higgins and Thompson’s I2 statistic

Higgins and Thompson’s (2002) I2 value indexes the proportion of variance due to between-study differences.

Higgins, J. P. T., & Thompson, S. G. (2002). Quantifying heterogeneity in meta-analysis. Statistics in Medicine, 21, 1539–1558. doi:10.1002/ sim.1186

Moderation effects: Identifying sources of systematic differences between studies.

Individual studies are treated as rows of a data matrix.

Individual study effect sizes are the scores that are analyzed.

Each column contains either an effect size dependent variable or a potential moderating variable.

t-tests, analysis of variance, or regression analysis may be used to examine the relationship between the dependent variables and potential moderators.

Since the dependent variables in these analyses are measures of relationships, a significant t, or F, or significant r in a regression analysis is an indication of moderation.


Examples of results of meta-analyses:

From . . .

Most people report the operational validity when estimating a population correlation for selection situations.

They report the true-score correlation when estimating a population correlation in contexts other than selection.


Estimate of population correlation adjusted for predictor and criterion reliability.

For Selection purposes, estimate of population correlation adjusted only for criterion reliability.

Weighted mean of observed, unadjusted correlations.

Example 2 from . . .


Ignore this – it refers to parts of the table not copied here.

THE ROLE OF EMOTIONAL INTELLIGENCE IN LEADERSHIPEFFECTIVENESS: A META-ANALYSIS

A Thesis Presented for theMaster of Science Degree

The University of Tennessee at Chattanooga

Ashleigh D. FarrarMay 2009

AbstractLeaders are an essential element of the business world. While good leaders can provide many benefits for an organization, unsuccessful leaders can be detrimental. The notion that emotional intelligence plays a part in whether a leader is effective or not effective has recently been introduced. This study sought to unify the literature evaluating the possible link between emotional intelligence and leadership effectiveness. Meta-analytic techniques were used to analyze this relationship. Results revealed that overall, there is a positive relationship between emotional intelligence and leadership effectiveness. Also, while the type of emotional intelligence measure used served as a moderator to this relationship, a second and third meta-analysis supported the overall positive relationship of emotional intelligence and leadership effectiveness for each type of EI.

ResultsThe central aim of the present study was to examine the overall relationship of EI and leadership effectiveness. The initial meta-analysis was conducted using all of the included studies. The results of this meta-analysis are provided in Table 1. A total of 20 correlations were used from 20 studies, with a total sample size of 3,295. After correcting for unreliability in both EI and leadership effectiveness measures, the sample-size-weighted mean rho linking the constructs was .458. The 80% credibility interval did not include zero, indicating that there was a relationship between EI and leadership effectiveness. These results supported Hypothesis 1.

Ashley’s data


Ashley’s results

Table 1: All Studiesk 20Total Sample Size 3295Mean Rho 0.457 Correlation of EI with Leadership EffectivenessVariance of Rho 0.02880% Credibility .24-.67

Table 2: EI Mixed Model Measuresk 12Total Sample Size. 2265Mean Rho 0.427Variance of Rho 0.03080% Credibility .20-.65

Table 3: EI Ability Model Measuresk 8Total Sample Size. 1030Mean Rho 0.536Variance of Rho 0.01380% Credibility .39-.68


From Frank Schmidt’s presentation at the 2012 RCIO conferenceTable 1. Selection methods for job performance

Selection procedures/predictors

Operationalvalidity

(r)

Multiple R Gain in

validity% gain in validity

StandardizedRegression weights

GMA Supple-ment

GMA testsa .65Integrity testsb .46 .78 .130 20% .63 .43Employment interviews (structured)c .58 .76 .117 18% .52 .43Employment interviews (unstructured)d .60 .75 .099 15% .48 .41Conscientiousnesse .22 .70 .053 8% .67 .27Reference checksf .26 .70 .050 8% .65 .26Biographical data measuresg .35 .68 .036 6% .91 -.34Job experience h .13 .67 .023 4% .66 .17Person-job fit measuresi .18 .67 .020 3% .64 .16SJT (knowledge)j .26 .67 .018 3% .76 -.19Assessment centersk .37 .66 .014 2% .78 -.19Peer ratingsl .49 .66 .013 2% .55 .16T & E point methodm .11 .66 .009 1% .65 .11Years of educationn .10 .66 .008 1% .65 .10Interestso .10 .66 .008 1% .65 .10Emotional Intelligence (ability)p .24 .65 .007 1% .70 -.11Emotional Intelligence (mixed)q .24 .65 .005 1% .63 .09GPAr .34 .65 .004 1% .71 -.10Person-organization fit measuress .13 .65 .004 1% .64 .07Work sample testst .33 .65 .003 0% .69 -.07SJT (behavioral tendency)u .26 .65 .000 0% .64 .03Emotional Stabilityv .12 .65 .000 0% .64 .02Job tryout procedurew .44 .65 .000 0% .63 .02Behavioral consistency methodx .45 .65 .000 0% .64 .02Job knowledgey .48 .65 .000 0% .65 -.01

Table 2. Selection methods for training performance

Selection procedures/predictors

Operationalvalidity

(r)Multiple

RGain invalidity

% gain in validity

StandardizedRegression weights

GMA Supple-ment

GMA testsa .67Integrity testsb .43 .78 .109 16% .65 .40Biographical data measuresc .30 .74 .073 11% 1.04 -.50Conscientiousnessd .25 .73 .061 9% .69 .29Employment interviewse .48 .72 .051 8% .57 .28Reference checksf .23 .71 .038 6% .67 .23Years of educationg .20 .70 .029 4% .67 .20Interestsh .18 .69 .024 4% .67 .18Peer ratingsi .36 .67 .002 0% .70 -.06Emotional Stabilityj .14 .67 .001 0% .66 .03Job experience (years)k .01 .67 .000 0% .67 .01

Note. Operational Validity estimates in parentheses are what is reported in Schmidt and Hunter (1998, Table 2). Selection procedures whose operational validity is equal to and greater than .10 are listed in the order of gain in operational validity.Unless otherwise noted, all operational validity estimates are corrected for measurement error in the criterion measure and indirect range restriction (IRR) on the predictor measure to estimate operational validity for applicant populations.


!

Over GMAwith GMA

Over GMAwith GMA

McAbee & Oswald’s results – Prediction of gpa Skipped to end of lecture in 2015, 2016



Quasi-Experiments Not in 2016, 2017True Experiment

Design in which individual participants are randomly assigned to conditions.

Quasi-Experiment

Anything else.

Most often: Designs in which different treatments are assigned (perhaps randomly) to already existing groups.

Always: Designs comparing Subject variables, such as gender, graduate program, age, any prior condition

Sometimes: Designs for which participants could be randomly assigned but are not for one reason or the other.

Some authors differentiate designs for which assignment of conditions to groups is possible (e.g., Training programs to different buildings) from those for which it is not (e.g., Gender).

Pretest-Posttest with Nonequivalent Groups Design Also called the Nonequivalent Control Groups Design with Pretest (NECG with Pretest Design)

The most frequently discussed Quasi-Experimental Design

The design involves pretests on both groups, making one the experimental group and the other the control group, then taking a post observation of both.

Diagrammed as

Pre Condit Post

O1 XE O2

-------------------------------- The line signifies nonequivalence.O1 XC O2

The true experimental counterpart is the Randomized Groups Design. (RG Design)

O1 XE O2

RO1 XC O2

Note that in the Randomized Groups design, a pretest is not necessary, because the groups are theoretically equivalent prior to the administration of research conditions.

But it's certainly possible and advisable to take a pretest in the Randomized Groups design. With the appropriate analysis, having a pretest can increase power of the comparison between the groups.


Campbell & Stanley Cook & Campbell Shadish, Cook, & Campbell

Examples

1. Program to compare efficacy of a new diet program vs. an existing diet program.

O1 is weight prior to beginning the diet

O2 is weight after the diet

XE is the new diet

XC is the old diet

Obviously, if participants could be randomly assigned to the diet type, then this would be a Randomized Groups design.

But if individual participants could not be randomly assigned, then the design would be NECG.

Pre Condit Post

Prior Weight New Diet Post Weight------------------------------------------------------------ :Prior Weight Old Diet Post Weight

2. Comparison of I-O vs. Research program students in statistics.

O1 is knowledge of statistical concepts taught in the course prior to the start of the semester

O2 is the same test of knowledge of statistical concepts at the end of the course

XE is the I-O program (or it could be the Research program)

XC is the Research program (or it could be the I-O program)

Pre Condit Post

Pretest I-O Posttest--------------------------------------Pretest Research Posttest


A few outcomes of the NECG design and their interpretations . .Assuming pretest and posttest are commensurable

1. Best possible Outcome

Analysis: Between Subjects / Within Subjects analysis of variance

Group Factor: Level 1 = Experimental GroupLevel 2= Control Group

Time Factor. Level 1 = Pretest measureLevel 2 = Posttest measure

1. Main Effect of Time Factor could be found if both groups increase performance from pre- to posttest.2. Main Effect of Group Factor could be found depending on the specific results.3. But Interaction is the key. If the interaction is significant and of the form shown below, this would indicate that the Experimental group increased significantly more than the Control group.

Moreover, since the Experimental group performance was below that of the Control group on the pretest, any explanation of the result in terms of pre-existing differences would be difficult.


C2C1

E2

E1

Pre PostTreatment

2. 2nd best possible outcome

One Between Subjects / One Within Subjects analysis: Significant effects

1. Main Effect of Time Factor might be found if both groups increased from pre- to posttest.2. Main Effect of Group Factor might be found in some circumstances.3. Interaction is the key result here, though.

Again, an explanation in terms of pre-existing groups would be difficult to accept in view of the similarity of performance on the pretest.


E

E

C C

3. The Dark side of the NECG with Pretest design: Regression to the Mean or Maturation

One Between Subjects / One Within Subjects analysis:

Significant effects

1. ME of Time possibly2. ME of Group, possibly3. Interaction is significant.

The point being made here is that you can't blindly follow the significant effects. You must consider the pattern of differences. Even if the interaction were significant, it might be argued that the interaction was due to regression to the mean or to maturation effects.

Maturation Explanation

RTTM ExplanationChildren enrolled in Head Start program increased more from pretest to posttest than children not enrolled. But the result may have been due to differential regression to the mean. The Head Start children may have been from the lower tail of the distribution on the pretest. They would be expected to score higher on the posttest simply due to regression to the mean. Those in the control group may have been more likely to be near the middle of the distribution, and thus would not be expected to change much from pretest to posttest.


E

E

C C

E

E

C

C

C

4. Main effects only outcome

One Between Subjects / One Within Subjects analysis: Significant effects

1. ME of Time possibly2. ME of Group, possibly3. Interaction not significant.

Again, you can't blindly follow the significant effects. You must consider the pattern of differences. This outcome signifies nothing. The control and treatment groups increased by the same amount. The difference between them at pretest was the same as at posttest.

Example from Shadish, Cook, & Campbell (2002).

Carter, Winkler, & Biddle (1987) evaluated the effects of the National Institutes of Health (NIH) Research Career Development Award, a program designed to improve the research careers of promising scientists.

Those who received the RCDA did better on the posttest than those who did not receive the award.

But, the difference on the pretest was about the same as the difference on the posttest,

So the final difference may have been due to pre-existing differences –

Those who got the award were better to start with (which is why they got the award) and they were about as much better at the end of the study.


E

E

C

C

E

Documents

Quasi-Experiments€¦ · Web viewLecture 13 - Meta-Analysis and Quasi-Experiments From The American Heritage College Dictionary. 3rd Ed. (1997). Houghton Mifflin Company. The analysis