MANOVA/ANOVA (July 2014 updated)

July 2014 updated

Prepared by Michael Ling Page 1

QUANTITATIVE RESEARCH METHODS

SAMPLE OF

ANOVA/MANOVA ANALYSIS

Prepared by

Michael Ling

Reference: McElroy, J. C, & Crant, J. M. (2008). “Handicapping: The effects of its

source and frequency,” Journal of Applied Psychology, Vol. 93, 893-900.

July 2014 updated


I. INTRODUCTION

Prior to the event of performance, handicapping meant setting up to “deflect

blame away” in case of failure and “accept credit for success by having overcome

difficulty” in case of success. Handicapping was considered a “direct impression

management tactic” in relation to the “credit or blame attributed to and level of affect”

allocated towards the actor by an observer.

Past research in handicapping was limited to single instance of self-

handicapping. This paper contributed to handicapping research by extending a

single instance self-handicapping event into a much broader framework where

multiple instances of handicapping and indirect handicapping were evaluated.

The research question was to which extent that frequency and sources of

handicapping could influence reactions of an observer and his/her perceived

credibility about the actor concerned. These effects were further examined in the

context of success or failed performance. Three hypotheses were proposed and

ANOVA/MANOVA procedures were used as the research method.

July 2014 updated


II. SUMMARY

A 2x2x2 factorial ANOVA/MANOVA design was used to examine the main and

interaction effects of three factors - handicap source (self or other), frequency (once

or multiple times) and performance (success or failure) – on credit/blame,

interpersonal affect and perceived credibility. The sample size was 246 with an

average cell size of 28 to 34, and participants were randomly assigned to one of

eight cells. Manipulation checks were conducted on the measurement scales to

ensure their consistency and reliability.

Partial support was found in hypothesis 1, which stated that frequency

moderated the source of handicapping and the observer impressions where (i)

impressions (credit/blame and interpersonal affect) were more favourable in third-

party handicaps than self-handicaps; and (ii) increased frequency reduced

impressions more strongly in self-handicaps then third-party handicaps. The results

showed that there was (i) a significant Source*Frequency effect (p < .05) on the set

of dependent variables (MANOVA); (ii) a significant Source*Frequency effect (p<.05)

on interpersonal affect (ANOVA); and (iii) a moderating effect of Frequency on

Source and Interpersonal affect. However, no significant Source*Frequency

interaction effect was found on credit/blame (ANOVA).

Partial support was found for hypothesis 2, which stated that frequency

moderated the source of handicapping and the perceived credibility of the

handicapping information where (i) handicaps were more credible in third-party

handicaps than self-handicaps; and (ii) increased frequency reduced credibility more

strongly in self handicaps than third-party handicaps. The results showed that there

July 2014 updated


was Source*Frequency interaction effect (p < 0.05) on Credibility (ANOVA). Despite

the significant interaction effect, Eta-squared (η2) was 2 percent. Contrary to

expectations, self-handicaps were found more credible than third-party handicaps in

single handicaps.

Partial support was found for hypothesis 3, which stated that performance

moderated the source and frequency of handicapping and the observer impressions

and perceived credibility where (i) impressions were more favourable following failed

than successful performance; and (ii) perceived credibility was more favourable

following failed than successful performance. It was found that multiple handicaps

decreased credibility for all handicaps except in the case of third-party handicaps

following failed performance.

July 2014 updated


III. CRITIQUE

The use of ANOVA/MANOVA procedures were appropriate because it

examined the main and interaction effects of independent categorical variables

(Source, Performance and Frequency) on multiple dependent interval variables

(Credit/blame, Interpersonal affect and Credibility) by comparing group differences.

The sample design suggested that a balanced design was adopted. In

MANOVA, the cell sizes should be roughly equal because normality of the dependent

variables was important.

The coefficient alpha values for the measurement scales were high and the

outcomes of the manipulation checks were satisfactory.

A key weakness of the paper was that it did not specify the assumptions of the

ANOVA/MANOVA tests. For example,

i. No results were provided for univariate and multivariate normality that

the dependent variables, and their combinations, were distributed

normally. No Scatterplots were checked for linear relationships among

the dependent variables.

ii. No results of multicollinearity were provided to examine the correlations

of the dependent variables.

iii. No results of multivariate outliners, such as Maximum Mahalanobis

Distance, were provided.

iv. No results of homogeneity of the covariance matrices, such as Box’s M

test, were provided. If Box’s M test showed the covariance matrices

were significantly different across levels of the independent variables, it

July 2014 updated


indicated an increased possibility of Type I error and hence there was a

need to use a smaller error region than p < .05.

v. No evidence was provided for the independence of observations. As

the questionnaires were completed by the respondents in their

workplaces, it was probable that the respondents might have discussed

the questions amongst themselves. Data quality would be a potential

issue.

Hypothesis 1

Despite that the Source*Frequency interaction effects were reported

significant in Hypothesis 1, there were a few areas of concerns:-

i. Wilks’s λ was a measure of the percent of variance in the dependent

variables that was not explained by differences in the level of the

independent variable. Wilks’s λ for Source*Frequency was .97 in the

MANOVA test, which meant that 97 percent of variance was still

unexplained.

ii. The Eta-squared for Source*Frequency was 3 percent, which meant

the percent of total variance in the dependent variable explained by the

variance between groups formed by the independent variable was only

3 percent, which not very impressive despite a significant result.

iii. If the MANOVA omnibus test was significant, it was common practice to

conduct separate ANOVAs. However, considerations such as

Bonferroni adjustment should have been given to adjust the

July 2014 updated


significance cut-off level of ANOVAs in order to minimize the probability

of Type I error.

Hypothesis 2

An area of concern was that credibility had not been included in the MANOVA

omnibus test. The authors argued that they “felt that these reactions to handicapping

were conceptually distinct enough to preclude us from including all of the dependent

variables in just one… (MANOVA)”. This argument was not relied upon any

substantive literature but was based on what the authors “felt” it should be. On the

contrary, it was possible that credibility, credit/blame and affect were correlated. If

this was the case, the independent ANOVA would have ignored their interrelations

and substantial information would be lost. The resultant p values for the 1-way

independent ANOVA would have been incorrect.

The Bartlett’s Test of Sphericity could have been used to test the hypothesis

that the population correlation matrix was an identity matrix. If the determinant was

small, independence of the variables would be rejected and there was a need for

MANOVA.

Again, the Eta-squared for Source*Frequency was 2 percent, which was not

impressive at all despite a significant result.

No discussions were provided on the adjustments in relation to the treatment

of experiment-wise and testwise errors in the univariate ANOVA tests.

Hypothesis 3

July 2014 updated


The MANOVA omnibus result failed to find significance in the three-way

interactions effect of Performance*Source*Frequency on impressions. A three-way

interactions effect was found on credibility in the ANOVA test. Despite the interaction

effects were reported significant, a concern was that Eta-squared (η2) of the three-

way interactions effect was very low (0.02) in the ANOVA result.

Although all forms of handicaps were found more credible following failure

than success, their mean differences were very small in the range of 0.13 to 0.71, as

shown in the table below. A larger sample size could have been considered to test if

reasonable effect sizes could be achieved. Proper measures of effect sizes such as

Cohen’s d, as the difference in group mean divided by the pooled standard deviation,

should have been provided to measure effect sizes.

Success

Performance

Failure

Performance

Mean

Difference

Self-handicaps Single 4.11 4.82 0.71

Self-handicaps Multiple 3.65 3.92 0.27

Other handicaps Single 3.98 4.11 0.13

Other handicaps Multiple 3.61 3.92 0.31

Note: the mean values above were extracted from Figure 3.

Overall Assessment

Though the authors reported partial support of the three hypotheses, the low

values of Wilks’s λ and Eta-squared created some concerns. A large portion of the

variance was not accounted for in the model. Not all relationships, despite their

importance, suggested in the hypotheses were found significant. The assumptions of

July 2014 updated


ANOVA/MANOVA procedures were left untested. The exclusion of Credibility in the

MANOVA test was not adequately supported. No consideration was given to the

need for adjustment of significance criteria in ANOVA. Contrary evidence was found

with respect to self-handicaps and multiple handicaps in hypotheses 2 and 3

respectively. As a result, the conclusions drawn about the hypotheses were not

totally trustworthy.

July 2014 updated


III. CONCLUSION

The contribution of the paper rested on advancing a theoretical framework in

handicapping research, from single instance self-handicapping into multiple

instances of handicapping and indirect handicapping.

The authors started off on the right track by using a full factorial design and

ANOVA/MANOVA procedures as the research method to establish their hypotheses.

Unfortunately, the results were not quite satisfactory as all hypotheses were only

partially supported and the variances explained by the independent variables were

very negligible.

The research could have improved by addressing the concerns raised in this

critique. In particular, assumptions of the ANOVA/MANOVA procedures needed to

be tested and data quality needed to be reinforced, especially independence of

samples; a redesign of the scenario-based experiment methodology, for example, by

replacing questionnaire with testing respondents in a laboratory setting. Given that

there was only partial support of the three hypotheses, an important step would be a

review of the handicapping model from the theoretical perspective.

Business

MANOVA/ANOVA (July 2014 updated)