BRIEF REPORT

INTER-INFORMANT AGREEMENT ON THE MOTIVATION ASSESSMENT SCALE: ANOTHER FAILURE TO

REPLICATE

Sarah Thompson and Eric Emerson

Hester Adrian Research Centre, University of Manchester, Oxford Road, Manchester M13 9PL

Abstract The present study evaluated inter-informant agreement on the Motivation Assessment Scale across 42 topographies of challenging behaviour in five children with severe intellectual disabilities. The results revealed unacceptably low levels of inter-rater reliability on all sub-scales of the MAS and unacceptably low levels of inter-informant agreement on the purported behavioural function of the rated challenging behaviours.

A diverse range of indirect, observational and experimental methods have been devised to determine the behavioural functions of challenging behav- iours shown by people with severe intellectual disabilities (see Emerson, 1995). The Motivation Assessment Scale (MAS: Durand & Crimmins, 1988, 1992), probably the most commonly cited indirect assessment pro- cedure, purports to identify the general nature of reinforcement contin- gencies responsible for maintaining an individual’s challenging behaviour.

The MAS is a 16-item 7-point Likert scale in which the informant rates the probability of specific behaviours occurring in contexts which are indicative of the operation of particular maintaining contingencies (e.g. ‘Does the behaviour occur whenever you stop attending to himher?’). Each item contributes to one of four subscales designed to measure the general processes of sensory or perceptual reinforcement (Sensory), nega- tive reinforcement (Escape), positive social reinforcement (Attention) and

0952-9608/95/03/0203-6$1 O.OO/O MENTAL HANDICAP RESEARCH

0 1995 BILD Publications Vol. 8, No. 3, 1995

203

204 MENTAL HANDICAP RESEARCH

positive material reinforcement (Tangible). While originally developed and validated in relation to the functional assessment of self-injurious behav- iour (Durand & Crimmins, 1988), the authors have more recently re- commended its use for the assessment of a wide variety of challenging behaviour (Durand, 1990; Durand & Crimmins, 1992).

In recent years, conflicting evidence has accumulated concerning the utility of the MAS as a clinical tool. Two studies have reported good cri- terion validity (Durand & Carr, 1992; Durand & Crimmins, 1988) and another two have reported that the scale possesses a coherent factor struc- ture (Bihm et al., 1991; Singh et al., 1993). However, several studies have failed to replicate Durand & Crimmins’ (1988) original reports of accept- able levels of inter-rater reliability (Lawrenson, 1993; Newton & Sturmey, 1991; Sigafoos, Kerr & Roberts, 1994; Zarcone et al., 1991). The present study sought to extend previous research in two ways. Firstly, it investi- gated levels of inter-informant agreement on the MAS with regard to the identified behavioural function (or behavioural diagnosis) of target behav- iours. Secondly, it was applied across multiple forms of challenging behav- iours shown by children with severe learning disabilities and severely chal- lenging behaviours.

Method

Five children (three girls and two boys) participated in the study. All children were attending a behaviourally-orientated residential special school for children with severe learning difficulties and severe challenging behaviour after being excluded from at least one ‘mainstream’ special school. The mean age at assessment was 12 (range 8-16). Between 5 and 12 distinct topographical forms of challenging behaviour were identified for each child. These included various forms of self-injury (1 7 behaviours), aggression (1 7 behaviours), disruption (4 behaviours) and stereotypy (4 behaviours) . Two informants independently completed the MAS regarding each of the child’s behaviours in two settings: the school classroom and the residential areas of the school. This resulted in the administration of the MAS across 42 distinct behaviours in two settings by a total of 10 informants. All informants had worked directly with the child for a mini- mum of 12 months.

Scoring and interpretation of the MAS results followed the recommen- dations of Durand & Crimmins (1992). That is, behavioural function was ascribed on the basis of the highest mean subscale score. Multiple func- tions were ascribed on the basis of ties between the highest subscale scores or if the highest mean score was within 0.50 of the second highest mean subscale score. An additional scoring method proposed by Emerson &

MOTIVATION ASSESSMENT SCALE 205

Bromley (in press) was also applied. In this approach function was ascribed if the mean subscale score was equal to or greater than a cut-off point of three which would be equivalent to the behaviour occurring half the time in situations indicative of a particular function (Emerson & Bromley, in press).

Inter-rater reliability was evaluated by calculating intra-class correlation coefficients for each of the MAS’S four sub-scales (Dunn, 1989). Inter- informant agreement on the purported behavioural function of the rated behaviours was analysed by calculating the percentage agreement between raters. Overall agreement was calculated on the basis of exact correspon- dence between the two raters on the function or multiple functions of a given behaviour. Function-specific agreement was calculated for the occur- rence and non-occurrence of each of the four potential functions. That is, an agreement was scored if the two raters agreed that a particular function did (occurrence) or did not (non-occurrence) occur, regardless of the pres- ence of any additional behavioural functions identified by either rater. Per- centage agreement for both occurrence and non-occurrence was calculated by dividing the number of agreements between raters by the number of agreements plus disagreements.

Using the Durand & Crimmins (1992) scoring method 23%-33% of behaviours (range across raters and settings) were allocated to the Sensory category, 43%-67% to Escape, 27%-38% to Attention and 54%-83% to Tangible. Between 48%-67% of behaviours were considered multifunc- tional. Results using the alternative cut-off scoring method were Sensory (13%-20%), Escape (37%-54%), Attention (24%-35%), Tangible (50%- 7 2 %) , multifunctional (43 %-6 3 %) .

Inter-rater reliability and inter-informant agreement across subscales and settings are presented in Table 1.

As can be seen, unacceptably low levels of inter-rater reliability were obtained for all four sub-scales for all behaviours and, separately, for self- injurious and aggressive behaviours. Overall agreement between informants did not exceed 30%. Function specific occurrence agreement ranged from 0% to 80% and percentage agreement on the non-occurrence of specific behavioural functions ranged from 8% to 89%. Notable is the variation in levels of both occurrence and non-occurrence agreement across the four subscales (see also Newton & Sturmey, 1991).

206 MENTAL HANDICAP RESEARCH

Table 1 Inter-rater reliability and inter-informant agreement across topographies, settings and scoring methods

Sub-scale or Ascribed Function

Attention Sensory Tangible Escape Overall

Inter-Rater Reliability (Intra-class correlation coefficients)

all behaviours, all settings 0.0402 0.4208 0.6475 0.5254 nla all behaviours, classroom setting -0.0010 0.3443 0.6918 0.6218 nla all behaviours, residential setting 0.0728 0.4974 0.6059 -0.1028 nla self-injury, all settings -0.2114 -0.1342 0.4232 0.1647 nla aggression, all settings -0.1323 0.1675 0.6582 0.5310 nla

Percentage Inter-Informant Agreement (Occurrence): Durand & Crimmins scoring method

all behaviours, classroom setting 14.3% 42.3% 69.7% 46.7% 27.3% all behaviours, residential setting 20.0% 54.5% 58.3% 46.7% 27.3%

all behaviours, all settings 17.1% 47.9% 62.9% 46.7% 27.3%

self-injury, all settings 6.3% 25.0% 80.6% 58.8% 22.5% aggression, all settings 15.8% 43.4% 44.8% 22.7% 20.6%

Percentage Inter-Informant Agreement (Occurrence): Emerson & Bromley scoring method

all behaviours, classroom setting 27.8% 62.5% 47.0% 35.6% 18.2% all behaviours, residential setting 0.0% 68.4% 52.8% 72.0% 20.0%

aggression, all settings 13.0% 67.8% 64.2% 53.8% 20.6%

all behaviours, all settings 14.3% 65.1% 50.0% 50.9% 19.0%

self-injury, all settings 0.0% 16.7% 36.1% 41.7% 10.8%

Percentage Inter-Informant Agreement (Non-Occurrence): Durand & Crimmins scoring method

all behaviours, all settings 58.0% 61.5% 44.4% 45.8% d a all behaviours, classroom setting 56.1% 54.5% 54.5% 44.8% nla all behaviours, residential setting 60.0% 68.8% 34.5% 46.7% d a self-injury, all settings 61.5% 75.7% 36.4% 30.0% nla aggression, all settings 48.4% 45.8% 27.3% 41.3% nla

Percentage Inter-Informant Agreement (Non-Occurrence): Emerson & Bromley scoring method

all behaviours, all settings 56.4% 75.0% 30.0% 51.8% nla all behaviours, classroom setting 56.4% 77.7% 35.7% 58.6% nla all behaviours, residential setting 56.4% 72.4% 22.8% 44.4% nla self-injury, all settings 75.0% 88.9% 8.0% 48.1% nla aggression, all settings 35.5% 40.0% 37.5% 40.0% nla

MOTIVATION ASSESSMENT SCALE 207

Discussion

The present study supports the recent findings of low levels of inter- rater reliability on the MAS (Lawrenson, 1993; Newton & Sturmey, 199 1; Sigafoos, Ken- & Roberts, 1994; Zarcone et al., 1991). These results were obtained for both self-injurious and aggressive behaviours. As such, serious doubt must be cast over the use of the MAS in the behavioural assessment of challenging behaviours shown by people with severe intellectual dis- abilities. This is particularly so given that the present study investigated agreement between informants with regard to the ‘behavioural diagnosis’ resulting from the use of the instrument, i.e. the very information which practitioners are encouraged to use in the development of treatment strategies. At this time, it would appear that indirect assessment approaches to identifying the behavioural functions which control or main- tain challenging behaviours should only contribute to, and certainly not replace, techniques based upon direct observation and/or (experimental) functional analysis.

References Bihm, E. M., Kienlen, T. L., Ness, M. E. and Poindexter, A. R. (1991) Factor

structure of the motivation scale for persons with mental retardation. Psycho- logical Reports 68, 1235-8.

Dunn, G. (1989) Design and Analysis of Reliability Studies. New York Oxford University Press.

Durand, V. M. (1 990) Severe Behavior Problems: A Functional Communication Train- ing Approach. New York: Guilford Press.

Durand, V. M. and Carr, E. G. (1992) An analysis of maintenance following func- tional communication training. Journal of Applied Behavior Analysis 25, 777-94.

Durand, V. M. and Crimmins, D. B. (1988) Identifying variables maintaining self- injurious behaviour. Journal of Autism and Developmental Disorders 18, 99-1 17.

- (1992) The Motivation Assessment Scale. Topkepa, KS: Monaco & Associates. Emerson, E. (1995) Challenging Behaviour: Analysis & Intervention in Learning Dis-

abilities. Cambridge: Cambridge University Press. Emerson, E. and Bromley, J. (in press) The form and function of challenging

behaviours. Journal of Intellectual Disability Research. Lawrenson, H. J. (1993) Concurrent validity and inter-rater reliability analysis of

the Motivation Assessment Scale. M.Sc. Thesis in Clinical Psychology, Univer- sity of Manchester.

Newton, J. T. and Sturmey, P. (1991) The Motivation Assessment Scale: Inter- rater reliability and internal consistency in a British sample. Journal of Mental Deficiency Research 35, 472-4.

Sigafoos, J., Kerr, M. and Roberts, D. (1994) Interrater reliability of the Motiv- ation Assessment Scale: Failure to replicate with aggressive behavior. Research in Developmental Disabilities 15, 333-42.

Singh, N. N., Donatelli, L. S . , Best, A., Williams, D. E., Barrera, F. J., Lenz, M. W., Landrum, T. J., Ellis, C. R. and Moe, T. L. (1993) Factor structure of

208 MENTAL HANDICAP RESEARCH

the Motivation Assessment Scale. Journal of Intellectual Disability Research 37,

Zarcone, J. R., Rodgers, T. A., Iwata, B. A., Rourke, D. A. and Dorsey, M. F. (1991) Reliability analysis of the Motivation Assessment Scale: A failure to repli- cate. Research in Developmental Disabilities 12, 349-60.

65-74.