6
Brief report The accuracy of the Child and Adolescent Intellectual Disability Screening Questionnaire (CAIDS-Q) in classifying severity of impairment: a brief report A. L. Murray 1 & K. McKenzie 2 1 Centre for Cognitive Ageing & Cognitive Epidemiology, Department of Psychology, University of Edinburgh, Edinburgh, UK 2 Community Child Health, NHS Lothian, Edinburgh, UK Abstract Background Severity of intellectual disability (ID) is associated with a range of outcomes for the indi- vidual and having an indication of severity can help inform support needs. Previous research has not evaluated whether screening tools can accurately ascertain severity category in addition to providing a red flag for the presence of ID. Methods We used multi-category receiver operating characteristic (ROC) analysis to assess whether the Child and Adolescent Intellectual Disability Screen- ing Questionnaire (CAIDS-Q) could be used clini- cally to classify individuals (n = 191) aged between 12 and 18 according to British Psychological Society (BPS) categories of severity of impairment. Results The volume under the surface statistic (VUS) was 0.59. The optimal cut-points estimated based on the ROC surface and Youden Index pro- vided correct classification probabilities for the severe, significant and non-ID groups of 0.44, 0.63 and 0.86 and 0.79, 0.29 and 0.88 respectively. Conclusions While the CAIDS-Q can accurately discriminate between those with and without ID, and provides a heuristic for severity of ID, the results indicate that it does not reliably identify whether an individual falls into the severe or signifi- cant category of intellectual impairment. Keywords CAIDS-Q, intellectual impairment, screening, severity Introduction Intellectual disability (ID) refers to a heterogeneous group of people who differ in respect of their par- ticular strengths and support needs. Attempts have been made to capture some of these differences in terms of the severity of an individuals’ ID, with classifications such as mild to severe ID (World Health Organization 1996; American Psychiatric Association 2000). In the UK, two classifications of intellectual impairment are recommended for use in clinical practice: significant impairment (IQ level of Correspondence: Ms Aja L. Murray, Centre for Cognitive Ageing & Cognitive Epidemiology, Department of Psychology, University of Edinburgh, Edinburgh, EH89JZ, Scotland, UK (e-mail: a.l.murray-2@sms.ed.ac.uk). Journal of Intellectual Disability Research doi: 10.1111/jir.12115 volume 58 part 12 pp 11791184 december 2014 1179 © 2014 MENCAP and International Association of the Scientific Study of Intellectual and Developmental Disabilities and John Wiley & Sons Ltd

The accuracy of the Child and Adolescent Intellectual Disability Screening Questionnaire (CAIDS-Q) in classifying severity of impairment: a brief report

  • Upload
    k

  • View
    213

  • Download
    1

Embed Size (px)

Citation preview

Page 1: The accuracy of the Child and Adolescent Intellectual Disability Screening Questionnaire (CAIDS-Q) in classifying severity of impairment: a brief report

Brief report

The accuracy of the Child and Adolescent IntellectualDisability Screening Questionnaire (CAIDS-Q) inclassifying severity of impairment: a brief report

A. L. Murray1 & K. McKenzie2

1 Centre for Cognitive Ageing & Cognitive Epidemiology, Department of Psychology, University of Edinburgh, Edinburgh, UK2 Community Child Health, NHS Lothian, Edinburgh, UK

Abstract

Background Severity of intellectual disability (ID)is associated with a range of outcomes for the indi-vidual and having an indication of severity can helpinform support needs. Previous research has notevaluated whether screening tools can accuratelyascertain severity category in addition to providing ared flag for the presence of ID.Methods We used multi-category receiver operatingcharacteristic (ROC) analysis to assess whether theChild and Adolescent Intellectual Disability Screen-ing Questionnaire (CAIDS-Q) could be used clini-cally to classify individuals (n = 191) aged between12 and 18 according to British Psychological Society(BPS) categories of severity of impairment.Results The volume under the surface statistic(VUS) was 0.59. The optimal cut-points estimatedbased on the ROC surface and Youden Index pro-vided correct classification probabilities for the

severe, significant and non-ID groups of 0.44, 0.63

and 0.86 and 0.79, 0.29 and 0.88 respectively.Conclusions While the CAIDS-Q can accuratelydiscriminate between those with and without ID,and provides a heuristic for severity of ID, theresults indicate that it does not reliably identifywhether an individual falls into the severe or signifi-cant category of intellectual impairment.

Keywords CAIDS-Q, intellectual impairment,screening, severity

Introduction

Intellectual disability (ID) refers to a heterogeneousgroup of people who differ in respect of their par-ticular strengths and support needs. Attempts havebeen made to capture some of these differences interms of the severity of an individuals’ ID, withclassifications such as mild to severe ID (WorldHealth Organization 1996; American PsychiatricAssociation 2000). In the UK, two classifications ofintellectual impairment are recommended for use inclinical practice: significant impairment (IQ level of

Correspondence: Ms Aja L. Murray, Centre for Cognitive Ageing& Cognitive Epidemiology, Department of Psychology, Universityof Edinburgh, Edinburgh, EH8 9JZ, Scotland, UK (e-mail:[email protected]).

Journal of Intellectual Disability Research doi: 10.1111/jir.12115

volume 58 part 12 pp 1179–1184 december 20141179

bs_bs_banner

© 2014 MENCAP and International Association of the Scientific Study of Intellectual and Developmental Disabilities and

John Wiley & Sons Ltd

Page 2: The accuracy of the Child and Adolescent Intellectual Disability Screening Questionnaire (CAIDS-Q) in classifying severity of impairment: a brief report

between 55 and 69) and severe impairment [IQ ofless than 55: British Psychological Society (BPS)2000].

Increased severity of ID is associated with greaterhealth care needs (see Allerton et al. 2011 for anoverview); increased likelihood of displayingbehaviours that challenge (see Royal College ofPsychiatrists, British Psychological Society andRoyal College of Speech and Language Therapists2007 for an overview), and decreased life expec-tancy (Bittles et al. 2002). In this context, profes-sionals may often be called upon by parents of achild with an ID to provide information about thelikely future prospects of their son or daughter(Stein et al. 2011).

Diagnosis and identification of the severity of IDcurrently requires assessment of intellectual andadaptive functioning and developmental history(BPS 2000). This process can be time-consuming(Ryan et al. 2007) and is constrained by the needfor appropriately qualified professionals to conductthe assessments (BPS 2000). Screening tools (e.g.McKenzie et al. 2012) currently offer a means ofidentifying those who may be likely to have ID andthere has been an increasing interest in whetherthey might also offer an indication of severity of ID.The present study, therefore, aimed to explore if ascreening tool, the Child and Adolescent Intellec-tual Disability Screening Questionnaire (CAIDS-Q:McKenzie et al. 2012) was able to accurately classifyindividuals, not only as whether they had or did nothave ID, but, in addition, to place them in thecorrect category of either significant or severeimpairment.

Methods

Participants

The CAIDS-Q covers ages 8–18, but is stratifiedinto two age categories. For the planned analysesin the present study, sufficient data were onlyavailable for ages 12 to 18 years, and it is to thissubsample that all subsequent results refer. Thesample utilised in the current study was that usedin the development of the CAIDS-Q (McKenzieet al. 2012), plus additional cases collected sincethe publication of the initial CAIDS-Q evaluationstudy. A total of 156 12- to 18-year-olds were

included in the initial evaluation study and a totalof 191 12- to 18-year-olds were included in thecurrent study. Ethical approval had previouslybeen given by the relevant Caldicott Guardians.Participants had all been referred to Child andAdolescent Mental Health (CAMH) or ID servicesand diagnosis of the presence or absence of IDwas determined by the independent clinicians’opinion based on the three criteria of ID: signifi-cant impairment in intellectual and adaptive func-tioning and childhood onset (BPS 2000). Becauseof insufficient data from any one measure of adap-tive functioning, ‘severity’ was based on intellec-tual functioning only (non-ID, significantimpairment or severe impairment). Informationon the sample is provided in Table 1.

Measures

Data from the CAIDS-Q were used. This is a short(seven item), dichotomously scored questionnaire,developed as a screening tool to help identify thosepeople who are likely to have an ID. The CAIDS-Qhas been found to have good psychometricproperties and to correlate significantly with bothintellectual (McKenzie et al. 2012) and adaptivefunctioning (McKenzie & Murray 2013). In a two-category receiver operating characteristic (ROC)analysis (ID vs. non-ID) the tool has yielded sensi-tivity and specificity values at the chosen cut-off inclinical samples of 96% and 85% respectively, forthe older age group (McKenzie et al. 2012).

Assessing classification accuracy

ROC analysis is commonly applied to clinicalinstruments to determine whether that instrumentcan effectively separate individuals with a disorderfrom individuals without the disorder. The methodcan also be extended to more than two ordered cat-egories when, for example, there are multiple diag-nostic categories dependent on the severity or stageof a disorder. Methods for assessing the ability of ameasure to correctly place individuals in diagnosticcategories when there are three or more orderedcategories are described in detail in Nakas &Yiannoutsos (2004), Xiong et al. (2006) and Nakaset al. (2010). We applied these methods to evaluatewhether the CAIDS-Q can successfully classify

1180Journal of Intellectual Disability Research volume 58 part 12 december 2014

A. L. Murray & K. McKenzie • Classification accuracy of the CAIDS-Q

© 2014 MENCAP and International Association of the Scientific Study of Intellectual and Developmental Disabilities and

John Wiley & Sons Ltd

Page 3: The accuracy of the Child and Adolescent Intellectual Disability Screening Questionnaire (CAIDS-Q) in classifying severity of impairment: a brief report

individuals as ‘no ID’, ‘significant impairment’ or‘severe impairment’ using the DiagTest3Grppackage in R (Luo & Xiong 2012).

We first examined the distributional characteris-tics of the CAIDS-Q score within and across cat-egories. First we checked whether mean CAIDS-Qscores increased from the severe category throughthe significant category to the non-ID category. Inaddition, we checked for deviations from normalitywithin groups. If non-normality is violated withincategories this suggests that non-parametric estima-tion methods should be employed.

In the current application there were three pos-sible diagnostic categories: non-ID [Full Scale IQ(FSIQ) ≥ 70], significant (55 ≤ FSIQ < 70) andsevere (FSIQ < 54). This means that there werethree possible true classification rates, referring towhen an individual is placed in the correct category.In a two-category ROC analysis plotting the sensi-tivity against 1-specificity gives the ROC curve inthe unit square. Sensitivity is the true positive rate,i.e. the proportion of individuals with the disorderwho have a positive result on the test. Specificityis the true negative rate i.e. the proportion of

individuals who do not have the disorder who havea negative result on the test.

The area under the curve (AUC) can be used asa measure of the overall classification accuracy ofthe test and an optimal cut point chosen that pro-vides an acceptable trade-off between sensitivity andspecificity. Multi-category ROC extends these ideasto instances in which there are more than two diag-nostic categories.

For three categories, the classification accuracy –rather than being described by a curve in twodimensions – can be described by a ROC surface inthree dimensions. The volume under this surface(VUS) can be taken as a measure of overall classifi-cation accuracy. Specifically, VUS can be inter-preted as the probability that the measurementsfrom three participants, one each taken from thenon-ID, significant and severe group will be classi-fied in the correct order. For three possible catego-ries, VUS can range from 1/6 (approximately 0.167)to 1, where a value of 1 indicates perfect classifica-tion (Nakas & Yiannoutsos 2004). The variance ofthe VUS statistic can be estimated using U-statisticsor a bootstrap method which can then be used to

Table 1 Descriptive statistics for Childand Adolescent Intellectual DisabilityScreening Questionnaire (CAIDS-Q) andFull Scale IQ (FSIQ) by diagnosticcategory for ages 12+ sample

Category

Gender Male Female

No intellectual disability 63 27Significant impairment 34 22Severe impairment 28 17

n Mean (SD) Min. Max. Skew Kurtosis

CAIDS-Q

No intellectual disability 90 80.33 (18.56) 14.00 100.00 −1.15 1.62Significant impairment 57 36.01 (28.78) 0.00 100.00 0.51 −0.61Severe impairment 45 18.10 (20.77) 0.00 57.00 0.81 −0.76

FSIQ

No intellectual disability 90 83.41 (12.32) 70.00 125.00 1.12 0.64Significant impairment 57 62.84 (4.92) 55.00 69.00 0.01 −1.54Severe impairment 45 44.76 (5.24) 30.00* 54.00 −0.06 0.17

* Estimated IQ by clinician.

1181Journal of Intellectual Disability Research volume 58 part 12 december 2014

A. L. Murray & K. McKenzie • Classification accuracy of the CAIDS-Q

© 2014 MENCAP and International Association of the Scientific Study of Intellectual and Developmental Disabilities and

John Wiley & Sons Ltd

Page 4: The accuracy of the Child and Adolescent Intellectual Disability Screening Questionnaire (CAIDS-Q) in classifying severity of impairment: a brief report

construct a statistical significance test for the VUSvalue.

Optimal cut-points can be selected either basedon the ROC surface or by use of the Youden Index.Optimal cut-points can be computed based on theROC surface by finding the co-ordinate on thesurface that minimises the Euclidean distance fromthe point (1,1,1) representing perfect classificationfor all categories. Alternatively, use of the YoudenIndex selects the pair of cut-points that maximisesthe overall classification accuracy by maximising theequation:

12

1( )x y z+ + −

where x, y and z are the respective probabilities ofcorrect classification for each of the diagnostic cat-egories. The Youden statistic takes values between0 and 1, with 1 representing a test with perfect diag-nostic accuracy. The optimal cut-points suggestedby these two methods do not tend to be the sameunless x, y and z are equal, therefore, a good strat-egy is to compute optimal cut-points based on bothmethods and select those that provide the best clas-sification accuracies for a particular application. Forexample, it may be of particular concern to maxim-ise classification accuracy for the ‘severe ID’ groupfor the purposes of triage. On other occasions itmay be more important to maximise classificationaccuracy for one of the other two groups.

Results

Descriptive statistics

Descriptive statistics by diagnostic category are pro-vided in Table 1 and the distribution of CAIDS-Qscores across the categories shown in Fig. 1. Theseresults indicate that the mean CAIDS-Q scoreincreases from the ‘severe impairment’ categorythrough ‘significant impairment’ to ‘no intellectualdisability’. This is consistent with expectation and anecessary condition for the subsequent ROC analy-ses. The plots in Fig. 1 suggest that there is a sepa-ration of the distribution of scores from those withan ID and from those without an ID (both signifi-cant and severe impairment) consistent with thepreviously reported sensitivity and specificity valuesof 0.96 and 0.85 for the two-category classificationcase. They also show, however, that the distribu-tions of scores from the two severity categories (sig-nificant and severe) show a high degree of overlapand will likely prove difficult to discriminate basedon CAIDS-Q scores alone.

There was only a small deviation from normalitywithin groups, therefore, parametric estimationmethods were used for the ROC analyses. The VUSwas estimated as 0.59 [95% confidence interval(CI) = 0.51−0.69]. We judged this value to be toosmall for a test to be useful in clinical practice forthe three-category classification case. The optimalcut-points estimated based on the ROC surface

Figure 1 Kernel density plots for Childand Adolescent Intellectual DisabilityScreening Questionnaire (CAIDS-Q)scores across diagnostic categories. Thedotted line represents the ‘severe’ group,the dashed line represents the ‘significantgroup’ and the solid line represents the‘non-ID’ group. ID, intellectual disability.

1182Journal of Intellectual Disability Research volume 58 part 12 december 2014

A. L. Murray & K. McKenzie • Classification accuracy of the CAIDS-Q

© 2014 MENCAP and International Association of the Scientific Study of Intellectual and Developmental Disabilities and

John Wiley & Sons Ltd

Page 5: The accuracy of the Child and Adolescent Intellectual Disability Screening Questionnaire (CAIDS-Q) in classifying severity of impairment: a brief report

were 0% and 71% with correct classification prob-abilities for the severe, significant and non-IDgroups of 0.44, 0.63 and 0.86. Optimal cut-pointsbased on the Youden Index were 34.9% and 59.9%with correct classification probabilities for severe,significant and non-ID groups of 0.79, 0.29 and0.88. Again, these values suggested that the testdoes not discriminate effectively enough betweenthe three categories to justify its use as a measure ofseverity of ID in clinical practice for the age groupassessed in the present study.

Discussion

Previous studies have suggested that the CAIDS-Qcan effectively discriminate between individuals withand without and ID. The present study aimed toexplore if the CAIDS-Q, could also place individ-uals in the correct classification category related toseverity of ID. This was prompted by an interestfrom users in determining whether cut-off scorescould be calculated which would identify likely levelof severity of ID more formally. The present analy-ses, however, suggest that the CAIDS-Q does notproduce effective discriminations between the severeand significant categories and should not be usedfor this purpose.

This finding may be for a number of reasons.Overall severity of ID is ideally determined by boththe individual’s intellectual and adaptive function-ing, however, because of insufficient available dataon adaptive functioning, the present study basedclassification of severity on intellectual functioningonly. This has two main limitations. First, currentstandardised intellectual assessments are unable toaccurately measure the lowest levels of IQ(Whitaker 2010). Thus, the diagnostic categoriesbased on FSIQ are themselves subject to a degreeof error. In addition, while there is a positive rela-tionship between intellectual and adaptive function-ing, the two are imperfectly correlated (e.g. Murrayet al. 2013). Thus any given individual may differ inthe level of severity on each, making an overall clas-sification of ID difficult (BPS 2000). This is likelyto be particularly challenging in the case of childrenbecause they may develop different skills at differentages (Bornholt et al. 2004). Therefore, although it ispossible that the CAIDS-Q under-performed

because of the fact that classifications of severitywere based on intellectual functioning alone, itremains a challenge in general for clinicians to inte-grate adaptive and intellectual functioning informa-tion in order to place individuals in a single severitycategory.

Conclusion

While previous research has found that theCAIDS-Q can accurately discriminate betweenthose children and young people who do and do nothave an ID, the present study indicates that it maynot accurately discriminate between the severe andsignificant categories of intellectual impairment andshould not be used for this purpose.

Conflict of interest

The second author was involved in the developmentof the CAIDS-Q and receives a small income fromany sales.

Sources of funding

None.

References

Allerton L. A., Welch V. & Emerson E. (2011) Healthinequalities experienced by children and young peoplewith intellectual disabilities: a review of literature fromthe United Kingdom. Journal of Intellectual Disabilities15, 269–78.

American Psychiatric Association (2000) Diagnostic andStatistical Manual of Mental Disorders, Text Revision, 4thedn. American Psychiatric Association, Washington,DC.

Bittles A. H., Petterson B. A., Sullivan S. G., Hussain R.,Glasson E. J. & Montgomery P. D. (2002) The influ-ence of intellectual disability on life expectancy. Journalsof Gerontology Series A: Biological Sciences and MedicalSciences 57, 470–2. doi: 10.1093/gerona/57.7.M470.

Bornholt L. J., Spencer F. H., Ouvier R. A. & Fisher I. H.(2004) Cognitive screening for young children: develop-ment and diversity in context. Journal of Child Neurology19, 313–17.

1183Journal of Intellectual Disability Research volume 58 part 12 december 2014

A. L. Murray & K. McKenzie • Classification accuracy of the CAIDS-Q

© 2014 MENCAP and International Association of the Scientific Study of Intellectual and Developmental Disabilities and

John Wiley & Sons Ltd

Page 6: The accuracy of the Child and Adolescent Intellectual Disability Screening Questionnaire (CAIDS-Q) in classifying severity of impairment: a brief report

British Psychological Society (BPS) (2000) Learning Dis-ability: Definitions & Contexts. British PsychologicalSociety, Leicester.

Luo J. & Xiong C. (2012) DiagTest3Grp: an R packagefor analyzing diagnostic tests with three ordinal groups.Journal of Statistical Software 51, 1–24.

McKenzie K. & Murray A. L. (2013) The convergentvalidity of the Child and Adolescent Intellectual Disabil-ity Screening Questionnaire with a measure of adaptivefunctioning: a brief report. Journal of Intellectual andDevelopmental Disabilities. doi: 10.3109/13668250.2013

.854321.

McKenzie K., Paxton D., Murray G., Milanesi P. &Murray A. L. (2012) The evaluation of a screening toolfor children with an intellectual disability: the Child andAdolescent Intellectual Disability Screening Question-naire. Research in Developmental Disabilities 33, 1068–75.

Murray A. L., McKenzie K. & Murray G. C. (2013) Towhat extent does g impact on conceptual, practical andsocial adaptive functioning in clinically referred chil-dren? Journal of Intellectual Disability Research. doi:10.1111/jir.12092.

Nakas C. T. & Yiannoutsos C. T. (2004) Orderedmultiple-class ROC analysis with continuous measure-ments. Statistics in Medicine 23, 3437–49.

Nakas C. T., Alonzo T. A. & Yiannoutsos C. T. (2010)Accuracy and cut-off point selection in three-class clas-sification problems using a generalization of the Youdenindex. Statistics in Medicine 29, 2946–55.

Royal College of Psychiatrists, British PsychologicalSociety and Royal College of Speech and LanguageTherapists (2007) Challenging Behaviour: A UnifiedApproach. Royal College of Psychiatrists, London.

Ryan J. J., Glass L. A. & Brown C. N. (2007) Administra-tion time estimates for Wechsler Intelligence Scale forChildren-IV subtests, composites, and short forms.Journal of Clinical Psychology 63, 309–18.

Stein D. S., Blum N. J. & Barbaresi W. J. (2011) Develop-mental and behavioral disorders through the life span.Pediatrics 128, 364–73.

Whitaker S. (2010) Error in the estimation of intellectualability in the low range using the WISC-IV and WAIS-III. Personality and Individual Differences 48, 517–21.

World Health Organization (1996) ICD-10 Guide forMental Retardation. World Health Organization, Geneva.

Xiong C., van Belle G., Miller J. P. & Morris J. C. (2006)Measuring and estimating diagnostic accuracy whenthere are three ordinal diagnostic groups. Statistics inMedicine 25, 1251–73.

Accepted 12 December 2013

1184Journal of Intellectual Disability Research volume 58 part 12 december 2014

A. L. Murray & K. McKenzie • Classification accuracy of the CAIDS-Q

© 2014 MENCAP and International Association of the Scientific Study of Intellectual and Developmental Disabilities and

John Wiley & Sons Ltd