Click to edit Master text styles

Click icon to add picture

Click to edit Master title style

On the Adaptive Measurement of Suicide Risk

Robert D. Gibbons Blum-Riese Professor of Biostatistics

University of Chicago

Supported by NIMH Grants R01-MH-66302, R01-MH-100155 Dr. Gibbons founded Adaptive Testing Technologies (www.adaptive testingtechnologies.com) which

distributes mental health computerized adaptive tests.

May 2017

Classical vs. IRT Measurement

Classical Measurement Model

Classical vs. IRT Measurement IRT

Arithmetic Algebra Calculus

What is CAT?

Imagine a 1000 Item Math Test

Bi-Factor IRT Model

,)()(1

= 11221

11

1=2=

θθθθαα

θαθαγdgdgP vv

jvu

jvj

vjvjjn

j

d

v

−−

−−Φ∏∫∏∫

∞

∞−

∞

∞−

.)()()(1=),|(=ˆ11

*

2=1

1211 θθθθθθθθθθ

θθdgdgL

PE vvviv

v

d

vi

idiiiii

∫∏∫u

.)()()()ˆ(1=),|( 11*

2=

211

121 θθθθθθθθθθ

θθdgdgL

PV vvviv

v

d

vii

idiiii

− ∫∏∫u

Gibbons and Hedeker, 1992, Psychometrika Gibbons et.al., 2007, Applied Psychological Measurement

CAT

• Traditional – all subjects get all items

• CAT - Subjects get different items based on severity

• Smallest number of items for fixed precision

• Develop large item banks that completely characterize a disorder such as depression

• Select items dynamically based on responses

• Maximize precision – Minimize burden

Diagnosis and Measurement are fundamentally different things

CAD-MDD – Decision Tree

Preliminary Results • Correlations with total bank scores

• Depression r=0.95 – 12 items – 389 item bank • Anxiety r=0.94 – 12 items – 431 item bank • Mania r=0.91 – 12 items – 88 item bank

• Emergency Department U of Chicago (n=1000)

• 26% MDD positive screens (>50% confidence) • 22% MDD positive screens (>90% confidence) • 7% MDD Positive + moderate or severe CAT-DI • 3% suicide screen positive • 3-fold increase in ED visits in past year in moderate/severe • 4-fold increase in hospitalizations in past year in moderate/severe • None of these patients had a psychiatric indication

• Primary Care Spain and US Latino Samples (n=1000)

• 33% MDD positive screens (>50% confidence) • 25% MDD positive screens (>90% confidence) • 9% MDD Positive + moderate or severe CAT-DI

High Frequency Measurement

Source: Rush University Medical Center, Drs. Sepehr Sani and Joao Busnello

•Adult (English and Spanish) •Depression* •Anxiety* •Mania/Hypomania* •Suicidality* •Psychosis*** •Substance Abuse** •Functional Impairment*** •Quality of Life*** •PTSD**** •Functional status and well being (Thyroid Cancer Survivors)**

•Perinatal (English and Spanish) •Depression* •Anxiety* •Mania/Hypomania*

•Child and adolescent (child and parent ratings) Ages 7-17 – English only (at least currently) •Depression** •Anxiety** •Mania/Hypomania** • ADHD** •Conduct disorder** •Oppositional defiant disorder** •Suicidality**

*Completed, validated and available today **Completed and being validated ***Completed and awaiting validation ****Under development

What can the CAT-MH Measure?

Current Uses of the CAT-MH • Rapid screening for depression, anxiety, mania, suicide

– Primary Care – Emergency Medicine – Psychiatric clinics – Children and Adults – Foster Care – Jails and Juvenile Justice

• Longitudinal assessments during treatment – Mental Health care – Ketamine – Deep Brain Stimulation – Identify onset of perinatal depression

• Triage for assessment of severity at intake – Prioritize wait-list based on need rather than waiting time – Identification of patients that might benefit from mobile CBT – Stepped Care Model

• Pharmaceutical – Identification of patients severe enough for RCTs – Boosting the signal to noise ratio in measurement of patient reported outcomes – Large-scale post-marketing

The Making of a Child Suicide CAT

Robert D. Gibbons David A. Brent

and Members of the Y-CAT (Kiddie-CAT) Team

The Dataset

• Y-CAT Study (Gibbons and Brent PI’s) • Depression Domain

– 177 items – Suicide subdomain – 10 items – 801 subjects ages 7-18

Methodology

• Identify suicidal children – Fit a unidimensional IRT model to 10 suicide items – Fit a mixture of two normals to the EAP scores – Classify membership in elevated component as

suicidal

• Map remaining 167 depression items to suicide classification using logistic regression

• Select items with OR>2.5 for the bank

Methodology

• Fit a bifactor model to the identified depression items and 10 suicide items using subdomains of depression and suicide.

• Simulate CAT from subjects with complete data. • Tune the CAT based on:

– Termination SE – Information threshold – Probability of item selection – 2016 simulated CATs

• Select CAT with fewest items with r>0.93 with total score

Results Unidimensional IRT Model – Suicide items

Item Factor Loadings 0.63 I made a suicide attempt. 0.80 I thought about death or about being dead myself. 0.94 I thought life wasn't worth living. 0.83 I thought my family would be better off without me. 0.94 I thought about killing myself. 0.92 I had a plan for how I would kill myself. 0.92 I said I was going to kill myself. 0.95 I thought that I would be better off dead. 0.95 I felt everyone would be better off without me. 0.97 My life wasn't worth living.

Results Mixture Distribution

2.01.40.70.1-0.6

200

160

120

80

40

0

Freq

uenc

y

Score

74%

26%

Results Crosswalk between Depression and Suicide

OR 2.5 - I wanted to be alone. 3.0 - I felt like I didn't have any friends. 3.5 - I didn't enjoy anything at all. 4.0 - I felt lonely. 4.5 - I felt listless compared to how I used to be. 5.0 - I felt blah and low down. 6.0 - I felt worthless. 7.0 - I felt like I was losing it. 7.5 - I stopped enjoying anything.

Results Bifactor Model

Item Threshold Loading Text_______________________________________ 28 -0.47 0.65 I wanted to be alone. 14 -0.17 0.80 I felt lonely. 131 0.02 0.84 I felt blah and low down. 59 0.18 0.82 I felt worthless. 132 0.18 0.88 I felt like I was losing it. 2 0.33 0.71 I felt like I didn't have any friends. 115 0.36 0.88 I felt listless compared to how I used to be. 127 0.45 0.93 I stopped enjoying anything. 68 0.45 0.74 I thought my family would be better off without me. 20 0.57 0.82 I didn't enjoy anything at all. 174 0.70 0.86 My life wasn't worth living. 69 0.90 0.88 I thought about killing myself. 70 1.19 0.83 I had a plan for how I would kill myself. 30 1.65 0.75 I made a suicide attempt.

Results Mixture Distribution

210-1-2

30

24

18

12

6

0

Freq

uenc

y

Score

32%

52%

16%

Results Simulated CAT

• Termination SE = 0.36 • Information threshold 0.90 • Probability to select most informative item = 0.5 • Average # of items administered = 7 • Min = 2 • Max = 18 • Correlation with 64 item score r=0.94

Results: Live YCAT-SS Iter Question Response Score Se Info 1 My life wasn't worth living. 1 25.5 13.6 5.4 2 I felt awful or unhappy. 1 14.7 12.0 3.8 3 I felt I couldn't do anything right. 1 11.5 11.5 1.9 4 I felt grouchy and in a bad mood. 1 7.5 11.1 1.7 5 I felt grumpy and upset. 1 4.5 11.0 1.5 6 I wanted to be alone. 1 2.4 11.0 1.1 7 I had a harder time getting started than usual. 1 0.3 10.6 0.9

Iter Question Response Score Se Info 1 My life wasn't worth living. 5 73.9 10.0 5.4 2 I said I was going to kill myself. 5 84.4 8.6 7.3 3 I felt listless compared to how I used to be. 5 89.8 7.7 5.1 4 I had a plan for how I would kill myself. 5 92.0 7.6 3.5 5 I didn't enjoy anything at all. 5 96.1 7.5 3.2 6 I thought there was nothing good for me in the future. 5 99.1 7.3 1.8 7 I no longer cared about anything. 5 100.0 7.2 1.4

Iter Question Response Score Se Info 1 My life wasn't worth living. 1 25.5 13.6 5.4 2 I felt awful or unhappy. 5 51.2 8.3 3.8 3 I felt worthless. 5 58.1 7.5 6.7 4 I felt I was no good anymore. 3 54.3 7.1 6.4 5 I felt listless compared to how I used to be. 4 66.8 4.9 6.2

Adult CAT Suicide Scale - Validation

• Validation UC and UMass EDs – n=290.

• CAT-SS: 58% no risk, 31% low risk, 11% high risk.

• CSSRS: 15% ideation (9% active, 6% intent/plan).

• 52-fold increase in clinician rated suicidal ideation on C-SSRS across range of CAT-SS.

• No risk vs. high risk (CAT-SS) vs any ideation (C-SSRS) sensitivity=1.00, specificity=0.95, kappa=0.81

• Active ideation, sens=1.00, spec=0.92, kappa=.66

• JCP in press

Relevant Publications Gibbons R.D., & Hedeker D.R. Full-information item bi-factor analysis. Psychometrika, 57, 423-436, 1992. Gibbons R.D., Bock R.D., Hedeker D., Weiss D., Segawa E., Bhaumik D.K., Kupfer D., Frank E., Grochocinski V., Stover A. Full-Information Item Bi-Factor Analysis of Graded Response Data. Applied Psychological Measurement, 31, 4-19, 2007. Gibbons R.D., Weiss D.J., Kupfer D.J., Frank E., Fagiolini A., Grochocinski V.J., Bhaumik D.K., Stover A. Bock R.D., Immekus J.C. Using computerized adaptive testing to reduce the burden of mental health assessment. Psychiatric Services, 59, 361-368, 2008. Gibbons R.D., Weiss D.J., Pilkonis P.A., Frank E., Moore T., Kim J.B., Kupfer D.K. The CAT-DI: A computerized adaptive test for depression. Archives of General Psychiatry, 69, 1104-1112, 2012. Gibbons R.D., Hooker G., Finkelman M.D., Weiss D.J., Pilkonis P.A., Frank E., Moore T., Kupfer D.J. The CAD-MDD: A computerized adaptive diagnostic screening tool for depression. Journal of Clinical Psychiatry, 74, 669-674, 2013. Gibbons R.D., Weiss D.J., Pilkonis, P.A., Frank E., Moore T., Kim J.B., Kupfer D.J. Development of the CAT-ANX: A computerized adaptive test for anxiety. American Journal of Psychiatry, 171, 187-194, 2014. Achtyes E.D., Halstead S., Smart L., Moore T., Frank E., Kupfer D., Gibbons R.D. Validation of computerized adaptive testing in an outpatient non-academic setting. Psychiatric Services, published on-line. Beiser D., Vu, M., Gibbons, R.D. Test-retest reliability of a computerized adaptive depression test. Psychiatric Services, on-line. Gibbons R.D., Computerized adaptive diagnosis and testing of mental health disorders. Annual Review of Clinical Psychology, 12, 83-104, 2016. Kim J.J., Silver R.K., Elue R., Adams M.G., La Porte L.M., Cai L., Kim J.B., Gibbons R.D. The experience of depression, anxiety and mania among perinatal women. Archives of Women’s Mental Health, published on-line ahead of print. Gibbons R.D., Kupfer D., Frank E. Moore T., Boudreaux E. Development of a computerized adaptive suicide scale., Journal of Clinical Psychiatry, in press. Gibbons R.D., Beiser D., Boudreaux E., Kupfer DJ. Einstein, measurement and prediction. Journal of Affective Disorders, in press.