1.Outcome Measurement in Mental Health John Brazier Director of the Economic Evaluation of Health and Care Interventions Policy Research Unit (EEPRU)School of Health and Related Research The University of Sheffield, UKCEMPH Conference 7 November 2013

2. Background Growing use of generic outcome measures in people with mental health problems: Surveys (e.g. ESEMed, PMS in UK) Use in clinical trials Assessing cost-effectiveness to inform reimbursement decisions (e.g. NICE) Routine outcome monitoring (e.g. IAPT, PROMS programme, PBR) Things have changed since the Gilbody review! 27/11/2013 The University of Sheffield2 3. 3What should we be measuring? Quality of life (QoL) or well-being are ill-defined and there are different ways of conceptualising them: functionings, capabilities, wellbeing etc. The World Health Organization (1948) declared health to beA state of complete physical, mental and social wellbeing, and not merely the absence of disease and infirmity QoL means different things to different people and this is why we need the views of mental health service users in developing and testing measures 4. Types of measure Generic measures: those instruments designed for use on any population (EQ-5D, ICECAP etc.) Condition specific measure (CSM): those instruments designed for use in a specific population (CORE-10, PHQ9, GAD etc.) Both types of measure are standardised and come with a scoring algorithm (that may or may not be preference based for calculating QALYs) 5. Quality Adjusted Life years (QALY) QALYs combines both quantity and quality of life into a single measure QALYs can be used across all health care interventions for all patient groups 6. 6By placing a tick in one box in each group below, please indicate which statements best describe your own health state today. Mobility I have no problems in walking about I have some problems in walking about I am confined to bed Self-Care I have no problems with self-care I have some problems washing or dressing myselfScored using UK TTO values ( Dolan et al,1997) with a range minus 0.54 (worst impairment) to 1 (full health) Preferred by NICE and used in DH PROMS programme 5 level version now availableI am unable to wash or dress myself Usual Activities (e.g. work, study, housework, family or leisure activities) I have no problems with performing my usual activities I have some problems with performing my usual activities I am unable to perform my usual activities Pain/Discomfort I have no pain or discomfort I have moderate pain or discomfort I have extreme pain or discomfort Anxiety/Depression I am not anxious or depressed I am moderately anxious or depressed I am extremely anxious or depressed 7. How do we know if a measure like EQ-5D is measuring the right thing? Does it cover the important dimensions of (mental) health related quality of life and relevant ranges (Content validity) Does it reflect known group differences and correlate with other indicators of quality of life (Construct validity) Does respond to known changes quality of life (Responsiveness) Assessing validity is problematic due to the absence of a gold standard and validity being a question of degree 8. 8Overview of talk Based on the findings of an MRC funded study looking at the appropriateness of EQ-5D and other generic measures in people with mental health problems Psychometric evidence based on a systematic review and further analyses Qualitative evidence based on a review and indepth interviews with mental health service users Implications for existing measures: EQ-5D, SF6D and the new kid on the block the SWEMWBS 9. Systematic review To assess the appropriateness of the EQ-5D and the SF36 family in terms of their validity and responsiveness in five mental health conditions: Schizophrenia, Bipolar disorder, Personality disorders, Depression and anxiety Ten health databases were searched Studies were appraised and data extracted using a standardised template. Performance: Construct validity: known group differences with general population control or between severity groups and convergent validity Responsiveness to known changes 27/11/20139 10. 10PRISMA FLOW DIAGRAM Citations retrieved by search of electronic databases (n=7,095) Unique citations (n=4,115)Titles and/or abstracts potentially relevant (n=266)Studies satisfying inclusion criteria (n=154)PDs (n=9)Schizophrenia (n=33)27/11/2013 The University of SheffieldBipolar disorder (n=22)Duplicate citations (n=2,980)Citations excluded after screening (n=3,849)Full papers excluded after screening (n=124) 12 references from reference list checkDepressive & anxiety disorders (n=23 not SF-36) 11. 11Depression and Anxiety EQ-5D x SF-6D Known group: Casecontrol Convergent validity-ResponsivenessxKnown group: SeverityWhere indicates results in support of validity or responsiveness and x indicates an inconsistent or non-significant results by test (and not an individual study); 12. 12Depression and anxiety EQ-5D and SF-6D: good known group and convergent validity, and responsiveness. More highly correlated with depression scales in patients with anxiety than with clinical anxiety scales Convergent validity in patients with anxiety may be driven by aspects of depression within anxiety disorder and presence of co-morbid depression. 27/11/2013 The University of Sheffield 13. 13Personality Disorders EQ-5D Known group: severity Known group: case-control Convergent validity ResponsivenessSF-36 xSF-12 SF-6D ------- 14. 14Personality disorders EQ-5D: responsive, KGV and CV Limited and mixed SF-36 evidence (related to two studies only) Very little evidence on SF-12 and none for SF-6D 15. 15Schizophrenia EQ-5DSF-36Known group: Severity--Know group: Case-control- - x x x x x x x xxxx--xConvergent validity Responsiveness x x x x x xSF-12 SF-6D 16. 16Schizophrenia KGV: Yes, but crude measures CV & R: Mixed results Clinical assessment of symptoms e.g. Positive and Negative Symptoms Scale (PANSS), functioning measures (e.g. Global Assessment of Functioning) and schizophrenia specific measures of HRQL (e.g. QLS). Some evidence that EQ-5D reflected depression rather than other symptoms. 17. 17Bipolar disorder EQ-5D SF-36 xKnown group: Severity Know group: Case- x controlConvergent validity ResponsivenessSF-12 SF-6D - x-x x x x -- 18. 18Bipolar Disorder Majority evidence on the SF-36 Crude KGV CV goodEQ-5D (4 studies) Mixed results for CV and R EQ-VAS performed better than the index. Properties demonstrated for depression but not mania 27/11/2013 The University of Sheffield 19. Overview of quantitative evidence19 EQ-5D and SF-36 achieved adequate levels of performance in depression and to some extent anxiety and personality disorder (but limited evidence) Results were more mixed in schizophrenia and bi-polar with a suggestion that results in some studies may reflect differences in depression These findings were supported by analyses of further data sets undertaken as part of the researchBut: Need more evidence using better indicators for testing validity and responsiveness is required. Need to assess content validity using qualitative research to better understand the findings27/11/2013 The University of Sheffield 20. Qualitative evidence 21. 21Why Qualitative Research? Content and face validity Perspective of individual important Required by FDA for measures being used to support labelling claims (and generally good practice)27/11/2013 The University of Sheffield 22. 22Two research studies Systematic Review and synthesis of qualitative research Primary research interviews of service users27/11/2013 The University of Sheffield 23. 23Study 1: Systematic review 13 studies Canada/UK/Sweden/USA/Australia/NZ Occupational Therapy (5); Nursing (4); Psychology (2); Psychiatry (1) Social worker (1) Schizophrenia (3); Bi-polar (3); Panic Disorder (1); Mixed (6) Framework analysis used to identify common and variable patterns of themes Limitations: focus on severe mental health schizophrenia/bi-polar 27/11/2013 The University of Sheffield 24. 24Study 2: Service User Interviews 19 interviews Broad range diagnosis and severity Severe and enduring (CMHT x 2) -Schizophrenia; Bipolar; Personality Disorder; -Severe Depression/Anxiety; PTSD Mild to Moderate (IAPT) Anxiety/Depression Themes from review made up initial themes of framework Limitations gatekeepers/diagnostic range 27/11/2013 The University of Sheffield 25. 25Conceptual Difficulties Quality of life as a positive concept Responses tend to focus on negative aspects of condition including clinical symptoms Symptoms being separate from QoL Review - overlap of QoL with other concepts Recovery/lived experience Overlap of domains within QoL e.g. Self-stigmatization (Belonging or self perception) 27/11/2013 The University of Sheffield 26. 26Findings QoL domainsAutonomyWell-Being Ill-BeingSelf PerceptionBelonging ActivityHope Physical health27/11/2013 The University of Sheffield 27. 27Well-Being Ill-Being Positive adds qualityNegative takes quality awayOverall sense of well-beingOverall feelings of distressFeeling calm/relaxedAnxiety/worry/fearFeeling safeLow mood/boredomEnjoymentLack of energy/feeling tired(Happiness)Lack of concentration(If there was one thing you could change to improve your life, what would it be) I would lose this anxiety that I seem to be constantly carrying with me. I dont know why, I dont know how, although I can remember when me and my sister spoke she says, at that time, they always used to say I was very highly strung, its a term that you dont really hear now, but I was always a bit like that when I was very young anyway and I seem to have carried this anxiety and nervousness with me ever since (IAPT Panic attacks) 28. 28Physical Health Physical health problems affect mental health Mental health affects physical health Presence of both makes life difficult to cope with it feels physical as well as mental . my body aches and like I think I just become really tense and that is what makes my body ache and I feel like erm I feel like my chest is being crushed and erm I cant breath and things like that and erm I just want to be asleep all the time to escape but I cant sleep (CMHT Severe depression/Socio-affective PD) 27/11/2013 The University of Sheffield 29. 29Self Perception Positive adds qualityNegative takes quality awayCoherent sense of selfLack of self identityPositive self identityLow self esteemSelf acceptanceLack of confidence Feelings of failure/uselessnessThe worse part is the lack of self worth and having to accept that one is disabled one has a stigma which stops you from doing anything else really other than being worthless (CMHT Schizophrenia) 27/11/2013 The University of Sheffield 30. 30Relationships and Belonging Positive adds qualityNegative takes quality awayAccepted and understoodLack of understandingSupportStigmaCompanionship/camraderieRejection/exclusionLove and affectionLoneliness/isolationTrustAbuseFeeling part of societyFeeling alien to societyI have feelings of erm not belonging to the human race, like, I feel very-, its not an outcast, I just dont feel a connection erm I dont know how else to describe that, its being like an alien, thats the only way I can describe that, and I know that sounds weird but thats the only way I can describe the feeling of it, I dont feel akin with anybody, I am very guarded and things like that I would just like to be supported by other people all working to a common cause err helping other people, thats all I have ever wanted to do (CMHT Severe Depression/Anxiety) 31. 31Autonomy Positive adds qualityNegative takes quality awayChoiceLack of choiceOpportunityLack of opportunityControl/self determinationControl (excessive)CopingNot coping - overwhelmedI seem a bit of a control freak, I want everything to be worked out before I decide to do a certain thing, you know, I want everything to be fairly straightforward and I mean, you cant, in a way, you cant live life like that, and yet I still want to live life like that, do you know what I mean? its about the stress, erm having faith or taking this, stepping out of your comfort zone, whatever you want to call it, yeah (Schizophrenia)27/11/2013 The University of Sheffield 32. 32Activity Positive adds qualityNegative takes quality awayEnjoyment well beingStress (exacerbates symptoms)Rewarding - esteemFailure - esteemMeaning and purpose Belonging social activity StructureDistraction from problems I went on a years course at engineering and I was absolutely scared about going on that, but I did it and I did it, you know, quite well and at that time, I was really happy in my life and I thought well Ive done something, Ive achieved something here doing this I think thats because I had the drive and a purpose of getting up and going out every morning and doing what, you know, normal people do, sort of, you know, I got into a routine which was very good, so I was happy and more stable, I just felt a little bit more worthwhile, you know. I dont like to feel useless really (Anxiety/panic attacks) 27/11/2013 The University of Sheffield 33. 33Hope and Hopelessness Positive adds qualityNegative takes quality awayPositive view of futureNegative view of futureOptimismPessimismDreams and goalsFeelings of lossThinking aheadDemoralizationThe one thing that I used to do a lot is not think about the future, Id think a couple of days ahead and then not think about, you know, any further than that cos now, one thing thats different from when I started going through this process, is that Im more willing to think further ahead, you know, Im more willing to say, well in a years time Id like to be at this place, before I did this I wouldnt, but theres still a long way to go, and still a lot of obstacles that Ive got to overcome (Depression/trans-gender) 27/11/2013 The University of Sheffield 34. 34Overview of qualitative findings Measuring QoL in mental health complex QoL covers negative as well as positive aspects Similar themes to personal recovery also service user driven: Boardman review for DH Connectedness, Hope and optimism, Identity, Meaning and Empowerment (CHIME) (Leamy et al, 2011)27/11/2013 The University of Sheffield 35. Discussion: implications 36. Content validity of EQ-5D Well-being/Ill-beingDepression and anxiety - negative only, excludes calm, happiness, energy , safety etcPhysical healthMobility, self care, usual activity and pain or discomfortSelf-perceptionNoneRelationships and belongingUsual activities? - Excludes affection, trust, support and loneliness, stigma, abuse etc. (the quality of contact)AutonomyNoneActivityUsual activity (and perhaps self-care) - Nothing on stress/failure/meaning/ purpose etc.Hope and hopelessnessNone27/11/2013 The University of Sheffield36 37. Content validity of SF-6D Well-being/Ill-beingDepression and anxiety, energy - Calmness and happiness in SF-36Physical healthPhysical functioning, role limitation, painSelf-perceptionNoneRelationships and belongingSocial functioning - Excludes affection, trust, support and loneliness, stigma, abuse etc. (the quality of contact)AutonomyNoneActivityRole limitationHope and hopelessnessNone - General health perception in SF-36 asks about future health expectation, but very limited27/11/2013 The University of Sheffield37 38. 3827/11/2013 The University of Sheffield 39. 39Adoption of SWEMWBS for PBR27/11/2013 The University of Sheffield 40. The Short Warwick-Edinburgh Mental Well-being Scale (S-WEMWBS) Below are some statements about feelings and thoughts. Please tick the box that best describes your experience of each over the last 2 weeksNone of the timeRarelyIve been feeling optimistic about the future12345Ive been feeling useful12345Ive been feeling relaxed12345Ive been dealing with problems well12345Ive been thinking clearly12345Ive been feeling close to other people12345Ive been able to make up my own mind about things12345STATEMENTSSome of Often the timeAll of the timeShort Warwick Edinburgh Mental Well-Being Scale (SWEMWBS) NHS Health Scotland, University of Warwick and University of Edinburgh, 2008, all rights reserved. 27/11/2013 The University of Sheffield40 41. Content validity of S-WEMWBS Well-being/Ill-beingFeeling relaxed, thinking clearlyPhysical health Self-perceptionFeeling usefulRelationships and belongingFeeling close to other peopleAutonomyMake up my own mind about thingsActivityDealing with problemsHope and hopelessnessFeeling optimistic27/11/2013 The University of Sheffield41 42. Implications for research42 Further testing of construct validity and responsiveness of EQ-5D and SF-6D using better indicators of HRQL Comparative testing of SWEMWBS on populations with mental health problems Extend qualitative research to conditions not well covered (e.g. OCD), recruit through different channels and extend to other countries and cultures Develop a mental health specific generic preference-based measure? 43. 43Developing a generic preferencebased mental health measure Questionnaire Confirm dimensions Develop items to reflect dimensions from qualitative work Test properties of items in range of populations Scoring Value using Time trade-off or discrete choice experiments with duration General public and people with mental health problems 27/11/2013 The University of Sheffield 44. 44References: Brazier JE, Connell J, Papaioannou D, Mukuria C, Mulhern B, OCathain A, Barkham M, Knapp M, Byford S, Gilbody S, Parry G. Validating generic preference-based measures of health in mental health populations and estimating mapping functions for widely used specific measures. Health Technology Assessment (in press)Connell J, Brazier JE, O'Cathain A, Lloyd-Jones M, Paisley S. Quality of life of people with mental health problems: a synthesis of qualitative research Health and Quality of Life Outcomes 2012, 10:138.Papaionnou D, Brazier JE, Parry G. How to measure quality of life for cost effectiveness analyses in personality disorders? A systematic review. Journal of Personality Disorder 2013; 27(3):383-401Papaionnou D, Brazier J, Parry G. How valid and responsive are generic health status measures, such as the EQ-5D and SF-36, in schizophrenia? A systematic review. Value in Health 2011, 14(6):907-920.27/11/2013 The University of Sheffield