Upload
haracha
View
53
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Approaches to Evaluating and Measuring Outcomes in Integrated Care Key Issues for Consideration. Dr Nick Goodwin CEO, International Foundation for Integrated Care Senior Associate, The King’s Fund Paper to: Health Quality and Safety Commission New Zealand - PowerPoint PPT Presentation
Citation preview
Approaches to Evaluating and Measuring Outcomes in Integrated Care
Key Issues for Consideration
Dr Nick GoodwinCEO, International Foundation for Integrated Care
Senior Associate, The King’s Fund
Paper to: Health Quality and Safety Commission New Zealand Workshop: Towards Integrated Care in New Zealand
Wellington, New Zealand, November 14th 2013
What is a ‘programme evaluation’?
• A systematic method of collecting, analysing and using information to answer questions about projects, policies or programmes
• In health care systems, they are particularly concerned with quality of care, patient safety, system efficiency and/or cost effectiveness
• They also seek to examine whether programme goals are, of have been, appropriate and/or useful so can be used to change and adapt strategic directions
• They tend to utilise both qualitative as well as quantitative methods
• They can be ‘formative’ or ‘summative’
Typical components of a ‘programme evaluation’
• Assessment of the need for the programme– Needs assessment/gap analysis/population health planning
• Assessment of the design and/or theory and logic of the programme in supporting its desired influence– Are the assumptions of the programme justified?
• Assessment of how the programme is being implemented– Process evaluation - is it going to plan?
• Assessment of the programmes outcome or impact; cost and efficiency– Outcome level, outcome change, programme’s effect
Rossi, Lipsey and Freeman (2004) Evaluation: a systems approach, Sage
Understanding what to evaluatein an integrated care programme
What are you evaluating – some key questions
• Who and what is the programme seeking to influence?
Need to clarify aim and design of the integrated care intervention by looking at the needs of patients/users
• What is the timeframe over which outcomes are expected to be achieved?
Given this timeframe, which categories of outcomes have the potential to be improved?
• Is there sufficient opportunity in a given population to achieve this targeted improvement in outcomes?
• How can you measure the impact? How can you ensure attribution?
What are you evaluating – some key questions
Before developing questions and/or survey instruments to examine the experience and impact of integrated care from a person’s perspective, there is a need to understand four things:
the programme theory of change – what are the assumptions that lie behind the programme (why?)
the (set of) problems to be addressed (where and who?) the (set of) interventions best suited to address the problem
(what and who?) the strategy best suited to develop, implement, and
evaluate the (set of) interventions (how, when and who?)
What are you evaluating – some key questions
For integrated care to be successful, it needs to execute the following three functions:
accurate identification of individuals within target population (e.g. reliable predictive modelling, health risk assessment, medication list and/or laboratory values from EMRs);
individuals must be enrolled and actively participate in the program for a meaningful period of time (e.g. readiness to change, motivational interviewing, incentives);
the program must include a set of interventions that modify or close deficits in participant and provider behaviour (e.g. tailoring to needs).
Key Points to Consider Baseline data Define a comparison group Define nature and structure of integrated care being
implemented Include measures of the professionals’ perspective where care
is delivered through multidisciplinary teams Identify what good looks like from a patients’ perspective and
evaluate this through user feedback Include analysis of utilisation and costs of care
Experiences, care outcomes, utilisation & costs
Understanding what to measurein an integrated care programme
Measurement Types - 1Care Outcome Measures Patient outcomes
– e.g. mortality, morbidity, functional status, quality of life Cost and utilisation outcomes
– e.g. hospital admissions, bed days, LOS, nursing home placementsCare Process Measures Occurrence of recommended care activities
– e.g. presence of a care plan; patient follow-up – often processes that are set out in best practice guidance
The measurement tools we have on outcomes and processes are mostly disease-specific - for people with multiple needs, the process of care is less well understood
Examples of outcome measures Hospital utilisation
– Emergency admissions; hospital readmissions; lengths of stay; number of bed days etc …– Disease-specific hospital admissions etc …
‘Social care’ utilisation– Levels of home care support packages– Rates of long-term nursing home/residential home stays
Mortality and disease-specific mortality Short-term clinical outcomes
– e.g., glycated haemoglobin levels for diabetic patients Functional status
– e.g., for CHF patients Quality of life
– e.g., functional dependence Other patient outcomes
– e.g., missed school days for children due to illness; experiences with system Treatment and service adherence
– e.g., remaining in contact with services for mentally ill patients
Measurement Types - 2
Measures of care co-ordination Information exchanges and transfers Relational co-ordination between organisations and professionals
– levels of awareness/interaction among participants– is there a common understanding of care activities and goals– shared culture and mission?
Co-ordination Mechanism Measures - Approaches
Direct observation Interviews and staff surveys (self-report) Medical record audits on information transfer Measures of inter-professional collaboration within teams and
organisations
Examples:ITMA - Integrated Team Monitoring and Assessment Tool – see http://www.readiness-tools.com/tool-full.aspx?toolguid=0d6382ad-f017-4623-8d10-93f2f314e346POET – Partnership Outcomes Evaluation Toolkit – see www.dhcarenetworks.org.uk/asset.cfm?aid=1479
Care Co-ordination Measures Atlas
McDonald KM, Schultz E, Albin L, Pineda N, Lonhart J, Sundaram V, Smith-Spangler C, Brustrom J, and Malcolm E. Care Coordination Atlas Version 3 AHRQ Publication No.11-0023-EF. Rockville, MD: Agency for Healthcare Research and Quality. November 2010. http://www.ahrq.gov/qual/careatlas/careatlas.pdf and http://www.ahrq.gov/qual/careatlas/careap4.pdf - 64 different survey tools
Domains for measuring care co-ordinationCo-ordination activity: Establish accountability/negotiate
responsibility Communication – informational and
inter-personal Facilitate transitions – e.g. across
settings or as coordination needs change
Assess multiple needs and goals Pro-active care planning Monitor, follow-up, review Support self-management Link or refer to community resources Align resources to meet individual or
community needs
Service delivery approaches: Care management Medicines management Healthcare at Home Multi-disciplinary teams ICT-enabled integrated care (e.g.
telehealth)
Perspectives:Family/patient
ProfessionalSystem/organisation
Measurement Types - 3Patient, carer, family-reported perceptions PROMs
• do you feel better? • have your symptoms improved?
PREMs• recommend to a friend? • how satisfied are you?
* PACIC patient assessment of chronic illness care http://www.improvingchroniccare.org/downloads/2004pacic.doc.pdf
* Care Transition Measures (CTM) – patient assessment of discharge from hospital http://www.caretransitions.org/documents/CTM-15.pdf
The more ‘robust’ measures are those that ask about specifics of care co-ordination rather than generalities, and which link data sets together to understand association between responses. Two key things: measures that are specific to a particular group of people; measures that translate to actionable service improvements
Patient Reported Outcome Measures in England
Since 2009, PROMs have been collected for four elective procedures: hip surgery, knee surgery, hernia repair and varicose veins – these costs the English NHS £800m/year
Cataract surgery was to be included, but concerns on methods of data collection
250,000 patients a year invited to complete survey – all NHS providers (100%) collect data
PROMS being extended to cover: anxiety and depression, cancer care, asthma, COPD, diabetes, epilepsy, heart failure, stroke
It’s a careful process – identify the right PROMs instrument; pilot before roll-out; implement data collection; evaluate programme
Uses EQ-5D – part 1 about mobility, self-care, usual activities, pain/discomfort, anxiety/depression; part 2 gives overall assessment of health on 0-100 scale
NOTE: Not sensitive to looking at people with multiple conditions/needs
Some Conclusions on Measuring Integration
Many different tools available: Need to define the client group Need to understand the goal in terms of outcomes to patients and service users Need to create ‘measurable’ outcomes and experiences Measures need to mean something – i.e. that actions can follow
Patients and users tend to understand the term ‘care co-ordination’ or ‘continuity of care’ – e.g. to what extent they feel that care is co-ordinated around their needs
Baseline on measures required on which to base progress over time Link measures to other data – e.g. on clinical outcomes, utilisation, costs Where possible, benchmark performance with others or investigate with a
matched ‘control’ Use data in ‘real time’ to monitor progress and drive performance
Case Example
Integrated Care Metrics NHS London (2012)
Activity metrics currently used in IC systems in London, across all settings of careMetric Definition Data source ICS Emergency admissions (No.)
Number of unplanned admissions for ACS conditions that should not usually require hospital admissionNumber of emergency admissions for the IC cohort (over 65/75/ top ICD 10 codes/ community ward patients)Case file audit results for reviews of avoidable ED admissions
SUS / HA data NWL, ONEL, NCL, ONEL,ELC, KHP
Emergency bed days (No.) Number / total number of emergency hospital bed days for IC cohort (over 65/75/ top ICD 10 codes/ community ward patients)Emergency bed days associated with multiple acute hospital admissions
SUS/ HA data ONEL, NWL, ELC
A and E attendances (No.) Number / Total number of A and E attendances for the IC cohort (over 65/75/ top ICD 10 codes/ community ward patients)
SUS data NWL, ONEL, NCL, KHP, ELC
Acute re-admissions (No/ %)
All emergency re-admissions that occur within 30 days of discharge for IC cohort (over 65/75/ top ICD 10 codes/ community ward patients)
SUS / HA data NWL, ONEL, KHP
Length of stay (No.) Difference between the discharge time and the admittance time for both elective and non elective episodes divided by the total number of spells for IC cohort Average Length of Stay for patients in the IC cohort (over 65/75/ top ICD 10 codes/ community ward patients)
Dr Foster/ SUS / HA data
NWL, ONEL, NCL, ELC
Permanent admissions to residential and nursing care homes (No./ %)
Number of patients in long term care homes as a proportion of the total number of patients in the pilot (%) Number of new permanent admissions to residential and nursing care homes, 65+/ -65Case file audit results for reviews of recent care home admissions
Referral Assessment Provision (RAP) stats
KHP, NWL, Greenwich, NCL, ONEL
Number of people completing re-ablement (No.)
Proportion of service users independent following re-ablement Readmission within 1 year for patients who have completed re-ablement
Referral Assessment Provision (RAP) stats
Greenwich
Community nursing hours per person (No.)
Total number of community nursing visits or units delivered divided by number of patients in pilot
RiO NWL
People supported at home with low/med/high care packages (No.)
Number of people 65+supported at home with low/med/high care packages Referral Assessment Provision (RAP) stats
KHP, Greenwich
Urgent GP response DH survey (Q14-17 seen on same day within primary care 'how convenient' score)
QOF KHP
Process metrics currently used by IC systems in London
Metric Definitions Source ICS
Number of health assessments completed (No.)
Number of assessments complete versus target Outcomes of health assessments Emergency admissions of patients who had health assessments
Local ICS collection KHP
Number of people case managed (No.)
Numbers of case managed patients Caseload, analysed by length of stay, source of referral and state of care plan % currently case managed patients with no ED contact ‘Referral bounce’ for case managed patients
Local ICS collection KHP
Number of care plans completed (No.)
Numbers of patients with personalised care plans Number of people in cohort on a care plan as a proportion of total number of people that should be on a care plan
Local ICS collection ONEL , NWL
Adherence to care plan (No.) Number of patients with one or more delay or incomplete actions as a proportion of the total number of patients in the pilot (%)
Local ICS collection NWL
MDT operation (No.) Numbers of case managed patients discussed at MDT for whom follow up actions took place
Local ICS collection KHP
Attendance at MDT meetings (No.)
Number of meetings scheduled / held / fully attended Number of clinicians attending monthly MDT case conferences and quarterly review meetings as proportion of total number expected at these meetings (%)
Local audit KHP
Frequency of MDT meetings (No.)
Virtual Ward: number of monthly MDT meetings held Local ICS collection ELC
Degree of joint working Attendance rates at joint meetings Exception reporting – local ICS collection
ONEL
Metric Definition Source ICS Control measures Patients with T1/T2 DM with HbA1c less than or equal to 10
% of patients with blood pressure targets achieved as per NICE guidance Percentage of patients with cholesterol less than or equal to 5BMI less than or equal to 30proportion of patients receiving medication review who should
QOF NWL
Hard outcomes Speed of referral for recognised foot complications, Amputation rate below the ankle, Falls rate among the frail elderly, Number of fractures
QOF NWL
Outcome metrics currently used by IC systems in London
Patient and staff experience measures used by IC systems in London
Metric Definition Source ICS PROMS Measure of quality from the patient perspective covering
four clinical procedures: hip replacements, knee replacements, hernia, varicose veins
NHS Information Centre NWL, ONEL
PREMS Measure of patients experience based on range of different possible metrics: patient recommendation, overall satisfaction, complaints, patient confidence
Patient surveys and questionnaires, focus groups, complaints data, one on one patient interviews
NWL , ONEL
Patient satisfaction surveys
Results of ICS service user survey Percentage of people with LTCs who feel supported to manage their condition The proportion of people who use services who feel safe
DH GP survey 032 KHP, ONEL
LTC6 survey 1. Did you discuss what was most important for you in managing your own health?
2. Were you involved as much as you wanted to be in decisions about your care or treatment?
3. How would you describe the amount of information you received to help you to manage your health?
4. Have you had enough support from your health and social care team to help you to manage your health?
5. Do you think the support and care you receive is joined up and working for you?
6. How confident are you that you can manage your own health?
Local health observatory
Staff satisfaction Staff survey to determine satisfaction of IC pilot, to determine number of staff responding very satisfied and satisfied to survey as proportion of total number of staff surveyed
Local ICS collection NWL
Contact
Dr Nick GoodwinCEO, International Foundation for Integrated Care
nickgoodwin@integratedcarefoundation.orgwww.integratedcarefoundation.org
@goodwin_nick @IFICinfo
Appendix 1: From a measure to an indicator that can be used for quality improvement
In developing ‘indicators’ to judge comparative performance, the following criteria are important:
Statistical validity Accurate – measures what is says it measures! Reliable – can be tracked over time Consistent – data collection robust and reproducable Avoids bias
Data considerations Data source – sample or full population; existing or new data source? Unit of assessment – country, region, locality, practice etc Client groups covered – what is the targeted population? account for bias/case mix? Significance – smaller groups, reduced statistical power Patient reported data is indirect – how cross-compare to other data (multi-methods) Coverage – health, health & social care, health, social care and housing ….
Face validity Meaningful to public; clinically credible; Potential to support quality improvement; Cost and value for money
Appendix 2: Key Resources on Care Co-ordination MeasuresMcDonald KM, Schultz E, Albin L, Pineda N, Lonhart J, Sundaram V, Smith-Spangler C, Brustrom J,
and Malcolm E. Care Coordination Atlas Version 3 AHRQ Publication No.11-0023-EF. Rockville, MD: Agency for Healthcare Research and Quality. November 2010. http://www.ahrq.gov/qual/careatlas/careatlas.pdf ttp://www.ahrq.gov/qual/careatlas/careap4.pdf - 64 different survey tools
King et al - 1995/2004 - Measures in the process of care – MPOC http://www.canchild.ca/en/measures/mpoc56_mpoc20.asp
Flocke SA. Measuring attributes of primary care: development of a new instrument. J Fam Pract. 1997 Jul;45(1):64-74.
Safran DG, Kosinski M, Tarlov AR, et al. The Primary Care Assessment Survey: tests of data quality and measurement performance. Med Care. 1998 May;36(5):728-39.
Cassady CE, Starfield B, Hurtado MP, Berk RA, Nanda JP, Friedenberg LA. Measuring consumer experiences with primary care. Pediatrics. 2000 Apr;105(4 Pt 2):998-1003.
Starfield B, Cassady C, Nanda J, Forrest CB, Berk R. Consumer experiences and provider perceptions of the quality of primary care: implications for managed care. J Fam Pract. 1998;46(3):216-26.
Grimmer K, Moss J. The development, validity and application of a new instrument to assess the quality of discharge planning activities from the community perspective. Int J Qual Health Care. 2001 Apr;13(2):109-16
McGuiness C, Sibthorpe B. Development and initial validation of a measure of coordination of health care. Int J Qual Health Care. 2003 Aug;15(4):309-18.
Radwin L, Alster K, Rubin KM. Development and testing of the Oncology Patients' Perceptions of the Quality of Nursing Care Scale. Oncol Nurs Forum. 2003 Mar- Apr;30(2):283-90.
Billings J, Coxon K, Alaszewski A. Empirical research methodology for ‘Procare’ version 3. University of Kent at Canterbury: Centre for Health Services Studies; 2003. Available from: http://www.kent.ac.uk/chss/docs/procare_version3.pdf.
Morita T, Hirai K, Sakaguchi Y, Maeyama E, Tsuneto S, Shima Y. Measuring the quality of structure and process in end-of-life care from the bereaved family perspective. J Pain Symptom Manage. 2004 Jun;27(6):492-501.
Coleman EA, Mahoney E, Parry C. Assessing the quality of preparation for posthospital care from the patient's perspective: the care transitions measure. Med Care. 2005 Mar;43(3):246-55. (also CTM website)
Glasgow RE, Wagner EH, Schaefer J, Mahoney LD, Reid RJ, Greene SM. Development and validation of the Patient Assessment of Chronic Illness Care (PACIC).Med Care. 2005 May;43(5):436-44
Drewes et al (2012) PCIC+ at http://www.ijic.org/index.php/ijic/article/view/862/1896 Haggerty J, Roberge D, Freeman G, Beaulieu C, Breton M. Review. When patients encounter
several providers: validation of a generic measure of continuity of care. Annals of Family Medicine (In press) and Breton et al at http://www.ijic.org/index.php/ijic/article/view/682/1534
Hibbard JH, Stockard J, Mahoney ER, Tusler M. Development of the Patient Activation Measure (PAM): Conceptualizing and Measuring Activation in patients and consumers. Health Services Research 2004: 39:1005-1026.
Powell, R. Powell, H. Baker, L. & Greco, M. (2009) Patient Partnership in Care: A new instrument for measuring patient–professional partnership in the treatment of long-term conditions. Journal of Management & Marketing in Healthcare. Vol. 2 No. 4. PP 325–342.
Long Term Conditions 6 (LTC 6) (part of the QIPP programme) Department of Health and also Living with your Long Term Condition (LWYLTC)
Safran DG, Karp M, Coltin K, Chang H, Li A, Ogren J, et al. Measuring Patients' Experiences with Individual Primary Care Physicians Results of a Statewide Demonstration Project. Journal of General Internal Medicine. 2006;21(1):13-21.
Adair CE, McDougall GM, Mitton CR, Joyce AS, Wild TC, Gordon A, et al. Continuity of Care and Health Outcomes Among Persons With Severe Mental Illness. Psychiatric Services. 2005;56(9):1061-9.
King M, Jones L, Nazareth I. Concern and continuity in the care of cancer patients and their carers: A multi-method approach to enlightened management. London: National Coordinating Centre for NHS Service Delivery and Organisation 2006.
Dolovich LR, Nair KM, Ciliska DK, Lee HN, Birch S, Gafni A, et al. The Diabetes Continuity of Care Scale: the development and initial evaluation of a questionnaire that measures continuity of care from the patient perspective. Health Soc.Care Community. 2004;12(6):475-87
Kowalyk KM, Hadjistavropoulos HD, Biem HJ. Measuring Continuity of Care for Cardiac Patients: Development of a Patient Self-report Questionnaire. Canadian Journal of Cardiology. 2004;20(2):205-12.
Hadjistavropoulos H, Biem H, Sharpe D, Bourgault-Fagnou M, Janzen J. Patient Perceptions of Hospital Discharge: Reliability and Validity of a Patient Continuity of Care Questionnaire. International Journal for Quality in Health Care : Journal of the International Society for Quality in Health Care / ISQua. 2008;20(5):314-23.
Borowsky SJ, Nelson DB, Fortney JC, Hedeen AN, Bradley JL, Chapko MK. VA community-based outpatient clinics: performance measures based on patient perceptions of care. Med Care. 2002;40(7):578-86.