Embed Size (px)
Citation preview
Using propensity score matching to understand what works in reducing re-offending
GSS Methodology Symposium
Sarah French & Aidan Mews, Ministry of Justice
1st July2015
What will be covered
• What is propensity score matching (PSM)?
• History of using PSM in MoJ
• Justice Data Lab aims, history and how it works
• Methodology
• Key outcomes
• Developments
3
• We find individuals who were not treated, but who were very similar at the point treatment started.
TREATMENT
CONTROL
?
Observational Studies
4
What Propensity Scores Are, and Why They Are Important
Propensity Score = Pr (Treated | background info)
• Mimics an RCT in that the treatment is purely random for individuals with similar values of the background variables
• Groups of subjects with similar propensity scores can be expected to have similar values of all of the background information, in the aggregate.
Propensity Score Matching
Scale of propensity score
0 1
Control offenders
Treatment offenders
Key
Matched offenders
Control offenders are matched to treatment offenders if the control offender’s propensity score are within a specified range away from the treatment offender’s propensity score.
History of propensity score matching in MoJ
PSM first used in 2010 Compendium of re-offending to compare the effectiveness of two sentences
Since then used:
.. for further comparisons of sentences & sentence requirements .. to look at the impact of various interventions to reduce re-offending
.. to evaluate the relationship between employment and re-offending
Aim of the Justice Data Lab
Launched in April 2013
..to improve the evidence base on successful rehabilitation..
..by giving organisations working with offenders secure and legal access to aggregate re-offending data
..enabling them to better assess the impact of their work on re-offending
Why do we have the Justice Data Lab?
In 2012 we identified that charitable organisations in particular found it difficult to access re-offending data on their clients…
… this meant that they could not understand how effective their services were at rehabilitating offenders…
… and they were therefore unable to understand how their services could be improved, or have the evidence for further funding
It soon became clear that there was intense interest in this initiative from both public and private sector organisations too
How does the Justice Data Lab work?
Individual level data sent securely to MoJ
Provider organisation
MoJ
Analysis and Matching
Aggregate data return
Process overview
• Data upload template (60 minimum)
• Match to MoJ/DWP data to get treatment group– find sentences for offenders from PNC
– match to DWP/HMRC data
• Matched control group via PSM (30 minimum)– Many to one matching, radius matching with replacement
• Assess quality of matched control group by analysing standardised mean differences
• Significance testing on re-offending measures compared between treatment and control groups
Key characteristics of ‘research reports’
• Binary re-offending rate (overall, resulting in custody, severe, 1 year, 2 year)
• Frequency of re-offending
• Survival (or death) analysis
• Representativeness of matched treatment group
• Quality of matching
• Sensitivity analysis
• Caveats (e.g. unobserved characteristics not included in matching process)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0 30 60 90 120 150 180 210 240 270 300 330 365
Days into follow-up period
Immediate Custody (1-4 years)
Immediate Custody (less than 12 months)
‘Death’ analysis
Re-
offe
nd
ing
rate
What is provided to Justice Data Lab users?
• One year re-offending rate
• Frequency of re-offending
• Time to re-offending
• Information on characteristics of both the treatment and control groups
0%
10%
20%
30%
40%
50%
60%
WYJS Participants (82offender records)
Matched ControlGroup (41,403 offender
records)
On
e ye
ar p
rove
n r
e-o
ffen
din
g r
ate
The best estimates for the one year proven re-offending rate for offenders who received an intervention from WYJS, and a matched
control group.
Key Justice Data Lab outcomes
Of the 125 reports published so far:
• 29 reports indicated statistically significant reductions in re-offending on the one year proven re-offending rate
• 89 reports indicated insufficient evidence to draw a conclusion about the effect on the one year proven re-offending rate
• Of these 89, 11 reports detail statistically significant reductions in the frequency of re-offending
• 7 reports indicated a statistically significant increase in re-offending on the one year proven re-offending rate
Developments in PSM use within MoJ
• Move away from 1-1 to 1-many matching
• Use of more information in matching process
• More thinking about methodological improvements to reduce bias
• Wider range of outcome measures
• Post matching regression analysis
Justice Data Lab Developments in Progress
• Providing additional information on the re-offending outcomes, such as severity of re-offending
• Enhancing understanding of the criminogenic needs of individuals – through the use of Offender Assessment (OASys) data
• Understanding more about individuals that are not matched in an analysis
• Official Statistics methodology review
• Supporting other government departments on potential Data Labs
Contact Details
Email: [email protected]
Accessing the Justice Data Lab service:https://www.gov.uk/government/publications/justice-data-lab
Published reports:www.gov.uk/government/collections/justice-data-lab-pilot-statistics
