Anders, JD; Dorsett, R; (2017) A review of the matching process for the impact analysis of the HMP Peterborough Social Impact Bond. Ministry of Justice: London, UK.

Preview

Text peterborough-social-impact-bond-cohort-1-methodological-review.pdf - Published Version Download (228kB) | Preview

Abstract

In 2010, the world’s first Social Impact Bond (SIB) was launched at Peterborough Prison. It was used to fund an intervention – ‘The One Service’ – aimed at reducing the reoffending among prisoners discharged after serving a sentence of less than 12 months. Under the terms of the SIB, investors are paid according to how successful the One Service is in reducing reconvictions. If a minimum threshold of a 7.5% reduction in reconviction events is reached across the pilot, payment is triggered. Additionally, there is an option to trigger an early payment if a 10% reduction is noted in the number of reconviction events in individual cohorts. A propensity score matching (PSM) approach was used to estimate the impact. For cohort 1, the impact was estimated, by a previous team of independent assessors, to be a reduction in reconviction events of 8.4% (Jolliffe and Hedderman, 2014). This report reviews the PSM approach, prompted in part by the desire to understand the reasons behind the differences in reconviction rates between prisoners discharged from HMP Peterborough and prisoners discharged from other prisons. Its primary aim is to identify whether there is a need to revise the approach taken before it is applied to cohort 2 (and the final cohort - the weighted mean of cohort 1 and cohort 2). To do this, various amendments to the methodology were explored. It was not possible to replicate the results of Jolliffe and Hedderman (2014). This should perhaps be no surprise given the difficulties often encountered with replication attempts. However, the difference between the replication result and the Jolliffe and Hedderman (2014) result was not statistically significant. Since this review did not identify any clear improvement in the matching process, the conclusion is that the cohort 1 approach be maintained. We also recommend an adjustment to the sample selection in cohort 2. It is important to note that this recommendation is based on the analysis of cohort 1 data and is not informed by cohort 2 reoffending data. In more detail, the review began with a replication of Jolliffe and Hedderman (2014) and then explored whether various changes might give better estimates: changing the set of variables included in the propensity score model; changing how propensity score matching was implemented; allowing for unobserved prison- and/or area-specific differences in outcomes. The analysis used data for cohort 1 plus a number of years prior to the introduction of the One Service. The results suggest: it may be possible to alter the set of variables in the propensity score model in order to achieve a somewhat better fit, but this did not lead to any overall improvement in the methodology; changing the implementation of matching did not achieve any overall improvements in the model; controlling for historic differences in outcomes between prisons is problematic since the mean number of reconviction events at HMP Peterborough has followed a different trend from other prisons over the 2006-2009 pre-treatment period. In view of these findings, the recommendations for cohort 2 are as follows: Use the same matching variables as for cohort. While some advantages to altering the variable set were found, these were not sufficient to justify a change; Use the same matching approach. The cohort 1 approach of (up to) 10:1 matching without replacement, within a 0.05 caliper performed as well as alternative approaches; Estimate effects only for men aged 21 or above. This reflects the fact that, in cohort 1, there were almost no under-21s at HMP Peterborough. If under-21s are similarly absent at HMP Peterborough in cohort 2, the recommendation is to exclude them from comparator prisons too. This will help make the composition of individuals from other prisons more similar to that of HMP Peterborough; Report standard errors of impact estimates. This is suggested as a means of providing some sense of the statistical significance of the estimated impacts. The protocol for cohort 2 and the final cohort is included as an appendix.

Download activity - last month

Export as CSVXMLJSON

Download activity - last 12 months

Export as CSVJSONXML

Downloads by country - last 12 months

Export as CSVJSONXML

Archive Staff Only

View Item