Conducting RCTs in prisons presents numerous practical and methodological challenges. Simply gaining access to prisons to conduct research and to deliver an intervention can be a difficult and lengthy process. There are uncertainties about the number of people who might be eligible to participate, the level of acceptability and the level of compliance with the intervention and research procedures. There are additional challenges when trying to follow up recently released prisoners in the community. Low retention rates are a common problem for studies that involve following up recently released prisoners.28 Our preparatory work for the programme demonstrated our ability to follow up 65% of released prisoners at 3 months after release.29 Some people experience homelessness when released from prison, and for people who do have accommodation this is often temporary, which makes contacting them post release difficult. Even when researchers are able to contact people after release, people need to be willing to meet up with a researcher, answer questions and complete outcome measures. There are also fundamental uncertainties about how to measure the effectiveness of a complex intervention, as this population frequently has a complex array of inter-related problems. Therefore, it is not clear which problems are the most important, which should be the target of an intervention and, once these are decided, how to measure them as outcomes in the context of a RCT.
When the Engager team began this programme of work, there were, to the best of our knowledge, no published studies of evaluations of through-the-gate mental health interventions for incarcerated men with common mental health problems. Therefore, alongside the intervention development work in workstream 1, we conducted a separate workstream to develop the trial science methodology to evaluate the intervention.
This work programme aimed to address the following methodological questions:
- What outcome measures should be used to evaluate a through-the-gate intervention for incarcerated men with common mental health problems?
- What trial procedures are required to deliver a RCT of a through-the-gate intervention for incarcerated men with common mental health problems?
- How can we measure the process of care of a collaborative care intervention designed to support incarcerated men with common mental health problems near to, and following, release from prison?
Addressing these questions in the first phase of the programme was a prerequisite for continuing to a full trial. Specifically, to proceed to a full trial we needed to demonstrate that we could achieve a follow-up of at least 50% and also demonstrate intervention acceptability in a pilot trial.
Development of a set of outcomes for a randomised controlled trial to evaluate the effectiveness of the Engager intervention
Our aim was to select a set of outcome measures that captured the most important areas of outcome using measures that were acceptable to the target population and, as far as possible, met scientific validity criteria.
We adopted a four-stage approach, involving:
- a single-round Delphi survey to identify the most important outcome domains
- a focused review to identify potentially suitable measures within the three most highly rated outcome domains
- testing of these measures in the target population to assess acceptability and the psychometric viability of the measures43
- a consensus panel meeting to select the primary outcome measure for the RCT and key secondary outcome measures.
More detailed description of these stages can be found in Report Supplementary Material 7.
We actively sought the input of our Peer Research Group, with 10 members of the group completing the Delphi survey and four members of the group attending the consensus meeting.
In the single-round Delphi exercise, although the most important areas that people thought we should be capturing outcome data for were mental health symptoms, substance misuse and social inclusion, many of the other outcome domains were rated as important. Although the outcomes of this exercise were clear and, we believed, unlikely to change much in a second round, we recognised that not all outcome domains would be relevant to all participants or amenable to change through an intervention.
To address this, we tested using a measure that captured outcomes across multiple domains and two measures that captured personalised goals. Specifically, we included the Camberwell Assessment of Need – Forensic Version (CAN-FOR),44 which captured outcomes across multiple domains and has been used as a composite measure for offenders with mental health problems. We also included the PSYCHLOPS measure45 and the Goal Attainment Scale (GAS),46 both of which allow the participants to specify personalised goals and, therefore, enabled the capturing of idiographic outcomes that were not specific to one domain.
When we tested potential measures on men in prison who were close to release, we found that we were unable to identify a measure of social inclusion that had the psychometric properties necessary to be included as an outcome measure in the RCT. The use of PSYCHLOPS was problematic in that participants tended to identify problems that were primarily due to their situation of being in prison, and these problems no longer existed once they had been released. Similarly, the GAS was also problematic in that goals were often poorly specified and, therefore, measuring change in a systematic and standardised way at follow-up was difficult. Consequently, PSYCHLOPS and the GAS were not included in our considerations for the primary or secondary outcome measures.
At the consensus meeting, we first discussed measures by domain before focusing on the primary outcome. Of the measures tested within the mental health symptom domain, the Clinical Outcomes in Routine Evaluation – Outcome Measure (CORE-OM)47 was selected as the preferred measure over the General Health Questionnaire-1248 because of its wider range of constructs. The Patient Health Questionnaire-949 and General Anxiety Disorder-750 were not selected as many individuals were subthreshold on one or the other measure. The data we collected using the CAN-FOR also indicated that it would be a psychometrically viable option to be included as an outcome measure. No other composite measures were considered.
Two-thirds of participants reported having a substance misuse problem and, although it was decided that a substance misuse measure should not be the primary outcome, two short measures were included as secondary outcomes, with one capturing information on which substances were being used and the other capturing level of dependence.
After discussion about outcome measure criteria for trials and close reading of the measures, and with peer researchers supported to ask questions, the CORE-OM and CAN-FOR received the same number of votes to be the primary outcome measure. We opted for the CORE-OM as the primary outcome measure, as it had marginally superior psychometric properties and could be administered in a highly scripted fashion that would reduce researcher bias.
The four-stage process we conducted, incorporating the opinions of our Peer Research Group, enabled us to be confident that the outcome measures we selected focused on important areas of outcome and were acceptable to our target population. The outcome measures also demonstrated psychometric properties that indicated that they would be suitable for evaluating the effectiveness of the Engager intervention. However, some questions arise on this latter point. None of the selected instruments was specifically designed for, or validated on, prison populations and, therefore, may not accurately reflect the acutely stressful effects of custodial environments. Moreover, the transition from custody to community may itself have a significant impact on mental health, perhaps ‘swamping’ effects brought about by the intervention. As most of our data testing was undertaken when the participants were still in prison, we did not establish the efficacy of the measures in community settings. Although the above are clearly limitations of this study, it should be added that trials are usually expected to use validated and established instruments, which greatly restricts choice, and that any trial measuring change ‘through the gate’ is likely to be complicated by recovery among both arms simply as a consequence of leaving prison.
Finally, we considered using outcomes based on practitioner records, such as those from the Offender Assessment System (OASys) and other assessments entered by offender managers on nDelius, the Probation Service case management system (URL: https://data.gov.uk/dataset/8dae5fb2-82a2-4232-ae79-87e9c9fcfe46/ndelius; accessed 1 August 2022). However, it quickly became clear that these records were not recorded in a sufficiently consistent way to merit inclusion: they were not undertaken at set time points, were often subjective in terms of focus and they suffered from missing data. Above all, mental health was an issue that received relatively little attention.
Development and piloting procedures for a randomised controlled trial
This work is published.43
Alongside the work undertaken in selecting a set of outcomes, we piloted RCT procedures to test the feasibility of a larger RCT. A key objective was to demonstrate that we could achieve a retention rate of > 50% when following up participants at 3 months post release from prison. Additional aims of the pilot RCT were to test out trial procedures, such as recruitment, consent and randomisation, to assess the levels of participation with the intervention, to test our ability to maintain research blinding and to model workforce requirements for the research team for a larger trial.
We reviewed the records of 864 individuals and identified 182 records of individuals who met the initial eligibility criteria. Of these individuals, we screened 110 potential participants to identify 60 eligible participants who were randomised at a ratio of 2 : 1 to receive either the Engager intervention plus usual care (i.e. the intervention group) or usual care alone (i.e. the control group). We achieved follow-up rates of 73% at 1 month and 47% at 3 months post release from prison.
No participants withdrew from the trial between consent and up to, and including, the point of randomisation, indicating that the trial procedures relating to recruitment, screening and randomisation were acceptable. However, maintaining researcher blinding was problematic and by the time of the 1-month follow-up the researchers were aware of the allocation of practically every participant. This was usually a consequence of participants sharing their allocation with the researchers who they frequently came across in the closed prison environment.
There were very high levels of data collection and completion across all the measures used at baseline and at follow-up, with few missing data. However, during the pilot trial we were still testing a range of potential outcome measures to use in the full trial and, therefore, not all participants completed the same set of outcome measures. Given the relatively small number of participants that completed each set of outcomes, we were unable to establish whether the intervention was able to bring about change on the measures, and whether or not such a change would be discernible over and above any change in scores that may have been due to whether the participants were in prison or the community. This issue was a key limitation of the pilot trial.
With regard to intervention acceptability, 90% of participants allocated to the intervention group received some of the intervention. Of the four participants who did not receive any intervention, three were either released or transferred to another prison before being seen by a practitioner and one decided the intervention was not for him prior to having contact with the practitioners. Of the 36 participants who did receive the intervention, 28 met with their practitioners in the community following their release from prison. There was strong indication that receiving support on the day of release led to ongoing contact in the community.
Overall, we found the trial procedures and intervention to be feasible to deliver, as well as acceptable to participants. With regard to contamination between the intervention and usual care, we found no evidence of this and, as the Engager practitioners were a stand-alone team, with many not working within the prison prior to the Engager intervention, there was little risk of contamination. The one major exception was in maintaining researcher blinding, which was a problem that was almost universal at both sites. We considered possible solutions to this problem, but it was felt that any solution created potentially more challenging problems. For example, we considered using a different researcher to collect data at follow-up, but felt that this might have a negative impact on follow-up rates, as participants may not be willing to meet a researcher they had not built up trust and rapport with. Ultimately, we opted to accept that unblinding would occur in the trial and to focus attention on reducing biases by training researchers to apply measures consistently, while continuing to allow flexibility for the whole schedule.
The retention rate for the 3-month follow-up was 46%, which was lower than hoped. To address this for the full trial, we broadened the window of time during which follow-ups could be completed. Participants often led chaotic lives and were very difficult to contact for periods of time and so it was felt that giving researchers longer to make contact with and meet participants in the community would increase the follow-up rate. In addition, we also decided to offer ‘thank you’ vouchers to compensate participants for their time when we interviewed them in the community, and we received approval from the Ministry of Justice to do this. Last, many of the participants worked with a range of services on their release from prison, and the introduction of the community rehabilitation companies (around the time of the pilot trial) meant that all people leaving prison were to be provided with support from probation for 12 months following their release. We developed closer working practices with services in the community, including establishing information-sharing agreements, to help researchers contact participants in the community.
The whole of the feasibility study also allowed us to test and develop the intervention and to model researcher workforce requirements and recruitment rates for small or more extensive geographical follow-up areas and different participant eligibility criteria, and this was critical for both assessing viability of and planning for the full trial.
Developing a process of care framework
In addition to developing a set of outcome measures for a RCT to evaluate the Engager intervention, we sought to develop a framework for capturing the care that prisoners received during their participation in the trial from the Engager intervention and standard services. A detailed description of the work we undertook in developing methods to capture the process of care is provided in Report Supplementary Material 8.
Measuring routine care
We recognised that the RCT was going to take place at a time of considerable change within the probation service, as well as at a time of ongoing changes in the NHS environment. Therefore, it was also important to capture what ‘usual care’ comprised. One of the key components of the Engager intervention was to support participants to work with services that could help them work towards achieving their goals. This might involve the Engager practitioners identifying appropriate organisations, liaising with those organisations and supporting the participant in meeting with those organisations. Therefore, it was anticipated that the Engager intervention could increase the amount of care received from other organisations and, potentially, improve the effectiveness of that care in meeting the participants’ needs.
To capture the care received from organisations, we adapted the service use elements of the Client Service Receipt Inventory.51 Specifically, we expanded the list of organisations to include third-sector organisations and organisations providing social support (e.g. accommodation, employment and benefits). To help participants recall the services they had been in contact with, we split the services up by the type of service and asked about each type of service contact at relevant points in the interview schedule. For example, service use relating to accommodation was captured immediately after asking participants about their current accommodation situation and level of need in relation to accommodation. This also created a more coherent conversation and reduced annoyance induced by revisiting topics.
In addition, we included questions on how helpful the contact with each organisation was as a proxy measure of the effectiveness of the support received from each organisation. We asked participants about who initiated contact with each organisation (e.g. self, CJS, health service or the Engager practitioner) to provide an indication of how much service contact was initiated by the Engager practitioner.
Last, we asked participants to complete the Brief Inspire,52 which is a tool designed to assess a service user’s experience of the support they receive. We supplemented the five questions used in the Brief Inspire with a further nine items that were identified in workstream 1 as important for participants to have a good experience with a service (e.g. participants can trust the service and participants feel that the service listens to them). Although participants were likely to engage with many services, we decided to ask participants to provide ratings of their experiences of working with services in general, using the Brief Inspire to reduce participant burden.
Quantifying care from the Engager intervention
We sought to capture the amount of support participants received from the Engager practitioner and information regarding the response to this support. This was important to assess fidelity to the intervention and in understanding whether or not the intervention worked in the way we anticipated from the logic model.41
To capture the process of care provided by the Engager practitioners, we developed a range of tools. First, to capture the amount (i.e. extent and dose) of contact between participants and the Engager practitioners, practitioners were asked to complete timesheets detailing every contact they had with the participant, providing details of the length of each contact and the location (i.e. prison or community). In addition, the Engager practitioners wrote clinical notes for each session they had with participants. The notes acted as a confirmation of the information recorded in the timesheets and provided details of the nature of the meetings between the practitioner and participant, as well as which components of the intervention were delivered (e.g. practical or psychological).
Last, to capture information regarding which components of the intervention were delivered, and whether or not they had the intended consequences, we developed a series of ‘if–then’ statements. The ‘if’ statement captured information relating to how often a component was delivered and the ‘then’ statement captured the extent to which it had its intended consequence, according to the Engager programme theory. The ‘if–then’ statements were rated by practitioners for each participant they had on their caseload, and were also rated by each participant. More expansive qualitative comments were also recorded.
The programmatic nature of the grant allowed us to focus on developing bespoke ways of capturing usual care and intervention content, which is required both for interpretation of the trial and understanding the way in which the intervention might work. However, such development work often requires iterative mixed-methods approaches and we were not able to fully test and refine the measures for the Engager intervention.
Economic modelling
This part of the economic analyses comprised conceptual model development, decision model specification and assessment of the feasibility of conducting estimated value of perfect information analysis (see Report Supplementary Material 10).
Development and testing of a simulation model to estimate the outcomes of the Engager intervention
A discrete event simulation model of the short-term outcomes and costs of the Engager intervention was developed based on (1) two previous relevant models (i.e. the revolving doors agency financial analysis model53 and the Home Office’s model for the economic and social costs of crime54) and (2) a systematic review of economic evaluations and cost-of-illness studies relating to the commonly overlapping problems that many prisoners and ex-prisoners face, such as drug or alcohol misuse. For both treatment strategies, that is, the Engager intervention and usual care, the model simulates an individual’s pathway through different events, recording the events experienced and calculating the total costs and outcomes associated with that pathway. The model assumed that prisoners had 4 months in prison before release, and then incorporated contact with health services in the community, medications, non-fatal self-harm events and deaths.
For people receiving the Engager intervention, there would be a probability of an improvement in mental health within prison once treatment with the Engager intervention started. After release from prison, there would be an increase in the probability of planned contact with health services (with probabilities differing depending on the type of health service) and a subsequent decrease in the probability of unplanned contact with services.
Modelled outcomes
This initial stage of modelling was based on a model with service use for mental health and drug misuse, as well as general practitioner (general practice) attendances in the pilot work. The challenges of obtaining and using data that could potentially be available from the full trial to enable these basic models to generate plausible cost and cost-effectiveness were expected to be considerable. The challenges would certainly prevent the possibility of the kinds of cross-sectoral estimated value of perfect information analysis that were originally planned. Along with the shift to a definitive RCT, the challenges informed the decision to make the economic evaluation of the main trial a combination of three cost-effectiveness analyses (using the three main trial outcomes) alongside a cost–consequence analysis.