The objectives of the mixed-methods process evaluation, with data collected in parallel to the trial, were to:
- determine the extent to which the intervention was delivered as outlined in the Engager manual
- evaluate the extent to which the core components and mechanisms of the intervention produced the intended outcomes
- support the interpretation of the RCT
- identify aspects of the intervention and delivery that could be improved
- explore unintended consequences of the intervention
- understand how to deliver the intervention in real-world settings (including training, supervision, meta-supervision).
The process evaluation carried out the following analyses:
- A fidelity analysis that quantified the extent of the intervention delivery.
- A thematic analysis that demonstrated the wider issues that may have supported or inhibited the delivery of the intervention.
- An analysis of the extent to which individuals and practitioners experienced the intervention components as having been delivered and having a positive impact.
- A realist-informed in-depth case study analysis that explored:
- how the quality of delivery of various components of the intervention related to individual responses
- how key outcomes were or were not generated through practitioner activity.
Each analysis is summarised below. Data were collected from a range of sources and a table detailing which data sources were used for each analysis is shown in Report Supplementary Material 11.
Fidelity analysis
More detail on the fidelity analysis can be found in Report Supplementary Material 12.
The fidelity analysis aimed to quantify the extent to which the intervention was delivered as hypothesised. The primary objectives were to examine (1) adherence to delivery of the components of the intervention, as specified in the logic model, and (2) delivery of the team model throughout the trial. Data were extracted from practitioner logs and records for 129 of the 140 intervention participants. We were able to identify participants who had received the minimum dose of the intervention thought necessary for impact (i.e. two prison sessions and eight community sessions) and the extent of delivery for other participants.
Less than half (n = 62, 48%) of the participants received the minimum dose and just over one-third (n = 48, 36%) of the participants received any release day contact. Variations in intervention delivery were observed across sites. Information on the content of sessions was available for 45% (n = 332) of prison sessions and 47% (n = 701) of community sessions. Based on the content of records, just over two-thirds (67%) of the sessions were assessed as practically focused (e.g. focused on housing issues), 23% contained both practical and therapeutic content and just 10% were predominantly of a therapeutic nature (e.g. focus on thoughts, feelings and behaviours).
Overall, the fidelity analysis based on records highlighted that intervention delivery was not as therapeutically focused as was intended. Staff turnover, long-term sickness and the prison context (e.g. regime and lockdowns) presented barriers to the planned team model delivery of the intervention.
Thematic analysis of barriers and facilitators
This thematic analysis aimed to explore the individual- and organisational-level barriers to and facilitators of intervention delivery and implementation. A full description of the methods and results can be found in Report Supplementary Material 13 and 14.
Five key cross-cutting themes were identified.
Perceived usefulness of Engager resources
A shorter version of the manual could have more effectively supported daily delivery. More regular training sessions tailored to the competency gaps of practitioners and focused on the most complex aspects of the intervention (e.g. MBA and shared understanding/action plan) could have improved quality of delivery.
Intrateam and interteam dynamics
Intervention delivery was compromised, at times, by long-term sickness, practitioner turnover and unhelpful team dynamics.
Individual contributions of practitioners and clients
Practitioners’ lack of experience in working with this client group, and in a psychologically informed way, limited their ability to deliver the intervention. Some participants were difficult to get hold of and hard to engage, and this influenced delivery.
Context within the prison and community
The prison environment was a volatile context to deliver the key values of the intervention. Outside the mandated atmosphere of the prison environment a chaotic community context, with participants less likely to attend appointments, was also often challenging for ongoing intervention delivery and for planning formal endings.
Extent of delivery and impact of key intervention components
This component of the process evaluation provided more granularity as to the extent to which intervention components were delivered and whether or not these had an impact. The development of the practitioner and participant perspective checklists are reported in Report Supplementary Material 11. The two ‘if–then’ checklists were based on the realist-informed programme theory and were used to record ‘if’ individual components of the intervention had been delivered and, as a result of that delivery, whether or not the hypothesised outcome had ‘then’ been achieved.
Data from the practitioner perspective were available for 121 out of 140 intervention participants; however, because of late introduction of this measure, low follow-up rates at 12 months and participant burden, data from the participant perspective are reported for only 35 participants. The key findings were as follows.
Trust and practical support
Practitioners reported that they treated participants with respect (n = 120, 99%) and showed that they cared by helping participants and offering practical support (n = 103, 85%), which led to participants having trust in practitioners in most cases (n = 96, 81%). All 35 participants reported that they felt respected, 32 (91%) said that practical support was given and 28 (86%) reported that this led to trust.
Mentalisation-based approach
Delivery of the key mentalisation techniques (i.e. stop/listen/look, micro-slicing, rewind) was reported by practitioners as being delivered regularly (n = 50, 40%) and as not being achieved at all (n = 33, 25%). When using strengths-based work, practitioners reported participants becoming more self-reliant in 38% (n = 45) of cases. Seventy-three per cent (n = 25) of participants reported that being able to talk through things meant that they were more able to understand their thoughts, feelings and behaviours, and 50% (n = 17) of participants said that it helped them to get on with others.
Resource mobilisation
Practitioners ‘always’ or ‘frequently’ liaised with other services to galvanise support in almost two-thirds of cases (n = 70, 58%). Practitioners reported that 53% (n = 61) of participants maintained engagement with other services as a result.
A positive ending
According to practitioners, less than half (n = 46, 45%) of the participants received support around ending the intervention, often due to loss of contact. However, of the 46% (n = 13) of participants who reported a planned ending, all said that they would be more able to continue working towards their goals after the Engager intervention ended.
Although there was greater practitioner than participant data completion, the multiperspectival approach contributed towards a degree of data triangulation by adding weight to the health records-based fidelity analysis that practitioners were empathic and provided practical support, but delivered psychological interventions less often.
Realist-informed case studies
Twenty-four longitudinal realist-informed in-depth case studies were conducted to examine how participants engaged and responded to intervention offers. Two papers reporting this part of the process evaluation have been published.58,59 Report Supplementary Material 15 presents more details of the ‘if–then’ checklist and Report Supplementary Material 16 details the purposive sampling strategy that was used to select case study participants and provides an overview of the data collection strategy and interview schedules.
Typologies of engagement and response to the intervention
We showed that the intervention was delivered with varying degrees of fidelity. We identified five typologies of participants as follows.
Crises but coping
The participants in the ‘crises but coping’ group engage with the intervention and appreciate Engager intervention support, but continue to experience problems. The participant makes positive incremental changes over time, despite setbacks, leading to a better life situation. However, it is of note that the positive outcomes achieved by these individuals did not correlate with the standard measures when the case study outcomes and trial outcomes were compared after both analyses had been completed.
Resigned acceptance
Participants in the ‘resigned acceptance’ group initially engaged with the intervention, but problems and challenges continued and the participants became resigned that their circumstances would not change and concluded that they were not able to pursue goals.
Crises and in chaos
Participants in the ‘crises and in chaos’ group were unsure about their ability to make changes, but appreciated offers of support and initially engaged with the intervention. However, this was short-lived and was not accompanied by the development of a shared understanding. When confronted by problems and challenges, these participants soon became overwhelmed by their circumstances and disengaged from the intervention.
Wilful withdrawal
Participants in the ‘wilful withdrawal’ group declined support early on. These participants received minimal contact with their Engager practitioner in prison and so trust was never established.
Honeymoon
Participants in the ‘honeymoon’ group were motivated to make changes at the outset; however, soon after their release, they believed that they could manage without support. Therapeutic work was, therefore, ineffective as Engager practitioners appeared to struggle to challenge the overoptimistic participants to develop a realistic understanding of their ability to manage their situation. Subsequently, participants would hit crisis soon after release and would not be able to sustain motivation to pursue their goals.
Effect to cause analysis
We attempted to understand what enabled, or limited, the intervention mechanisms (i.e. practitioner resource offers and offender responses) in achieving the intended outcomes. The main areas explored are as follows.
Emotional competencies (motivation and agency)
Practical and therapeutic support offered by the practitioners did help participants to feel motivated towards changes. However, often the support was too practical and not informed by a shared understanding of the underlying issues that would help participants manage challenges after their release and of the skills they needed to develop to sustain change.
Psychological competencies (coping, emotion regulation, coping strategies)
Participants who received more intensive therapeutic support continued to engage with their practitioner during setbacks and were helped to mentalise about where things had gone wrong. However, as above, therapeutic work was often absent, shallow and superficial, and the resources offered to participants were often practical and did not offer opportunities to reflect on dysregulated thinking.
Social skills and capital (behaviours, building positive relationships)
Participants needed repeated opportunities to reflect before mechanisms had a sustained effect. For some participants, initial engagement with the intervention was a means to foster a favourable impression to other services; however, this is not to say that participants were deliberately disingenuous about wanting to make changes with their lives. Participants believed (fallibly) that their own willpower and motivation would be enough to cease compulsive behaviours. Practitioners were sometimes met with guarded responses and found it difficult to challenge a participant’s overly optimistic perception of life outside prison to enable participants to develop a more realistic perception of the challenges that they faced.
Synthesis 3
The five analyses from the parallel process evaluation were brought together, assessed and discussed by team members. Details of this informal synthesis and analysis can be found in Report Supplementary Material 17. The results were used to further develop the Engager intervention theory. The three main refinements identified were:
- increased support for practitioners to understand fluctuating emotional states
- making mentalisation more central
- developing epistemic trust between the participant and the practitioners as a key mechanism to improve the effectiveness of the therapeutic work.